This is the manual.
Thought of the day: "Our job would be so much easier, if it wasn't for those damn end users."
0) Basics
0.0 Introduction.
0.0.0 Why the scripts and their history. The idea behind those scripts started when I wrote
a guide to crops for DF Fallout Equestria mod. Those two tables have proven to be pretty tedious to do by hand. Both reading plant RAWs for values and writing all that bbcode to make tables on the forum took hours. Then someone asked me, if I had a script to do it. Eventually I wrote a 2kb perl script to convert a csv file to a bbcode table. It took me an evening and was encouraging, but the problem of fishing for all the values to fill the table remained.
Then I decided that if the mod updated (which it sadly didn't, at least when it comes to plants), it would be handy to have a script that could read RAWs and print crops in human-readable form nearly instantaneously. Heck, it could even be used by modders to make manuals for farming. So I wrote it and had a working version within a week, then goofed around with generating hints and colouring crops.
After that I figured that hey, I've got the "infrastructure" to read a subset of DF RAWs and print tables in several formats, so why not use it for something else. Since then I've been working on an off on various scripts for a while. I usually can produce a script for a new section of RAWs pretty quickly, but then continue to add features, polish and test it on various mods for a while, so it takes time before I publish something. On the plus side, those scripts tend to be pretty stable from all that extra effort.
0.0.1 Why perl. Well, a scripting language was a natural choice for processing text files and printing text output (html, wikicode and bbcode are really text with some "special words" in it). I've used perl before and even worked with it at my second job. So while Ruby, Python or Lua might be a better choice, I stuck with what was familiar. That said, it is a horrible language. Even after writing thousands lines of code, I have to keep looking up seemingly simple things and getting them to work by trial and error.
0.0.2 What are these scripts good for? My scripts are useful to give feedback to the modder. It simply helps to see all (or at least much) of that data as a table and compare different crops / creatures / weapons / whatever. The scripts can also be used to produce manuals for the user, so long as you're willing to edit their output a bit.
They can all output html (preferable for just looking at the data), bbcode (used by forums) or csv (can be imported into a spreadsheet) and I'm pretty happy about this versatility. The scripts also provide multiple options for controlling the output, for example which columns to print. Run any of them with --help to get a brief explanation.
0.0.3 Why not integrate this with dfhack? Having a similar functionality within dfhack would be useful, but:
* I've never written a dfhack plugin and have little idea how to fish for RAW data with it. But I guess I could overcome this difficulty.
* Standalone scripts don't require a running copy of Dwarf Fortress. While dfhack commands might be more convenient, standalone scripts don't require reloading Dwarf Fortress after changing something in the RAWs.
* Now that I have these scripts, I don't feel compelled to port them to dfhack and it wasn't requested.
0.1 Getting Perl
To run my scripts you need to have Perl interpreter installed. Just type this into the console to check if you already have it.
$ perl -v
If you're using Linux, there is a very good chance you have perl. If not, use Synaptic packet manager to get it. Simple as that.
If you're on Windows, there is
Strawberry Perl, or the outdated version that comes with
MSYS. It runs my scripts, but is otherwise worth it only if you need a Unix shell under Windows. If in doubt, go for Strawberry. Finally, Urist McTeellox, a Perl professional, recommended
DWIM Perl for having Padre IDE and lots of CPAN modules.
My scripts are very backwards compatible, run on a version of Perl that's over 10 years old and require no additional CPAN modules. I've tested them for perl 5.14 and perl 5.6.
0.2 On using the command line.
My scripts have no graphical user interface and use the console instead. If you're not familiar with concepts such as current directory or standard input and output, I strongly suggest that you familiarise yourself with a shell tutorial. This one is pretty good.
http://community.linuxmint.com/tutorial/view/100 Either way, this is what you need to know to use my scripts:
Any time you see something like this
$ ls -l
it is a shell command. You can paste that into the console and see the result. You skip the dollar sign, which is called command prompt and means "execute the thing after the dollar in the console". I use the names console and shell as synonyms. You have to press enter to run any command you type or paste into a console.
A few times I'm using # as the command prompt, mostly for dfhack getplants command. Anyway, a # means it is a dfhack command, not shell. Under Unix it may also mean running a command as a superuser, but the # isn't used in that context in this manual.
Globs are short-cuts for multiple file names, commonly used as convenient short-cuts for shell commands. The important things to remember about globs is that they are expanded to a list of files as separate arguments, * stands for any character string and ? stands for any one character. Don't put globs inside quotation marks or they won't be expanded. You can check what they do with ls command. For example.
$ ls *plant*.txt
$ dir *plant*.txt
Should print things like plant_grasses.txt, plant_standard.txt and any other plant-themed files you may have.
Personally I just use glob *.txt which makes the scripts read all the RAW files (skimming most of them). It takes only a few seconds anyway.
Any time you see something inside <these>, it means "substitute this to a suitable value". For example a --sort=<column> is not to be used verbatim, but instead becomes --sort=ID or --sort=NAME or similar. The <> signs are to be removed when you do the substitution, they are just markers.
A console program (or in this case, a shell script) is normally executed with a series of parameters. You might for example see something like this
$ grep -l "short sword" *
This first thing the shell does is to expand the glob * into a list of matching files in the current directory (in this case all files inside the raw/objects). It then runs the grep program, giving it a list with "-l", "short sword" and any found files. It is up to grep to interpret what it gets. In this case it sees "-l" and interprets it as an option to just list files containing the pattern. Then it sees "short sword", which doesn't match any of it's recognized options, so it gets interpreted as the pattern to look for. Everything else is treated as a list of files to check.
The quotation marks around "short sword" are important, because otherwise it would get interpreted as two separate parameters.
$ grep -l short sword *
Now grep gets a list of parameters with "-l", "short", "sword" and a list of files. It then looks for "short" and considers "sword" to be a file name. This is especially important with long file names or options containing custom text, such as "--extra=To do". These parameters are also not the same and get interpretted differently: --a--b, --a --b, "--a --b". Basically, the shell copies any parameters inside quotation mark verbatim (well, not quite and there are three kinds of quotation makrs, but this is outside the scope of this tutorial), so if you need a space o a * or some other non-alphanumeric signs as parameters, surround them with quotation marks.
The minuses at the beginning of options are just a convenient convention to differentiate options from file names. If some file has a name like -f, you can use -- to tell the script "everything from now on is a file, even if it starts with a minus".
When using the console, you'll often find yourself typing long commands. If you want tot repeat them, or use something similar, there is a convenient way to do it.
By using up and down arrows, you can scroll the history of recently used commands, select the one you want, edit it and run with enter. Oh and when you're redirecting output stream be careful where you point it. You can redirect standard output of a console program or script to a file by appending the command with >out.txt where "out.txt" is the name of the file you want to save to. Just be mindful that if you have write permission to that file, it will be overwritten.
There are a few shell commands that will make your use of my script (and hopefully dealing with mods) easier. The most important are:
Display files in current directory $ ls
Change current directory $ cd
List files containing pattern: $ grep -l "Thunder" *
Show files with patterns and context: $ grep -C 10 "SADDLE" *
0.3 Script, RAWs and directories.
Dwarf Fortress RAWs are text files inside <DF directory>/raw/objects You'll need to pass (some of) them as parameters to my scripts. I usually like to cd to raw/objects, then just use globs, such as *.txt.
The rest of this manual assumes you are inside the RAW directory and both the scripts and output files are somewhere else.
While the values in tables are taken from the RAWs, the spacing, comments, hints, etc. may be imperfect, so you are encouraged to edit these tables before using them as part of some manual. Ether way, the scripts do the bulk of the tedious work for you.
0.4 Output types and txt/csv.
While at first I just wanted to generate tables with bbcode to post them on the forum, several other output options turned out to be attractive. My scripts can write tables in txt, csv, bbcode, wikicode and html formats. I briefly considered LaTeX for an intermediate step in producing pdfs, but didn't do it. My scripts handle multiple output types by checking for parameters such as --html. They then set several variables controlling the output, such as $bbold and $ebold. For example under txt and csv those are empty strings, under bbcode they become \[b\] and \[\/b\], while wikicode uses ''' and '''. This way output method is abstracted from the rest of the script and for example
print $bbold . "Size" . $ebold . " is in cubic centimeters." . $n;
doesn't need to care if I'm printing html or bbcode or what. On the downside, this sometimes produces too many or too few empty lines in some output formats.
The plain text output is there if you for some reason like plain text files. I made it synonymous with csv, because I don't really see much of a difference. Well, I guess I could count the length of each cell, then pad cells with spaces and tabs so that it looks nice, but I find html more convenient for looking at the data anyway.
The CSV stands for comma separated values. The basic idea is that each line is a row of a table and the columns are separated by commas. At least in theory, because things like commas inside quoted strings are implementation dependent. On it's own, this format is just a variation on a plain text file, but it can be imported into a spreadsheet. Just save the script output into a file with .csv extension (this is important) and open it with a spreadsheet program, such as Open Office Calc. Import the data, using the comma as a separator and you now have a spreadsheet. You'll probably have to move some things around, resize columns and delete some text, but if you know how to use a spreadsheet, you should feel at home.
Wikicode is for pasting tables in the magmawiki. I haven't tested it much and html works too, but someone suggested that option, so I added it.
bbcode is what many forums use, including bay12.
html turned out to be arguably the most useful. Just save the file with a .html extension, open it in a web browser and refresh it every time you re-run the script. On a wide screen this also seems to provide the most space to view data.
LaTeX ouptut was dropped because html works well enough, but if someone really wants to make pdfs, I can see about adding it. I guess, I'd just emit the tables to be imported into another LaTeX document.
1) Plant script
The plant script does two things. It prints a series of dfhack getplants commands and, more importantly, prints tables with crops. It also colour-codes
good and
bad crops and some of their properties.
1.0 Options:
Options in general are case-insensitive, so usually doesn't matter if you use small or capital letters.
-h --help /h
Print help and exit. Obviously not the default behaviour, but help is also printed if you misspell an option.
-d --debug --nodebug
Enable or disable printing extra output, such as all the plant data. Off by default.
--colour --color --nocolour --nocolor
Enable or disable colouring the plant tables. On by default and doesn't work for text and csv outputs.
--nogetplants
Without this option, the script will print a few dfhack getplants commands to designate cutting of trees or collecting bushes. They are printed by default and exclude dye-only plants (which can be added with --yesdye).
--nocrops
Won't print crop tables. Printed by default. Unless you only want to see getplants macors, don't use this option.
--align --aligns
Print alignment of plants (Good / Evil / Savage / All). Since this is "All" for almost all plants, it is off by default. If enough people are interested, I can change it to print all biome RAW tokens, but they tend to occupy a lot of space.
--yield --yields
Print crop yields per year. (There are 12 months, 28 days each.) Off by default.
--yesdye --yesdyes
Include dye-only plants in getplants macros. Off by default.
--freq --freqency
Print frequencies of crops. The higher the frequency, the more wild plants of this type will grow. Off by default. Somewhat useful for plant gathering.
--o=<txt|csv|bbcode|html|wiki> --<txt|csv|bbcode|html|wiki>
Set the output type. The default is txt / csv
--
End options. Everything else is treated as a RAW file.
There is no option to set output file, so redirect the output stream instead.
1.1 Columns:
Name is the name of the plant, as seen in the game. It is green for plants that seem worth farming (high yeield, high value, high drink value, good products) and red for plants that seem inferior. This may be deceiving when those "bad" plants are used in custom reactions to make elixirs or something.
ID is the plant's ID, as used by the game internally. This is useful for # getplants command and for modders.
Value is the plant's value. It is taken into account when the plant is eaten or cooked and obviously also counts for buying plants from caravans.
Grows is growing duration in days.
Seasons displays the seasons when this plant can grow. N/A? means the plant has no seed template and therefore (probably?) can only be gained from plant gathering or trading. See the comment below about yields.
Yield is the maximum theoretical yearly yield of this plant. Because planting and harvesting isn't instantaneous, this value will never be achieved even for plants that grow all year. Yield of a plant is simply the number of season it grows times 28*3 (three months, 28 days each), divided by it's growdur in days. There are 12 growdur units per days and 1008 per season and the Grows column gives you growing duration in days. Note that unless said plant grows all year, the actual yield per year is capped by the integer part of theoretical yield; the fractions just give you some more wiggle room in planting and harvesting. Some mods may introduce other constraints, for example in Masterwork Mod most plants can only be planted in Spring. Just look at growdur and seasons and use common sense. It may be a good idea to leave the farm plot fallow in seasons when the plants won't have enough time to grow. You need to use --yield option to get this column.
Align is the alignment of the plant. It can be "Good", "Evil", "Savage" or "All" and is "All" for almost all the crops (that was a pun). The plants are already separated into aboveground and subterranean and I didn't include detailed biome information. You must use --align to get this column.
Freq is the frequency of this plant in the wild. You must use --freq to get this column.
Eat says if this plant is edible raw. Yes / No.
Cook says if this plant can be cooked. Yes / No.
Brew (Value) (Cook) is the name of the drink made from this plant as well as it's value. (Y) at the end means the drink can be cooked. N/A means there is no drink.
Biofuel This column is specific to Fallout Equestria mod and won't show up for mods where it doesn't exist. Basically alcohol can be distilled into biofuel which can be refined into flame-thrower fuel or coke for the furnaces.
Products (Value) Any products such as Extract (Flask or Barrel), Flour, Dye, Thread, Sugar, Leaves, Oil, Soap and Pressed are listed here, along with their values.
Hints columns was supposed to give hints what to do with this plant once you have it, but it doesn't work all that well. Take them with a grain of salt and edit them if you want to post a guide to your crops.
1.2 Comments:
The dye-only plants (that is inedible, unbrewable and with no products other than dye) are considered to be useless. Of course this isn't true and some mods may even add reactions that make good use of those plants, but I've developed a dislike for them in my forts. When I want dye, I can get it from caravans in sacks of 10, while milling gives me sacks of 3-5 and is cumbersome. Besides, I like to gather plants for drink variety and my plant stockpiles end up clogged with those damn Dimple Cups. This is why I don't include them by default in the # getplants macros and colour them red in the crop tables.
# getplants is a dfhack command for mass designating trees to be cut and plants to be gathered by ID. The manual is
here and I highly recommend reading through that entire document. Basically you can specify plant IDs or use options like "cut all trees of all kinds everywhere". This command is powerful, but you may want to use it with caution and even go as far as to manually remove designations or only select one plant type, because you don't want to end up with 2k+ jobs all over the map which will take years to complete.
Plants normally only grow in their native biomes, but crops can be planted in most (all?) biomes, at least I never had trouble farming non-native plants in temperate biomes.
1.3 Recommendations:
TODO
2) Creature script
Intro
2.0 Options:
OK, I really went overboard with synonyms for options for this one.
options and their rationale
-h --help /h
Print help and exit.
-d --debug
--nodebug
Off
Print debug information. Don`t print debug information.
--pdesc --desc
--nopdesc --nodesc
Off
Print descriptions of creatures in last column (verbose).
--pfake --fake
--nopfake --nofake
On
Print fake creatures table (transformations, etc. NOT foolproof)
--pvermin --vermin
--nopvermin --novermin
Off
Print table of vermin
--pmisc --misc
--nopmisc --nomisc
--pverbose_size --verbose_size
--nopverbose_size --noverbose_size
Off
Print the whole size information, not just maximum size.
--pfarm --farm
--nopfarm --nofarm
On
Do not print milk, yarn and egg columns.
--pmaxage --maxage
--nopmaxage --nomaxage
Off
Print max age
--empty --pempty
--noempty --nopempty
Off
Do not print empty columns, checked separately for each table.
--notes --pnotes
--nonotes --nopnotes
On
Do not print notes at the end of tables.
--sepname
--nosepname
Off
Print creature name in separate column, not below ID.
--syn --psyn --syndromes --psyndromes --syndrome --psyndrome --int --pint --interactions --pinteractions --interaction --pinteraction
--nosyn --nopsyn --nosyndromes --nopsyndromes --nosyndrome --nopsyndrome --noint --nopint --nointeractions --nopinteractions --nointeraction --nopinteraction
Off
Print some (basic) info on syndromes and interactions.
--freq --pfreq
--nofreq --nopfreq
On
Do not print frequency of creatures.
--biome --pbiome
--nobiome --nopbiome
Off
Print biome.
--skills --pskills --skill --pskill
--noskills --nopskills --noskill --nopskill
Off
Print natural skills (if any) for creatures.
--compact
--nocompact
No
Combine some columns to preserve space.
--man --men --pman --pmen
--noman --nomen --nopman --nopmen
On
Do not print animal men (ID ends with _MEN)
--giant --pgiant
--nogiant --nopgiant
On
Do not print giant variants (ID starts with GIANT_)
--robot --robots
--norobot --norobots
Off / separate
Merge animal and robot tables.
--sort
--sort=<column>
--nosort
"NAME"
Sort by column. NAME by default. Valid: ID, MAX_SIZE, SPEED, PETVALUE, NAME, FREQ.
--allskills --pallskills --allskill --pallskill --nobestskills --nopbestskills --nobestskill --nopbestskill
--noallskills --nopallskills --noallskill --nopallskill --bestskills --pbestskills --bestskill --pbestskill
On / short
Print all natural skills, even if they repeat for castes.
--na=<text>
"--"
Text for empty columns. -- by default.
--extra=<text> --pextra=<text>
Off
Print an extra column filled with text (TODO) at end of tables.
--all --pall
$pdesc = 1; $pmisc = 1; $pfarm = 1; $pfreq = 1; $psyn = 1; $pbiome =1; $pskill =1; $pmaxage =1;
$pfake = 1; $pvermin = 1; $pnotes = 1; $compact =1; $pmen = 1;
Print everything.
--eval=<text> --expr=<text>
Eval <expr> for each creatures and print only those that match.
--o=<txt|csv|bbcode|html|wiki> --<txt|csv|bbcode|html|wiki>
--
End options. Everything else is treated as name of DF RAW file.
2.1 Columns:
can change which columns get printed or even glued together, but not their order. The order is sane and it was easier to write this way.
2.2 Comments:
examples for --expr
2.3 Suggested uses:
Suggestions:
--desc for looking for spelling mistakes in creature descriptions.
3) Weapon script
Highlights errors - use a search and replace on color tags to get rid of it.
3.0 Options
3.1 Columns
3.2 Comments
3.3 Suggested uses:
Author
Topic: Perl scripts that read RAWs and present tables of data (plants, creatures) (Read 11258 times)