Bay 12 Games Forum

Dwarf Fortress => DF Modding => Utilities and 3rd Party Applications => Topic started by: Roses on August 19, 2013, 06:30:46 pm

Title: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Roses on August 19, 2013, 06:30:46 pm
HAS NOT BEEN UPDATED TO .40x! WILL BE UPDATED SHORTLY.

So I have written a couple different python utilities for analyzing and modifying dwarf fortress raws. I figured I would start uploading them as I make them public worthy.

Modding with Excel!
 
Modding with Excel is a new set of python scripts I have been working on that allows you to move data between .txt and .csv (comma separated value) files. This allows you to add entries via Excel (or your favorite CSV reader) easily and cleanly, allowing for sorting and quick analysis of entries. Now working for items and inorganics. There are included csv files so you can see how to set them up.
Screen shot for MWE:
Spoiler (click to show/hide)

PLEASE NOTE: In the columns with more than one entry the python program is expecting enters between the entries (i.e. \n). If people feel this is too limiting I can have it acknowledge the entries in a different way.

MWE_ITEMS: This utility will handle all item types (Weapons, Ammo, Armor, Gloves, Helm, Pants, Shoes, Food, Instruments, Tools, Toys, Siege Ammo, and Trap comps) by making separate files for each. To use, simply place the python script in your raws folder and run. It will prompt you if you wish to copy from Raw to CSV or CSV to Raw, choose by entering 1 or -1.

MWE_INORGANICS: This is a preliminary utility. It has all the function you need to create inorganics in Excel, but it doesn't break up the inorganic tokens by what they are commonly used for (i.e. there is a column for MEAT_NAME for all them). I am planning on adding some more logic to the program so that the csv files aren't so large, but currently it works fine (also supports syndromes in your inorganics). Functions the same as MWE_ITEMS.

MWE_PLANTS: Another preliminary utility that suffers from the same drawbacks as MWE_INORGANICS (although it does successfully parse the plants into trees, grasses, and crops) which makes the csv files more manageable.

Both MWE_INORGANICS and MWE_PLANTS works better to generate raws from then as a means for sorting data. More work needs to go into making the csv files generated useful.

MWE_CREATURES: This utility differs from the others in that it only works in one direction (namely to go from CSV files to RAW files). Because of the complexity of creatures I have also introduced a new feature, Templates! (I will be adding this feature into the other MWE scripts as well). Templates allow you to group your creatures with similar bodies, attacks, castes, and traits, while still giving you the ability to modify them. A sample attack template is provided below;

Code: [Select]
[TEMPLATE:BITE]
[ATTACK:BITE:BODYPART:BY_CATEGORY:#ARG1]
[ATTACK_SKILL:BITE]
[ATTACK_VERB:bite:bites]
[ATTACK_CONTACT_PERC:#ARG2]
[ATTACK_PENETRATION_PERC:#ARG3]
[ATTACK_FLAG_EDGE]
[ATTACK_PRIORITY:#ARG4]
[ATTACK_FLAG_CANLATCH]

you can reference this template by providing a line in the CSV file under the attack column BITE:BEAK:100:100:MAIN. Templates are able to handle an infinite number of arguments, but for ease of use it is best to keep your arguments limited in number. An example CSV file (creatures.csv) is included to show case the use of this script. In addition four template files are included (templates_ATTACK.txt, templates_BODY.txt, templates_CASTE.txt, templates_TYPE.txt) each with example templates that are referenced in the CSV file. This script allows fast an easy creation of new creatures. (NOTE: Currently only the ATTACK and CASTE templates support variable arguments, the BODY template uses specific arguments for materials and tissues, and the TYPE template has no argument support. This will be fixed in a later version)

Misc Scripts

ADD_CASTES: This utility adds caste information to creatures raws. It only modifies files specified and adds castes that are user specified to the creatures in those files (UPDATE: There is now an external file named creature_iist.txt that lets the user specify which creatures are actually modified). This allows adding things like giant variations directly to the creature instead of having them as a separate creature. This script requires input from the user. To use open the script with some text editor (I personally use VIM) and modify the required fields (there is short comments that explain what to do). Because of the extra level of complication with this script files in the objects folder will not be overwritten, instead a new file will be created in the raws folder which can then be checked to make sure it did what you wanted, and then put into the objects folder.
Screen shot for ADD_CASTES:
Spoiler (click to show/hide)

ADD_SIZE: This utility reads all of the creature raws and puts the max size of the creature (as listed in the creature raws) in the creatures description. This utility currently has several limitations. It does not add a size to any creature that doesn't have a size listed in the raws (most creatures that use the creature variation templates, like GIANT and ANIMAL_PERSON, don't have their own size listed), and it only lists the largest size in the raws. Also, subsequent runs will not change the values in the description, even if the values in the raws are changed. To run, simply put the python file in the raws folder and run it, it will modify all creature files in the objects folder. As always, it is a good idea to back up your files before running any utility that directly modifies files.
Screen shot for ADD_SIZE:
Spoiler (click to show/hide)

ORGANIZE_ENTITY: This utility reads the entity files and separates each entity into their own files. In addition it organizes the file into different components (e.g. Equipment, Items, Behavior, Ethics, Flavor, etc...) This allows for easier navigation of the entity files. To run, simply put the python file in the raws folder and run it, it will modify all creature files in the objects folder. As always, it is a good idea to back up your files before running any utility that directly modifies files.
Screen shot for ORGANIZE_ENTITY:
Spoiler (click to show/hide)

SPLIT_CVARIATIONS: This utility reads through all of the creatures raws and seperates out the ANIMAL_PERSON and GIANT creature variations into their own files. This allows for easy deletion of these creatures for those that don't like having them in the game, or for easy manipulation.  To run, simply put the python file in the raws folder and run it, it will modify all creature files in the objects folder. As always, it is a good idea to back up your files before running any utility that directly modifies files. (UPDATE: Fixed a bug that was brought to my attention by CryptoCactus)
Screen shot for SPLIT_CVARIATIONS:
Spoiler (click to show/hide)

SPLIT_REACTIONS: This utility reads through all of the reaction raws, and separates them into individual reaction files. I use this for taking apart mods where I just want to use one specific building. To run it all you need to do is place the python file in the raws folder and it will spit out an individual file for each building mentioned in the raws. I have personally used it with Masterwork and Genesis Reborn with no issues. Please note that some mods make use of hundreds of buildings, so it will generate hundreds of text files. Only use if you are wanting a specific buildings reactions separated out.
Screen shot for SPLIT_REACTIONS:
Spoiler (click to show/hide)

SPLIT_VERMIN: Same as SPLIT_CVARIATIONS except it pulls out vermin.

More utilities will be added as I make them fit for public use.
http://dffd.wimbli.com/file.php?id=7916 (http://dffd.wimbli.com/file.php?id=7916)

Python 3+ is required for use. Please let me know if you run into an trouble with any of the utilities
Title: Re: Python Utilities for DF (v1.0)
Post by: Meph on August 20, 2013, 02:08:31 pm
I think I do like that screenshot quite a bit. With some extra work I could seperate different features of the mod and offer them as a stand-alone download for vanilla DF.
Title: Re: Python Utilities for DF (v1.0)
Post by: Roses on August 20, 2013, 02:20:59 pm
I think I do like that screenshot quite a bit. With some extra work I could seperate different features of the mod and offer them as a stand-alone download for vanilla DF.

The python code itself is fairly simple and can easily be extended to other raw files (I've experimented with splitting out Mega and Semi-Mega beasts from creature files, economic vs war metals, and  above ground vs under ground plants). Pretty much anything that has a distinct raw token (or tokens) can be extracted and split into separate files.
Title: Re: Python Utilities for DF (v1.0)
Post by: Meph on August 20, 2013, 02:56:01 pm
Thats nice. Would help me sort stuff for my UI. But I dont think that I can export complete features, because too many things interlock.

But yeah, sorting only aboveground vs underground plants/creatures, and only megabeasts, only creatures with interactions... thats all interesting. :)
Title: Re: Python Utilities for DF (v1.0)
Post by: Roses on August 20, 2013, 03:10:53 pm
Thats nice. Would help me sort stuff for my UI. But I dont think that I can export complete features, because too many things interlock.

But yeah, sorting only aboveground vs underground plants/creatures, and only megabeasts, only creatures with interactions... thats all interesting. :)

I am actually working on a more robust version that can look through all the raws and grab all of the dependencies. For example, say I want to use your runesmith reactions, it would go through the raws, give me the buildings required, any inorganic tokens, items, materials, and other things needed. I am just having trouble with how to approach reaction classes, and how to make sure everything needed gets included (first run through might tell me x is needed, but x might need y and y might need z etc...)
Title: Re: Python Utilities for DF (v1.1)
Post by: Roses on August 20, 2013, 03:28:17 pm
A new utility has been added, also a bug in the STRIP utility has been fixed.
Title: Re: Python Utilities for DF (v1.1)
Post by: Meph on August 20, 2013, 03:39:58 pm
A while ago I tried to make a flowchart for the library... but it was impossible.

Ok, you can write on paper, vellum or papyrus. Thats 3 sources from 3 buildings, from plants, leather, or wood. So do I include woodcutting into the chart? How about treefarms as a source of wood? Or caravans? The Merchant Stalls I added can also sell you wood. Or you raid elves, they have wooden items to steal. Papyrus comes from plants... so herbalism/gathering? And farm-plots. And the Herbalists Lab can also make plants. Where do the dependencies end? ^^

Anyway, good luck :) I am still curious about your "soul system / kill dragon learn dragonfire - private mod" ;)
Title: Re: Python Utilities for DF (v1.1)
Post by: Roses on August 20, 2013, 03:48:42 pm
Anyway, good luck :) I am still curious about your "soul system / kill dragon learn dragonfire - private mod" ;)

Yeah, it can get really really complicated, right now I am thinking of just doing the absolute minimum (i.e. all tokens are present, and DF won't give you any errors), which would mean as long as they aren't special leather, plants, or wood, they wouldn't get included in the extraction of the library.

I have a couple python scripts for adding souls to all creatures and such that I am prettying up and getting rid of errors so that I can post them. Trying to come up with a reasonable automation system so that you don't have to go through all the creatures and assign their soul level individually. It is pretty straight forward for some creatures (you can just say all Megabeasts have a level 5 soul, Semi-Mega Beasts have level 4) but other creatures aren't quite so easy. I suppose I should post my own raws at some point, but I keep making changes and additions, and never seem to be quite happy with them.
Title: Re: Python Utilities for DF (v1.1)
Post by: Meph on August 20, 2013, 03:59:01 pm
If you figure out a nice way to use corpses, let me know... its driving me mad atm.

I want to use corpses of invaders in specific reactions, but I cant target them... I have to give them itemcorpses, which works. Then I wanted to use an itemcorpse of a civ-member, but it still counts as corpse, not as item. I even transformed a dwarf into a completely different creature using TrueTransform, then kill it, then use its itemcorpse... but no, its still a corpse that gets buried. So that idea I have to scrap completely.

Same thing goes for using DISPLAY_TILE:TILE on civ-members. It seems that the profession graphics overwrite the added display tile from syndromes. So instead a dwarf miner will look like a dwarf miner, even if he has DISPLAY_TILE:TILE:64, which is the tile humans use. Damnation and curses on DF for doing stuff like that. Have to scrap three different ideas I had because of that.  ::)
Title: Re: Python Utilities for DF (v1.2)
Post by: Roses on September 11, 2013, 11:49:42 am
New scripts added
Title: Re: Python Utilities for DF (v1.3)
Post by: Roses on November 04, 2013, 12:54:46 am
New scripts added and a fix to a previous script brought to my attention by CryptoCactus.

Please let me know if you have any ideas for new scripts or any trouble with these.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.0)
Post by: Roses on December 01, 2013, 05:45:13 pm
New script added, the first of many in the Modding with Excel set! Read the first post for more information.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.1)
Post by: Roses on December 02, 2013, 08:45:53 pm
Modding with Excel now handles inorganics as well as items. Sample csv files have been included so that you can see how to better utilize these tools.

Up next are plants!
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Roses on December 03, 2013, 10:09:35 pm
Plants and inorganics are now both preliminarily handled. (See first post for more information about the drawbacks and difficulties)

Next up are the much dreaded creatures. They will work slightly differently by supplying templates (e.g. specifying BIRD will put in all the raws that are in the BIRD template). Will make assigning creatures and their materials much easier, but will not allow RAW -> CSV conversion (i.e. will be CSV -> RAW only)
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Maklak on December 05, 2013, 11:17:30 am
I'm working on an off on a set of similar RAW analysers in Perl. They only print tables of data, though; I have no ambitions to have them aid modding directly. CSV is one of my output formats. http://www.bay12forums.com/smf/index.php?topic=126953.0

My question is, how do you handle a list of something in a CSV file? For example an item has a set of properties (ID, name, size, whatever) that I can separate by commas. What do I do to get a column that has multiple things, like Attacks?  What did you do to display them so nicely in a spreadhseet?
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Roses on December 05, 2013, 11:38:27 am
If you make a sublist like {Attack1,Attack2,Attack3} and then join the sublist with new line commands (i.e. \n) then in a pure data format they will look like Attack1\nAttack2\nAttack3 but in a visualizer like Excel or Open Office they will look like;

Attack1
Attack2
Attack3

The only down side to this method is that it can sometimes cause the default width of a column to be large (but just re-sizing it in Excel fixes that problem)
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Maklak on December 05, 2013, 11:55:27 am
I just tried a CSV file with

Code: [Select]
ID, name, Attack
AXE, Axe, Att1\nAtt2\nAtt3
HAMMER, Hammer, Att4\nAtt5


And it didn't work the way I wanted it to in Open Office. Would you please paste a few sample lines from your csv weapon files? It may be just a problem with my Opern Office being old.

EDIT: Never mind, I found it. I run OO, copied a "\n" from an open text document into a spreadsheet cell by selecting end of line and beginning of next line, saved it as a CSV and it now works.

Code: [Select]
"ID","Name","Att"
"AXE","Axe","Att1
Att2
Att3"
"HAMMER","Hammer","Att4
Att5"

It looks stupid and I'd rather have a more elegant way to do it, but this is definitely something I want to patch into my scripts. It needs those quotation marks too.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Roses on December 05, 2013, 12:12:22 pm
I believe you need to put quotes around the attacks, I'm not at my computer right now, but I will paste when I get there.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Roses on December 05, 2013, 02:56:56 pm
EDIT: Never mind, I found it. I run OO, copied a "\n" from an open text document into a spreadsheet cell by selecting end of line and beginning of next line, saved it as a CSV and it now works.

Code: [Select]
"ID","Name","Att"
"AXE","Axe","Att1
Att2
Att3"
"HAMMER","Hammer","Att4
Att5"

It looks stupid and I'd rather have a more elegant way to do it, but this is definitely something I want to patch into my scripts. It needs those quotation marks too.

Yeah, that's how it will look when you open it up in notepad or something like that. If you open it in OO though it should look just like mine does.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Maklak on December 05, 2013, 04:03:43 pm
> Yeah, that's how it will look when you open it up in notepad or something like that. If you open it in OO though it should look just like mine does.
It does work in OO, but I needed to know how it looks in notepad to be able to write output text like that. I now have another reason to treat txt and csv tables differently.
Even if your set of scripts is better for the ability to put changes back into RAWs, I still want to write my own RAW2table scripts and improve the output in any sensible way I can. For example I add explanations and comments under the tables for most output formats, but they just don't work out for CSV.

> Please let me know if you have any ideas for new scripts.
Well, I did plants, creatures and weapons for my scripts. What remains are (at least) armour, materials, reactions (sorted by buildings), buildings (with items required) and interactions / syndromes. Then the idea is to refactor and merge everything and have a big, modular script that has all kinds of complicated hash tables to store all this (plus entities and whatever else seems like a good idea), then print a whole bunch of html files with tables with links, so that for example walking along a chain of interactions is easy. It shouldn't be that hard to always put an anchor or a link on any ID anywhere.
If you do any of those with editing, more power to you.

BTW, this is in some ways similar to what you or I are doing, but targeted at players, not modders: http://www.bay12forums.com/smf/index.php?topic=131301

So well, thanks for making me realise, multiline cells in a spreadsheet from a CSV are possible and that quotations are crucial. Sorry for just stealing some spotlight in this thread and not being of help.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Roses on December 05, 2013, 04:22:35 pm
No problem, glad to be of help. Sounds like what you are doing is pretty different from what I intended. Definitely sounds useful though, especially the html files with links and such.

 I mostly just made these scripts because I like working in spreadsheets, and I think it makes adding in new items and such easier than parsing through the raws and finding things of similarity (also decreases the chance that I will somehow screw up the raws with copying and pasting).
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.2)
Post by: Meph on December 07, 2013, 05:37:26 am
Watching with interest :)
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Roses on December 09, 2013, 11:23:44 pm
Modding with Excel now includes Creatures!

MWE_CREATURES: This utility differs from the others in that it only works in one direction (namely to go from CSV files to RAW files). Because of the complexity of creatures I have also introduced a new feature, Templates! (I will be adding this feature into the other MWE scripts as well). Templates allow you to group your creatures with similar bodies, attacks, castes, and traits, while still giving you the ability to modify them. A sample attack template is provided below;

Code: [Select]
[TEMPLATE:BITE]
[ATTACK:BITE:BODYPART:BY_CATEGORY:#ARG1]
[ATTACK_SKILL:BITE]
[ATTACK_VERB:bite:bites]
[ATTACK_CONTACT_PERC:#ARG2]
[ATTACK_PENETRATION_PERC:#ARG3]
[ATTACK_FLAG_EDGE]
[ATTACK_PRIORITY:#ARG4]
[ATTACK_FLAG_CANLATCH]

you can reference this template by providing a line in the CSV file under the attack column BITE:BEAK:100:100:MAIN. Templates are able to handle an infinite number of arguments, but for ease of use it is best to keep your arguments limited in number. An example CSV file (creatures.csv) is included to show case the use of this script. In addition four template files are included (templates_ATTACK.txt, templates_BODY.txt, templates_CASTE.txt, templates_TYPE.txt) each with example templates that are referenced in the CSV file. This script allows fast an easy creation of new creatures. (NOTE: Currently only the ATTACK and CASTE templates support variable arguments, the BODY template uses specific arguments for materials and tissues, and the TYPE template has no argument support. This will be fixed in a later version)
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: milo christiansen on December 12, 2013, 11:54:19 am
Just a suggestion: If you were to use Rubble as a backend (by generating your raws with Rubble template calls embedded, then calling Rubble to finish the job) you could get a powerful template system, a module system, and compatibility with a broad assortment of raw tweaks and libraries, for basically "free".

It wouldn't even be hard to do, for the bite example:
Code: [Select]
{!TEMPLATE;ATTACK_BITE;bp;contact;penetrate;priority;
[ATTACK:BITE:BODYPART:BY_CATEGORY:%bp]
[ATTACK_SKILL:BITE]
[ATTACK_VERB:bite:bites]
[ATTACK_CONTACT_PERC:%contact]
[ATTACK_PENETRATION_PERC:%penatrate]
[ATTACK_FLAG_EDGE]
[ATTACK_PRIORITY:%priority]
[ATTACK_FLAG_CANLATCH]
}
Then insert calls to ATTACK_BITE like so: {ATTACK_BITE;BEAK;100;100;MAIN} and let Rubble do the work :)
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Roses on December 12, 2013, 04:03:19 pm
It's already compatible with Rubble in a way, but I don't want to require another program since It already required Python. Also, I am planning some new features that won't really fit with Rubble.

One thing I have always been interested in is a method for making raws more readable (indenting child tags automatically and such). Is that a feature you have thought about for Rubble? I was planning on doing it in Python, but if you already have a system in place theres no reason for me to reinvent the wheel.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: milo christiansen on December 13, 2013, 11:12:06 am
Nope, no pretty printer.
My philosophy about pretty printers is that if you need one, something is wrong with your generation. Of course if you are generating EVERYTHING instead of just sections some kind of pretty printer is a must I guess.

I did have one (for another project), but it was VERY basic and only supported two levels of indentation. It would be trivial to write a better one than what I had.

I understand that Blast had one, it's written in perl but you should be able to port it to python without too many problems :)
(The thread for Blast is near the end of the second page of this board)
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Roses on July 15, 2014, 12:57:32 pm
Note sure if anyone besides me uses any of these scripts, but in the lull between the release and DF once again being playable, I have begun updating them for the new version.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Urist McGoombaBrother on January 19, 2015, 02:26:52 pm
That project sounds plain awesome. But I have never worked with python and you said the update to DF v40.x isn't ready so far anyway.

However, you also stated something about testing. Just in case you exported already the RAWs of following files in Excel during testing, do you mind uploading those excel-files somewhere? That would be a tremendous help. Thx.

plant_crops.txt
plant_garden.txt
plant_grasses.txt
plant_new_trees.txt
plant_standard.txt
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Roses on January 19, 2015, 03:50:27 pm
Oh man, I'm going to be perfectly honest, I had completely forgotten about this project. But it is one I want to start up again. I think I left it in a decent state, but I know I haven't added all of the new 40x stuff. Luckily all the other scripts should still work, its just the MWE scripts that I will need to look through again. Consider it added to my todo list.
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: jason0320 on July 17, 2015, 10:13:16 am
Hello? Is this going to be continued?
Title: Re: Python Utilities for DF now with Modding with Excel! (v2.3)
Post by: Roses on July 22, 2015, 10:25:20 am
Yes it will be. Continued and improved eventually. Unfortunately I am very swamped with work and expect to be for a while, plus I have several other things on my to-do list. So I can't really give a good time frame, sorry!