Bay 12 Games Forum

Dwarf Fortress => DF Modding => Utilities and 3rd Party Applications => Topic started by: madk on May 18, 2015, 07:37:53 pm

Title: PyDwarf 1.1.4: Werebeast
Post by: madk on May 18, 2015, 07:37:53 pm
(http://pineapplemachine.com/pydwarf/logo_transparent.png)

Download PyDwarf 1.1.4 (https://github.com/pineapplemachine/PyDwarf/releases/tag/v1.1.4)
View PyDwarf on GitHub (https://github.com/pineapplemachine/PyDwarf)


For players, PyDwarf is an easy way to manage your preferred mods. Never worry about incompatible or obsolete mods again! (Or, at least, not nearly as often.) And for modders, PyDwarf is your new best friend. PyDwarf's mod manager exposes a very handy API for interacting with the raws. Here's an example of how effortless it is to customize raws even from a python console:

Code: [Select]
>>> import raws
>>> df = raws.dir('raw/objects')
>>> elf = df.getobj('CREATURE:ELF')
>>> description = elf.get('DESCRIPTION')
>>> print description
[DESCRIPTION:A medium-sized creature dedicated to the ruthless protection of nature.]
>>> description.setarg('A medium-sized creature undeserving of life.')
>>> df.write('raw/objects')

client-170:PyDwarf pineapple$ grep 'CREATURE:ELF' raw/objects/creature_standard.txt -A 4
[CREATURE:ELF]
    [DESCRIPTION:A medium-sized creature undeserving of life.]
    [NAME:elf:elves:elven]
    [CASTE_NAME:elf:elves:elven]
    [CREATURE_TILE:'e'][COLOR:3:0:0]

Here's just a few examples of the mods included with this release of PyDwarf:
Normally you'd be bound to run into compatibility issues when trying to install so many of these mods at once, and you'd surely have to wait for authors to update their mods before reinstalling everything whenever Dwarf Fortress updates. Not so with PyDwarf. PyDwarf is highly capable of avoiding conflicts between mods, and it cares little about which version of Dwarf Fortress you're modifying. Not only does PyDwarf make it much easier for the player to install mods, it also makes it easier for the modder to craft something excellent. Instead of editing text files on your own, PyDwarf allows you to write scripts which edit them for you: What this means for you is an enormous reduction in tedium.

Here's a step-by-step tutorial (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/introduction.md) to guide you through your first time configuring and running PyDwarf, and here's another tutorial (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/modding.md) to help you get started creating your first mod. Also, here's a handy list of scripts (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/scripts.md) with which PyDwarf comes packaged. And here's where all those scripts are actually located (https://github.com/pineapplemachine/PyDwarf/tree/master/scripts), in case you'd like to see what their innards look like.

PyDwarf is cross-platform. Only Python 2.7 (https://www.python.org/downloads/release/python-2713/) is required to run PyDwarf.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Max™ on May 18, 2015, 09:51:37 pm
Looks pretty neat, naturally just as I'm getting comfortable screwing around in lua I'm looking at this trying to see how it works like:
Spoiler (click to show/hide)
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Putnam on May 18, 2015, 10:57:08 pm
Yeah, this'll probably be way better than find in files.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 19, 2015, 03:14:09 am
Quote from: Max™ on May 18, 2015, 09:51:37 pm
Looks pretty neat, naturally just as I'm getting comfortable screwing around in lua I'm looking at this trying to see how it works like:

Ha, as far as mod creation goes it may seem a bit complex but that's partly due to the fact I haven't made much in the way of documentation yet. If you know the basics of Python - which are easier to pick up than Lua, in my personal opinion - then PyDwarf should be really very easy to use as a modder. I'd like to also make things easier on users, too, though I'm not yet sure how I'm going to do that.

Quote from: Putnam on May 18, 2015, 10:57:08 pm
Yeah, this'll probably be way better than find in files.

I should hope so!
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: DragonDePlatino on May 19, 2015, 01:14:52 pm
Ohh! I spend a lot of time doing repetitive Notepad++ replace functions, so this could certainly come in handy. I'm not really a programming guy, so Python is the only language I've ever gotten a grasp on.

Here's a suggestion...why not structure your documentation like the PyGame documentation (http://www.pygame.org/docs/)? You list a command, what the output is, what it's best use is and possible values for your arguments. It's really straightforward for newbies.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 19, 2015, 01:43:05 pm
Quote from: DragonDePlatino on May 19, 2015, 01:14:52 pm
Ohh! I spend a lot of time doing repetitive Notepad++ replace functions, so this could certainly come in handy. I'm not really a programming guy, so Python is the only language I've ever gotten a grasp on.

Here's a suggestion...why not structure your documentation like the PyGame documentation (http://www.pygame.org/docs/)? You list a command, what the output is, what it's best use is and possible values for your arguments. It's really straightforward for newbies.

This will definitely be better-suited than find/replace. And making proper documentation is a priority, but I want to make the querying functionality more final before I start to document it, otherwise I'll be changing it too frequently to be practical. I'll mind to look into PyGame for an example of how to structure the docs.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 20, 2015, 05:34:01 am
I added a much more complex example script to help show just how powerful PyDwarf can be: https://github.com/pineapplemachine/PyDwarf/blob/master/scripts/pineapple/pydwarf.stoneclarity.py

Using the default settings, stoneclarity will cause ores and gems and such to assume standard appearances. Using this script, there'll be no more confusions for beginners like "Is that coal or is it a gem?" and "Why does cobaltite look like a metal?"
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Putnam on May 20, 2015, 01:29:00 pm
Quote from: madk on May 20, 2015, 05:34:01 am
"Why does cobaltite look like a metal?"

(mostly because it is)
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 20, 2015, 01:45:55 pm
"Why does cobaltite look like a metal?"

"Why does cobaltite look like something I can smelt?"
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: notfood on May 21, 2015, 12:24:38 am
This is very handy. Thanks for sharing. I'm often overwhelmed by the amount of search and replace I have to do, this is going to make things easier.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Smurfton on May 23, 2015, 07:04:25 pm
Quote from: madk on May 19, 2015, 03:14:09 am
'd like to also make things easier on users, too, though I'm not yet sure how I'm going to do that.
To do this, create two functions, void install(string filename) and void uninstall(string filename). Both of them will need to be part of an object (Python allows objects, right?) for ease of use purposes. The object should contain a private variable which has a list of mods installed. You'll also need a copy of the raws for reverting purposes.
install() would accept a filename, open that file (which hopefully contains the mod), read it, and eval() the mod contents. Once the mod is installed, append the filename to the private list variable.

uninstall() would accept a filename, make sure that was in the list of installed mods, remove it from the list, copy the list to a temporary variable, delete the remainder of the list, copy the unmodified raws over, then run install() on every item. Alternate solution: have a version of install() that doesn't append to the installed list, and call that.

As an extra, you can overload (Python does that, right?) both so that if no arguments are given, install() is run on a specific folder, and uninstall copies over unmodified raws, and clears the list.

Probably a better way to do uninstall(), but that should work, as a start.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Putnam on May 23, 2015, 07:13:04 pm
Python is object-oriented.

"Overloading" there seems to be referring to duck typing, which Python has. Or it refers to one function being able to operate on different types by defining a function of the same name for multiple types of arguments, which is not necessary since Python is dynamically-typed.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: PeridexisErrant on May 23, 2015, 08:12:00 pm
PTW.  If easier for users is a goal, maybe this could be incorporated as an upgrade to PyLNP's mod merger? (http://)  The only requirement is that it take whole modified raw files as inputs and outputs.

Quote from: Smurfton on May 23, 2015, 07:04:25 pm
<snip>
Setting up default arguments is certainly a good idea, but the rest of the psudeocode is more C (or Java?) than Python.  A detailed response would be more nitpicks than productive.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: lethosor on May 23, 2015, 08:26:00 pm
Quote from: Putnam on May 23, 2015, 07:13:04 pm
"Overloading" there seems to be referring to duck typing, which Python has. Or it refers to one function being able to operate on different types by defining a function of the same name for multiple types of arguments, which is not necessary since Python is dynamically-typed.
It sounds to me like Smurfton is referring to functions taking a variable number of arguments (which is also possible).

PTW
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Putnam on May 23, 2015, 08:31:15 pm
Yeah, with *args and **kwargs.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 23, 2015, 09:27:08 pm
Scripts are already allowed to take whatever arguments that Python will allow them to take, though there's not currently a mechanism for allowing *args. (But that's okay, because it's exactly the same thing to just pass a list or other iterable as a named argument.) I don't believe it's practical to have an "uninstall" functionality - making the changes is much easier than reversing them. Instead the user would be expected to keep backups, and if they decide they want to remove some mod or change the load order they would need to re-run the manager with the updated settings.

I understand the documentation is shitty - I'm working on that - but it you might look into how it works before speculating about how to change it. You may find that many of the things you're asking for are there already.

For me, concern regarding ease-of-use is more to do with making it so that users won't need to have programming knowledge to use the mod manager. Eventually I'd like this to mean having easy ini or ini-like configuration files as an option, perhaps even a more graphical interface that could abstract aspects of configuration and management. For usability from the perspective of modders, I believe what I have is a solid foundation. There are improvements I want to make but the essentials are present if not a little buggy in places. Also worth nothing that it's my on the todo list to write diff-based mod functionality into PyDwarf as a mod itself that accepts existing diff-based mods as input and merges them as best it can. (I believe this would actually be simple compared to the querying functionality that's at the core of PyDwarf.)
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 26, 2015, 08:38:56 am
I'm interested in helping modders to rewrite their mods using PyDwarf. If you have a mod you'd like to end up in the official repo, please let me know! I'll work with you closely to make sure everything gets done smoothly.
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: Button on May 26, 2015, 10:17:10 pm
Ignore me, I asked a dumb question that I answered myself by looking at the code more
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 27, 2015, 01:49:42 pm
I wrote a PyDwarf script which is able to fairly intelligently merge modified raws provided the DF versions all line up. (Understanding changes based on disparate vanilla raws would be a highly nontrivial problem.) It should be listed as the first script to run and given, as an argument, paths to each file or directory that should be merged and applied.

https://github.com/pineapplemachine/PyDwarf/blob/master/scripts/pineapple/pydwarf.diff.py
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on May 29, 2015, 09:42:04 pm
There's a tutorial now! And I've generally gone through a lot effort to make PyDwarf more accessible to users and modders.

https://github.com/pineapplemachine/PyDwarf/blob/master/tutorial.md
Title: Re: PyDwarf: Edit your raws with the power of Python
Post by: madk on June 03, 2015, 06:35:45 am
(http://pineapplemachine.com/pydwarf/terminal_example.gif)
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: Button on June 03, 2015, 11:00:33 am
Hey madk,

Does this support changing individual tokens within tags?
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: madk on June 03, 2015, 11:21:02 am
Quote from: Button on June 03, 2015, 11:00:33 am
Hey madk,

Does this support changing individual tokens within tags?

Could you elaborate? I always understood the terms "token" and "tag" to be synonymous when speaking of the raws.

Do you mean changing the arguments of a token? i.e. a token being like [VALUE:ARGUMENT_0:ARGUMENT_1:...:ARGUMENT_n]? If so then absolutely. A raws.token object has a value attribute, corresponding to the first item in the list delimited by colons, and an args attribute which is a list of all the other items. There are also helper methods like token.nargs(), token.getarg(), and token.setarg() for things like convenience and input sanitization.
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: Button on June 03, 2015, 11:37:17 am
Quote from: madk on June 03, 2015, 11:21:02 am
Could you elaborate? I always understood the terms "token" and "tag" to be synonymous when speaking of the raws.

Do you mean changing the arguments of a token? i.e. a token being like [VALUE:ARGUMENT_0:ARGUMENT_1:...:ARGUMENT_n]? If so then absolutely. A raws.token object has a value attribute, corresponding to the first item in the list delimited by semicolons, and an args attribute which is a list of all the other items. There are also helper methods like token.nargs(), token.getarg(), and token.setarg() for things like convenience and input sanitization.

Yes, that's what I meant. I can't speak to the nomenclature, but my understanding has been that a tag is a full [...], and a token is anything between the [ and ], separated by :s, including the part you've designated VALUE.

Are the arguments identified only by their index? How would one go about, say, setting all the time ranges in all GROWTH_PRINT tags to ALL? Or is the range identified as a single "range-type" argument, even though it's got a semicolon in?
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: madk on June 03, 2015, 11:50:42 am
Quote from: Button on June 03, 2015, 11:37:17 am
Are the arguments identified only by their index? How would one go about, say, setting all the time ranges in all GROWTH_PRINT tags to ALL? Or is the range identified as a single "range-type" argument, even though it's got a semicolon in?

As an effort to make PyDwarf itself as version-agnostic as possible, there is very little implicit handling of anything. None so far of arguments: Everything separated by a colon is a separate argument as far as PyDwarf is concerned, neither does PyDwarf try to discern names or purposes for arguments; they are identified only by their indexes. Here's an example of how to do what you're thinking of:

Code: [Select]
>>> import raws
>>> df = raws.dir(path="raw")
>>> quarrybush = df.getobj('PLANT:BUSH_QUARRY')
>>> growthprint = quarrybush.get('GROWTH_PRINT')
>>> print growthprint
[GROWTH_PRINT:0:6:7:0:0:NONE]
>>> print growthprint.args
['0', '6', '7', '0', '0', 'NONE']
>>> growthprint.args[4] = 'ALL'
>>> growthprint.args[5] = '1'
>>> print growthprint
[GROWTH_PRINT:0:6:7:0:ALL:1]
>>> print growthprint.args
['0', '6', '7', '0', 'ALL', '1']

On nomenclature: "token", "value", "args" seemed to me like the most appropriate terms, coming from my experience with parsers.
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: Button on June 03, 2015, 01:45:15 pm
Quote from: madk on June 03, 2015, 11:50:42 am
Quote from: Button on June 03, 2015, 11:37:17 am
Are the arguments identified only by their index? How would one go about, say, setting all the time ranges in all GROWTH_PRINT tags to ALL? Or is the range identified as a single "range-type" argument, even though it's got a semicolon in?

As an effort to make PyDwarf itself as version-agnostic as possible, there is very little implicit handling of anything. None so far of arguments: Everything separated by a colon is a separate argument as far as PyDwarf is concerned, neither does PyDwarf try to discern names or purposes for arguments; they are identified only by their indexes. Here's an example of how to do what you're thinking of:

Code: [Select]
>>> import raws
>>> df = raws.dir(path="raw")
>>> quarrybush = df.getobj('PLANT:BUSH_QUARRY')
>>> growthprint = quarrybush.get('GROWTH_PRINT')
>>> print growthprint
[GROWTH_PRINT:0:6:7:0:0:NONE]
>>> print growthprint.args
['0', '6', '7', '0', '0', 'NONE']
>>> growthprint.args[4] = 'ALL'
>>> growthprint.args[5] = '1'
>>> print growthprint
[GROWTH_PRINT:0:6:7:0:ALL:1]
>>> print growthprint.args
['0', '6', '7', '0', 'ALL', '1']

OK, although that doesn't really address my question because you started with NONE and went to ALL, I see from your example that I could accomplish what I would be looking for with a slice.

Your example does bring up another question though - what about growths with multiple GROWTH_PRINTs?

Code: [Select]
[GROWTH:LEAVES]
[GROWTH_NAME:apple leaf:apple leaves]
[GROWTH_ITEM:PLANT_GROWTH:NONE:LOCAL_PLANT_MAT:LEAF]
[GROWTH_DENSITY:1000]
[GROWTH_HOST_TILE:BRANCHES_AND_TWIGS]
[GROWTH_HOST_TILE:SAPLING]
[GROWTH_TIMING:0:300000]
[GROWTH_PRINT:0:6:2:0:0:0:209999:1]
[GROWTH_PRINT:0:6:6:0:1:210000:239999:1] autumn color
[GROWTH_PRINT:0:6:4:0:1:240000:269999:1]
[GROWTH_PRINT:0:6:4:0:0:270000:300000:1]
[GROWTH_DROPS_OFF]
There are 4 growth prints which vary only in their args. Can I change the color (args 2-4 inclusive 0-based) of the 2nd GROWTH_PRINT without touching the others?

Quote
On nomenclature: "token", "value", "args" seemed to me like the most appropriate terms, coming from my experience with parsers.

Based on my experience with parsers, I've always considered a token to be the smallest unit of meaning. Each argument in a function is considered a separate token, so it seems odd to talk about a token containing args. A tag is more like a statement than a token. But you do you. :)
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: Putnam on June 03, 2015, 02:03:45 pm
Quote from: Button on June 03, 2015, 11:37:17 am
Quote from: madk on June 03, 2015, 11:21:02 am
Could you elaborate? I always understood the terms "token" and "tag" to be synonymous when speaking of the raws.

Do you mean changing the arguments of a token? i.e. a token being like [VALUE:ARGUMENT_0:ARGUMENT_1:...:ARGUMENT_n]? If so then absolutely. A raws.token object has a value attribute, corresponding to the first item in the list delimited by semicolons, and an args attribute which is a list of all the other items. There are also helper methods like token.nargs(), token.getarg(), and token.setarg() for things like convenience and input sanitization.

Yes, that's what I meant. I can't speak to the nomenclature, but my understanding has been that a tag is a full [...], and a token is anything between the [ and ], separated by :s, including the part you've designated VALUE.

Are the arguments identified only by their index? How would one go about, say, setting all the time ranges in all GROWTH_PRINT tags to ALL? Or is the range identified as a single "range-type" argument, even though it's got a semicolon in?

Token and tag are synonymous. The VALUE token/tag has one argument, which is the word you're looking for.
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: Button on June 03, 2015, 02:34:45 pm
Quote from: Putnam on June 03, 2015, 02:03:45 pm
Quote from: Button on June 03, 2015, 11:37:17 am
Quote from: madk on June 03, 2015, 11:21:02 am
Could you elaborate? I always understood the terms "token" and "tag" to be synonymous when speaking of the raws.

Do you mean changing the arguments of a token? i.e. a token being like [VALUE:ARGUMENT_0:ARGUMENT_1:...:ARGUMENT_n]? If so then absolutely. A raws.token object has a value attribute, corresponding to the first item in the list delimited by semicolons, and an args attribute which is a list of all the other items. There are also helper methods like token.nargs(), token.getarg(), and token.setarg() for things like convenience and input sanitization.

Yes, that's what I meant. I can't speak to the nomenclature, but my understanding has been that a tag is a full [...], and a token is anything between the [ and ], separated by :s, including the part you've designated VALUE.

Are the arguments identified only by their index? How would one go about, say, setting all the time ranges in all GROWTH_PRINT tags to ALL? Or is the range identified as a single "range-type" argument, even though it's got a semicolon in?

Token and tag are synonymous. The VALUE token/tag has one argument, which is the word you're looking for.

Well, if that's the definition then that's the definition, but it's an extraordinarily unhelpful one for performing lexical analysis.
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: madk on June 03, 2015, 03:00:51 pm
Quote from: Button on June 03, 2015, 01:45:15 pm
OK, although that doesn't really address my question because you started with NONE and went to ALL, I see from your example that I could accomplish what I would be looking for with a slice.

Your example does bring up another question though - what about growths with multiple GROWTH_PRINTs? ... There are 4 growth prints which vary only in their args. Can I change the color (args 2-4 inclusive 0-based) of the 2nd GROWTH_PRINT without touching the others?

I'm not sure I understand what you mean, then. Can you provide some sort of pseudocode example of what you would expect to be able to do?

As for multiple GROWTH_PRINTs, they could be handled like this:

Code: [Select]
>>> pomegranate = df.getobj('PLANT:POMEGRANATE')
>>> print pomegranate
[PLANT:POMEGRANATE]
>>> growths = pomegranate.allprop('GROWTH_PRINT')
>>> print growths
[
                [GROWTH_PRINT:0:6:2:0:0:0:209999:1],
                [GROWTH_PRINT:0:6:6:0:1:210000:239999:1],  autumn color
                [GROWTH_PRINT:0:6:4:0:1:240000:269999:1],
                [GROWTH_PRINT:0:6:4:0:0:270000:300000:1],
                [GROWTH_PRINT:5:5:4:0:1:60000:119999:2],
                [GROWTH_PRINT:'%':'%':4:0:1:120000:200000:3]]
>>> growths[1].args[2] = 'some value'
>>> growths[1].args[3] = 'some other value'
>>> growths[1].args[4] = 'some other value again'
>>> print growths
[
                [GROWTH_PRINT:0:6:2:0:0:0:209999:1],
                [GROWTH_PRINT:0:6:some value:some other value:some other value a
gain:210000:239999:1],  autumn color
                [GROWTH_PRINT:0:6:4:0:1:240000:269999:1],
                [GROWTH_PRINT:0:6:4:0:0:270000:300000:1],
                [GROWTH_PRINT:5:5:4:0:1:60000:119999:2],
                [GROWTH_PRINT:'%':'%':4:0:1:120000:200000:3]]
It might be helpful to see PyDwarf's querying functionality as a bunch of methods that look for tokens in a linked list, starting with some particular token, that match some particular conditions, and ceasing to accumulate matching tokens on some condition. Because that's exactly what it is. All of the querying methods are built on top of a generalized query method that takes filter objects and an iterable collection of tokens as arguments. For example, the method allprop as used in the above code starts from the current token and returns everything with a value of 'GROWTH_PRINT' until it encounters the next token with a value of 'PLANT'. And each of the token objects in the list returned can be operated on in any way from adding more tokens before or after, removing the token, or accessing/modifying its attributes e.g. its arguments.

If you navigate a terminal to your PyDwarf directory, try entering python and entering the same commands as above. (Don't forget to import raws and df = raws.dir(path="..."). You will need to change the path in raws.dir(path="...") to refer to actual raws on your machine.) Screw around with them as you see fit, and the workings should become more apparent.

Quote from: Button on June 03, 2015, 01:45:15 pm
Well, if that's the definition then that's the definition, but it's an extraordinarily unhelpful one for performing lexical analysis.

Given the raws snippet [CREATURE:DWARF] a lexical scanner would break it down into these elements: '[', 'CREATURE', ':', 'DWARF', ']'. These are formally called lexemes, but are typically referred to as tokens in the context of lexical scanners. In no context that I know of would tokens conventionally refer to 'CREATURE' and 'DWARF' only. The term I've most frequently seen to refer to the tags/tokens in raws is token, for example this wiki page (http://dwarffortresswiki.org/index.php/DF2014:Raw_file). The first item in the list delimited by colons represents the purpose of the token, and value seemed to me a reasonable term to use. The others elaborate on the first in much the same way as arguments passed to some procedure.
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: milo christiansen on June 03, 2015, 03:28:17 pm
Quote from: Button on June 03, 2015, 02:34:45 pm
Well, if that's the definition then that's the definition, but it's an extraordinarily unhelpful one for performing lexical analysis.

Quote from: madk on June 03, 2015, 03:00:51 pm
Given the raws snippet [CREATURE:DWARF] a lexical scanner would break it down into these elements: '[', 'CREATURE', ':', 'DWARF', ']'. These are formally called lexemes, but are typically referred to as tokens in the context of lexical scanners. In no context that I know of would tokens conventionally refer to 'CREATURE' and 'DWARF' only. The term I've most frequently seen to refer to the tags/tokens in raws is token, for example this wiki page (http://dwarffortresswiki.org/index.php/DF2014:Raw_file). The first item in the list delimited by colons represents the purpose of the token, and value seemed to me a reasonable term to use. The others elaborate on the first in much the same way as arguments passed to some procedure.

Generally "token" is used to refer to a lexeme with attached meta-data (location in the file, type, ect).
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: madk on June 03, 2015, 03:34:49 pm
Quote from: milo christiansen on June 03, 2015, 03:28:17 pm
Generally "token" is used to refer to a lexeme with attached meta-data (location in the file, type, ect).

Quote from: Wikipedia
A lexeme is a string of characters which forms a syntactic unit.[2]

Some authors (for example, [1], [2]), just call this a token, using 'token' interchangeably to represent (a) the string being tokenized, and also (b) the token datastructure resulting from putting this string through the tokenization process.

http://en.wikipedia.org/wiki/Lexical_analysis#Lexeme

Personally I've never heard someone actually refer to them as lexemes.
Title: Re: PyDwarf: A powerful modding utility and mod manager
Post by: madk on June 04, 2015, 04:07:31 pm
Holy shit, just look at all that documentation! (https://github.com/pineapplemachine/PyDwarf/blob/master/raws/token.py)
Title: Re: PyDwarf 1.0.0: A New Hope
Post by: madk on June 07, 2015, 08:51:07 am
Finally put out a first release! https://github.com/pineapplemachine/PyDwarf/releases

A list of scripts included with the current release:
Spoiler (click to show/hide)
Title: Re: PyDwarf 1.0.1: Strange Mood
Post by: madk on June 22, 2015, 07:06:05 am
Here's the second release, v1.0.1: Strange Mood (https://github.com/pineapplemachine/PyDwarf/releases/tag/v1.0.1). Mostly it adds a lot of documentation and fixes/improves stuff that was already there.

I'm still trying to gain traction for PyDwarf! I believe the best way to do this is to add more mods. If you have a mod you'd like to see rewritten for PyDwarf, please let me know and I'll work with you to get it done. PyDwarf is very good at this sort of thing! Mods that might previously have been limited to a small range of Dwarf Fortress versions, and might have been troublesome to merge with other mods, can now be written in such a way that compatibility becomes almost a non-issue.

Changelog:
Spoiler (click to show/hide)
Title: Re: PyDwarf 1.0.1: Strange Mood
Post by: notfood on June 22, 2015, 11:54:37 am
Maybe it would be good to start with the popular graphic packs. The more complex do some generic changes to the raws.
Title: Re: PyDwarf 1.0.1: Strange Mood
Post by: madk on June 26, 2015, 01:38:00 pm
Coming in the next release: A major revising of the way PyDwarf handles files will provide modders with the ability to add or modify DFHack scripts, to work together with stonesense, to tinker with DF init settings and with speechfiles, and to add images for use with graphics packs or even just new tilesets, and still loads more. All of this with the ease and flexibility of Python!
Title: Re: PyDwarf 1.0.1: Strange Mood
Post by: madk on July 06, 2015, 04:32:10 pm
Recent mod additions which take advantage of the aforementioned major revision include omniclasm's State of Decay (https://github.com/pineapplemachine/PyDwarf/tree/develop/scripts/omniclasm), pkdawson's df-vegan (https://github.com/pineapplemachine/PyDwarf/tree/develop/scripts/pkdawson), and two more of umiman's smallthings mods (https://github.com/pineapplemachine/PyDwarf/tree/develop/scripts/umiman). Expect a new release quite soon!
Title: Re: PyDwarf 1.0.1: Strange Mood
Post by: expwnent on July 06, 2015, 07:21:27 pm
Posting to watch.
Title: Re: PyDwarf 1.0.2: Magma Forge
Post by: madk on July 07, 2015, 04:55:35 pm
PyDwarf 1.0.2: Magma Forge is released! (https://github.com/pineapplemachine/PyDwarf/releases/tag/v1.0.2)
Title: Re: PyDwarf 1.0.2: Magma Forge
Post by: madk on July 29, 2015, 07:17:53 am
I've been hard at work on PyDwarf and a new release is coming soon: Prepare yourself for improved documentation and a GemSet installation script! Many thanks go to dragondeplatino for working with me to get that script working.
Title: Re: PyDwarf 1.0.2: Magma Forge
Post by: madk on August 07, 2015, 12:58:23 pm
Haha, "soon".

This is going to be the last major release of PyDwarf so I'm spending a lot of time making the documentation as helpful as I can and hunting down any remaining bugs. How does it go, the last 10% of the code takes 90% of the development time? But this polish is important: I understand that PyDwarf can look like a confusing mess and that's why I'm working hard to make it as accessible and as easy to use as possible. For anyone with any programming experience who wants to get into modding I believe PyDwarf is their best option by far. Included with the next release will be several tutorials for various aspects of PyDwarf, thorough documentation on its more complicated parts, and summaries and example usages for as many functions and methods as I can cover.

Want to help PyDwarf to attain its final form? I can always use help discovering bugs, writing code examples, making sure everything actually makes sense to someone who isn't me.
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: madk on September 22, 2015, 07:09:32 am
PyDwarf v1.1.0: Chalcedony (https://github.com/pineapplemachine/PyDwarf/releases/tag/v1.1.0) has been released!

Here's a step-by-step tutorial (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/introduction.md) to guide you through your first time configuring and running PyDwarf, and here's another tutorial (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/modding.md) to help you get started creating your first mod. Also, here's a handy list of scripts (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/scripts.md) with which PyDwarf comes packaged. And here's where all those scripts are actually located (https://github.com/pineapplemachine/PyDwarf/tree/master/scripts), in case you'd like to see what their innards look like.

Here's just a few examples of the mods included with this release of PyDwarf:
Normally you'd be bound to run into compatibility issues when trying to install so many of these mods at once, and you'd surely have to wait for authors to update their mods before reinstalling everything whenever Dwarf Fortress updates. Not so with PyDwarf. PyDwarf is highly capable of avoiding conflicts between mods, and it cares little about which version of Dwarf Fortress you're modifying. Not only does PyDwarf make it much easier for the player to install mods, it also makes it easier for the modder to craft something excellent. Instead of editing text files on your own, PyDwarf allows you to write scripts which edit them for you: What this means for you is an enormous reduction in tedium.
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: madk on December 15, 2015, 07:50:16 am
Just a PSA that PyDwarf is 100% functional for DF 42.x. It's possible that some mods written for it may be incompatible with raws changes made, and mods depending on other utilities won't work until those have been updated - but PyDwarf itself requires no update.

All of the mods packaged with PyDwarf should work with DF 42.x without any changes, though some depend on DFHack or TWBT also being up-to-date.
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: cryzed on December 26, 2015, 09:49:56 am
I'm just stopping by to tell you how cool and impressive your project is. I'm a Python enthusiast myself (https://github.com/cryzed/) and have seen your project a few times on the forums and always thought so. I think it's unfortunate that this thread seems relatively inactive, which makes your continued progress and enthusiasm with the project even more impressive. Thanks for contributing really cool tools to Dwarf Fortress (even though I am not actively using them myself)!
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: Xinael on January 22, 2016, 02:15:01 pm
This project is awesome. This is going to make my life so much easier.

One question though, or suggestion for the documentation. I'm a definitely Python newbie, I've dabbled enough to know the syntax but I've never developed anything serious with it. I understand how to call scripts through manager.py well enough, so I think the documentation is great there, btw :)

But a big gap for me is understanding the best workflow to use when I'm developing a script. I want to be checking that my script is loading the right entities, amending them in the right way, and leaving them the way I want them to be after it runs. I could call the script over and over with manager.py but that seems inefficient and slow and it requires me to have a dev version as well as my real version of pydwarf (to have a different config). It will make debugging a nightmare too.

So if you could suggest a workflow for script development, that would be awesome. I assume you have some good advice to give since you've developed all the scripts that come with pydwarf. I assume that running raws.dir() to load the directory and then running scripts and functions against that to check it's working as I expect would be the best method - but I'd really appreciate any advice, help and tips you can share about ways to set that up that you found helpful.

I'm in the situation now where I have a good idea what I want my script to do, but I don't have a good process for making it happen and checking that it's working, except writing something I think will work in Sublime and then running manager.py over and over. Any help would be greatly appreciated :)
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: madk on January 22, 2016, 03:48:20 pm
Quote from: Xinael on January 22, 2016, 02:15:01 pm
One question though, or suggestion for the documentation. I'm a definitely Python newbie, I've dabbled enough to know the syntax but I've never developed anything serious with it. I understand how to call scripts through manager.py well enough, so I think the documentation is great there, btw :)

Thanks so much for your interest!

Developing the scripts that come with PyDwarf was a lot of trial and error for me, too, though I like to think I usually got things right the first time. (I didn't.)

To test scripts, in a console located at PyDwarf's root directory I ran python, then import raws, pydwarf; df = pydwarf.df(raws) - that pydwarf.df command should load your settings from your configuration files, though you can also manually specify settings by passing them as arguments to that function. Then I'd call one of my scripts using syntax like this: print pydwarf.scripts.pineapple.noexotic(df), or maybe I'd just step through some code manually. (You can pass named arguments to your script when you run it that way, I should mention.) And then I'd print information to the console that ought to have been changed by the script so I could check up on how things were working. I could also run df.reset() and then run it all over again, perhaps after making some manual changes to the raws to see how the script contended with them. (But you'll have to restart the python console and start over again if you make changes to your script. As inconvenient as this is, I couldn't find another reliable way to reload modified scripts!)

For example, on one occasion that I was testing a script this way, my console looked something like this:

Code: [Select]
>>> import raws, pydwarf; df = pydwarf.df(raws)
>>> print df.getobj('CREATURE:TIGER').list(range=12, skip=False)
[CREATURE:TIGER]
    [DESCRIPTION:A huge, striped predator.  It is found in almost any climate on a lone hunt for its prey.]
    [NAME:tiger:tigers:tiger]
    [CASTE_NAME:tiger:tigers:tiger]
    [CHILD:3][GENERAL_CHILD_NAME:tiger cub:tiger cubs]
    [CREATURE_TILE:'T'][COLOR:6:0:1]
    [PETVALUE:200]
    [PET_EXOTIC]
    [MOUNT_EXOTIC]
    [TRAINABLE]
>>> print pydwarf.scripts.pineapple.noexotic(df)
SUCCESS: Replaced 303 PET_EXOTIC and 150 MOUNT_EXOTIC tokens.
>>> print len(df.all(value_in=['PET_EXOTIC', 'MOUNT_EXOTIC']))
0
>>> print df.getobj('CREATURE:TIGER').list(range=12, skip=False)
[CREATURE:TIGER]
    [DESCRIPTION:A huge, striped predator.  It is found in almost any climate on a lone hunt for its prey.]
    [NAME:tiger:tigers:tiger]
    [CASTE_NAME:tiger:tigers:tiger]
    [CHILD:3][GENERAL_CHILD_NAME:tiger cub:tiger cubs]
    [CREATURE_TILE:'T'][COLOR:6:0:1]
    [PETVALUE:200]
    [PET]
    [MOUNT]
    [TRAINABLE]

I turned this into a unit test. You can find that exact snippet, except for the top line with the imports and such, in docs/examples/scripts.pineapple.txt. You can make your own file in that directory containing unit tests for your own scripts, and then run all the tests in that directory using docs/bin/verify.py. I'm not sure whether I actually documented anywhere how this works, but I can summarize:

I copied that console text wholesale, save for the topmost line, and pasted it into that unit test file. Violà, it became an actual unit test where the commands are run and the output I verified here is compared to the output given after, say, I made some changes to the code or to the script. The verify.py script will helpfully let you know whether the test passed or failed and, if it failed, it will tell you where and why. I used this to do some test-driven development, as well as testing code that was already written.

There's some special formatting going on mainly so that the code snippets in these unit test files can also be used in the HTML documentation, but you won't need to worry about that much. It's the three lines you see above every unit test in those files. Your three lines will probably look like this - sans the comments.

Code: [Select]
# The name of the script you're testing goes here
pydwarf.scripts.pineapple.noexotic
# For thoroughness' sake I listed all the API commands I called here in case it was useful in the generated docs, but you can leave this line blank
raws.queryableobj.getobj raws.queryable.all raws.tokenlist.__len__ pydwarf.registrar.__getattr__
# Putting "reset" here, as opposed to a blank line, means the DF directory is reloaded so that the following test has a clean slate to work with. You probably want to do this for every test.
reset

I last worked with this code months ago and my memory is fuzzy and I can almost guarantee I've made some mistakes and terribly mislead you regarding how to best test your scripts. But I hope I've helped answer your question and put you on the right track, and I am more than happy to work with you if you run into any more obstacles.

Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: Xinael on January 26, 2016, 03:22:55 pm
Should be really useful, thanks :)
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: Eagleon on February 01, 2016, 08:59:23 am
This will finally realize my lifelong dream of recursively feeding the description, name, etc. tags through a Markov chain chatbot like pyborg! So that's something. Will let you know how it goes if I don't get randomly distracted by some other project like usual.
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: madk on February 01, 2016, 09:23:29 am
Quote from: Eagleon on February 01, 2016, 08:59:23 am
This will finally realize my lifelong dream of recursively feeding the description, name, etc. tags through a Markov chain chatbot like pyborg! So that's something. Will let you know how it goes if I don't get randomly distracted by some other project like usual.

I look forward to seeing what you make!

Fetching a list of all the description tags, for example, could hardly be easier, using the raws package included with PyDwarf:

Code: [Select]
import raws
df = raws.dir('df_path/raw/objects')
descriptions = [token.args[0] for token in df.all('DESCRIPTION')]
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: Xinael on February 01, 2016, 03:31:55 pm
Just got round to playing around with this: import pydwarf fails because as __init__.py runs, it imports quick, which imports session, which imports config, which finally imports yaml - but this fails because it can't find the yaml library. I had to move it from /libs/ into /pydwarf/ and then it'd run. Maybe this is a weirdness about my setup, but that's what I had to do so I thought I'd let you know :)
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: madk on February 01, 2016, 03:44:02 pm
Oh, shame on me for not having caught that. Copying the lib/yaml/ dir into the pydwarf/ dir will work perfectly well, but you can also run pip install pyyaml or add PyDwarf's lib/yaml/ directory to your PYTHONPATH.

edit: Actually, I pushed a cleaner fix to the PyDwarf repository's develop branch (https://github.com/pineapplemachine/PyDwarf/tree/develop) as to avoid this problem in the future. Thank you for pointing it out!
Title: Re: PyDwarf 1.1.0: Chalcedony
Post by: madk on May 11, 2016, 05:16:53 pm
Just letting everyone know that with the DF update PyDwarf should continue to function as expected, and off the top of my head none of the recent changes should break any of the scripts packaged with PyDwarf.

I've been thinking about ways to make PyDwarf more accessible by creating a UI for configuring and running it for people who are scared of a CLI, and maybe I'll get to that soonish. (But I'm not making any promises.) Keep an eye out!
Title: Re: PyDwarf 1.1.1: Goblins
Post by: madk on May 05, 2017, 07:22:59 am
Here's a surprise: PyDwarf has just received a new minor release (https://github.com/pineapplemachine/PyDwarf/releases).

I'm getting back into playing DF after a long hiatus. I went to install my customary mods, and was very pleased to find that PyDwarf was still working as intended! ...With the notable exception of an issue arising from how PyDwarf determined script compatibility with Dwarf Fortress versions. This release isn't a very elegant fix, but it will suffice, and it makes PyDwarf less clunky to use for 0.40+ DF versions.
Title: Re: PyDwarf 1.1.1: Goblins
Post by: madk on May 21, 2017, 06:03:25 am
Extended Agriculture (http://www.bay12forums.com/smf/index.php?topic=157966.0) is now part of PyDwarf! You can view the mod online here (https://github.com/pineapplemachine/PyDwarf/tree/master/scripts/azerty), converted to work with a broad range of DF versions and to be more compatible with other plant-modifying mods.

With this and some other additions and improvements, a new v1.1.2 release of PyDwarf is now available for download (https://github.com/pineapplemachine/PyDwarf/releases)!

The next step is probably to add some kind of a graphical interface to make PyDwarf easier for players to install mods, since the current CLI interface might be a bit intimidating for newcomers. Is there anyone who might be interested in working with me to make something like that?
Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: Bearskie on September 18, 2017, 01:45:21 am
I've tried many times, but I just can't seem to get the pineapple.diff script to work. For starters, I'm not even sure whether I'm passing the right args. It would be nice if there was a usage case example somewhere for what is probably the most interesting script in the bunch... (diff-based mod merging)
Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: madk on September 18, 2017, 07:11:55 am
Quote from: Bearskie on September 18, 2017, 01:45:21 am
I've tried many times, but I just can't seem to get the pineapple.diff script to work. For starters, I'm not even sure whether I'm passing the right args. It would be nice if there was a usage case example somewhere for what is probably the most interesting script in the bunch... (diff-based mod merging)

Hi, I'm busy as heck this week but I will see if I can't put a proper tutorial together when I get the chance.

pineapple.diff isn't more spectacularly presented because it's so limited. Diff-based merging of mods is also a lot more failure-prone than PyDwarf's normal mod tools. So some things to keep in mind:

- pineapple.diff absolutely must be the first mod in your list. It cannot be run after any other mod, or be run multiple times and still function correctly. (You can pass more than one mod path to merge, though.)

- The mods you input must have been made for the same version of Dwarf Fortress that you're playing, otherwise the raws differences from one version to another will screw up the merging.

- Looking at the script I can't immediately remember whether the paths to mods should be paths to raws directories or paths to directories like "df_osx_40_23/" with the mod's raw directory inside. If you've got the other things right and it's still not working, try messing around with the mod paths.

- Also if all of that isn't enough to figure it out, give me more information about how it's failing and I'll do what I can to help. e.g. is PyDwarf giving you an error? Is it making the wrong changes to your DF install? Is it making no changes at all?



Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: Bearskie on September 19, 2017, 04:36:33 am
Oh hi, didn't expect such a quick response. To clarify,  it's the only script I'm running, with args: {'paths':['C:\<yadda yadda yadda>\mod_folder']}. The mod folder contains a small set of raws, conflicting and non-conflicting. The console claims to have 'Merged 1 mod successfully with no conflicts', but none of the files are found in the df-output directory. Nothing seems to have happened at all.

No rush needed, I suspect its my args at fault, but I've tried many different alterations and am still stumped.
Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: madk on September 19, 2017, 09:01:13 am
Could you give me a link to download your mod_folder?
Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: Bearskie on September 19, 2017, 09:13:46 am
Sure, check your PM.
Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: CharlesDorf on October 01, 2017, 05:46:18 pm
I believe I have done everything but when I go to run the python manager.py it just flickers and turns off. Did I miss anything?
Title: Re: PyDwarf 1.1.3: Flux Stone
Post by: madk on October 02, 2017, 03:07:15 am
Quote from: CharlesDorf on October 01, 2017, 05:46:18 pm
I believe I have done everything but when I go to run the python manager.py it just flickers and turns off. Did I miss anything?

How are you running it? If you're running it normally in a command line it shouldn't do that, and it should tell you what the program did and what went wrong, if anything.

Here is one short tutorial on how to do this: http://pythoncentral.io/execute-python-script-file-shell/ See the section named "Run a Python script under Windows with the Command Prompt".
Title: Re: PyDwarf 1.1.4: Werebeast
Post by: madk on October 02, 2017, 05:22:09 am
Today I released a new version of PyDwarf! This is a fairly small update, but I think a lot of people who've been interested in PyDwarf's mod merging capability will be happy to know that it now works correctly with the latest PyDwarf version, and that there is a new tutorial just explaining how to use this feature.

Visit PyDwarf's releases page (https://github.com/pineapplemachine/PyDwarf/releases) to download the most recent version and please feel welcome to check out the pineapple.diff tutorial (https://github.com/pineapplemachine/PyDwarf/blob/master/docs/pineapple.diff.md) that explains how to get started merging normally manually-installed mods using PyDwarf.