Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[PeridexisErrant]

Really really old thread, ancient code.

[Max™]

I am very amused by that depiction of dfhack as Der Nibelung of some sort.

[Abadrausar]

... At first there was chaos. Many young hackers did the same and did it differently. It was good time, because df was stuck in one version.

But soon one of those hackers has risen. He chose the hard path: to join everyone under a dfhack banner (and the nice performance gain by being in same address space) and that person was Peterix. And thus the Dfhack has risen from the chaos and is thriving since that day.

The second big revolution was "Lua". From the depths of internet came a person named "ag" (in these forums). He took the idea i toyed around and made it beautiful and powerful. And thus we could do anything (well almost) without any recompile.

The third big revolution was "Vmethod interposing". Again big thanks to "ag". This allowed us to do "MAGIC" with dfhack. Like catch when df calls some functions and replace them VERY EASILY. And also the gui was born.

Though this is only one half of it. During all that time there were many people - Like Peterix and Quietust - that worked on very important systems. Like the dfhack architecture itself and reversing the df...

Very good lyrics! I almost can feel the Iluvatar music weaving into the shining reality of today DFHACK 
You must have a poet soul to transform a relevant bunch of boring and precise technical jargon into an interesting saga song 

[Warmist]

Very good lyrics! I almost can feel the Iluvatar music weaving into the shining reality of today DFHACK 
You must have a poet soul to transform a relevant bunch of boring and precise technical jargon into an interesting saga song 

Thanks, on a rare full moon i try to bard things up

[Roses]

Has anyone managed to figure out how to get two entities to go to war with eachother, or declare peace if they are already at war, or just to change their diplomatic standing with each other at all? I've been looking through everything I could think of but haven't really found anything useful.

EDIT: Follow up question, does anyone have a script that reads what the name of something is? Whether a unit or entity or region or anything that has the name vector with words, parts of speech, and language?

[mifki]

EDIT: Follow up question, does anyone have a script that reads what the name of something is? Whether a unit or entity or region or anything that has the name vector with words, parts of speech, and language?

dfhack.TranslateName(something.name, true)

true - to translate to English, if you want that

[lethosor]

Also, if you decide to print the result of that to the console (or any other CP437-encoded text from DF), use

Code: [Select]

print(dfhack.df2console(dfhack.TranslateName(...)))
If you don't pass it through df2console, special characters won't be displayed correctly on all platforms.

[Dirst]

Also, if you decide to print the result of that to the console (or any other CP437-encoded text from DF), use

Code: [Select]

print(dfhack.df2console(dfhack.TranslateName(...)))
If you don't pass it through df2console, special characters won't be displayed correctly on all platforms.

Thanks for that tip, echoing civ names to the console worked just fine under Windows.  Didn't realize that would not be the case everywhere.

[HavingPhun]

So I started reading a book on reverse engineering software and I couldn't help but think of DFHack. I've come up with a few questions:

-Beyond the custom DLL, how does DFHack work? How is the DLL used to add DFHack's functionality?
-Can something like DFHack be made for any game/program?
-Is there a term for what DFHack does? Would it be DLL or Code Injection perhaps?

[lethosor]

The way DFHack hooks in is a bit different on the three platforms, but basically, we hook into four core SDL functions:

- SDL_Init: DF calls this on startup, so we implement our own version that does any initialization work we need and then calls the "original" SDL_Init
- SDL_Quit: similar, but called when DF exits cleanly, so we handle any necessary cleanup here and call the original SDL_Quit
- SDL_PollEvent: used to implement keybindings. We call the original function, then check for any keypress events that match DFHack keybindings. If any do match, the DFHack keybindings are invoked; otherwise, the event is passed on to DF.
- SDL_NumJoysticks: probably the most important one. DF calls this every game tick (simulation tick, not graphical frame), so we can essentially pause DF briefly and allow anything to modify DF data safely.

On OS X and Linux, it's fairly simple to make a library that just intercepts these four functions, and doesn't interfere with any other SDL functions at all. However, on Windows, we have to make a new SDL.dll that implements every SDL function DF needs and call the corresponding function in SDLreal.dll (which is just a copy of the original SDL.dll).

There are some parts of DFHack that would be easy to make work with other programs, and some that would be very hard (e.g. the hooks above). I think Peterix was wanting to make DFHack work with other games at some point, but I'm not sure how much progress was made there.

I'm not really sure what a proper term is, but I suppose "DLL injection" fits.

[HavingPhun]

The way DFHack hooks in is a bit different on the three platforms, but basically, we hook into four core SDL functions:

- SDL_Init: DF calls this on startup, so we implement our own version that does any initialization work we need and then calls the "original" SDL_Init
- SDL_Quit: similar, but called when DF exits cleanly, so we handle any necessary cleanup here and call the original SDL_Quit
- SDL_PollEvent: used to implement keybindings. We call the original function, then check for any keypress events that match DFHack keybindings. If any do match, the DFHack keybindings are invoked; otherwise, the event is passed on to DF.
- SDL_NumJoysticks: probably the most important one. DF calls this every game tick (simulation tick, not graphical frame), so we can essentially pause DF briefly and allow anything to modify DF data safely.

On OS X and Linux, it's fairly simple to make a library that just intercepts these four functions, and doesn't interfere with any other SDL functions at all. However, on Windows, we have to make a new SDL.dll that implements every SDL function DF needs and call the corresponding function in SDLreal.dll (which is just a copy of the original SDL.dll).

There are some parts of DFHack that would be easy to make work with other programs, and some that would be very hard (e.g. the hooks above). I think Peterix was wanting to make DFHack work with other games at some point, but I'm not sure how much progress was made there.

I'm not really sure what a proper term is, but I suppose "DLL injection" fits.

Thanks for the explanation, it was very informative. You can easily modify those functions and createa modified DLL because SDL is open source, right? If one were to do the same with a closed source library would they have to search through the assembly version of it and modify it or do you know of any other ways?

[lethosor]

We're not actually modifying the functions - we're just making our own versions that do their own thing, then call the actual SDL functions. The SDL headers are enough for that. (And we actually don't use the SDL headers for the functions we need to call - we just check the SDL headers for those functions' signatures, then put them in our own header.)

Of course, if SDL weren't open-source, that could potentially lead to licensing issues, but that's not the case.

[figment]

In theory one could use something like detours [1] to do the same thing that DFHack does.  It wouldn't really require the dll to be open source.  You just intercept the function and replace it with something that calls the original function.   This alternate approach is sometimes called API Hooking but DLL injection is probably more common.  Detours used to be free and 2.0 may still be for win32 but they charge and unreasonable amount for Win64 support. 

The real issue tends to be that you need your code to be loaded in the host process.   Having the process load your dll instead of having to attach to the process like a debugger is simpler in many cases especially for something like dfhack.

[1] http://www.microsoft.com/en-us/research/project/detours/

[lethosor]

To Iri from freenode: yes, we are working on 64-bit DFHack. You would have found this out if you had stayed around for more than 78 seconds. (For anyone else who asks a question on IRC - yes, there are around 70 people in the channel, but spread across many timezones, and among those that are awake, not many actively watch the channel. If you wait around for a while, chances are someone will see your question.)

It looks to me like Detours is doing some kind of code injection, which isn't at all how DFHack's SDL hooks work. (You could argue that DFHack's virtual method hooks do some, but even those don't rewrite code in memory, at least as far as I'm aware.) It  seems kind of Win32-API-specific too, although I could be wrong about that.

Anyway, 64-bit Linux has made some progress yesterday and today (from "nothing at all" to "some useful stuff"), although there are issues with getting some globals and vtables located.

[Quietust]

Quick summary of DFHack history:

It started out as a memory I/O library - you loaded a DLL and it gave you functions for retrieving the addresses of important variables (from a "memory.xml" file, whose contents were very similar to what Dwarf Therapist currently contains in its INI files) and other functions to read/write them, and over time it got extended to add high-level functionality to make utilities simpler. It worked sort of like a debugger, reading and modifying DF's memory from outside the process, making it rather slow (a Reveal on a full-sized map could take several seconds) and error-prone (if something went wrong, it could leave DF stuck in a "suspended" state, requiring a secondary utility to unlock it). It also had a history of setting off virus scanners. However, it had the benefit that you could update a utility to work with a newer version of DF just by editing memory.xml.

Later, after it was observed that certain operations like figuring out item types and materials was decidedly non-trivial (some values were at different addresses based on item type, and some could only be determined by executing code), we came up with the idea of injecting a DLL into DF's address space - with this, we could not only directly modify its internal memory as if it were our own (making certain operations massively faster, especially Reveal), but we could even execute DF's own code in order to perform complex operations that were otherwise very difficult (such as formatting item names exactly the same as they appear in-game). When this happened, we abandoned the idea of having offsets to individual fields and reverse-engineered all of the game's data structures (so we could access them in C++ as structs and classes), memory.xml became "symbols.xml" (containing only the locations of global variables), and the utilities all became "plugins" that were invoked from within the game from a special console window. This made some things a lot simpler, but it also meant that when newer versions of DF came out, DFHack and all of its plugins needed to be recompiled in order to handle new structure layouts (i.e. if new fields were added or old ones were removed).

The rest (Lua scripting, vtable interposing, etc.) has already been covered above.