Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 8 9 [10] 11 12 ... 373

Author Topic: DFHack 0.34.11 r3  (Read 1403456 times)

peterix

  • Bay Watcher
    • View Profile
    • Dethware
Re: DFHack 0.31.25 r6 (current)
« Reply #135 on: September 26, 2011, 05:30:15 pm »

thanks for the answer.

if you say it's not such a good idea, i'll take your word for it.  :D
It would be a great idea if Qt wasn't this silly about having a central event dispatch thing that pretty much limits all interaction with the GUI to a single thread... Anyway, I fixed the build problems. If you want to take a look at it, you should be able to do so now. It's just a gutted frontend for a ray tracer and doesn't really do much. It shows a window though :D
i will get back to my first idea which is to make a gui in c# .net and then in a first attempt use files in folders to communicate between my gui and a dfhack plugin running in a thread like stonesense or dfusion waiting for files to process.
since communications between the 2 programs should be scarse and spaced in time because a dwarf management tool is not so intensive as a stonesense, it should be a viable mean of communication for a start.
Interesting idea. It should work I guess. You can use Windows Mutex locks for syncing the files.
later, some kind of client/server data exchange service should allow for more "industrial" communications, but i'm not there yet and as much as i could implement it quickly in c#, i'm not so confident for the c++ part  :(, especially since i only dev web applications where communicating by webservices is only as hard as putting some attribute on class methods and letting visuall studio do al the work of creating the technical wires for both server and client...
In this case, it's the pipes that are missing. Anything too OS-specific will be swiftly kicked out of the dfhack project though.
Files sound nice and universal. Locking them with Mutex on Windows and a file lock on *nix makes a lot of sense too. It's actually not too different from shared memory. There are some slight differences in semantics for shared memory between OSes, but nothing that can't be handled with a few #ifdefs and maybe a cleanup function. I had something along those lines before, but kinda broken due to bad design.

My current plan in this area is using ØMQ together with Google protobuf for a cross-platform, network transparent way to talk to external apps. Some code is available on pastebin, written by a dude who's already using it for talking to DF.

See:
http://pastebin.com/hNPwfGSS
http://pastebin.com/Fdyvs9MU
http://pastebin.com/gvS81H8P

It's not complete on pastebin, but basically a client registers for updates of the local DF map and the server in DF sends them out over a ØMQ socket. The official external API for DFHack would be more of a request/response kind of thing, with the protocol extensible by plugins.

Anyway, whatever you come up with, keep me posted :)

rampaging-poet

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #136 on: September 26, 2011, 08:02:12 pm »

What steps are involved in making a new DFHack plugin? I was thinking of making a flood-fill traffic designation utility, but I couldn't figure out where to start.
Logged
Lame excuse? 'Having a drink instead' is the dwarfiest reason to not get something done, short of accidentally flooding your home with magma. Or intentionally flooding your home with magma.

toasteur

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #137 on: September 27, 2011, 05:55:26 am »

My current plan in this area is using ØMQ together with Google protobuf for a cross-platform, network transparent way to talk to external apps. Some code is available on pastebin, written by a dude who's already using it for talking to DF.

well i already looked into omq myself, and it seems really nice because it exists in so much langages variant, and seems pretty powerful but yet easy to use. but it's such a big library, i though maybe it would be a little overkill for what we want, but i'm glad i wasn't the only one to get it as a viable candidate amongst all other available solutions.

i looked into google protobuf when you mentioned it a while ago, but it seems complicated enough to put together, and since i'm from a web background i tough about xml, mainly because in .NET all it takes to produce an xml from a bunch of objects is to put a one-line [Serializable] attribute, and it's done. sadly, i didn't find anything approaching it on c++ side, so i was considering looking into yaml since apis exists in lots of languages and it could be more lightweight to put in place than google protobuf.

if we can manage to put some simple framework to allow communication between df and an arbitrary external program, i guess it should be worthwile to port it as some additional service for the dfhack console, where external programs could trigger random plugin commands instead of just one particular plugin. this way, the looping thread could be shared and plugins wouldn't have to implement each it's own communication protocol and loops.
Logged

Pseudo

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #138 on: September 28, 2011, 07:59:44 pm »

Is there (or could someone either make one or show me how to) a way to set jobs (as in for a building, not a dwarf) in DFHack?

It might be helpful for helping to make an optional add-on for Quickfort (see here), specifically for levers. It would be useful to be able to auto-connect levers (to say the least!), and the only other option (that I can see) is to scan through until the cursor co-ordinate is the same as the saved co-ordinate, which I think would get very slow fairly quickly...

I've been looking through the source and haven't found anything of the sort so far. Is it primarily because of missing memory vectors, or is it more complex then that?

On a side note, I can't believe that you actually managed to integrate DFHack and DF. The only other thing of the sort that I have ever encountered was TA Demo Recorder, and that was in many ways much less complex. (And yes, I do still occasionally play Total Annihilation... It may not have the best graphics, but it is a great game.)

What steps are involved in making a new DFHack plugin? I was thinking of making a flood-fill traffic designation utility, but I couldn't figure out where to start.

Flood-fill traffic designation? As in automating the first thing I always do (designating hallways as high traffic and dead-ends as low or restricted traffic depending on size)? ...yay!

Disclaimer: I have never written a plugin for DFHack, but I would imagine that it would be like writing any other C++ code. Why don't you take a look at the plugins folder of DFHack? You could start with tubefill, as it it (I think) among the closest to what you want to do. I haven't seen any mention of traffic designations yet in DFHack however.

EDIT: I just noticed this.
« Last Edit: September 28, 2011, 08:07:06 pm by Pseudo »
Logged
The lady the dog the paper the boy writ hit bit knit.

English is weird.

rampaging-poet

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #139 on: September 29, 2011, 01:26:23 am »

Thanks, that should help a lot!  I'm reading tubefill right now, and the explanation of the basic architecture in the compile document was highly informative.  I'll probably get started tomorrow evening.

EDIT: So if I've got this right, plugins need to export initialization and finalization procedures, and the initialization procedure returns (well, stores in a shared location) a command name, description, and the main command method.  I might get the basic skeleton filled in tonight...
« Last Edit: September 29, 2011, 01:32:30 am by rampaging-poet »
Logged
Lame excuse? 'Having a drink instead' is the dwarfiest reason to not get something done, short of accidentally flooding your home with magma. Or intentionally flooding your home with magma.

peterix

  • Bay Watcher
    • View Profile
    • Dethware
Re: DFHack 0.31.25 r6 (current)
« Reply #140 on: September 29, 2011, 04:26:05 am »

What steps are involved in making a new DFHack plugin? I was thinking of making a flood-fill traffic designation utility, but I couldn't figure out where to start.
Well. Let's see. The only plugin that uses flood fills is the vein digger. You can see it here.

This file is compiled into a library. DFHack then loads the library and requires some functions to be present:
plugin_name should always return the same name as the resulting plugin library file name. (see the bit about adding the plugin to the build system later)
plugin_init allows your plugin to declare which commands it exports to the dfhack console. There's an extra 'bool' parameter that declares if the command is interactive - accepts input from console. Such commands can't be used from a hotkey.
plugin_shutdown is called right before the plugin is unloaded by dfhack. You *must* clean up here. Any threads you might have started and are executing code from your plugin must be finished before this function returns (this applies to things like stonesense).
A command - in this case it's the vein digger command. It uses flood fills and shows how to use the map cache, which is a higher-level abstraction built on top of the Maps module. Using the Maps module directly would be much harder than using the cache, because the cache basically treats the DF map as one continuous space, hiding the underlying structures to an extent. This is very good for doing flood fills :)

A plugin can optionally include a plugin_onupdate function. In this case, it's the one from the reveal plugin, making it impossible to unpause if hell is revealed (to my great relief, not having to handle all the problems that caused for people).

Now a bit about synchonizing with the game :)

What happens is that the game has its ows thread where it executes everything interesting. This thread calls a function in DFHack, which in turn calls all the plugin_onupdate functions. That means that you don't have to synchronize with the game in this function.
The command functions are a bit different - they are tied to the console thread. When you type a command into the console, your command function will be called. There, you must call Core::Suspend to wait for the next time DF is idle and not doing stuff. After you are done with whatever you need to do, you have to call Core::Resume. This hands off the DF thread to the next command waiting, or DF itself. A lot of thread synchronization magic is hidden behind the DFhack API - Ideally, you won't ever have to deal with it directly.

You can use the Console object to mess with the console. A command can print to the console any time, but if you need interactivity, it has to be declared like this to prevent some breakage. The console implements a line editor and is thread safe. Only one thing can read from it at a time though, and only safe way to do it is with the lineedit method. The liquids tool has some examples of this.

To make it compile, you put your .cpp file in plugins/ and add it to the build system like this. Then rerun cmake to generate new project files.

Where you go from this point is up to you. Ideally, it will become a part of DFHack that can be maintained. If you can, fetch the dfhack source tree with git and work on top of that, even better, fork the project on github and publish your changes there.

And stop by in #dfhack on freenode. There's plenty of friendly people there :)

Rose

  • Bay Watcher
  • Resident Elf
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #141 on: September 29, 2011, 05:04:13 am »

And stop by in #dfhack on freenode. There's plenty of friendly people there :)

And me!
Logged

toasteur

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #142 on: October 01, 2011, 05:14:33 am »


Anyway, whatever you come up with, keep me posted :)

ok, so after several hours of trial and error in msvc 2008, i got libzmq 2.1.9 compiled, protobuf 2.4.1 compiled, and then i used the simple server wait and respond sample from zeromq website running both in c++. then it took me some times to generate the appropriate class helper for the sample addressbook message and got it running between c++ client and server.

to validate communication between different environments, i generated a c# class helper for addressbook, and got in a few minutes a perfectly running .NET client for the c++ server. so now i'm trying to integrate the c+ server into the dfhack project, but i have some problems gtetting the project to compile, it won't take my addressbook.h file complaining it can't find it even if i added it manually to a new plugin project which compiles fine by itself...

since it takes me like 100 times to code c++ than it take for c#, if it's not too much to ask, could you create some blank plugin project in the repository, which the basic wiring to launch a thread, wait for a kill signal like stonesense?

for now, i still have to define proper messages struct to exchange data, so i still have much work before just integrating into a dfhack plugin.
Logged

peterix

  • Bay Watcher
    • View Profile
    • Dethware
Re: DFHack 0.31.25 r6 (current)
« Reply #143 on: October 01, 2011, 10:15:22 pm »

ok, so after several hours of trial and error in msvc 2008
This won't work for dfhack. You need msvc 2010 for binary compatibility with DF.
I have some problems gtetting the project to compile, it won't take my addressbook.h file complaining it can't find it even if i added it manually to a new plugin project which compiles fine by itself...
Don't use the MSVC gui to add files. The real project files are written in CMake:

CMakeLists.txt - the main one. Just a bunch of setup stuff.
library/CMakeLists.txt - this says how the main lib is compiled.
Notice the setting of properties on lists of header files -- you can use the same for including any kinds of files and marking them as non-build (resources, images, linuxy stuff that shouldn't build on windows). I didn't really bother with this much, because I only use MSVC as a compiler button.
lua/CMakeLists.txt - building a static version of LUA for DFusion
plugins/CMakeLists.txt - building the plugins

The syntax should be easy enough to pick up :)

The stonesense one:
plugins/stonesense/CMakeLists.txt
This is a bit more crazy because I use my own precompiled static libs here... due to too many problems caused by the official build of allegro being just plain bad, or non-existent on linux.

Anyway, I guess I'll have to take zmq and protobuf and wrap them in cmake magic, just like I did with LUA for DFusion :D

since it takes me like 100 times to code c++ than it take for c#, if it's not too much to ask, could you create some blank plugin project in the repository, which the basic wiring to launch a thread, wait for a kill signal like stonesense?
Sure, I wanted to make one anyway. Just didn't have the reason to do it immediately :)
for now, i still have to define proper messages struct to exchange data, so i still have much work before just integrating into a dfhack plugin.
Hmm. I wanted to use protobuf for that... as for the protocol, I guess that will go through a few iterations before people are happy with it... I'd like to get the people that write visualizers and other tools involved.

toasteur

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #144 on: October 02, 2011, 04:38:27 am »

This won't work for dfhack. You need msvc 2010 for binary compatibility with DF.
that's not a problem, it's just that on my vm at work i only had vs2008. making the code in vs2008 or 2010, that's still c++ right  :D

i converted it easily in vs2010 version back to my home, no problem here.

Don't use the MSVC gui to add files. The real project files are written in CMake:
i hear you, but as a first quick and dirty version i just wanted it to compile to establish the principles. at worst other people would be able to make a clean version given the code.

Sure, I wanted to make one anyway. Just didn't have the reason to do it immediately :)
thanks, you save my life, all easy samples i found for now were using windows specific implementation...

Hmm. I wanted to use protobuf for that... as for the protocol, I guess that will go through a few iterations before people are happy with it... I'd like to get the people that write visualizers and other tools involved.

we didn't understand each other, my test used zeromq, AND protobuff with the sample message "addressbook" just to validate the principle.
so now, i have to define what i put in my messages, so define some proto files and generate corresponding classes for c++ and c#.

i'll define my own messages struct for my purpose, but it won't prevent others to do theirs also  ;D

if you want, here is the server and client code:
http://pastebin.com/r1ws8ebn

nothing fancy here, just proving how easy it is to setup communication between to applications in zmq which does all the dirty work for us.

the most time i spent was just how to manipulate the frigging strings in c++ with balancing between literal types and pointers, i though i was gonna kill myself  :o, that's why i asked you if you could do the technical wiring so my c++ coding would be limited to loops and simple dfhack objects usage.
Logged

Ieb

  • Bay Watcher
  • A Breakdancing Ogre
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #145 on: October 02, 2011, 05:35:07 am »

Do you still take suggestions on what tools to add?
Back in 40d days, whatever was the utility again that compared to this thing had a nifty function which allowed you to change one type of stone to another. Tired of microcline? Turn it all into magnetite!

Would that be possible to add to this package?
Logged

Zeehorse

  • Escaped Lunatic
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #146 on: October 02, 2011, 02:17:57 pm »

Is there a pre-compiled version of DFhack? I don't quite understand the intricacies of compiling fully, an the whole compiler software makes my head hurt so I figured I'd ask.
Logged

Rumrusher

  • Bay Watcher
  • current project : searching...
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #147 on: October 02, 2011, 02:28:08 pm »

Do you still take suggestions on what tools to add?
Back in 40d days, whatever was the utility again that compared to this thing had a nifty function which allowed you to change one type of stone to another. Tired of microcline? Turn it all into magnetite!

Would that be possible to add to this package?
I think it was named Dtil
it had a wicked hex editor and tile changer.
Logged
I thought I would I had never hear my daughter's escapades from some boy...
DAMN YOU RUMRUSHER!!!!!!!!
"body swapping and YOU!"
Adventure in baby making!Adv Homes

peterix

  • Bay Watcher
    • View Profile
    • Dethware
Re: DFHack 0.31.25 r6 (current)
« Reply #148 on: October 02, 2011, 08:10:26 pm »

Is there a pre-compiled version of DFhack? I don't quite understand the intricacies of compiling fully, an the whole compiler software makes my head hurt so I figured I'd ask.
PROTIP: read the first post again ;)

Do you still take suggestions on what tools to add?
Back in 40d days, whatever was the utility again that compared to this thing had a nifty function which allowed you to change one type of stone to another. Tired of microcline? Turn it all into magnetite!

Would that be possible to add to this package?
Sure. It's quite possible to do that :)
« Last Edit: October 02, 2011, 08:32:45 pm by peterix »
Logged

rampaging-poet

  • Bay Watcher
    • View Profile
Re: DFHack 0.31.25 r6 (current)
« Reply #149 on: October 02, 2011, 08:46:53 pm »

* Highly useful instructions *

Thank you very much for the explanation.  There's a few more features I'd like to add, but the basic flood fill is working.

I noticed the occupancy structure has three bits reserved for "building" and that you'd noted it should be an enumeration.  The current version of my plugin just checks to see if that field is non-zero, but it would be nice to know what each of those values actually means.  Is more research required to find the meaning of each value, or has it just not come up yet?
Logged
Lame excuse? 'Having a drink instead' is the dwarfiest reason to not get something done, short of accidentally flooding your home with magma. Or intentionally flooding your home with magma.
Pages: 1 ... 8 9 [10] 11 12 ... 373