Bay 12 Games Forum
Dwarf Fortress => DF Modding => Utilities and 3rd Party Applications => Topic started by: ragundo on July 08, 2019, 07:31:55 am
-
A dfhack plugin (yes, it's a dfhack plugin) similar to gm-editor but with a convenient GUI and some cool features:
(https://i.imgur.com/OpAncPP.png)
Features
- Graphical viewer for all Dwarf Fortress structures known from DFHack.
- Open any individual structure in a new window for easy understanding.
- All structure types supported (vectors, arrays, pointers, enums, unions, bitfields, etc).
- Hexadecimal memory viewer for Dwarf Fortress process memory.
- Show comments and 'refers-to' (if defined) attributes for fields in structures.
- Show the dfhack xml file where each structure is defined.
- Stores opened df structures so you can track evolution of the structures in time (rather primitive right now).
- Automatic decoding of enums, bitfields, coordinates and language names.
- Decoding of derived classes showing its hierarachy.
Supported operating systems
- Windows 64 bit
- Linux 64 bit
More information in the plugin repository https://github.com/ragundo/DwarfExplorer (https://github.com/ragundo/DwarfExplorer)
Download: https://github.com/ragundo/DwarfExplorer/releases (https://github.com/ragundo/DwarfExplorer/releases)
Linux version ready. No Mac support.
-
Does the hex memory viewer allow for binpatchs of yore?
Also, I guess you meant hier, not hyer-archy?
-
Unless I'm blind, the binaries are stored somewhere than the github repo linked above that containing the source code and more info.
It would be useful with a link(s) to where you'd actually find the binaries.
Edit:
OK, by comparing to where DFHack provides its releases I found the link: https://github.com/ragundo/DwarfExplorer/releases (https://github.com/ragundo/DwarfExplorer/releases). It would still be useful to provide links to that location both from this thread and from the documentation in the source repo.
Edit2: And I'd add the installation instruction to the first post of this thread as well, as that's a mandatory step for anyone who'd use it.
-
Yeah. sometimes I assume people know too much anouy github.
I will fix that.
My apologies for the typo. I'm not a native english speaker.
About the binpatches I don't understand your question.
-
Assuming too much is a common issue when you know (too) much about something ;)
A couple of further issues:
- Warning: Plugin dwarfexplorer compiled for DFHack 0.44.12-r2-24-XXXX, running DFHack 0.44.12-r2-0-XXXX
- "No such plugin or Lua script: qapplication"
I suspect you may have compiled the plugin for a (development) DFHack version rather than the released one?
Trying the one DFHack version I've used for development it claims to be version 0.44.12-r2-38-XXXX (I haven't updated it today: it's probably a week old, but still no qapplication, although it might not have been set to compile locally).
Are you sure qapplication is part of the standard DFHack package, and so doesn't require an additional installation step to get hold of that?
-
Yes. It's compiled with a later vesion on purpose. There's a lot of interesting changes in DF structures commited since the last official DFHack release.
The plugin only uses standard dfhack features that are in the code long time ago(CoreSuspender). That's why it works against the "official" DFHack release providing they are in the same major revision number.
About the running issues, I installed it in my laptop with a standard 0.42 LNP and worked. Did you copy the Qt5 dlls to the Dwarf Fortress directory?
There are also some DLL needed (MSVC 2017 runtime dlls, like msvcp140.dll), but they are present in the Dwarf Fortress directory with a LNP installation.
EDIT: You are right. Of course, the qapplication plugin is needed also. I updated the download link with the missing dll.
Hope that helps
-
About the binpatches I don't understand your question.
For example, angrilov made a binpatch for weapon racks/armor stands, back in 34.11, mentioned on bugtracker (http://www.bay12games.com/dwarves/mantisbt/view.php?id=1445#c23685) and also imported into DFHack version of that time.
Doc says it has been unavailable since 34.11 (https://dfhack.readthedocs.io/en/stable/docs/Plugins.html#fix-armory). (Though there's been changes (http://www.bay12forums.com/smf/index.php?topic=139553.msg6021243#msg6021243) to it.)
-
About the binpatches I don't understand your question.
For example, angrilov made a binpatch for weapon racks/armor stands, back in 34.11, mentioned on bugtracker (http://www.bay12games.com/dwarves/mantisbt/view.php?id=1445#c23685) and also imported into DFHack version of that time.
Doc says it has been unavailable since 34.11 (https://dfhack.readthedocs.io/en/stable/docs/Plugins.html#fix-armory). (Though there's been changes (http://www.bay12forums.com/smf/index.php?topic=139553.msg6021243#msg6021243) to it.)
This viewer only knows about structures defined in codegen.out.xml. If those patches were merged in the dfhack repository, they will be reflected.
-
No dice. Both my "official" and "inofficial", week old, DF with DFHack crash when the "enable qapplication" command is given ("crash" = both the DFHack and the DF windows disappear). I did only copy the qapplication DLL from the new download, as I assumed the other files were unchanged.
It can also be noted that the ZIP now contains 5 files, rather than 4 (That part of the text was left unchanged).
Have you tested it out by applying your own instructions to a DF with only the official DFHack version installed? That's what people will use unless they're in a fairly hardcore "beta testing" squad.
-
Sorry to hear that it doesn't work.
I'll repeat the process again with a new LNP.
-
OK.
I just downloaded the latest LNP, copied the plugin dlls and the Qt5 libraries and it works.
dumpbin the dll lists the following dependencies:
- SDL.dll
- Qt5Widgets.dll
- Qt5Gui.dll
- Qt5Core.dll
- KERNEL32.dll
- MSVCP140.dll
- VCRUNTIME140.dll
- api-ms-win-crt-runtime-l1-1-0.dll
- api-ms-win-crt-heap-l1-1-0.dll
- api-ms-win-crt-string-l1-1-0.dll
- api-ms-win-crt-stdio-l1-1-0.dll
- api-ms-win-crt-filesystem-l1-1-0.dll
- api-ms-win-crt-convert-l1-1-0.dll
- api-ms-win-crt-math-l1-1-0.dll
All those DLLs, except KERNEL32.dll are in the LNP default installation. Could you check that you have all those dlls? Are you in a 64 bit windows?
Thanks
-
This viewer only knows about structures defined in codegen.out.xml. If those patches were merged in the dfhack repository, they will be reflected.
Ah, not quite what I meant. When I said allow, I meant "could the hexadecimal viewer also edit the hex?"
Edit: I'm a dumbass who didn't read line "Right now you can't change values in Dwarf Fortress memory." >_> Nevermind this.
-
Hm. I'm using the LNP for my "main" DF installation, with the working one being just DF + DFHack.
I'm using 64 bit Windows (as that's the only on LNP supports), although I'm on the second latest LNP (r5 vs r6), but none of these items should be relevant here.
DT works fine, so it shouldn't be a fundamental QT issue.
I've verified the DLL's except the 3 Windows ones (the computer is busy with other things, and slow at searching).
However, when verifying that the QT DLL's were there, I found a file called qt_log.txt which contains this:
mån jul 8 17:29:26 2019 [info] Qt Init
mån jul 8 17:29:27 2019 [warning] Could not find the Qt platform plugin "windows" in ""
mån jul 8 17:29:27 2019 [fatal] This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
That indicates there's a need for an additional file somewhere, and Clément has probably resolved it for DT (Dwarf Therapist\platforms\qwindows.dll?). I tried copying that file both to the DF directory and to hack/plugins, but neither seemed to work.
-
Hmmm
Do you have visual studio 2017 runtime installed?
Qt is compiled against that runtime.
Another thing that can happens is that your path finds another Qt dll instead of the ones I supply (5.13.0).
Dwarf Therapist uses Qt 5.6
I'm on mobile now but I'll try later with a windows vm
-
Take your time: I'm not in a hurry.
I've only got VS 14,0, as that's what DF uses. I'm not sure if something else may have installed VS 2017 as part of itself, though, but C:\Program contains 11.0 and 12.0 (while D:\Program contains 14.0, due to the stupidity of leaving the install path to point to the OS partition by the PC supplier), but no (20)17 at either location.
I don't think I've got anything else using Qt except DT, and if it screws up the path you ought to suffer from it as well, as you've got the LNP.
-
ragundo, use windeployqt to copy Qt dlls, it will include all the required Qt plugins (platform, image formats, ...). You will find it in the bin directory of the Qt SDK you used to build the dfhack plugins.
-
Yes. I know about windeployqt, but the help page says that it's for deploying exe files, not dll ones.
I think that only qwindows.dll is needed to run the plugin in other systems (in my system Qt is not listed in the PATH variable and it works by deploying only the Qt5**.dlls).
I'm researching about the possibilty of copy qwindows.dll in the Dwarf Fortress directory by using a qt.conf file. I wouldn't like to add a platforms subdirectory to store qwindows.dll
-
Hmmm
Do you have visual studio 2017 runtime installed?
Qt is compiled against that runtime.
Another thing that can happens is that your path finds another Qt dll instead of the ones I supply (5.13.0).
Dwarf Therapist uses Qt 5.6
I'm on mobile now but I'll try later with a windows vm
Well, I updated the download file with qwindows.dll. It will go to a platforms subdirectory in the Dwarf Fortress directory as recommended by the Qt docs.
Could you try again to launch the plugin?. My local VMs are all 32 bits so pretty useless.
Thanks
-
Looks quite handy, wonder it this will be any of help for debugging efforts. Thank you for your work. Looking forward to the linux version.
-
Yes, that did the trick.
- It's a nice improvement to provide directories in the zip file. That makes "installation" that little bit easier (and harder to screw up).
- gui/gm-editor suppresses display of overloaded pointers because of the danger involved in following the incorrect interpretation. Is it possible to at least provide a warning in this situation (looking at the df.global.world.incidents.all - .unk_v42_1 pointers, where the pointer interpretation is controlled by the subtype field, but there's no way I know of to express the usage of this discriminant in the DF structures syntax, so it's provided only as a comment).
- Would it be possible to allow the user to invoke dwarfexplorer with a path parameter such you can do with gui/gm-editor? I'm thinking of things like referring to element 700000 of the hist fig vector, for instance, where you don't really want to manually scroll past a bazillion pages to find that element.
- Presentation of the comments can be very useful, especially if that inspires the writing of more comments. It might be useful to provide a popup for this field when hovered over to allow you to see the full comment (See e.g. df.global.world.world_data.feature_map).
Just some initial feedback after a quick look.
Edit: It's annoying that you have to start from scratch digging your way down into the structures every time you resume/suspend. I think it would be useful if the former view was remembered such that it could be recreated to the extent possible (and this goes for windows displaying sections as well: when suspended again, it would be useful to be able to "refresh" them, with the possible result of them becoming empty).
-
Yes, that did the trick.
- It's a nice improvement to provide directories in the zip file. That makes "installation" that little bit easier (and harder to screw up).
- gui/gm-editor suppresses display of overloaded pointers because of the danger involved in following the incorrect interpretation. Is it possible to at least provide a warning in this situation (looking at the df.global.world.incidents.all - .unk_v42_1 pointers, where the pointer interpretation is controlled by the subtype field, but there's no way I know of to express the usage of this discriminant in the DF structures syntax, so it's provided only as a comment).
- Would it be possible to allow the user to invoke dwarfexplorer with a path parameter such you can do with gui/gm-editor? I'm thinking of things like referring to element 700000 of the hist fig vector, for instance, where you don't really want to manually scroll past a bazillion pages to find that element.
- Presentation of the comments can be very useful, especially if that inspires the writing of more comments. It might be useful to provide a popup for this field when hovered over to allow you to see the full comment (See e.g. df.global.world.world_data.feature_map).
Just some initial feedback after a quick look.
Edit: It's annoying that you have to start from scratch digging your way down into the structures every time you resume/suspend. I think it would be useful if the former view was remembered such that it could be recreated to the extent possible (and this goes for windows displaying sections as well: when suspended again, it would be useful to be able to "refresh" them, with the possible result of them becoming empty).
I'm glad that you make it run. Many thanks for the time that you spend trying to make it work.!!
I've read some interesting articles about Qt deploying and now I understand why the plugin always works in the development machine the way I released originally and not in the client one. That will help me for the linux deployment in a few weeks.
About your comments:
Overloaded pointers. Yes I agree. This is something similar to some unions, where there's another field called type that could be uses to identify the union type. I'll see what I can do about that.
Invoking Dwarf Explorer with a path is the following thing I plan to do. It has its complications because the tree loading is lazy, and data only gets added to the tree as you expand the branch, but I'll manage to do it.
The tooltip with the comment data is a great idea. Noted!
And yes, the need to dig down the tree everytime you pause and resume is very annoying and the thing that I don't like at all of this release. This is related to the laziness loading of the tree. For now, I open the structure in another window and resume Dwarf Fortress. The window changes to the OUTDATED state and keeps a copy of the original data. When you pause again, I navigate to the same structure and comparte the old data with the new data. Not perfect I know.
Anyway, this is a 1.0 release and there's a lot of things to improve. I plan also for the next version to be able to follow refers-to attributes. That would improve the data navigation a lot.
Again, thanks for reviewing the plugin and don't hesitate to ask any feature that you would like to be implemente. This plugin was done for people like you.
-
Given the original description, it was clear it was going to be testing of an alpha or beta version, with a fair bit of development still being planned, and I have no problem with trying it and trying to provide some feedback. However, I'm not a great tester.
Overloading: Yes, the high level language concept of discriminants (as it's called in Ada) to determine how overloaded data should be interpreted isn't supported by High Level Assembly (a.k.a. C(++)) unfortunately. XML does have the capacity to describe it though, but most users of "XML" seem to ignore the built in features and use home brewed attributes instead, nor are XML schemas used to any extent to describe the legal syntax (that could then be validated to catch mistakes).
I believe it's necessary for the tool to load only the data that's to be displayed or starting would take forever and use up a huge chunk of memory, but there's a difference between laziness and complete amnesia... Remembering where you were is a kind of implicit expansion (with checks to prune references that do no longer exist). I agree this is a lot easier with the logic of gui/gm-editor, since you'd only need to keep track of a single expansion path, but it should definitely be doable with a reasonable effort.
The logic needed is probably similar to the one needed to open a given path, but with some additional complications, so I'd probably tackle the simpler problem first.
The ability to use an old and a new copy of some data and compare them, ideally with highlighted differences, would probably be very useful for field identification, as you'd be able to see what your in-game change resulted in (at that location) at a glance. Windows could possibly start outdated on a reload, with options to refresh or compare.
I agree the ability to follow references would be quite useful, but it can be noted that a lot of references are indicated only through field names (such as "hfid"). However, if there's a tool that can make intelligent use of such references, the drive to update the XML description with those attributes ought to be enhanced...
-
Regarding compare, another thing one may find useful is just recursively comparing differences between two structures - did this in a small (unposted) script myself for identification of flag meanings.
-
Recursion can be dangerous, as there are circular structures in DF (in particular linked lists, where the "previous" link points back to the previous element). However, comparison of the "same" view of the data at two different vector indices of the same vector could indeed be useful.
Edit: Some additional feedback:
It turns out comments using the "comment" attribute on compounds aren't picked up by dwarfexplorer (and they end up twice in the generated .h file (https://github.com/DFHack/dfhack/issues/1446 (https://github.com/DFHack/dfhack/issues/1446), but that's a different issue).
-
I'm honestly not sure what the intended usage of this (which if I understand right is just a way to directly view the stored data of DF?) and applications are, but I'm not that far into modding.
This sounds really cool, and vaguely reminds me of when I attempted to get into Cheat Engine, but I assume this might be a debugger tool?
-
I'm honestly not sure what the intended usage of this (which if I understand right is just a way to directly view the stored data of DF?) and applications are, but I'm not that far into modding.
This sounds really cool, and vaguely reminds me of when I attempted to get into Cheat Engine, but I assume this might be a debugger tool?
Similar to gui/gm-editor, but with a different take on it, so one would be better for some situations and the other of others (once it's 'finished').
The use would be for modding things, looking up things to know what to write into scripts (getting paths, fields, and types right. You can use the XML files for that as well, but a tool is often more convenient than finding the right file), investigations for structure identification (when DFHack has mapped structures, but their usage hasn't been identified), etc.
-
I'm honestly not sure what the intended usage of this (which if I understand right is just a way to directly view the stored data of DF?) and applications are, but I'm not that far into modding.
This sounds really cool, and vaguely reminds me of when I attempted to get into Cheat Engine, but I assume this might be a debugger tool?
As Patrick says, is like gm-editor on steroids. This is not a debugger, is for understanding how DF manages its data. You can use gm-editor for that, but I like more a tree structure as you don't miss the "locality" of your data as it happens with gm-editor. If you want to understand how the DF structures work I think that this is the ideal tool.
Also features like automatic translation of coordinates and language_names, be able to see the field comment and browse the memory make everything more snappy. Very useful also for undestanding lua scripts and C++ plugins
Things that will be added in future versions:
- Tooltips for comments
- Open a structure by typing its name.
- Search for fields
- Open refers-to attributes in new window
- Show the xml data that defines a structure
- Define new structures on the fly
- Compare changes of a structure as time goes by
- Changing data in the memory viewer
- Changing data in the tree
-
Linux release available!!!
https://github.com/ragundo/DwarfExplorer/releases (https://github.com/ragundo/DwarfExplorer/releases)
If you uses the Linux LNP, you need to download the version compiled for GCC4.8.
Installation:
- Install Qt using your package manager.
In Ubuntu:sudo apt-get install qt5-default - Unpack the file with the plugin in the /hack/plugins folder of your DF installation
- Launch dfhack
- Load your world
- type enable qapplication in the console
- type dwarfexplorer in the console
- Enjoy ;)
-
Looks like a very useful development tool already. I spent a while peeking through structures with lua commands. It should be much easier to figure out anon references with this.
-
In Ubuntu:sudo apt-get install qt5-default
qt5-default may be too much (it downloads development files), libqt5widgets5 should be enough for Ubuntu/Debian or whatever install libQt5Widgets.so.5 for your distribution.
-
Crazily great utility. It must be in LNP.
I found that i want - info about armies positions. Now, how can I export it using dfhack/lua/whatever?
Okey. Got that.
-
Crazily great utility. It must be in LNP.
I found that i want - info about armies positions. Now, how can I export it using dfhack/lua/whatever?
Odd question (at least I can't make out what's desired with any certainty). Noting the path to the data you want you can write a Lua script to print the parts you're interested in either to the DFHack console or to a file, or use it in the script itself for whatever you want to use it for (I assume it would be possible to write an overlay over the civ screen map to mark army positions, for instance).
-
Crazily great utility. It must be in LNP.
I found that i want - info about armies positions. Now, how can I export it using dfhack/lua/whatever?
Edited:Okey. Got that. Writed it based on legend_exports script.
-
Crazily great utility. It must be in LNP.
I found that i want - info about armies positions. Now, how can I export it using dfhack/lua/whatever?
Okey. Got that.
Glad that you found it useful.
My Job takes all my free time, so for now, this project is dormant, but not abandoned
-
New release!!
- Updated for dfhack 0.44.12-r3
- Added option to zoom the Dwarf Fortress window to coordinates (df::coord) selected in the tree of any structure
- Same for any building (df::building) selected in the tree
Download:
https://github.com/ragundo/DwarfExplorer/releases/tag/v1.1 (https://github.com/ragundo/DwarfExplorer/releases/tag/v1.1)
-
New release!!
- Updated for dfhack 0.47.03-beta1
- Linux build will be updated later
Download:
https://github.com/ragundo/DwarfExplorer/releases/tag/v1.2 (https://github.com/ragundo/DwarfExplorer/releases/tag/v1.2)
-
New release!!
- Updated for dfhack 0.47.04-beta1
- Linux build will be updated later
Download:
https://github.com/ragundo/DwarfExplorer/releases/tag/v1.2.2 (https://github.com/ragundo/DwarfExplorer/releases/tag/v1.2.2)
Thanks to the quarantine >:( I'm working hard on version 2 of the plugin, with lot of new features!
-
Thanks for the 47.04 :)
Good luck for V2 !
-
Thanks for the 47.04 :)
Good luck for V2 !
Uh I forgot that I need to publish it for Dfhack 0.47.04 release 1. :o :o
Still working on v2 with some problems with the hyperlink processing after some structural changes I made to the plugin.
-
Aye - tried it just after I posted on 47.04r1 thinking it would just be a warning for the minor mismatch, but it never can be easy ! Will be very useful for improving klokers accuracy when I next get a while to work on it, cheers.
-
Any plans to update this? Looks really useful but doesn't work with the current versions.
-
Any plans to update this? Looks really useful but doesn't work with the current versions.
Hi there
I'm waiting for DFHack r3 in order to update it.
-
Cool, glad to hear there's still plans for updates.
-
Probably I'll publish a beta of the version 2 of the plugin, with many enhancements, but also with some bugs.
-
Any chance this is still works and alive?
Sorry for necroposting, found out gui/gm-editor df.global.world
At least it's something.
-
DEL