Bay 12 Games Forum

Dwarf Fortress => DF Modding => Utilities and 3rd Party Applications => Topic started by: Morphyum on October 17, 2013, 08:14:39 am

Title: DwarfFamily
Post by: Morphyum on October 17, 2013, 08:14:39 am
DwarfFamily
(https://image.ibb.co/bWVBBQ/dwarf.png)
DFFD Download (http://dffd.bay12games.com/file.php?id=8053)

Description:
DwarfFamily is a tool, that let's you import your legends xml file and converts all dwarf to a .ged file, which you can use in FamilyTree programs like MyHeritage to display a family tree for the dwarfs.

Usage:

How to get legends xml:

Planned Features:

Credits:
Developed by Morphyum

Special thanks to Sallen for adding the interface and Ltan for making the ged compatible for more programs.

Support in Development Thread: http://www.bay12forums.com/smf/index.php?topic=132153.15
Title: Re: DwarfFamily
Post by: Sallen on October 17, 2013, 08:51:10 am
Damn you beat me to this! :)
If you need help with anything I'll be lurking around and willing to help. I'm very interested in this kind of tool.
Title: Re: DwarfFamily
Post by: Morphyum on October 17, 2013, 09:00:09 am
A few minutes ago i managed to create my first complete file, with family connections in it. Right now i am looking for some bugs that crash the program. If i will need help i will contact you ;-) What i would need in the future so is somebody, that creates a visual output, so we dont need to use another program to display the tree ^^

EDIT: found the bug, Seems like some dwarfs don't have a name in the xml? Why is that?
Title: Re: DwarfFamily
Post by: Covenant Ringthane on October 17, 2013, 09:03:42 am
PTW.

Good luck!
Title: Re: DwarfFamily
Post by: Morphyum on October 17, 2013, 09:42:39 am
Kinda have troubles finding good legends to test this. If i create long historys, it takes very long to convert it and in the shorter historys are not very much children from other dwarfs
Title: Re: DwarfFamily
Post by: Sallen on October 17, 2013, 09:44:18 am
Quote from: Morphyum on October 17, 2013, 09:00:09 am
A few minutes ago i managed to create my first complete file, with family connections in it. Right now i am looking for some bugs that crash the program. If i will need help i will contact you ;-) What i would need in the future so is somebody, that creates a visual output, so we dont need to use another program to display the tree ^^

EDIT: found the bug, Seems like some dwarfs don't have a name in the xml? Why is that?

I have no idea about anything related to DF-exported data, so I can't help you there.
What language/framework are you coding this in? I think I'm going to make a self-contained class/module/whateverapplies that visualizes GEDCOM files using the same platform so that you can integrate it with your code later on if you wish.

EDIT: On a second thought, you might want to PM me what kind of data structures you use for representing all the data inside the program so I can use those directly. Of course, that would be later on when you have everything figured out. You could also consider setting up a Github project. It all depends on how much you want to be left to work in peace or want other people to become involved. I don't want to get pesky, so I won't insist. So yeah, give me a heads up if you want. I can make the visual output side.
Title: Re: DwarfFamily
Post by: Morphyum on October 17, 2013, 09:56:49 am
Im Coding in java.
Found a legend, where we have a family that goes 2 steps down, thats all i wanted to test, so the familys get linked together right.
Whats missing now is a interface, so people can select wich legend file they want to use and also the program has troubles with large xml files because java runs out of memory so optimisations left.

I Added a link to the github in the main Post.
Title: Re: DwarfFamily
Post by: Morphyum on October 17, 2013, 01:05:11 pm
Thx to Sallen you have a minimalistic interface now and you can select the xml you want to use.
Title: Re: DwarfFamily
Post by: fricy on October 17, 2013, 03:53:49 pm
ptw
Title: Re: DwarfFamily
Post by: Morphyum on October 17, 2013, 04:00:57 pm
Right now there are two problems:

1.) It seems, that sometimes the xml of the legends gets exported with errors and isnt aknowledged by my program as an xml file.
2.) If the xml gets to big(Long history) java runs out of memory
Title: Re: DwarfFamily
Post by: Quietust on October 17, 2013, 04:51:59 pm
It might be worth writing a DFHack plugin to export the data in a more compact form (all of the data should be available) and then use that in your program; this might also allow exporting additional information that's not included in the Legends dump.
Title: Re: DwarfFamily
Post by: Putnam on October 17, 2013, 05:27:29 pm
Yeah, especially since familial relations are easy to find with DFHack. A historical figure has a vector of histfig links, which reference a target historical figure. For example, this function gets a unit's father in Lua:

Code: [Select]
function getUnitsFather(unit,numberIsNotFig)
 --pretend it's function overloading :V
    local fig =
        (df.unit:is_instance(unit) and df.historical_figure.find(unit.hist_figure_id) or
        df.historical_figure:is_instance(unit) and unit or
        type(unit)=="number" and numberIsNotFig and df.historical_figure.find(df.unit.find(unit)) or df.historical_figure.find(unit)
        )
    for k,link in ipairs(fig.histfig_links) do
        if df.histfig_hf_link_fatherst:is_instance(link) then return df.historical_figure.find(link.target_hf) end
    end
end

The fig stuff can be mostly ignored (I mostly wanted to create a working function that would accept any reasonable argument you could throw at it); this is the important part:
Code: [Select]
    for k,link in ipairs(fig.histfig_links) do
        if df.histfig_hf_link_fatherst:is_instance(link) then return df.historical_figure.find(link.target_hf) end
    end
Title: Re: DwarfFamily
Post by: Morphyum on October 17, 2013, 05:51:20 pm
yea you guys might be true, i think with the xml file attempt i will only come so far. Guess i have to check dfhack when i find the time.

Quote
A historical figure has a vector of histfig links, which reference a target historical figure.
Thats how it is in the xml too.
Title: Re: DwarfFamily
Post by: Morphyum on October 18, 2013, 03:08:17 am
Yea The xml files created by DF are just far to big and buggy to be handeled, so guess i have to use DFHack.

Also i will give StAX a chance before i give up.
Title: Re: DwarfFamily
Post by: Sallen on October 18, 2013, 08:30:04 am
Do dwarves adopt children? I'm seeing people with a mother (no father), but they are not listed as children of said mother. Or would this be a parsing bug?
Title: Re: DwarfFamily
Post by: Morphyum on October 18, 2013, 09:09:37 am
I will look into it, but its unlikely, that its a parsing bug, thats the kind of problems, that the xml has.
Title: Re: DwarfFamily
Post by: Morphyum on October 18, 2013, 11:24:26 am
1) I switched to StAx from DOm for the XML reading and that improved the speed extremly. It also solved the memory overflow.
2) i found the error in the xml dump:
<historical_figure>
      <id>-100</id>
      <name>autochop/config abbeybrew</name>
      <appeared>0</appeared>
      <birth_year>0</birth_year>
      <birth_seconds72>0</birth_seconds72>
      <death_year>0</death_year>
      <death_seconds72>0</death_seconds72>
      <breed_id>0</breed_id>
   </historical_figure>

Anyone knows what that is, and why it is in there?
Title: Re: DwarfFamily
Post by: Ruhn on October 18, 2013, 12:42:27 pm
If that is the only entry with a negative ID#, I would say it is hard coded to trigger special logic within DF?
You could add your own hard code to ignore that (if every game has the same entry).
Title: Re: DwarfFamily
Post by: Morphyum on October 18, 2013, 12:46:40 pm
The point is as far as i noticed it only appears in already played worlds. In newly generated ones its not in there.
Also i cant hard code an exception there, because my xml api doesnt recognize the file as UTF-8 anymore cause of this.
Title: Re: DwarfFamily
Post by: Putnam on October 18, 2013, 02:06:24 pm
That's code left by DFHack's persistent configuration. You'll want to just have it parse figs with positive IDs.

autochop/config abbeybrew means that it's the settings for autochop in the fortress called abbeybrew.
Title: Re: DwarfFamily
Post by: Morphyum on October 18, 2013, 02:24:44 pm
Yea apparently that wasn't the problem for the missencoding ::)
I just settled on reencoding all the legends xmls to ISO and now everything works.

Is there any special place where i should upload a first test version?
Title: Re: DwarfFamily
Post by: Sallen on October 18, 2013, 06:11:01 pm
I'd say the first post on this thread is as good a place as any. If you don't want to use Dropbox or something like that I can host it on my web server.

EDIT: I will probably begin work on the visualization this Sunday. Now that I finally got my home computer to behave like it should.
Title: Re: DwarfFamily
Post by: Morphyum on October 18, 2013, 07:20:25 pm
The program still has a problem with large xml files, but i dont know what the problem is. All the dwarfs get read but the linking doesnt happen o.O
Title: Re: DwarfFamily
Post by: Morphyum on October 19, 2013, 11:12:45 am
Found the error, apparently i had so many integers, that java decided to use different objects for same numbers. Ok so this version is testing ready, just need a place to upload now.
Title: Re: DwarfFamily
Post by: Morphyum on October 19, 2013, 11:47:02 am
Ok Edit on Entry Post, have fun testing ;-)
Title: Re: DwarfFamily
Post by: Meph on October 27, 2013, 01:40:17 pm
Good luck with this, it sounds great for succession forts. :)

Maybe you could add a screenshot to the first post, as an example of how the finished product would look for the player.
Title: Re: DwarfFamily
Post by: Morphyum on October 27, 2013, 03:53:44 pm
Sadly there's not much to screenshot yet, since the output is only a file. How it looks depends on the program you use to open the file.
Title: Re: DwarfFamily
Post by: Sallen on October 31, 2013, 04:59:12 pm
Quote from: Meph on October 27, 2013, 01:40:17 pm
Good luck with this, it sounds great for succession forts. :)

Maybe you could add a screenshot to the first post, as an example of how the finished product would look for the player.

I'm sorry. I am supposed to be working on an in-house data visualizer, but sadly my day job has been keeping my hands full. No matter how hard I work there's always more waiting for me >.<

I have not forgotten about this. As soon as I'm able I will dig into it.
Title: Re: DwarfFamily
Post by: rmblr on November 03, 2013, 05:30:52 am
I couldn't find the github link in this thread, but searching github revealed it :)

https://github.com/Morphyum/DwarfFamily
Title: Re: DwarfFamily
Post by: rmblr on November 03, 2013, 06:08:00 am
Hm, the XML parser is blowing up on some "invalid" unicode characters in my legends XML.

They might be values saved by dfhack, I'm not sure. Anyone encountered this?
Title: Re: DwarfFamily
Post by: Morphyum on November 03, 2013, 06:27:29 am
Hmm good question I don't think I change the XML at all, but I encountered Unicode problems with the XML too. That's why I had to switch from reading it in UTF8 to ISO.
Title: Re: DwarfFamily
Post by: rmblr on November 03, 2013, 07:38:12 am
It is because some dfhack plugins write binary data into the XML for persistent storage :(
Title: Re: DwarfFamily
Post by: Morphyum on November 03, 2013, 10:39:13 am
Yea but my program doesn't care normally. For me it works fine nomatter what the xml contains.
Title: Re: DwarfFamily
Post by: GM-X on November 03, 2013, 01:17:04 pm
Cool. Downloading.  :)
Title: Re: DwarfFamily
Post by: Meansdarling on November 05, 2013, 11:10:56 pm
PTW
Title: Re: DwarfFamily
Post by: Ruhn on November 30, 2013, 08:23:10 am
I finally used this and have a little feedback.  It would have been nice to have some type of status message or popup to announce that the import and export operations completed successfully.  Also it would have been cool if the GED file kept the region name from the imported XML.

The "View Dwarf List" portion is pretty cool, you should mention it in OP.  Here are some ideas for that:
* Have an option to hide non-parents
* Update the details pane when scrolling with arrow keys.  (Or update when hitting Enter)
* Clicking the name of a Mother or Father could jump over and select them on the list

It wasn't hard to figure out that MyHeritage tool, but a few notes would help since it is most likely what noobs like me will download.
1) Register
2) Create a project
3) Import GEDCON file created by DwarfFamily
Title: Re: DwarfFamily
Post by: VerdantSF on December 03, 2013, 02:48:21 pm
I just gave this a try and it was great to track through the family trees!  Thanks for creating this!
Title: Re: DwarfFamily
Post by: jaked122 on December 16, 2013, 08:51:38 am
Quote from: Morphyum on October 17, 2013, 08:14:39 am
  • Use the dwarf.ged in a program of your choice(For example MyHeritage (http://www.myheritage.com/family-tree-builder))
You have, a somewhat strange taste in geneological programs. That being said, this is fairly awesome.

It also seems to output broken GED files.
Title: Re: DwarfFamily
Post by: Sallen on December 27, 2013, 08:16:57 am
Issued a pull request. Nothing major yet.
Title: Re: DwarfFamily
Post by: Morphyum on January 09, 2014, 04:03:43 am
Havent gotten an email strangely, sorry for the late merge
Title: Re: DwarfFamily
Post by: Sallen on May 22, 2014, 07:00:44 am
Issued a pull request. This is what I changed:

Code: [Select]
Added .DS_Store to .gitignore
Dwarf names are now clickable hyperlinks in description for easier navigation.
Made progress bar be more informative when importing an XML file.
Gender is now displayed with proper casing.
Names in dwarf data are now displayed in bold.
Fixed: some clicks on the list would get ignored.
Changed export button label.

If you are busy you can tell me and I'll post a built JAR.
Title: Re: DwarfFamily
Post by: Morphyum on May 22, 2014, 08:37:05 am
Thanks Sallen, and thanks to all others that gave feedback im pretty bussy right now, so i couldnt work myself on it. But im Happy that Sallen keeps this going for now. I Uploaded the new Version und DFFD the download link is still on the first post :)
Title: Re: DwarfFamily
Post by: AoshimaMichio on June 03, 2014, 05:20:25 pm
Could you alter it to figure familytrees for other races? I'm partically interested about bloodline of megabeasts since I managed to capture a breeding pair of Rocs.
Title: Re: DwarfFamily
Post by: Sallen on June 09, 2014, 07:55:32 am
Quote from: AoshimaMichio on June 03, 2014, 05:20:25 pm
Could you alter it to figure familytrees for other races? I'm partically interested about bloodline of megabeasts since I managed to capture a breeding pair of Rocs.

That's a very good idea. I have no clue about how the XML file is structured, but I think we are loading that data at the moment anyway. I'll see if I can do something (not this week though, hands full).
Title: Re: DwarfFamily
Post by: Sallen on July 15, 2014, 07:27:41 am
Issued a pull request. This is the changelog:

Code: [Select]
· Open file dialog now filters and only allows XML files.
· In dwarf list, names list automatically jumps to selected dwarf when clicking on hyperlinks.
· Can now navigate dwarf list with arrow keys.
· Fixed bug that caused involuntary activation of hyperlinks.
· Added combobox to list window. You can browse any race on file.

The big thing is browsing any of the races on the XML file, of course. You can select the race you want right at the "dwarf list" window through a combo box. It defaults to dwarves.
Title: Re: DwarfFamily
Post by: Morphyum on July 15, 2014, 07:57:50 am
Hey Sallen having trouble to compile your newest version.

your using some extra librarys?
_xmlFileFilterImpl cannot be resolved to a type
debugwindow line 79
and control line 249
The method sort(Comparator<? super String>) in the type ArrayList<String> is not applicable for the arguments (Comparator<Comparable<? super Comparable<? super T>>>)
he cant do the sort
Title: Re: DwarfFamily
Post by: Morphyum on July 15, 2014, 08:18:58 am
Ok removed the FileFilter for now, everythings working now. GED export is still dwarfs only and after changing the race you have to close the details window to select another race else it wont update for the new race.

New download in start Post.
Title: Re: DwarfFamily
Post by: Sallen on July 15, 2014, 11:26:13 am
Crap, I forgot to stage the file for the filter, and that's why it wouldn't find it. It's not there! :D
GED export is untouched. But the details window should close and reopen automatically when selecting a new race. Mmmh. I only tested on OSX. I'll debug now that I'm at my Windows box.

EDIT: I see what you mean. Will try to fix.

EDIT 2: Done. Pull request issued. Sorry for pushing out crappy code.
Title: Re: DwarfFamily
Post by: Morphyum on July 15, 2014, 12:09:30 pm
Np atleast your working on it, since im still busy :-/ Will see if i can make the GED export available for all the races later this month.

Fixed version 0.3.1 is now available on the usual site in the first post.
Title: Re: DwarfFamily
Post by: Nagidal on July 15, 2014, 02:54:35 pm
PTW. Interesting ideas in here.
Title: Re: DwarfFamily
Post by: Sallen on July 16, 2014, 12:11:08 pm
I've done some stuff, but I feel the drawing of the tree is going to take a lot of work. I got the dwarves' names being drawn, but before I continue I think I'm going to refactor the code a bit (read: start over). That way I hope I'll get to wrap my head around the evil logic of recursive tree building.

I've had way to much tea already and need a shower...
If anyone reads this that is interested in helping out with this, by all means PM me or come see me: http://github.com/tjohnman/DwarfFamily (http://github.com/tjohnman/DwarfFamily). Warning: if you're an accomplished Java programmer this code will probably make you cry.
Title: Re: DwarfFamily
Post by: Morphyum on July 16, 2014, 12:35:44 pm
It's been growing a bit out of control I support his opinion yea xD
Title: Re: DwarfFamily
Post by: Sallen on July 17, 2014, 09:11:41 am
I'll see if I can rewrite some (most?) code this weekend before moving forward from a cleaner state.
Title: Re: DwarfFamily
Post by: Icefire2314 on July 18, 2014, 02:03:03 pm
Is it possible to make an in-program menu showing the family tree? Or do you export then do it?
Title: Re: DwarfFamily
Post by: Sallen on July 19, 2014, 02:10:19 pm
Quote from: Icefire2314 on July 18, 2014, 02:03:03 pm
Is it possible to make an in-program menu showing the family tree? Or do you export then do it?

The program currently lets you navigate the family tree in a non-visual way by clicking hyperlinked names of children and spouses. I'm working on a visual mode that draws the actual tree of a selected dwarf and lets you pan around looking at their relatives. It's not available in the last version yet though.

Currently the only way to look at family visually is to use the experimental GEDCOM export function.
Title: Re: DwarfFamily
Post by: Dutchling on July 30, 2014, 09:38:07 am
I assume this isn't updated for the new DF version?

Cause if it is, I have an error for you:
Spoiler (click to show/hide)
Title: Re: DwarfFamily
Post by: Morphyum on July 30, 2014, 09:59:55 am
Is the list in out application showing correctly? The GED export is experimental and might be outdated by now.
Title: Re: DwarfFamily
Post by: Dutchling on July 30, 2014, 10:02:12 am
You mean the Dwarf list? That seemed fine.

At least, it was a long list of Dwarf names :P
Title: Re: DwarfFamily
Post by: NothingClever on December 31, 2014, 07:21:39 pm
For future reference the 0 HEAD error from GenealogJ can be fixed by opening the dwarf.ged file in a text editor and adding 0 HEAD as the first line. I also had to add the line 0 TRLR as the last line. After that GenealogJ opens the file just fine.

It chokes on the DF date system but if you do a quick find and replace and replace the DF months with the first three letters of the appropriate real month everything works great, even the timeline.
Title: Re: DwarfFamily
Post by: Itnetlolor on January 01, 2015, 03:32:31 am
This might actually get interesting. Good for LP's and megaproject logging in-character and such. Adds more depth to story, if anyone's willing to go that far with the storytelling.
Title: Re: DwarfFamily
Post by: Ltan on May 11, 2017, 10:19:15 am
I really hate to necropost, but I made a few changes to the program and wanted to know if I could post them to git as a new branch perhaps?

Things I added:

All of the header and footer lines needed to work with gramps and other genealogy programs.
GIVN and SURN fields that split the NAME at the first and last words in the name string.
Added / around the surname/ last name in the string to flag that as the surname.

It is a bit cludgy as I don't really know Java and used Google to figure out what I needed for these additions.

Let me know if the changes are wanted or not?
Title: Re: DwarfFamily
Post by: Morphyum on May 11, 2017, 02:27:36 pm
Have you forked on github? Then just submit a merge request. Else pm me and I give you my email and look into the code. If everything else is still running fine I see no problem integrating your chances since I am not really working on this project anymore, considering I haven't started DF in over a year.
Title: Re: DwarfFamily
Post by: Ltan on May 11, 2017, 03:10:07 pm
Done. 

Oh...  I think that commit has it to where it will store the .ged file inside the legends folder where the .xml file is imported from.  It also names the file the same name as the legends file.
Title: Re: DwarfFamily
Post by: Morphyum on May 12, 2017, 10:14:03 am
Ok Version 0.4 is uploaded on DFFD and edited the description, thanks Ltan :)
Title: Re: DwarfFamily
Post by: Ltan on May 12, 2017, 02:40:52 pm
Not a problem at all Morphyum!  Was a nice little bit of learning for me, and a program that I will get some use out of :D