Post by: Mason11987 on July 06, 2010, 09:49:26 pm
Spoiler: Original Post (click to show/hide)
Post by: cameron on July 06, 2010, 09:56:16 pm
--edit found it
http://dffd.wimbli.com/file.php?id=2637
Post by: Mason11987 on July 06, 2010, 10:00:05 pm
now if only there was a link to it
--edit found it
http://dffd.wimbli.com/file.php?id=2637
I just copied from dffd, hilarious... Thanks.
Post by: Shima on July 06, 2010, 11:15:37 pm
Post by: Acanthus117 on July 06, 2010, 11:17:47 pm
Post by: Shima on July 06, 2010, 11:21:25 pm
Post by: Mason11987 on July 06, 2010, 11:23:49 pm
Now THIS is an interesting program.
This is great! Thumbs up from me!
Thanks!
I made some improvements: http://dffd.wimbli.com/file.php?id=2637
Think it might be possible to make a version that parses text data straight from DF's Legend's mode, so we can get a detailed Legends Mode mockup that can be reviewed out of the program? Or something like that, I'm no coder.
Perhaps in the future. I can't find an easy way to dump the rest of the info that's in legends mode outside those two files, but I'll take a look once I get the ability to view this stuff how I like.
Post by: MagicGuigz on July 06, 2010, 11:32:29 pm
Post by: Phenixmirage on July 06, 2010, 11:45:23 pm
A few suggestions:
Make the Civ list sortable by race and the Deity and Site lists sortable by Civ (and possibly race).
In the Deity list, it seems the elven deities descriptions just show up as "of", rather than "deity of (blah)". Also, would it be possible to list the civ the deity is attached to (similar to what's been done for the Sites) as well as gender?
Judging by a quick glance through the included world files, it looks like the program doesn't show accented letters. I don't know if this is by design or not, but I'd appreciate support for them.
Add an option to export info for one or more civs to a text file (all sites, rulers, deities, etc).
Rather than requiring the world info files to reside in the same folder as the program and have specific file names, allow the user to select the files that they'd like to load.
Keep up the good work, this program looks promising. :)
Edit: It looks like a new version came out while I was typing. Some of these suggestions may be redundant. ^^;
Post by: Mason11987 on July 06, 2010, 11:54:50 pm
The program worked great with the included world info files, but when I tried to use my own world files I get a "Run-time error '5': Invalid procedure call or argument" pop-up, and then it closes. :( I liked what I saw with the default files though. :)
Could you upload them to me? I find http://www.senduit.com/ to be useful, do day+ please.
A few suggestions:
Make the Civ list sortable by race and the Deity and Site lists sortable by Civ (and possibly race).
All sorts of sorts are on the horizon :)
In the Deity list, it seems the elven deities descriptions just show up as "of", rather than "deity of (blah)". Also, would it be possible to list the civ the deity is attached to (similar to what's been done for the Sites) as well as gender?
I see "force of", for some reason in the world gen, elven civs worship "forces" not "deities", do you not see "force of"? Civ on deity is on the list, I don't see the gender in those files though, am I missing it?
Judging by a quick glance through the included world files, it looks like the program doesn't show accented letters. I don't know if this is by design or not, but I'd appreciate support for them.
Yeah, that's a flaw with the language I'm using (VB6) and the controls it uses. There are ways around it, eventually I'll pursue one of them.
Add an option to export info for one or more civs to a text file (all sites, rulers, deities, etc).
Neat idea! I might do this.
Rather than requiring the world info files to reside in the same folder as the program and have specific file names, allow the user to select the files that they'd like to load.
Of course, this was simple to start with.
Keep up the good work, this program looks promising. :)
Thanks!
Edit: It looks like a new version came out while I was typing. Some of these suggestions may be redundant. ^^;
As you probably know, I like to make things with minimal functionality and throw it out there so I know whether it's worth bothering expanding it to something more useful based off of response, it looks like people might find this useful with more work. Thanks!
Post by: Phenixmirage on July 07, 2010, 12:26:14 am
Quote
The program worked great with the included world info files, but when I tried to use my own world files I get a "Run-time error '5': Invalid procedure call or argument" pop-up, and then it closes. :( I liked what I saw with the default files though. :)Here you go: http://senduit.com/ec1631 (http://senduit.com/ec1631) (I set it for 2 days.)
Could you upload them to me? I find http://www.senduit.com/ to be useful, do day+ please.
Quote
In the Deity list, it seems the elven deities descriptions just show up as "of", rather than "deity of (blah)". Also, would it be possible to list the civ the deity is attached to (similar to what's been done for the Sites) as well as gender?I tried the 1.1 version, and the elven forces still show up as just "of". (I'm on Windows XP 64, if it makes a difference.) As for deity gender, I thought I remembered deities showing up as "god" or "goddess" in the info files, but it looks like I got it mixed up with Legends mode. Sorry for the confusion. ^^;
I see "force of", for some reason in the world gen, elven civs worship "forces" not "deities", do you not see "force of"? Civ on deity is on the list, I don't see the gender in those files though, am I missing it?
Edit: Associated deities show up on the Civilizations tab just fine. Should I be seeing the Civ association under the Deities tab too?
Post by: Mason11987 on July 07, 2010, 06:02:59 am
QuoteThe program worked great with the included world info files, but when I tried to use my own world files I get a "Run-time error '5': Invalid procedure call or argument" pop-up, and then it closes. :( I liked what I saw with the default files though. :)Here you go: http://senduit.com/ec1631 (http://senduit.com/ec1631) (I set it for 2 days.)
Could you upload them to me? I find http://www.senduit.com/ to be useful, do day+ please.QuoteIn the Deity list, it seems the elven deities descriptions just show up as "of", rather than "deity of (blah)". Also, would it be possible to list the civ the deity is attached to (similar to what's been done for the Sites) as well as gender?I tried the 1.1 version, and the elven forces still show up as just "of". (I'm on Windows XP 64, if it makes a difference.) As for deity gender, I thought I remembered deities showing up as "god" or "goddess" in the info files, but it looks like I got it mixed up with Legends mode. Sorry for the confusion. ^^;
I see "force of", for some reason in the world gen, elven civs worship "forces" not "deities", do you not see "force of"? Civ on deity is on the list, I don't see the gender in those files though, am I missing it?
Edit: Associated deities show up on the Civilizations tab just fine. Should I be seeing the Civ association under the Deities tab too?
I had genned a huge, 1000 year world in the hopes of getting all possible scenarios in my 100 civs. Turns out lines can be "inherited", but not "inherited FROM" somewhere, which seems bizarre, but is now an acceptable scenario.
It now works with your files, if you want to test more and send me any sets that don't work, that'd be great.
I still can't see the deity just "of" situation, how bout a screenshot? Thanks Edit: nvm, I was able to recreate it
Post by: Mason11987 on July 07, 2010, 06:03:52 am
Added: List of civs for deities (turns out there can be several!)
Subtle changes. Double-clicking blue text, the sub lists in the "civilizations" tab, or the civ list in the deity tag will bring you more information about the item you're selecting. This isn't really intuitive enough as I'd like, but it's useful and later will become more obvious.
For example, go to civs tab, pick a civ with people, select a person who's worshipped, click the blue name of the object of worship.
You're brought to that item on the deities tab, double-click a civ, and you're brought to that item on the civ list.
Post by: Mel_Vixen on July 07, 2010, 07:56:21 am
Use:
You can check out the program as it by using my included world (a 1025 year world: Adreramul - The Past Plane), or by replacing the worldhistory.txt or the worldsites.txt file with your own world files.
When the program opens click "Parse New World", you can click "Remove Dead Civs" to remove all civs that don't have any existing sites.
I am surprised that you didnt use the XML export. Iirc it has all info in one file including beast (mega forgotten titan) and some other stuff.
Post by: Mason11987 on July 07, 2010, 08:08:34 am
Use:
You can check out the program as it by using my included world (a 1025 year world: Adreramul - The Past Plane), or by replacing the worldhistory.txt or the worldsites.txt file with your own world files.
When the program opens click "Parse New World", you can click "Remove Dead Civs" to remove all civs that don't have any existing sites.
I am surprised that you didnt use the XML export. Iirc it has all info in one file including beast (mega forgotten titan) and some other stuff.
It says "incomplete" and whenever I tried to use it (on a 1000+yo world) it didn't work. Due to the "incomplete" note, I assumed this wasn't completely user error. I'll try to work with it some more though and use it to supplement this info.
Also, I uploaded a new version so you can view important people in a full list.
Post by: BodyGripper on July 07, 2010, 08:11:27 am
Post by: Mel_Vixen on July 07, 2010, 08:21:18 am
Post by: Mason11987 on July 07, 2010, 08:29:37 am
This looks really cool, I'll check it out when I'm done with my current fortress.
One thing I did while testing is duplicate a save, and then abandon the duplicate fortress, then I can view the legends at the current time, I found that really interesting.
Well the Xml seems fine. My Xml-editor (the files are to big for the basic editor as it seems.) gives me a minor error thought.
Interesting, I'll see what I can do. Thanks!
Edit - Exported XML currently 210MB and growing... I'm growing skeptical of my ability to deal with this.
Post by: Mel_Vixen on July 07, 2010, 08:43:33 am
edit: I was actually thinking on moding a "open source" "desktop wiki" to get the info into a presentable form.
Post by: Mason11987 on July 07, 2010, 08:56:10 am
It stops at 620 MB roughly. just use the xml of an young world to get the tags.
edit: I was actually thinking on moding a "open source" "desktop wiki" to get the info into a presentable form.
Yeah, I had 480MB. That would be useful. I'm trying to do a little more then JUST what the legends mode show. For example: I'm going to get overall data like the total population of each civ, I'm going to try to get a visual of dynasties of various civs (it's possible to tie individuals to their parents through the Inheritance tag, and using math with birth/death) and a few other ideas. But I won't be pulling in full history (how someone died, for example). So something like what you're describing would be useful for that in particular.
Post by: Ratbert_CP on July 07, 2010, 09:15:17 am
Invalid property array index
Shows the name of a Kobold civ, but dies...
I've uploaded the world data files (http://senduit.com/d4d2e7) and set them for 2 days. This looks like a really interesting app, and I look forward to using it. :)
Post by: Phenixmirage on July 07, 2010, 10:33:18 am
Subtle changes. Double-clicking blue text, the sub lists in the "civilizations" tab, or the civ list in the deity tag will bring you more information about the item you're selecting. This isn't really intuitive enough as I'd like, but it's useful and later will become more obvious.
For example, go to civs tab, pick a civ with people, select a person who's worshipped, click the blue name of the object of worship.
You're brought to that item on the deities tab, double-click a civ, and you're brought to that item on the civ list.
I like the ability to cross reference things. :) Sometimes though, civ leaders worship megabeasts which don't show up under the civ deities. This might cause a crash if the program goes looking for info that isn't there. (I'm not at home at the moment, so I can't check on it myself.)
Post by: Mason11987 on July 07, 2010, 10:37:36 am
Run-time error '381':
Invalid property array index
Shows the name of a Kobold civ, but dies...
I've uploaded the world data files (http://senduit.com/d4d2e7) and set them for 2 days. This looks like a really interesting app, and I look forward to using it. :)
Fixed on most recent version, Kobold civs were special exceptions (because they just have the name in the world history file, no people) and having them earlier in the list caused a hiccup, now it shouldn't matter.
Subtle changes. Double-clicking blue text, the sub lists in the "civilizations" tab, or the civ list in the deity tag will bring you more information about the item you're selecting. This isn't really intuitive enough as I'd like, but it's useful and later will become more obvious.
For example, go to civs tab, pick a civ with people, select a person who's worshipped, click the blue name of the object of worship.
You're brought to that item on the deities tab, double-click a civ, and you're brought to that item on the civ list.
I like the ability to cross reference things. :) Sometimes though, civ leaders worship megabeasts which don't show up under the civ deities. This might cause a crash if the program goes looking for info that isn't there. (I'm not at home at the moment, so I can't check on it myself.)
Thanks,
That's funny, because I didn't realize that and that was one of the bugs that did come up and the files Phenixmirage uploaded to be investigated pointed out exactly that problem, which is now fixed.
I could have used you yesterday :P
Post by: Phenixmirage on July 07, 2010, 10:58:32 am
Thanks,
That's funny, because I didn't realize that and that was one of the bugs that did come up and the files Phenixmirage uploaded to be investigated pointed out exactly that problem, which is now fixed.
I could have used you yesterday :P
If I'd thought about it yesterday I would've mentioned it, but it was late and I was tired. :)
Post by: Mason11987 on July 07, 2010, 01:39:32 pm
Changes:
Various sorting options
View Site/Civ Populations
Post by: Knight Otu on July 07, 2010, 02:02:42 pm
Post by: Mason11987 on July 07, 2010, 02:20:05 pm
A bit unfortunate that it doesn't really support mods (yet?), but it looks good so far.
the background is being set up to allow for multiple different intelligent species. If you'd like to generate worldsites/worldhistory files and send them to me I could work on getting it compatible with that mod as well.
If a mod does something to those files besides adding new species.... I'm not sure how it'll work. I'd have to see. I don't use mods a lot.
Post by: nil on July 07, 2010, 02:20:36 pm
Post by: Mason11987 on July 07, 2010, 02:22:34 pm
If it doesn't support mods I definitely encourage you to hook that up, I couldn't have make my 40d mod without that version's world-gen parser (which wasn't nearly as slick as this).
http://www.senduit.com/
zip up some worldsites/world history files from some mods and post the link, I'll work on it.
Post by: Mason11987 on July 07, 2010, 10:28:29 pm
- Added a "Races" tab, which lets you view the information about the races in general. Can find out how many dwarves exist in the world, or how many human sites exist.
- Added a "World data" tab, which gives a few bits of world-wide cumulative information
- Drastically sped up sorting of all sorts.
It doesn't yet support mods with additional races, but it's fairly close.
Post by: Knight Otu on July 08, 2010, 01:41:18 pm
http://www.senduit.com/
zip up some worldsites/world history files from some mods and post the link, I'll work on it.
http://senduit.com/ebe6b7 (3 days)
The _DF files belong together (from my Direforged mod), as do the _Z files (another mod I'm creating). Hope they help.
Post by: Mason11987 on July 08, 2010, 08:18:28 pm
http://www.senduit.com/
zip up some worldsites/world history files from some mods and post the link, I'll work on it.
http://senduit.com/ebe6b7 (3 days)
The _DF files belong together (from my Direforged mod), as do the _Z files (another mod I'm creating). Hope they help.
Ran _df set, it didn't work, as expected.
I made some improvements until it parsed correctly.
I loaded my default set, it failed, I fixed the problems until it parsed correctly
Then I loaded your _z set for the first time, and it appeared to parse correctly on the first try! Success!
Version 1.5 is up, and it SEEMS like it supports worldhistory/worldsite files from modded DF versions.
Post by: Knight Otu on July 09, 2010, 02:15:42 pm
Post by: cephalo on July 10, 2010, 09:42:46 pm
Post by: Mason11987 on July 10, 2010, 10:35:12 pm
As I was going to say in another thread, I would sooo be willing to wait on a 400mb XML dump for a graphical world viewer. It would be awsome to be able to look stuff up while playing fort mode without having to abandon and back up saves.
I'll give it a shot, see what's possible for me to do. Maybe sometime next week I'll give it a shot.
Post by: cephalo on July 10, 2010, 10:49:46 pm
As I was going to say in another thread, I would sooo be willing to wait on a 400mb XML dump for a graphical world viewer. It would be awsome to be able to look stuff up while playing fort mode without having to abandon and back up saves.
I'll give it a shot, see what's possible for me to do. Maybe sometime next week I'll give it a shot.
Since DF also exports the world bitmap, all you really have to do graphics wise is set up a scrollable picture box and stick it in there. Then just calculate the coordinates vs. mouse location. Then on mouse clicks maybe open some child windows with the available info.
Post by: pickupsticks on July 11, 2010, 10:23:42 am
Post by: Knight Otu on July 11, 2010, 03:32:37 pm
Post by: Mason11987 on July 11, 2010, 04:03:04 pm
One problem - it seems that the program always assumes that the world is in the year 1050. Maybe there should be a way to input the actual year somewhere?
Ha, I need to keep track of all the temporary fixes I put into the program so I can make sure I fix them later.
I'll release a fix that lets you input the date a little later today, later it'll find the date and pull it out itself (if possible).
Thanks for catching this.
Edit: I also discovered the program doesn't handle a few other situations:
- Two Kobold civs in a row in the world history
- Kobold's being renamed in a mod
- Another Kobold-like civ (one that doesn't have leaders/worship anything) being added by a mod
Yeah, so that's a good deal of real problems, time to fix!
Post by: Mason11987 on July 11, 2010, 05:22:56 pm
One problem - it seems that the program always assumes that the world is in the year 1050. Maybe there should be a way to input the actual year somewhere?
Ha, I need to keep track of all the temporary fixes I put into the program so I can make sure I fix them later.
I'll release a fix that lets you input the date a little later today, later it'll find the date and pull it out itself (if possible).
Thanks for catching this.
Edit: I also discovered the program doesn't handle a few other situations:
- Two Kobold civs in a row in the world history
- Kobold's being renamed in a mod
- Another Kobold-like civ (one that doesn't have leaders/worship anything) being added by a mod
Yeah, so that's a good deal of real problems, time to fix!
V 1.6
Added a popup that asks for the year (simple temporary measure), also fixed all of the above problems. I need more people with different mods to give this a shot and make sure it's robust enough.
V 1.7
There is now a map/site viewer! Check it out!
Spoiler (click to show/hide)
Post by: cephalo on July 12, 2010, 08:38:38 am
Post by: Mason11987 on July 12, 2010, 09:13:55 am
Wow, when you said "next week" you meant 'Monday'. I'm going to gen a new huge world and put this through the paces.
Actually, it was up Sunday around 5p :).
The code itself doesn't have any (obvious) limitations on how big the xml should be, and the map is set up to allow scrolling if it's a huge map so that will work, I haven't yet tested it with a huge XML file, but it SHOULD work. Something I didn't anticipate is two sites being located on the same coordinates of the map. I don't know if that can happen, but if it can, it won't work here.
Let me know if you break it.
Post by: cephalo on July 12, 2010, 10:26:03 am
The error pops up immediately, not as if it's choking on a huge file.
EDIT: Ok nevermind the above ^^^ that was my fault for improperly renaming my files.
However I did get this when I corrected it.
'Run-Time Error '14': Out of string space.'
That sounds more like it's choking on file sizes. My xml file is 827 MB!
EDIT2: Ok, poking around with the standard functionality, I get an error when I'm looking at sites and I click on 'Remove NonDwarf Sites'. Run-Time Error '13': Type Mismatch.
Post by: Mason11987 on July 12, 2010, 10:56:56 am
Ok, when I try to import with map+XML I get 'Run-Time Error '5': Invalid Procedure Call or Argument.'
The error pops up immediately, not as if it's choking on a huge file.
EDIT: Ok nevermind the above ^^^ that was my fault for improperly renaming my files.
However I did get this when I corrected it.
'Run-Time Error '14': Out of string space.'
That sounds more like it's choking on file sizes. My xml file is 827 MB!
EDIT2: Ok, poking around with the standard functionality, I get an error when I'm looking at sites and I click on 'Remove NonDwarf Sites'. Run-Time Error '13': Type Mismatch.
Oh man, that's what I was worried about, interesting. I'll have to get my own xml and see if I can reproduce it.
Edit - Fixed your "Edit2" problem. That was from before I made the game figure out race names (before I made it work for mods), and I hadn't clicked it in a while. It'll work next release.
Post by: Knight Otu on July 12, 2010, 12:43:05 pm
Post by: Mason11987 on July 12, 2010, 01:09:49 pm
It seems that when you parse a new world when you previously parsed a world, some of the old data remains intact (for example, if you re-parse the old world, some content is doubled). This appears to be on the people and the races tabs only, though.
People: Found (obvious) problem, should be fixed.
Races: I don't see the same bug, and I couldn't recreate it. Let me know if it's still happening.
To be blunt - I haven't really done any cleaning up to allow safe re-parsing. To be safe (until I can look into it) just restart the program when parsing again or parsing a new world.
V 1.7.1
Map view should work for an arbitrarily large xml file.
Screenshot in OP/dffd shows a 400MB one I was able to explore in the map.
Post by: Knight Otu on July 12, 2010, 01:34:11 pm
Races: I don't see the same bug, and I couldn't recreate it. Let me know if it's still happening.Sorry, I should have been more specific there - On the races tab, the number of civilizations (to the right of the selection of races) of a given race is doubles (tripled...) when you reparse the world, whether dead or not.
To be blunt - I haven't really done any cleaning up to allow safe re-parsing. To be safe (until I can look into it) just restart the program when parsing again or parsing a new world.i wouldn't have done so normally, but in the case where I noticed this happening, I had first parsed the world without the XML information, and then with it, for a bit of a test of the XML parsing.
Post by: cephalo on July 12, 2010, 01:53:40 pm
EDIT: it looks like it partially worked, many site rectangles are on the map window, but there is no actual map in the window, perhaps it didn't get far enough to actually load the bitmap.
Post by: Mason11987 on July 12, 2010, 02:06:03 pm
Races: I don't see the same bug, and I couldn't recreate it. Let me know if it's still happening.Sorry, I should have been more specific there - On the races tab, the number of civilizations (to the right of the selection of races) of a given race is doubles (tripled...) when you reparse the world, whether dead or not.To be blunt - I haven't really done any cleaning up to allow safe re-parsing. To be safe (until I can look into it) just restart the program when parsing again or parsing a new world.i wouldn't have done so normally, but in the case where I noticed this happening, I had first parsed the world without the XML information, and then with it, for a bit of a test of the XML parsing.
no problem, found it. Fixed it. Thanks.
Understandable. If I only do the same things every run I'll only catch the same bugs :).
Ok I got a new error with my XML: 'Couldn't pull necessary sections of XML file' This was after the warning about the large xml file.
I knew putting in useful error messages would help me! Your xml was apparantly quite a bit bigger then my test one. We're getting close though, I think my next release should fix this.
In the meantime, take a look at this:
(http://imgur.com/94B4J.png)
I wanted some visual way to group sites belonging to the same civ together on the map, this is an ugly (but effective) way of doing that. Any good ideas on how to do this in a less ugly way? (I tried going behind the symbols, but I can't make it work).
Post by: Star Weaver on July 12, 2010, 02:22:02 pm
You could also probably find some algorythm online somewhere to remove interior edges in a 2d polygon, which would let you just draw a bordery line.
Looks awesome tho. Has the map view been tested with a square or other non-default tileset?
Post by: Mason11987 on July 12, 2010, 02:26:09 pm
Try drawing a circle around each site, and then offset the line start and end by that amount? That might require (simple) vector math and a normalizaiton (sqrt!) tho, to offset it in the right direction.
You could also probably find some algorythm online somewhere to remove interior edges in a 2d polygon, which would let you just draw a bordery line.
Looks awesome tho. Has the map view been tested with a square or other non-default tileset?
Not a bad idea, I'll give that a shot.
It hasn't, I didn't even think of that possibility. That might be a problem.
Post by: cephalo on July 12, 2010, 02:34:33 pm
Looks awesome tho. Has the map view been tested with a square or other non-default tileset?
It hasn't, I didn't even think of that possibility. That might be a problem.
People use alot of different character sets, and that's what gets exported. So you have to know the map size in some way versus the bitmap size, and divide by that in each direction to get the character width/height. Also keep in mind that it's possible to have non-square maps.
Post by: Star Weaver on July 12, 2010, 02:47:33 pm
If the map image is A pixels wide/tall, it is made of B tiles which are each C pixels wide/tall.
Just figure width and height seperately, for non-square maps :)
Spoiler (click to show/hide)
Post by: cephalo on July 12, 2010, 02:49:48 pm
Maybe you could draw one line to the nearest same-civ site.
Actually this would not work, I think the rule would have to be: One line to the nearest same-civ site that isn't already connected to me. Otherwise a two way connection could disconnect an empire. :D
Post by: Star Weaver on July 12, 2010, 02:57:48 pm
Hey, I just did some combination testing, and if my math is right, every possible dimension of the map is uniquely discernable to be a given tile size at given map size. This is CSV-style output, and can be read as:
If the map image is A pixels wide/tall, it is made of B tiles which are each C pixels wide/tall.
Just figure width and height seperately, for non-square maps :)Spoiler (click to show/hide)
Edit: To be clear, this is for tileset with dimensions ranging from 5 to 32 pixels, and it will not detect or misdetect other sizes. At some point there will have to be conflicts ... I think? Maybe power-of-two-plus-one is magical enough that it wouldn't. I don't think i've ever seen anything outside this range, though.
Post by: Mason11987 on July 12, 2010, 03:01:05 pm
anyways, here's my improvement on the lines
(http://imgur.com/CHJQt.png)
I took starweaver's idea of not hitting the site directly, made the lines brighter (just random not dark colors), and had every site just connect to one site (the central site is unremarkable, it's just the first in the worldsites file. Feel free to offer improvements suggestions on this. I'll release something in the next hour or so that is functioning (even with your larger xml cephalo)
Post by: JoRo on July 12, 2010, 03:14:44 pm
Post by: Mason11987 on July 12, 2010, 03:31:48 pm
I convert my world maps to .png as a matter of course, and the viewer only wants bitmaps. It's not a too big a hassle to just export the image again, but .png support would be handy.
I did a quick check and it seems like that isn't a simple addition, so I'll add it to the list.
V 1.8
Works with even larger xml files. Should be fine for almost anything now
added cave hiding and civ connecting to the map
added the ability to click the "Location:" coordinates on the "site" tab and have the map center on the location on the map. Now you can, for example... sort sites by population, click the largest site, and immediately see where it's located, so you can visit it in adventure mode quickly.
Post by: Star Weaver on July 12, 2010, 03:51:41 pm
Post by: Mason11987 on July 12, 2010, 03:56:21 pm
That's cool. Starting from their initial city (that is the first one listed, right?) is neat 'cause it gives you a clue how they moved in worldgen.
It makes sense it would be the first one, but then there aren't any unoccupied sites. I assume when people leave them they dissapear from that file at least. Then again I don't know that much about worldgen, so I couldn't be sure.
I've since set it up to have the "center" be the largest site in the civ, but if we find out the first one is always the "initial" one, then I'll leave it as it was.
Post by: BodyGripper on July 12, 2010, 05:56:07 pm
Post by: Mason11987 on July 12, 2010, 09:28:17 pm
Woah, a map? Sweet. I need to check back on this thread more. The original utility was cool, but the map will actually come in very handy.
Thanks!
V1.9
Something else I was planning is a way of automatically collecting data to show information about dynasties. It takes some work to get the information but form the worldhistory file you can find out who is the "father" or "mother" that are being referred to when it says a person inherited a post from them. I got all this information compiled for you:
Picture:
Spoiler (click to show/hide)
This I've already implemented in 1.9
Next thing I'm going to do is create an additional tab which will list all of the various dynastic lines, with cool depictions of how rule had been passed down over time. It's not perfect yet I'm sure, but I haven't found any bugs with who inheritied rule from who, but let me know if it makes any mistakes.
Post by: Sizik on July 13, 2010, 12:26:58 am
Post by: Mason11987 on July 13, 2010, 05:34:29 am
You should modify the text input so that accented characters show up correctly (or use the Terminal font).
Well that's cool. I didn't realize using the Terminal font would show all the text correctly. I'll make
V1.9.1
All the characters print correctly! Thanks Sizik!
Upcoming
Working on "dynasties", which are coming together much better then I expected!
Teaser Image:
(http://imgur.com/o1qYw.png)
Post by: pickupsticks on July 13, 2010, 08:19:25 am
(problem with the .zip on the most recent version, I had to use winrar to repair it before I could unpack it)
EDIT: I wonder if there's a way to get hold of the locations of the regions as well as the site names... my home-made maps of worlds that I was adventuring in always ended up with at least the mountain ranges and peaks listed, as well as things like rivers, oceans and deserts.
Post by: cephalo on July 13, 2010, 09:06:31 am
Also, it would probably help most people to use the windows zip function. Otherwise windows people will need to have a third party extractor.
Post by: Mason11987 on July 13, 2010, 11:10:29 am
Ok, still trying to use that same large file. This time I get, 'Run-Time Error '480': Can't create auto-redraw image'. Maybe the map is too big.
Also, it would probably help most people to use the windows zip function. Otherwise windows people will need to have a third party extractor.
Ah, thanks cephalo, my map was big, I guess yours was bigger and we've hit some limit I've never seen before. That's something I can test though, I'll just edit my image to be huge and see if I can't recreate it. Thanks!
I meant to use windows zip, I'm not sure why it zipped weird. This was the first one I didn't zip, then unzip and test, so I guess I'm paying the price for slacking on testing.
This is great!
(problem with the .zip on the most recent version, I had to use winrar to repair it before I could unpack it)
EDIT: I wonder if there's a way to get hold of the locations of the regions as well as the site names... my home-made maps of worlds that I was adventuring in always ended up with at least the mountain ranges and peaks listed, as well as things like rivers, oceans and deserts.
The names of the regions are there, and I could most likely pull them out of the XML, the problem is that I don't see any way that I can link locations to the map, to sites, to civs, or to people. The best I could do is just provide a listing of the region names with no other connection, which isn't that interesting I think.
If someone can fire up their xml viewers and show me the connection I'm missing I would love to add that, but so far I can't find anything.
Thanks for your testing and help everyone.
Post by: cephalo on July 13, 2010, 11:36:01 am
In order to crack this file, I think you have to write a program to pick out clues. Something that will pick out all the tags from historical events, and also all of the data in those tags that is not numerical and not names ( I don't believe there are names in historical events). Once you have a condensed list of possibilties, you might be able to recreate the legends associations *maybe*(The xml dump does say it is incomplete).
It would be great to be able to trace back a persons ancestry, not just royalty.
Post by: Mason11987 on July 13, 2010, 12:06:43 pm
That XML file is a beast. I used a large file text editor to look at it, seeing if I could figure it out. There are so many historical events that you cannot just open the file and learn by browsing. You might have a thousand of this event or that event in a row, way more than you would want to just read.
In order to crack this file, I think you have to write a program to pick out clues. Something that will pick out all the tags from historical events, and also all of the data in those tags that is not numerical and not names ( I don't believe there are names in historical events). Once you have a condensed list of possibilties, you might be able to recreate the legends associations *maybe*(The xml dump does say it is incomplete).
It would be great to be able to trace back a persons ancestry, not just royalty.
Yeah, the events are interesting but the sheer volume of text, and the difficulty in actually getting meaning from the information between two "historical_events" tags means that if it is attempted, it'll be far down on the list.
But what you're describing is exactly how my program works now with the sites portion of the XML. It pulls out the id/name/type/coordinates, then matches it to a site from the world history (matching id and name, just as a safety) and then adds the coordinates onto thate already created site. That way I can show where civs are on the map, even though that information doesn't actually exist directly in either file.
The same sort of thing is done in the worldhistory file, but it's more complicated. For getting who someone's parent is, for example:
I assign each person an array of all of their children's ages at their death.
I then go through every person, and if they inherited the throne from their father or mother (I don't deal with the paternal/maternal grandmother/father situation which DOES happen), then I find if there is a person who has a child born at the right time to be this person, and if they are from the same civ + race then I assume that the child is theirs.
Example:
John - died 200, children ages at death: 20, 30, 40 - CivA
Bob - inherited from father, born 180. - CivA
Since Bob was born in 180 and inherited from his father, he could (and almost certainly does) match the 20yo who survived John in 200.
This doesn't take into account the fact that Bob could be born in 180 and if John died in 200 Bob could technically be 19, but it appears the game doesn't take that into account either, so it doesn't matter.
In terms of ancestry that isn't royalty, I'll have to attempt to gen a world where "cull unimportant figures" is "off" and see what happens. So far I've actually never attempted that, but if it's in the files (worldhistory?) I'll do it eventually.
Post by: Shadowfury333 on July 13, 2010, 05:10:58 pm
Post by: Mason11987 on July 13, 2010, 05:49:56 pm
V 1.9.2
Fixed the zip also the beginnings of the dynasty feature.*
* - The included world didn't last long enough for any dynasties to form, so in the default set that'll be empty. In the future I'll hide the tab if there's nothing to go there.
Post by: Mel_Vixen on July 13, 2010, 06:03:33 pm
Post by: Mason11987 on July 13, 2010, 06:29:54 pm
If you want a small xml - just generate a pocket world over the course of say 20 years. Maybe you should also change the raws to lower the populations.
I have a small xml, that's how I got it, but 20 years meant no leader died and had an heir.
When I finish everything I want to do with the dynasties I'll gen a world that is small but shows everything the worldviewer can do.
Post by: LucasUP on July 13, 2010, 07:06:14 pm
It could maybe use a better/sleeker explanation on how to get world information and use it.
Post by: Star Weaver on July 13, 2010, 07:21:02 pm
Post by: Shadowfury333 on July 13, 2010, 07:41:21 pm
The files (http://rapidshare.com/files/406818558/WorldViewerData.zip) for debugging.
The default files work fine, by the way.
Post by: Mason11987 on July 13, 2010, 08:10:27 pm
This is cool. I don't have a world I'm particularly interested in at the moment though.
It could maybe use a better/sleeker explanation on how to get world information and use it.
Thanks! I'm going to work on more usability soon.
Hm, you could mod a race to become adult in 6 months and die after 4-7 years and see if that helps you test things with less data? :)
I have a huge world I test things on, I just don't want to upload a 400MB file to dffd when people can experience what the program does with a smaller xml, that's what it's for.
But that is a cool idea, I've never really modded anything though.
Argh, I'm still getting runtime error '5'. I put in the latest date, and the files are all in place from my world.
The files (http://rapidshare.com/files/406818558/WorldViewerData.zip) for debugging.
The default files work fine, by the way.
V 1.9.3
Somehow you genned a world that a had a site without any population? In all of my huge gens I never saw that so I didn't think it was possible. It was the last one though, so maybe a bug? I don't know. It should now work.
See!
Spoiler (click to show/hide)
Post by: Shadowfury333 on July 13, 2010, 08:11:51 pm
Spoiler (click to show/hide)
Thanks for the update.
Post by: Mason11987 on July 13, 2010, 08:19:12 pm
I think the site in question is my current fort (Roughnesschants) and my previous fort, which was a brief embark that may have gotten one wave of immigration and nothing else before it was abandoned without drama. The file doesn't seem to recognize the actual population, just the fact that it is dwarven.Spoiler (click to show/hide)
Thanks for the update.
That is exactly it. I genned a lot of worlds for this, but never genned, started a fort, abandoned it, then exported the legends. Since that is a lot of what people will use this for I have to make sure I do that more often.
I bet it doesn't have any population because you had to abandon the fort (everyone leaves) in order to get to legends.
All great info.
Edit - was Koganzedot yours too?
Post by: clc02 on July 13, 2010, 08:25:49 pm
Try curses_square 16x16 and see where they put the red squares. Perhaps have a option to ask for the tileset size if you can't figure it out somehow through the image.
Post by: Mason11987 on July 13, 2010, 08:36:10 pm
Not sure if this has been mention before but your 'overlay civs' option assumes that every tile set is your size, if you exported a map with a different sized tileset the positions are off.
Try curses_square 16x16 and see where they put the red squares. Perhaps have a option to ask for the tileset size if you can't figure it out somehow through the image.
Yup, someone brought it up and I can grab it automatically, just haven't done it yet. :)
Thanks though
V 1.10
Finished up most of what I wanted to do for dynasties, with sorting and all that.
Also, the map SHOULD work with different size tiles now (at least there is no longer any hard-coded references to 8 and 12).
I use a few (reasonable) assumptions to figure out the map size, because it's nowhere in any file. So if you see site markers twice the X or Y, or half the X or Y, let me know and send your xml file. I don't think this will happen though. If it does I'll use the table that was very usefully shared earlier.
Stuff I have on my to-do, listed in the order of what I'll handle next probably. These are some from requests, and some from me spotting my own bugs.
- Allow the user to select the files that they'd like to load.
- Give better instructions
- More Deity sorting options
- Manage worlds with "cull unimportant figures" off
- Export info for one or more civs to a text file (all sites, rulers, deities, etc).
- People listed twice, due to someone taking two important roles in the same civ within their lifetime
- Deal with sites with wrong race in it
- Parse other info out of legends (regions, events, era?)
- Support .png world image
edit -
V 1.10.1
There was a bug in the map tilesize auto-finding. I fixed it though. Should work fine now.
Post by: cephalo on July 14, 2010, 11:14:24 am
Post by: Mason11987 on July 14, 2010, 11:18:39 am
I'm still getting 'Run-Time Error '480': Can't create auto-redraw image'. I'd give you my files to test... but it's like a gig of data lol. This map I'm using is a large 'region', with no culling of historical figures. The 'region' vs. 'island' may be very important as a region probably has 4 times the playable surface that an island map has.
Right, I didn't specifically find + resolve your problem.
I've used a huge world though, so I'm not sure why it wouldn't work with your huge world... Could you just send your map? You can .png it and put it up on imgur, that should be fine. I can just replace my huge map with yours to see what happens.
Post by: cephalo on July 14, 2010, 11:49:02 am
I'm still getting 'Run-Time Error '480': Can't create auto-redraw image'. I'd give you my files to test... but it's like a gig of data lol. This map I'm using is a large 'region', with no culling of historical figures. The 'region' vs. 'island' may be very important as a region probably has 4 times the playable surface that an island map has.
Right, I didn't specifically find + resolve your problem.
I've used a huge world though, so I'm not sure why it wouldn't work with your huge world... Could you just send your map? You can .png it and put it up on imgur, that should be fine. I can just replace my huge map with yours to see what happens.
Here is a link to the png.
http://www.mediafire.com/?ynm2llifyq1tyny
Right now I'm genning another huge 'region' to see if it works.
Post by: Ratbert_CP on July 14, 2010, 11:56:33 am
Great tool! I love it!
Post by: Mason11987 on July 14, 2010, 12:05:06 pm
Here is a link to the png.
http://www.mediafire.com/?ynm2llifyq1tyny
Right now I'm genning another huge 'region' to see if it works.
Loads fine for me. You don't happen to have a particularly crappy computer do you? lol
(j/k, I'll add this to my list and will eventually fix it. Try genning slightly smaller worlds, and see what happens. thanks)
I just noticed that while your Civilization mapping calculations (boxes around sites, etc) is now working, the mouse position calculations seem to be off. You might want to normalize those as well... :)
Great tool! I love it!
Thanks! I think I just fixed it, try again.
The names of the regions are there, and I could most likely pull them out of the XML, the problem is that I don't see any way that I can link locations to the map, to sites, to civs, or to people. The best I could do is just provide a listing of the region names with no other connection, which isn't that interesting I think.
If someone can fire up their xml viewers and show me the connection I'm missing I would love to add that, but so far I can't find anything.
Thanks for your testing and help everyone.
Correction: I've read through the historical events (in the xml) and I think I understand them now. I can definitely pull out regions, and link regions to sites, people, and events. There's a lot we can get out of it which I'm pretty excited about. What would be awesome is a sort of visual timeline of sites/battels/civs, like what you can see at the end of a Civilization game. It can be done.
Post by: cephalo on July 14, 2010, 12:15:51 pm
Here is a link to the png.
http://www.mediafire.com/?ynm2llifyq1tyny
Right now I'm genning another huge 'region' to see if it works.
Loads fine for me. You don't happen to have a particularly crappy computer do you? lol
Actually, I do have a kinda crappy computer and this might be the issue. I'll update you on this next world gen.
Post by: cephalo on July 14, 2010, 12:46:32 pm
I have one more suggestion. It would be great if the map window grew along with the parent window. If I maximize the window, I wanna see more of the map.
Post by: Mason11987 on July 14, 2010, 01:51:15 pm
Ok, I just generated a large region with an XML file that is 1.4 GB, and it works! I don't know what the problem with the other files was...
I have one more suggestion. It would be great if the map window grew along with the parent window. If I maximize the window, I wanna see more of the map.
Good idea. Done
Only the map tab really does a resize though.
Post by: Shadowfury333 on July 14, 2010, 02:29:55 pm
I bet it doesn't have any population because you had to abandon the fort (everyone leaves) in order to get to legends.
All great info.
Edit - was Koganzedot yours too?
I haven't abandoned the fort (Dumatonshen "Roughnesschants") yet (check sig), that's the thing. Those files seem to be exported every time I play, but I guess they don't export populations when a fort is active.
Koganzedot was mine as well, and that was the quick embark which I abandoned after a season (I didn't like the embark point for my purposes). I'm surprised it doesn't have population, but I guess since nothing bad happened to it there was no reason to mark it? I don't know, recording of player forts is odd. Hopefully when the changes come down to make retired adventurers more stable similar changes will happen to fortresses so they can be put on hold or even left to work on their own, but not be destroyed by the player leaving.
Post by: Knight Otu on July 14, 2010, 02:45:31 pm
Spoiler: worldhistory (click to show/hide)
d. 5, Reign Began: 1), *** Original Line, Married
worldsites in next post
Post by: Knight Otu on July 14, 2010, 02:46:36 pm
Spoiler: worldsites (click to show/hide)
Post by: Mason11987 on July 14, 2010, 04:51:58 pm
Post by: Knight Otu on July 15, 2010, 05:29:30 am
Post by: Mason11987 on July 15, 2010, 08:07:49 pm
Doesn't seem to work. It might be because both rulers don't worship anyone (being demons), though the other Encar civ with only one leader that was a demon does get its worship list without problem.
V 1.10.4
Fixed it.
In my parsing I looked for the text " List", to know when I was done with the Worship List.
First deity in that civ was " Canci Listentruth". Ha.
I fixed it up so it shouldn't mess up like that again.
Would you mind trying it again?
Post by: Knight Otu on July 16, 2010, 04:51:07 am
Post by: Mason11987 on July 19, 2010, 06:54:19 pm
V 1.11
Not a major feature release but more of a "this should have came with the initial release" sort of thing.
Now no need to rename files, you can select the files you want to use.
(http://imgur.com/TTOmT.png)
Thanks for checking back :)
Post by: pickupsticks on July 19, 2010, 10:09:18 pm
Good work, thanks!
I was wondering what could be done with a detailed-world-map export, if someone made one of those available when parsing the world. Do you think you could get the sites and suchlike to line up on there as well? That would be a step towards incorporating evil and temperature and rainfall and all the other goodies, which some people might appreciate having along with everything else.
Watching this with interest :D
Post by: Mason11987 on July 19, 2010, 11:19:50 pm
That'll make things even easier.
Good work, thanks!
I was wondering what could be done with a detailed-world-map export, if someone made one of those available when parsing the world. Do you think you could get the sites and suchlike to line up on there as well? That would be a step towards incorporating evil and temperature and rainfall and all the other goodies, which some people might appreciate having along with everything else.
Watching this with interest :D
I had thought of it when I was first starting the map project and figuring out which was the "real" map. I haven't checked exactly how those work yet, but I wouldn't be surprised if I could have an option where you could flip between different maps.
I suspect this might be some fairly cool low-hanging fruit. I'll give it a shot.
I've been working on getting something out of huge historical event logs in the xml, but that's been buggy and slow (VB isn't all that great at throwing around 100's of MB in strings lol) so I might have to wipe all that and do it a different way, no big deal.
Post by: KaelGotDwarves on July 19, 2010, 11:23:14 pm
Post by: Mason11987 on July 19, 2010, 11:41:44 pm
Posting to follow the project. Good work.
Thanks!
That'll make things even easier.
Good work, thanks!
I was wondering what could be done with a detailed-world-map export, if someone made one of those available when parsing the world. Do you think you could get the sites and suchlike to line up on there as well? That would be a step towards incorporating evil and temperature and rainfall and all the other goodies, which some people might appreciate having along with everything else.
Watching this with interest :D
Just thought I'd give it a shot and see what happened by loading the graphic biome map.
And it... well, worked? You be the judge:
(http://imgur.com/2P1jT.png)
(I have a feeling that this might not work for different tileset sizes, haven't checked yet though)
In next:
- Fixed a bug where minimize will crash
- Fixed problem where clicking the location on a site didn't bring you to the site on the map (related to new support for different size tilesets)
Post by: Hectonkhyres on July 20, 2010, 12:07:52 am
Post by: Ratbert_CP on July 20, 2010, 10:09:28 am
I have a feeling that this might not work for different tileset sizes, haven't checked yet though
You shouldn't have a problem. These high-res maps use color-coding rather than tileset/graphics for the export. If you count the pixels, you should find the image dimensions to be a nice power-of-two multiple of the cell/tile dimensions of your world. And that should remain constant, regardless of your tileset. The issue with the low-res, tileset based export is that it's rendered with the tileset based on the underlying biome, and the rendered cell/tile is sized based on the size of the tileset rather than the fixed size of the cell.
Does that make sense? I think I may have confused myself... ;)
Post by: Mason11987 on July 20, 2010, 10:37:13 am
I have a feeling that this might not work for different tileset sizes, haven't checked yet though
You shouldn't have a problem. These high-res maps use color-coding rather than tileset/graphics for the export. If you count the pixels, you should find the image dimensions to be a nice power-of-two multiple of the cell/tile dimensions of your world. And that should remain constant, regardless of your tileset. The issue with the low-res, tileset based export is that it's rendered with the tileset based on the underlying biome, and the rendered cell/tile is sized based on the size of the tileset rather than the fixed size of the cell.
Does that make sense? I think I may have confused myself... ;)
so maps like the biome map will come out exactly the same regardless of what tileset you have installed?
If so I think we should be good.
We're moving slowly but consistently towards real parsing of the xml.
I've added basic "region" "entities" and "Historical figures" tabs.
The dump definitely is "incomplete", but things I expect to be able to get out of the sort of events you can read (http://df.magmawiki.com/index.php/User:Mason11987/XMLEvents):
Spoiler (click to show/hide)
It's possible some of these won't actually be in the program, but there are several that I find extremely interesting that I'm excited about (10, 12), and a few I think are necessary to connect everything that's relevant (1, 3, 5, 8, 9, 17, 20, 21). But any of the above are possible, certainly.
Post by: Star Weaver on July 20, 2010, 12:17:16 pm
Post by: timtek on July 26, 2010, 04:54:17 pm
Post by: Alehkhs on July 26, 2010, 05:34:52 pm
As an avid reader of my worlds' histories, this is greatly appreciated.
Post by: KrunkSplein on July 26, 2010, 06:00:05 pm
This is awesome. Well done!
Post by: Sizik on July 27, 2010, 02:35:33 am
I haven't really looked, but given that rivers and such form one-pixel lines on the export maps, I think they might be 1-pixel-per-embark-tile, or 16x16 pixels per world tile.
Seeing as towns are 3x3 pixel dots and take up 3x3 embark tiles, this is most likely the case.
Post by: Lightman on July 27, 2010, 04:14:33 pm
Post by: Deteramot on July 27, 2010, 04:36:25 pm
Post by: Mel_Vixen on July 27, 2010, 04:44:24 pm
This is only a educated guess thought.
Mason11987 are you working on a new version any new features?
Post by: jocan2003 on July 27, 2010, 06:24:06 pm
Post by: Mason11987 on July 28, 2010, 08:33:14 am
Post by: pickupsticks on July 28, 2010, 09:08:18 am
Hey guys, I'm working on this now, I have a version that already parses a lot of the XML for some cool connections, but I'm on vacation this week, that's why I haven't posted anything recently.
No rush :)
Looking forward to the next release, thanks.
Post by: Mason11987 on July 30, 2010, 12:43:00 pm
I just tried this program out and I like it. Is there any way to make the font and stuff bigger? When I resized the window it all stayed tiny. Also what does the overlay on the map do? Is it showing me where cities originate from? All in all, this program makes it a lot easier to view the data from legends mode. Thanks for making it.
Font size changing may or may not happen. I'm trying to fit a lot of info on the screen and it's difficult with larger text.
The overlay links all the sites of a single civ together, the "hub" is the largest site. In the future I MAY be able to show it as a progression from the original site, but that information may not completely exist in the xml.
Wow, this is fantastic! I can't believe I'd never seen this before...
As an avid reader of my worlds' histories, this is greatly appreciated.
Never before has a text-parser been so... necessary!
This is awesome. Well done!
Cool! This looks like a great program. I need to try it, straight away.
Thanks all. Let me know if you have any questions or any problems. Or anything you think might improve it!
Does this program support the latest version of DF? I'd imagine there isn't a lot of change between the versions with the Legends Mode, but better to ask than download a file I can't use.
I expect it to work up until (and perhaps past) the entity populations update the Toady is working on, no .31 update since my initial release has broken it so I expect it to work fine for a while. It's hard to tell exactly how resistent to change it will be. Changes to the xml shouldn't cause it to break, changes to the world history and world sites? Probably more likely to cause problems, because there aren't the tags around things like in the xml. Of course if you find it doesn't at some point, please let me know.
Genning in .31.12 to test... ... and it works fine! :)
What is awesome is i was going to make a new world for my tabletop top game in medieval setting and your tool just gave me way less trouble to set up, now mixed with DF i can generate a new world, have town, history cave, cavern, megabeast and so on already generated for me, all i need to do is add up some details and *convert* all creature using my RPG character sheet and add up some town detail like mage guild and so one but masicly more than 80% of the job is done with DF mixed with this utility GREAT!!
One thing mentioned earlier in this thread is to output files with information about groups/sites/civs in a readable form. If that's something you'd find useful let me know.
Post by: Mason11987 on July 30, 2010, 01:22:19 pm
V 2.0
- Added Historical Figures (HF), Entities, and Regions Tabs.
- HFs are linked to other figures that they worship, are worshipped by, or are married to
- HFs are linked to sites they've lived in over their life.
- Entities are linked to the sites they've founded (, and sites they were created at (for religions)
- Sites list when they were founded, and by what entity
- Added progress notes in the titlebar, for some of the larger XML files it could take some time.
- The non-xml details should be pulled out nearly immediately for any world, during the xml parsing portions you can now look through the rest of the information until everything is done.
- Probably a lot of other things
Pictures
Spoiler (click to show/hide)
Coming up soon!
Spoiler (click to show/hide)
Post by: Deon on July 30, 2010, 02:43:47 pm
I am going to include it in Genesis download and post a link to it in the mod thread. This is a must-have tool!
P.S. The link is just under my mod downloads. Now I will run off and try to make something cool and watch it with this great program :).
Post by: Mason11987 on July 30, 2010, 03:15:28 pm
THIS IS AWESOME!
I am going to include it in Genesis download and post a link to it in the mod thread. This is a must-have tool!
P.S. The link is just under my mod downloads. Now I will run off and try to make something cool and watch it with this great program :).
I haven't had nearly as many tests with mods, but I did try it with Genesis early on, so if you see some problems let me know and I'll see what I can find. Just as importantly if you don't find any problems let me know, so I can know what works often.
Thanks for adding it on your thread. I only used your mod a little bit about a month ago but I wasn't prepared at the time to relearn some of the bigger changes (iron/steel stuff) but I love big game changing mods.
Post by: Deon on July 30, 2010, 03:17:01 pm
Post by: Deon on July 30, 2010, 03:22:03 pm
I can upload the world info which caused it.
P.S. It's:
Code: [Select]
Runtime error '5'.
Invalid procedure call or argumentIt happens on 85% of XML parsing.
P.S. Here's the world data which is not being parsed.
http://dffd.wimbli.com/file.php?id=2862
Post by: Mason11987 on July 30, 2010, 04:54:16 pm
Okay, I've just got a "runtime error: Invalid *** argument 5" or something (I don't remember what *** was).
I can upload the world info which caused it.
P.S. It's:Code: [Select]Runtime error '5'.
Invalid procedure call or argument
It happens on 85% of XML parsing.
P.S. Here's the world data which is not being parsed.
http://dffd.wimbli.com/file.php?id=2862
This forced me to rework a few parts of the program.
V 2.0.1
Your files should now work, and it should support larger xml files. Still crashes on my 387MB one though. It appears the way I allocate memory to parse the XML files isn't efficient enough to keep me under some OS (or Visual Basic) memory limit. I've never had a need to throw around hundreds of MB in a program before though, so this is uncharted territory. Thankfully there is a lot of memory I can save if I do things smarter. I didn't bother earlier because I didn't know this limit even existed lol.
Let me know if it works for you and any questions you have.
Edit: I knew it would be a great idea to add the progress % on top, I knew what the problem was before I even ran your files :).
Post by: Deon on July 30, 2010, 05:17:53 pm
Post by: Mason11987 on July 30, 2010, 05:24:05 pm
Can you reupload it? I get only 101 kb out of the download.
Just uploaded it again, but the zip is only 98 KB or so.
Edit - Just downloaded myself and confirmed that it works.
Post by: Deon on July 30, 2010, 05:37:19 pm
(http://imgur.com/uXgI6.png)
And a couple of other errors like that.
It looks like it doesn't know "site taken over" and "razed structure".
P.S. It would be cool if "filter out" was toggleable, not a one-time action or something. After I clicked filters a bit I was unable to see any site in the "sites" list at all.
Post by: Mason11987 on July 30, 2010, 06:47:26 pm
That's what I get
(http://imgur.com/uXgI6.png)
And a couple of other errors like that.
It looks like it doesn't know "site taken over" and "razed structure".
P.S. It would be cool if "filter out" was toggleable, not a one-time action or something. After I clicked filters a bit I was unable to see any site in the "sites" list at all.
Those events I have noted and they'll be included in some form next version, if only enough to stop that messagebox.
Lol, yeah that was a really early feature I didn't do very well. I'm not sure how I'll mess with those, but yeah, they've become outdated/irrelevant especially after mod support.
Post by: Deon on July 30, 2010, 07:56:19 pm
Post by: Mason11987 on July 30, 2010, 08:44:03 pm
Also note the site plundering event. There may be more.
I already captured a large number of events, if you can screenshot what comes up it'll help me out though, there can't be that many more.
Post by: Mason11987 on July 30, 2010, 09:40:23 pm
- Added sorting options for regions, entities, and historical figs
- Fixed bug linking sites to entities
- Map now has civs expanding out from their original site, which I think is pretty cool
- Map shows founding entity and founding year
- site taken over and razed structure events handled, in the case of an undiscovered event coming up during world parsing, a new popup will save text directly to your clipboard so you can easily post and share here.
- Added kill counts and kill filters
- Added option to undo filters per Deon's request
- Added some summary information to the World Data tab that might be interesting.
Spoiler: Kills record! (click to show/hide)
Spoiler: World Data (click to show/hide)
Post by: Deon on July 31, 2010, 03:45:30 am
Post by: timtek on July 31, 2010, 03:46:16 am
Post by: Knight Otu on July 31, 2010, 05:49:52 am
Just tried version 2.1 of the world viewer. It got 61% of the way through world parsing and crashed out 50% of the way through the xml file. run time error 9: subscript out of range. my xml file is 263 MB (275,906,100 bytes). Hope this info helps ya out.Getting that even on a pocket world, so size isn't an issue here. Might be unhandled events?
Hmm, the program states it is at Parsing Events (3218) - 93%. Event id stops 3310 and starts at 93. Seems to point at around ~3084, but there doesn't seem to be a particularly odd event in the area.
I'll try to generate another pocket world, and will probably upload them to the DFFD if I can't figure it out.
Uploaded (http://dffd.wimbli.com/file.php?id=2865).
Post by: Mason11987 on July 31, 2010, 09:02:10 am
Just tried version 2.1 of the world viewer. It got 61% of the way through world parsing and crashed out 50% of the way through the xml file. run time error 9: subscript out of range. my xml file is 263 MB (275,906,100 bytes). Hope this info helps ya out.
Yeah, I have an idea where that's coming from, and is expected for larger files. My next fix (today?) will either deal with bigger files (I have a 400MB I test with) and work or gracefully fail and reduce functionality if it can't handle events. Both of these would be far better then a failure like that.
Just tried version 2.1 of the world viewer. It got 61% of the way through world parsing and crashed out 50% of the way through the xml file. run time error 9: subscript out of range. my xml file is 263 MB (275,906,100 bytes). Hope this info helps ya out.Getting that even on a pocket world, so size isn't an issue here. Might be unhandled events?
Hmm, the program states it is at Parsing Events (3218) - 93%. Event id stops 3310 and starts at 93. Seems to point at around ~3084, but there doesn't seem to be a particularly odd event in the area.
I'll try to generate another pocket world, and will probably upload them to the DFFD if I can't figure it out.
Uploaded (http://dffd.wimbli.com/file.php?id=2865).
Knight Otu,
Fixed both the same way, thanks!
Spoiler: Details (click to show/hide)
Post by: Mel_Vixen on July 31, 2010, 09:53:21 am
Post by: Mason11987 on July 31, 2010, 10:42:34 am
Cant you parse the Xml in chunks, sort the info and offload everything to own files? This way you dont need to keep everything in your memory.
Yeah, that's what I'm inevitably going to do. In the meantime my next update has some improvements which should prevent many crashes due to size, that way you can get the map, and lists of regions/historical figures/entities, even if it doesn't have very much info about them (due to skipping events because they are almost all of the XML files)
Post by: Still Standing on July 31, 2010, 11:10:22 am
Post by: Mason11987 on July 31, 2010, 11:19:37 am
I just tried to download this, but I get a 100kb corrupt archive. Not sure where the problem is. :(
Try again.
Post by: Still Standing on July 31, 2010, 01:44:11 pm
Post by: Mason11987 on August 02, 2010, 04:39:41 pm
V 2.2
Now lists information about abductions for HF that have been abducted.
Now lists when sites are destroyed, and by whom.
Plus a lot of "under-the-hood" stuff.
Post by: timtek on August 02, 2010, 08:01:31 pm
Post by: Mel_Vixen on August 04, 2010, 10:24:34 am
Quote from: Mason11987In legends: export xml, historical events, each has a type. I've discovered 33 different event types:
add hf entity link ? add hf hf link ? add hf site link ? attacked site ? body abused ? change hf job ? change hf state ? create entity position ? created site ? created structure ? created world construction ? creature devoured ? destroyed site ? entity created ? field battle ? hf abducted ? hf died ? hf new pet ? hf razed structure ? hf reunion ? hf simple battle event ? hf travel ? hf wounded ? impersonate hf ? item stolen ? new site leader ? peace accepted ? peace rejected ? razed structure ? reclaim site ? remove hf entity link ? remove hf site link ? replaced structure. If you could tell me how many there are, and/or what they are, that'd be fantastic. Thanks!
Some of them aren't in use:Spoiler (click to show/hide)
So how is it comming along?
Post by: einstein9073 on August 04, 2010, 01:47:03 pm
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
Post by: dree12 on August 04, 2010, 02:59:13 pm
Post by: Askot Bokbondeler on August 04, 2010, 04:23:24 pm
Post by: Mason11987 on August 04, 2010, 07:50:57 pm
For the case you did not see i yet:Quote from: Mason11987In legends: export xml, historical events, each has a type. I've discovered 33 different event types:
add hf entity link ? add hf hf link ? add hf site link ? attacked site ? body abused ? change hf job ? change hf state ? create entity position ? created site ? created structure ? created world construction ? creature devoured ? destroyed site ? entity created ? field battle ? hf abducted ? hf died ? hf new pet ? hf razed structure ? hf reunion ? hf simple battle event ? hf travel ? hf wounded ? impersonate hf ? item stolen ? new site leader ? peace accepted ? peace rejected ? razed structure ? reclaim site ? remove hf entity link ? remove hf site link ? replaced structure. If you could tell me how many there are, and/or what they are, that'd be fantastic. Thanks!
Some of them aren't in use:Spoiler (click to show/hide)
So how is it comming along?
Awesome! I hadn't seen that yet.
I've already done almost every connection I could out of these events, the ones I list below are already in.
add hf hf link - This is the "related HF" part you see. These are marriage/worship events.
change hf state - This is one way of marking which site a HF lives in. (more can be pulled out later)
change hf job - This is one way of marking which site a HF lives in (this gets the ones where HF spawned at world beginning in a site)
created site - This allows me to show which civ (and which group - entity) founded a site and when that site was founded
destroyed site - This allows me to show which group destroyed a site, and when.
entity created - This allows me to show at what site a religion (entity) was created, or a new group (entity) was created (these groups lead attacks or found new sites, or other things) and when it happened.
hf abducted - To list tie abductees to abductors. This could help you possibly see the history of that lone dwarf in the goblin raiding party attacking you
hf died - This lets me get kill counts and (in the future) age of HFs
new site leader - This lets me show you when a site was taken over by another civ, and who the leader of that new government is.
There is quite a lot still to do. And many of the tags don't have enough information to do anything. It seems like he's open to filling them out more though.
I just downloaded world viewer from the link on the first page of this thread, exported everything from my world, and ran it.
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
That means it didn't even get to the XML, could you post your world sites/world history? You can even just drop it inside spoiler tags right here if you like, they shouldn't be very long.
Thanks!
Mason, since you can place coloured boxes, can you make it so the boxes for each civ (instead of all being red) are a darker version of the connection colour? Because even with the civsplosion, some large civs (like the "Tusks of Ferocity") in some larger worlds have a crumpled mess as their are many sites. It would really help, as I can disable the connection for these civs, while still seeing which sites belong to who.
I liked red because it stood out against other bright backgrounds, but I'll see how this looks, you can expect it next release.
hey, toady is using this himself! congratulations. also, following
Using it himself? *GASP* Where'd you read that? Also, thanks :)
Post by: dree12 on August 04, 2010, 08:16:38 pm
Maybe tint it red? Like, I dunno, divide both G and B by 1.2 and keep R the same?Mason, since you can place coloured boxes, can you make it so the boxes for each civ (instead of all being red) are a darker version of the connection colour? Because even with the civsplosion, some large civs (like the "Tusks of Ferocity") in some larger worlds have a crumpled mess as their are many sites. It would really help, as I can disable the connection for these civs, while still seeing which sites belong to who.
I liked red because it stood out against other bright backgrounds, but I'll see how this looks, you can expect it next release.
Post by: Mason11987 on August 04, 2010, 08:41:49 pm
Maybe tint it red? Like, I dunno, divide both G and B by 1.2 and keep R the same?Mason, since you can place coloured boxes, can you make it so the boxes for each civ (instead of all being red) are a darker version of the connection colour? Because even with the civsplosion, some large civs (like the "Tusks of Ferocity") in some larger worlds have a crumpled mess as their are many sites. It would really help, as I can disable the connection for these civs, while still seeing which sites belong to who.
I liked red because it stood out against other bright backgrounds, but I'll see how this looks, you can expect it next release.
Spoiler: Thoughts? (click to show/hide)
Spoiler: Maybe thicker lines? (click to show/hide)
Spoiler: Or Circles? (click to show/hide)
Post by: Askot Bokbondeler on August 04, 2010, 09:58:08 pm
Using it himself? *GASP* Where'd you read that? Also, thanks :)
have you read the latest devlog entry?it certainly looks like it's your little utility (http://www.bay12games.com/dwarves/imgs/farm_map_det.png)
Post by: Mason11987 on August 04, 2010, 10:07:28 pm
Using it himself? *GASP* Where'd you read that? Also, thanks :)
have you read the latest devlog entry?it certainly looks like it's your little utility (http://www.bay12games.com/dwarves/imgs/farm_map_det.png)
Not my utility, that's a map that can be exported in legends mode after worldgen. My utility doesn't make the maps, just makes them sort of interactive.
That being said this update will CERTAINLY screw up a lot of stuff. But hopefully I'll fix it up fairly quickly.
Post by: timtek on August 05, 2010, 01:48:59 am
Post by: dennislp3 on August 05, 2010, 01:53:57 am
it is with a rather large XML file and I assume that is the issue...
also getting runtime error 9 (subscript error of some sort...forgot)
I am using Deons genesis medium region with 1000 years of gen...not sure why its not working...worked ok before I upgraded
First XML was over 800MB (one that gave error 14) and one that gave error 9 was roughly 500MB
Post by: James.Denholm on August 05, 2010, 04:38:47 am
This is awesome.
Post by: Knight Otu on August 05, 2010, 09:22:16 am
I just downloaded world viewer from the link on the first page of this thread, exported everything from my world, and ran it.
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
I edited my previous upload here (http://dffd.wimbli.com/file.php?id=2865) with two problematic worlds. Region 4 exhibits the quoted behavior, where I'm uncertain what the cause is. Region 3 is a pocket world genned to year 10000, and it evidently has too many leaders in the first civilization, as I get an Overflow error when I don't pare down that list.
Post by: Mason11987 on August 05, 2010, 09:51:22 am
I just downloaded world viewer from the link on the first page of this thread, exported everything from my world, and ran it.
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
I edited my previous upload here (http://dffd.wimbli.com/file.php?id=2865) with two problematic worlds. Region 4 exhibits the quoted behavior, where I'm uncertain what the cause is. Region 3 is a pocket world genned to year 10000, and it evidently has too many leaders in the first civilization, as I get an Overflow error when I don't pare down that list.
On Region 3 you were right. Integer --> Long
V2.2.1
- Fixed the above problem.
- Map now circles sites with civ-color instead of red squares
It's a pretty cool world, shame only one civ lived for any real length of time. Though there was a 17 generation dynasty, which is cool.
One thing I haven't done is put is cause of death under the leaders tab. Though this gave me the idea to get that set. Although your great PulleyTouched dynasty (7212-7960) was finally ended because Thikut Waxchamber never married, and died without children at the ripe old age of 169. I assume this was of old age. But I'll find out after I make my change. Good edge-case world though. Thanks.
I've tried a few times to rewrite the xml parsing to use less memory (a cause for MANY of the problems here), but it's slow going. It'll happen though, probably around when toady adds entity populations and screws it all up anyway ;).
Edit - Fixed a bug with hist figs dieing of old age (recently introduced)
Post by: Greiger on August 05, 2010, 01:27:41 pm
"Error: Site twice reclaimed" Which does not appear to be a fatal error because it continues for a bit afterward, and
"Run time error '6' Overflow" Which immediately exits the program.
Notes:
It does say the xml is very large at the start, and from what I'm seeing that appears to be a recurring issue. The world is modded with 2 extra civs, Dracon and an extra dwarf civ. Worldgen ends at year 500. And the world is being played currently but the files were all generated while the world was fresh if that matters.
Thinking you may be aware of this, but posting just in case. If you want the relevant files just give me a hollar and tell me how ya want them.
P.S. The .xml is 156 mb
Post by: Mason11987 on August 05, 2010, 01:43:23 pm
Very nice tool. I used it previously with an old version and it loaded the world fine after apperantly skipping a few entries, (windows calls that version 1.11.0.0 in the details box) but with the most recent version (2.2.1) it comes up with 2 errors while loading.
"Error: Site twice reclaimed" Which does not appear to be a fatal error because it continues for a bit afterward, and
"Run time error '6' Overflow" Which immediately exits the program.
Notes:
It does say the xml is very large at the start, and from what I'm seeing that appears to be a recurring issue. The world is modded with 2 extra civs, Dracon and an extra dwarf civ. Worldgen ends at year 500. And the world is being played currently but the files were all generated while the world was fresh if that matters.
Thinking you may be aware of this, but posting just in case. If you want the relevant files just give me a hollar and tell me how ya want them.
P.S. The .xml is 156 mb
Yeah, early on I completely ignored events. Since they are almost all of the xml file that addition broke some of the larger ones. I'm keeping a note of everyone who recently posted their XML being large and it breaking, I'll send you a PM when I expect that problem to be fixed if you wouldn't mind testing.
I had assumed a site could be reclaimed twice but I hadn't actually accounted for it (same with twice captured), so I put that note in to see if it happens, and also to let you know the history of the site isn't completely correct under those circumstances (in the sites tab).
If you don't mind, could you tell me what it shows in the titlebar when it overflows (text + percents)? That's probably NOT the same failure I'm working on right now though, especially if you see the site twice reclaimed error first.
After, I might put some debug stuff in a version for you so you don't have to upload 156MB for me to get something useful.
Post by: Greiger on August 05, 2010, 01:52:23 pm
Parsing World - Get Event Data (82%) - (651325) (55%) is where the overflow happens.
Parsing World - Get Event Data (82%) - (651325) (06%) is where the Site twice reclaimed happens just for completeness.
And yea, I'll be happy to test. I like poking around worldgen, so I'd be glad to help in any way I can.
Post by: Mason11987 on August 05, 2010, 02:09:13 pm
Ha seems it loads the xml ok now that I see that top bar has useful information. Or at least the first pass or whatever that is. :)
Parsing World - Get Event Data (82%) - (651325) (55%) is where the overflow happens.
Parsing World - Get Event Data (82%) - (651325) (06%) is where the Site twice reclaimed happens just for completeness.
And yea, I'll be happy to test. I like poking around worldgen, so I'd be glad to help in any way I can.
V2.2.2
- Put in some temporary debugging (Greiger, try this and paste what you get)
- Linked Leaders to their listing as historical figures (I'm not sure if I like this, but for the time being it works okay). Although some aren't linked because of this bug (http://www.bay12games.com/dwarves/mantisbt/view.php?id=2955).
- You can now see how your leaders died (if they're linked, and they died)
Post by: Mel_Vixen on August 05, 2010, 02:16:50 pm
Post by: Mason11987 on August 05, 2010, 02:18:46 pm
You should create a logfile for the parsing process. I guess it would help with debugging if you could know which function goes Bogus.
I agree, It's in the works ;).
Post by: Greiger on August 05, 2010, 02:29:45 pm
"Error: Site twice reclaimed" at Parsing World - Get Event Data (82%) - (651325) (06%)
and
"Error in get event info. Error: Overflow
Event data(auto-saved to clipboard)
<historical_event>
<id>361043</id>
<year>293</year>
<seconds72>-1</seconds72>
<type>hf died</type>
<hfid>27672</hfid>
<slayer_hfid>34497</slayer_hfid>
<slayer_item_id>-1</slayer_item_id>
<slayer_shooter_item_id>-1</slayer_shooter_item_id>
<site_id>-1</site_id>
<subregion_id>557</subregion_id>
<feature_layer_id>-1</feature_layer_id>
<cause>struck</cause>
</historical_event>"
It also appears that auto-save to clipboard doesn't quite work. Maybe because my clipboard already has something in it. I copied it exactly as it appeared I believe though.
EDIT: I origonally put 1 for seconds72. it is actually -1 sorry
Post by: Mason11987 on August 05, 2010, 03:00:39 pm
It also appears that auto-save to clipboard doesn't quite work. Maybe because my clipboard already has something in it. I copied it exactly as it appeared I believe though.
Yeah, it will now clear your clipboard.
V2.2.3
I found the problem though. Give it another attempt.
The logfile mentioned by Heph is made now but it's still empty :P.
Post by: Greiger on August 05, 2010, 03:32:13 pm
Uh oh...new problem.
Now it's getting overflows repeatedly. All on hf abductions. Judging from the data it outputs in the errors they are all different abductions. But there was no end in sight so I had to kill the process. Which seems to have killed the clipboard data as well. I'll transcribe one of them for you, but from a cursory glance it seems it's just the act of abductions screwing it up now.
First one: at 58% of event processing.
<historical_event>
<id>383809</id>
<year>309</year>
<seconds72>-1</seconds72>
<type>hf abducted</type>
<target_hfid>36374</target_hfid>
<snatcher_hfid>34649</snatcher_hfid>
<site_id>92</site_id>
<subregion_id>-1</subregion_id>
<feature_layer_id>-1</feature_layer_id>
</historical_event>
Something like 50 clicks of the OK button later: 65%
<historical_event>
<id>425765</id>
<year>339</year>
<seconds72>-1</seconds72>
<type>hf abducted</type>
<target_hfid>39692</target_hfid>
<snatcher_hfid>36728</snatcher_hfid>
<site_id>105</site_id>
<subregion_id>-1</subregion_id>
<feature_layer_id>-1</feature_layer_id>
</historical_event>
Seems to me it's having trouble with every single abduction after a certain point. Maybe my world is screwy... clipboard seems to work though if I paste before killing the process.
Post by: Mason11987 on August 05, 2010, 03:43:50 pm
<snip>
Seems to me it's having trouble with every single abduction after a certain point. Maybe my world is screwy... clipboard seems to work though if I paste before killing the process.
Yup, any historical figure above 32767 (2^15-1, this is the largest that can be stored as an integer). That limit should now be fixed.
Another try please :)
Post by: Greiger on August 05, 2010, 03:54:37 pm
<historical_event>
<id>468460</id>
<year>370</year>
<seconds72>-1</seconds72>
<type>hf died</type>
<hfid>41505</hfid>
<slayer_hfid>1981</slayer_hfid>
<slayer_item_id>-1</slayer_item_id>
<slayer_shooter_item_id>-1</slayer_shooter_item_id>
<site_id>-1</site_id>
<subregion_id>87</subregion_id>
<feature_layer_id>-1</feature_layer_id>
<cause>struck</cause>
</historical_event>
Same as last time, repeats with varying numbers repeatedly after the first one. the id field actually does skip a few numbers near the beginning though. And like last time the <type> entry are all the same thing, except this time instead of abductions they are all deaths.
I guess I have an inordinately large number of events or something? It's only a medium world..
Post by: Mel_Vixen on August 05, 2010, 04:00:01 pm
edit: excuse if i am a little bit pedantic on this ^^.
Post by: Mason11987 on August 05, 2010, 04:10:26 pm
Made it to 72% this time. Getting closer heh.
<historical_event>
<id>468460</id>
<year>370</year>
<seconds72>-1</seconds72>
<type>hf died</type>
<hfid>41505</hfid>
<slayer_hfid>1981</slayer_hfid>
<slayer_item_id>-1</slayer_item_id>
<slayer_shooter_item_id>-1</slayer_shooter_item_id>
<site_id>-1</site_id>
<subregion_id>87</subregion_id>
<feature_layer_id>-1</feature_layer_id>
<cause>struck</cause>
</historical_event>
Same as last time, repeats with varying numbers repeatedly after the first one. the id field actually does skip a few numbers near the beginning though. And like last time the <type> entry are all the same thing, except this time instead of abductions they are all deaths.
I guess I have an inordinately large number of events or something? It's only a medium world..
You seem to be at the point in xml files size that aren't too large to cause the memory failure, but are bigger then the one I've been using. This is valuable regardless because I would have to address these soon enough anyway.
When you zip (rar/7z/whatever) your xml file, what kind of size do you get? Maybe you could share it?
This is "short" or "short int". Integer itself has and means normaly 32 bits. I asume you turned to a unsigned short then (65.535)? If so i suggest that depending on thelanguage you use you go over to DWord or Double Word. Long int would be to much :P .
I'm using VB6 where integer is 16bit (short int in other languages), and long is 32bit, which should cover me up until someone has more than 2,147,483,647 events ;).
Post by: Mel_Vixen on August 05, 2010, 04:22:16 pm
Post by: Mason11987 on August 05, 2010, 04:58:15 pm
VB? Hmmmm i have done some VBa but normaly i do C++ Php or javascript (especially the last drives me mad and makes me happy at the same time.) Ayway go unsigned because you shouldnt have events with negative values.
Yeah VB has a lot of flaws (specifically the LACK of unsigned data types for example :-X), especially when you use an old and unsupported version like me (VB6), but I've done enough big personal projects on it that I can often whip up programs as fast as I can type, which means I get to see some results quickly which helps to stave off boredom for personal projects about managing data.
Post by: Greiger on August 05, 2010, 05:04:11 pm
Post by: Mason11987 on August 05, 2010, 05:10:12 pm
Oh wow, 3% compression ratio. the entire.xml is 4.71 MB when rar ed with winrar under the normal setting. So it looks like it would be easy to send in rar format. I assume you need all the other files Worldview asks for as well right?
Awesome yeah, just upload all of them somewhere.
I thought it might be small with all the repeated text in it.
I appreciate it.
Post by: Greiger on August 05, 2010, 05:14:39 pm
There ya go. Filename is Greiger region1-legends.rar Should have everything. Hope it's not too hard to fix. :)
As I said before, worldgen ends in 500 and it has 2 extra civs.
P.S. Forgot about the Orcs... 3 extra civs.
Post by: Mason11987 on August 05, 2010, 05:24:55 pm
http://www.sendspace.com/file/9c0bn2
There ya go. Filename is Greiger region1-legends.rar Should have everything. Hope it's not too hard to fix. :)
As I said before, worldgen ends in 500 and it has 2 extra civs.
P.S. Forgot about the Orcs... 3 extra civs.
Hmm... seemed to work for me.
I just recompiled but I don't think I've made any real changes since last time.
Testing with compiled version... works there too. I'll upload again.
Give it another shot.
Post by: Greiger on August 05, 2010, 05:30:08 pm
Spoiler: off topic (click to show/hide)
Post by: Mason11987 on August 05, 2010, 06:29:39 pm
Yea, it works now for me as well. Donno, what you did but it's working fine now. Thanks much!Spoiler: off topic (click to show/hide)
I've done a few games, and a big path finding/A* demonstration thing in VB myself.
I was using your save and looking at the map and I found that The Silver of Flight civ was centered on a site they didn't originally create, because my algorithm had it center a civ on the site which was founded first. Problem with that is that if a civ captured another civs first site then that site might be considered the "center" which didn't really sit well with me.
So I moved it so it was the site founded the earliest by the civ that owns it now, and also distinguished graphically between sites that were natural expansion (solid line), and ones that were forcefully taken or reclaimed (dotted line).
Spoiler: Check it out (click to show/hide)
Post by: Greiger on August 05, 2010, 06:56:46 pm
I'll have to take a peek to see what pissed them off.
Of course knowing the luck of most of the other civs of the world they were probably just smashed into submission by megabeasts early and The Silver of Flight just looked at it and said, "Oh hey, free town"
Very nice catch.
Post by: Mason11987 on August 05, 2010, 07:16:12 pm
Oh wow, even I didn't notice that. But yer right. Never saw dracon attack other dracon or humans before. Was always goblins, orcs, or elves.
I'll have to take a peek to see what pissed them off.
Of course knowing the luck of most of the other civs of the world they were probably just smashed into submission by megabeasts early and The Silver of Flight just looked at it and said, "Oh hey, free town"
Very nice catch.
Yeah, if you click a site connected with dotted lines (or any site) you will go to the site tab which has the full history.
Post by: shadow_slicer on August 06, 2010, 12:59:57 pm
Code: [Select]
<?xml version="1.0" encoding='UTF-8'?>
<DFLegend xmlns="http://www.bay12games.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
... Contents of XML dump from DF (without <?xml?> part) ...
</DFLegend>
Post by: einstein9073 on August 06, 2010, 01:50:41 pm
I tried, but got a message board error:I just downloaded world viewer from the link on the first page of this thread, exported everything from my world, and ran it.That means it didn't even get to the XML, could you post your world sites/world history? You can even just drop it inside spoiler tags right here if you like, they shouldn't be very long.
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
Thanks!
Quote from: SMF
The following error or errors occurred while posting this message:The Sites file is 80KB, and the history file is 99KB.
The message exceeds the maximum allowed length (40000 characters).
I would be happy to send them to you; would you PM me your email address?
Post by: Mason11987 on August 06, 2010, 01:53:16 pm
I tried, but got a message board error:I just downloaded world viewer from the link on the first page of this thread, exported everything from my world, and ran it.That means it didn't even get to the XML, could you post your world sites/world history? You can even just drop it inside spoiler tags right here if you like, they shouldn't be very long.
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
Thanks!Quote from: SMFThe following error or errors occurred while posting this message:The Sites file is 80KB, and the history file is 99KB.
The message exceeds the maximum allowed length (40000 characters).
I would be happy to send them to you; would you PM me your email address?
Sent PM, and if you zip (or .rar/.7z) the xml it should compress really well. Send that too (and the map) if you can.
Post by: MaDeR Levap on August 06, 2010, 03:59:30 pm
Yup, any historical figure above 32767 (2^15-1, this is the largest that can be stored as an integer). That limit should now be fixed.I advise to change every* int to long, just in case. With DF, no overflow is sufficiently far (http://www.bay12forums.com/smf/index.php?topic=61507.0). ;)
*Ok, not every, but still.
Post by: Mason11987 on August 06, 2010, 04:41:41 pm
I tried, but got a message board error:I just downloaded world viewer from the link on the first page of this thread, exported everything from my world, and ran it.That means it didn't even get to the XML, could you post your world sites/world history? You can even just drop it inside spoiler tags right here if you like, they shouldn't be very long.
I get the error "Run Time Error '5': Invalid Procedure Call or Argument" when the parsing reaches 22%.
My XML is a puny 105MB. Do you have any ideas?
Thanks!Quote from: SMFThe following error or errors occurred while posting this message:The Sites file is 80KB, and the history file is 99KB.
The message exceeds the maximum allowed length (40000 characters).
I would be happy to send them to you; would you PM me your email address?
v 2.2.4
Fixed your problem. It was some strange problem where a civ didn't have a leader.
Could you check the legends for your save for confederacy of tapering (under entities) and let me know what the hell happened to them that they didn't have any leaders. Crazy :). I had assumed that was a requirement.
Yup, any historical figure above 32767 (2^15-1, this is the largest that can be stored as an integer). That limit should now be fixed.I advise to change every* int to long, just in case. With DF, no overflow is sufficiently far (http://www.bay12forums.com/smf/index.php?topic=61507.0). ;)
*Ok, not every, but still.
I have some memory problems already, but I did change everything I even considered possible to be a >30k. Plenty of things can be ints though, like civ counts, number of leaders of a civ, end year, etc.
But there are a few that I put as long just in the "craziest possible circumstance", like number of Historical figures killed. IN case a megabeast goes on a rampage for 10000 years and kills 40k different people. lol
Post by: Knight Otu on August 07, 2010, 03:45:38 am
v 2.2.4I believe that humans and goblins do not necessarily create their leadership positions right away, so might might take up to year 5 or so for them to appear. If the civ is destroyed by a sudden case of megabeast before that, this might happen.
Fixed your problem. It was some strange problem where a civ didn't have a leader.
Could you check the legends for your save for confederacy of tapering (under entities) and let me know what the hell happened to them that they didn't have any leaders. Crazy :). I had assumed that was a requirement.
Post by: Mason11987 on August 08, 2010, 12:25:12 am
Fairly extensive re-write of my XML parsing, highlights:
- It works! I was able to parse, complete with events, a 387 MB xml file.
- It uses less memory during operation
- After parsing is done most of the memory is cleared
- Negative:Unfortunately, since I'm not taking all the events into memory at once I can't give a % done for events, some large files may have 1.5million or more, so be patient. It's slower then legends mode but I think it's worth it.
Every world with xml that was sent to me works as well, which is the best I can do for testing right now.
I'm sending out a few PMs to those who showed interest but had world's too large. If you wouldn't mind testing against it, that'd be great :).
Post by: dennislp3 on August 08, 2010, 04:15:56 am
V 2.4.0
Fairly extensive re-write of my XML parsing, highlights:
- It works! I was able to parse, complete with events, a 387 MB xml file.
- It uses less memory during operation
- After parsing is done most of the memory is cleared
- Negative:Unfortunately, since I'm not taking all the events into memory at once I can't give a % done for events, some large files may have 1.5million or more, so be patient. It's slower then legends mode but I think it's worth it.
Every world with xml that was sent to me works as well, which is the best I can do for testing right now.
I'm sending out a few PMs to those who showed interest but had world's too large. If you wouldn't mind testing against it, that'd be great :).
Awesome update I was able to parse a 784 MB XML file
Failed with a 1.02 GB XML file but...cant expect everything...nor do I expect GB XML files lol....I got carried away there >.>
Post by: timtek on August 08, 2010, 06:35:24 am
Happened around 59% xml parsing. Total file size is 263mb.
Post by: Mason11987 on August 08, 2010, 07:12:48 am
Run time error '6': overflow :(
Happened around 59% xml parsing. Total file size is 263mb.
Could you send me all 4 files, I PM'd you my e-mail address.
Post by: Mason11987 on August 08, 2010, 11:23:29 am
V 2.4.0
Fairly extensive re-write of my XML parsing, highlights:
- It works! I was able to parse, complete with events, a 387 MB xml file.
- It uses less memory during operation
- After parsing is done most of the memory is cleared
- Negative:Unfortunately, since I'm not taking all the events into memory at once I can't give a % done for events, some large files may have 1.5million or more, so be patient. It's slower then legends mode but I think it's worth it.
Every world with xml that was sent to me works as well, which is the best I can do for testing right now.
I'm sending out a few PMs to those who showed interest but had world's too large. If you wouldn't mind testing against it, that'd be great :).
Awesome update I was able to parse a 784 MB XML file
Failed with a 1.02 GB XML file but...cant expect everything...nor do I expect GB XML files lol....I got carried away there >.>
Fantastic. Hadn't tried one as large as 784 MB, glad it worked. There are a few things you'll notice that won't work. For example listboxes can only hold 32k items, so it won't list every hist fig you have, I'm not sure of a good way around this right now, but we'll see. Also, it's possible you'll find a leader that has "...More" at the bottom but the "link" doesn't work, that's because that person wasn't in the histfig listbox.
I'll let you know when I make more memory improvements, I still have a lot of options. How did you manage a file that big? thousands of years, no cull important figures, and huge world?
Run time error '6': overflow :(
Happened around 59% xml parsing. Total file size is 263mb.
V 2.4.1
I fixed this problem, apparently you have a 50 civ world, and just a single one of those civs had 93 sites (out of 600 total). I've never seen a civ with that many sites. Since it was humans they built a HUGE amount of roads, so much so that the road counter was over 32k (there are multiple roads that make up a connection between two sites on the map). That was one of those things I didn't even imagine could be so large, but it makes sense now. As I get larger and larger worlds I'll uncover huge world problems :).
Also, now I show the Year of the last checked event, so when you're running through 10k years of events, you can get an idea of how long it'll take.
Post by: MaDeR Levap on August 08, 2010, 11:56:16 am
Since it was humans they built a HUGE amount of roads, so much so that the road counter was over 32k (there are multiple roads that make up a connection between two sites on the map). That was one of those things I didn't even imagine could be so largeSee? I told you! :P
Post by: Mel_Vixen on August 08, 2010, 01:53:35 pm
edit: And considering the latest updates of the devlogs you should expect a fairly huge number of towns, villages, coatages, farms, bridges etc. being in the next version of df.
If this (http://www.bay12games.com/dwarves/imgs/farm_map_det.png) happens with all human civs you need robust dropdowns and sufficiently huge variables. Especially since i normaly play on large worlds with the ighest possible number of civs.
Post by: Mason11987 on August 08, 2010, 02:16:05 pm
Since it was humans they built a HUGE amount of roads, so much so that the road counter was over 32k (there are multiple roads that make up a connection between two sites on the map). That was one of those things I didn't even imagine could be so largeSee? I told you! :P
I listened, just not QUITE enough :P
A quick idea: Just make the listbox so that it shows a part of an array and shift the displayed part if the player scrolls up/down.
edit: And considering the latest updates of the devlogs you should expect a fairly huge number of towns, villages, coatages, farms, bridges etc. being in the next version of df.
If this (http://www.bay12games.com/dwarves/imgs/farm_map_det.png) happens with all human civs you need robust dropdowns and sufficiently huge variables. Especially since i normaly play on large worlds with the ighest possible number of civs.
That's crossed my mind. I'll have to think about how to actually do that using the controls that exist (I'd prefer to not have to package anything but an .exe), but thanks.
For all the stuff in the update, I'll have to see how in-depth the xml gets. I'm not going to bother setting something up until I can parse it correctly.
As long as all that stuff is just added to entitypopulations the program will just ignore it until I put out a new version.
having 100+ locations (farms/etc.) for each civ won't really make or break anything, that's like 10k or so max probably, instead of the ~1k or so max number of sites. I'm more concerned with their being many more events, or historical figures, neither of which will probably go up by 100x, so we're okay.
I am looking forward to this update though :).
Post by: Mel_Vixen on August 08, 2010, 02:54:07 pm
Post by: timtek on August 08, 2010, 03:21:15 pm
Post by: Mel_Vixen on August 08, 2010, 05:14:55 pm
Post by: James.Denholm on August 08, 2010, 08:25:34 pm
And maybe a bar that shows what at the current moment is parsed to indicate that the program didnt crash.
Or, if it's difficult working out the number things requring parsing, something that flicks backwards and forwards during the loading. Or a box that swaps from white to black with each thing parsed.
Or something.
Post by: dennislp3 on August 08, 2010, 08:36:13 pm
Fantastic. Hadn't tried one as large as 784 MB, glad it worked. There are a few things you'll notice that won't work. For example listboxes can only hold 32k items, so it won't list every hist fig you have, I'm not sure of a good way around this right now, but we'll see. Also, it's possible you'll find a leader that has "...More" at the bottom but the "link" doesn't work, that's because that person wasn't in the histfig listbox.
I'll let you know when I make more memory improvements, I still have a lot of options. How did you manage a file that big? thousands of years, no cull important figures, and huge world?
Actually I did it with a medium world....at only 2000 years old.
to get such large files I actually used Deons genesis mod....I believe since some of the added races are extremely aggressive they spit out lots of info (kills by HF and whatnot as well as constant conquests and stuff)
Hard to say really but I found Deons mod adds to the XML in an extreme way...if you want to test larger files WITHOUT having to tweak to much or gen for excessive amounts of time I would try some worlds with it
as for what I tweaked....bumped up the titans from 9 to 15 or so...and the civs from 20 to 30 I believe
Post by: timtek on August 08, 2010, 09:02:59 pm
And maybe a bar that shows what at the current moment is parsed to indicate that the program didnt crash.
It doesn't show it's progress up top for you?
Post by: tenach on August 09, 2010, 03:45:52 am
Post by: dennislp3 on August 09, 2010, 04:36:26 am
Post by: Mason11987 on August 09, 2010, 06:01:25 am
Well done mason! Everything loaded up and works with out any issues. On a side note, judging by everyone's posted files sizes, I think you should change the "loading >10mb file please wait" message. I don't think I've seen anyone with less than 100mb file.
I removed it outright, along with every other popup. Either it will work, or it'll crash once you put in the files (also removed the "last year" for xml parsing, because I'm going to grab it from the year of the last event.
And maybe a bar that shows what at the current moment is parsed to indicate that the program didnt crash.
It's at the top for most of the time periods that take much time.
And maybe a bar that shows what at the current moment is parsed to indicate that the program didnt crash.
Bar at the top does it, but I can expand on it with more details for slower computers/bigger worlds. But at almost every part you should be able to click on the civ/site/leader tabs and look around, if that works, it didn't crash. Also if you see the titlebar being updated, it didn't crash.
Or, if it's difficult working out the number things requring parsing, something that flicks backwards and forwards during the loading. Or a box that swaps from white to black with each thing parsed.
Or something.
At a couple points it might show "Not responding" but if it does give it a minute before trying to close.
Actually I did it with a medium world....at only 2000 years old.
to get such large files I actually used Deons genesis mod....I believe since some of the added races are extremely aggressive they spit out lots of info (kills by HF and whatnot as well as constant conquests and stuff)
Hard to say really but I found Deons mod adds to the XML in an extreme way...if you want to test larger files WITHOUT having to tweak to much or gen for excessive amounts of time I would try some worlds with it
as for what I tweaked....bumped up the titans from 9 to 15 or so...and the civs from 20 to 30 I believe
I'll get an immense XML file when I go to work today, to see what happens. Thanks.
Thanks for such a great program! :D It's quite awesome to see all this information for world gens. :3
Awesome, feel free to vote on dffd for it if you like it :). More is to come, including battles and similar interesting details if I can manage it.
ok managed a vanilla game with a XML of .97 GB....it failed...I am guessing that around the 800 MB mark is the limit for now
It also might crash depending on your RAM, but yeah, that looks like it for now. When I gen a huge world I"ll make it work for that too.
Post by: dennislp3 on August 09, 2010, 06:41:34 am
Also....with an XML in the 700+ MB range it doesn't parse HFs....which is a major bummer...but civs and basic function is still intact
Here is the last world gen parameters that worked for me and produced a 700+ MB file (changes each time) but it seems to skirt the limit
it was in Vanilla and I modded the humans to be playable (simple copy of the traps, adamantine and steel, and nobles) and I also have the higher learning mod. These should not affect the results to much...
Spoiler (click to show/hide)
Post by: Mason11987 on August 09, 2010, 07:11:42 am
Also....with an XML in the 700+ MB range it doesn't parse HFs....which is a major bummer...but civs and basic function is still intact
Interesting, I've yet to see it fail once it gets to hist figs, so that'll be good to see (since 2.4.1, when I moved events in front of histfigs). I'm on about year 500 of a 3k year gen using genesis. We'll see how it turns out. Hopefully sometime this week I'll have a version out that can handle this huge of an xml (although it will DEFINITELY take a long time to parse.)
Post by: dennislp3 on August 09, 2010, 07:24:27 am
any world with about 200,000 HFs and about 3 mil events comes out to be right around 700 - 800 MB....so use that as a gauge perhaps
Post by: FinalSin on August 09, 2010, 07:46:29 am
Alternatively, would it be possible to get some 'Export As...' functionality that could spit out a sqlite database, or something more manageable than the XML? I want to use the Legends data for a sideproject, but I dont' want to reinvent the wheel when it comes to parsing the Legends data raw.
Post by: dennislp3 on August 09, 2010, 07:58:53 am
Post by: Ratbert_CP on August 09, 2010, 08:25:23 am
Post by: cephalo on August 09, 2010, 08:36:22 am
I was just thinking this weekend that it would be nice if there was a way to export or save a world, so that you wouldn't have to parse the same data over and over again if you want to keep looking at the same world. This assumes that VB has some sort of reasonable serialize function... ;)
I second this. It would be awsome to process these massive files once, and then delete them in favor of something more sane.
Post by: FinalSin on August 09, 2010, 08:47:50 am
Hint. (not hinting)
Post by: tenach on August 09, 2010, 02:09:13 pm
<snip>
Awesome, feel free to vote on dffd for it if you like it :). More is to come, including battles and similar interesting details if I can manage it.
Ahead of you there! You've got a 5/5 from me :D
Post by: Mason11987 on August 09, 2010, 05:19:58 pm
Yeah when I parsed my world (about to test another) it just pops up with a message at the very end "did not parse HF"....no error code or message or anything....it just didn't do it.
any world with about 200,000 HFs and about 3 mil events comes out to be right around 700 - 800 MB....so use that as a gauge perhaps
Ah, I understand, I found where that problem comes from. I'll fix this next update.
Is there any chance of getting the source for this? I would totally understand if the answer was no.
Alternatively, would it be possible to get some 'Export As...' functionality that could spit out a sqlite database, or something more manageable than the XML? I want to use the Legends data for a sideproject, but I dont' want to reinvent the wheel when it comes to parsing the Legends data raw.
I'm not quite sure now. I like working on this myself right now, it's a fun diversion for me and if someone else took it over and went another route I could see it ruining the fun for me. But if I stop working on it for whatever reason, I'll release it.
Exporting is near the top of my "to do" list. I had planned to do a "export information about this civ" to text sort of thing, for people who mentioned using worlds for RP stuff.
One of the reasons I made this program was to do the connections that exist between stuff, things like getting the list of kills a HF had, or creating dynasties and (if I can convince Toady to add an XML event) family trees within leaders and historical figures. I'm not sure how exactly I can do this, I guess having a series of tables with foreign keys referring to hist figs and such might be useful, but it seems like an immense amount of work for something not drastically different from the XML.
So for a database export I'll need some help specifically with database design, I've done this sort of thing before but having input beforehand might make me more willing to go through all the work. This is all assuming it's possible
For text related (MUCH smaller than the XML) export I have a few ideas on how I could do it but feel free to offer any suggestions before I get started.
I'm thinking of doing small scale export (of civs, sites, leaders, dynasties, and battles) in a few different ways, but I hadn't initially considered a XML-replacement. Would that be valuable even if I didn't have any of the additional logical connections from the program?
I was just thinking this weekend that it would be nice if there was a way to export or save a world, so that you wouldn't have to parse the same data over and over again if you want to keep looking at the same world. This assumes that VB has some sort of reasonable serialize function... ;)
This is something I've been thinking about as well. I think I'm being lead to create a .dfworld file. Unless the entitypopulations update by Toady coming (soon?) is extremely complicated and interesting this will probably be my next project.
Also, exporting in a tasty format (like sqlite, not that I'm heavily hinting at anything) would allow other coders (again, not hinting, nope) to make use of the great work already done here, and write further utilities that use this simpler format. :)
Hint. (not hinting)
*Goes to read about sqlite again...*
Post by: FinalSin on August 10, 2010, 05:11:44 am
What I wanted to do was use the generated worlds to generate new content from (things like stories or jokes including people from the world, or even small games). It's an experiment, really, in using one game's content to create new things, that I wanted to hack out. I'm rather unfairly burdening you with a million requests beforehand though!
Text export would be great anyway, because I'm sure people would be interested in a more human-readable format. From my perspective, a database (or even a CSV-style file) would be a boon, but it's really up to you. You've done great work so far. :)
Post by: Kaelem Gaen on August 10, 2010, 05:15:33 am
I was parsing a history off a NrthAmer map, but I was trying to find info about a human city (Well ruin) to see how it fell, I forgot to turn on "Cull Unimportant Historic figures" (actually since it was all inhabiting I was just lazy I guess). But you can only remove Non-Dwarven Civ/sites.
Though I'm wondering if It didn't bring up the ruins as they'd be covered in the History bit that didn't get parsed.
Post by: Mason11987 on August 10, 2010, 07:10:35 am
Haha. To tell you the truth, I have little experience in database design myself, but even if they just replicated the XML layouts (one table for each top-level element) it would be so handy.
What I wanted to do was use the generated worlds to generate new content from (things like stories or jokes including people from the world, or even small games). It's an experiment, really, in using one game's content to create new things, that I wanted to hack out. I'm rather unfairly burdening you with a million requests beforehand though!
Text export would be great anyway, because I'm sure people would be interested in a more human-readable format. From my perspective, a database (or even a CSV-style file) would be a boon, but it's really up to you. You've done great work so far. :)
Doing a straight replacement of the xml with a table would be difficult on a few levels.
- Sites aren't tied to civs in the XML, they could be though through intelligent use of the events (in XML) though
- Using just the XML it's impossible to distinguish among races for anything
- The events table would need to have a large number of columns, most would be empty for any individual event.
- I'd need to be able to append items to the event table, because I can't put all the events in at once
The first two could be solved by intelligently using the XML and sites/history files together to make the database (like they are used in the program). The third isn't a big problem, but I don't know how manageable a 2.5million record x 15 field table would be. The last may or may not be possible. I suspect it is though.
I'll work first on a text export of some sort. It's just hard to know what's important to other people.
If you want a straight list of HistFigs/Sites/Regions stripped from the XML with their IDs that's not a problem. If you want the events, I'll have to think of a meaningful way to export them (perhaps by event type?)
Hopefully I didn't miss this when I jumped to the end but, I was wondering if you plan to allow a filter for any race.
I was parsing a history off a NrthAmer map, but I was trying to find info about a human city (Well ruin) to see how it fell, I forgot to turn on "Cull Unimportant Historic figures" (actually since it was all inhabiting I was just lazy I guess). But you can only remove Non-Dwarven Civ/sites.
Though I'm wondering if It didn't bring up the ruins as they'd be covered in the History bit that didn't get parsed.
If the events didn't get parsed then you wouldn't see what happened to it, but my understanding is that sites, once founded, are a permanent fixture on the world, so they should be in that list somewhere. If it was destroyed it wouldn't be listed as "Human" under the sites, because race is tied to the race of the currently occupying civ, which there wouldn't be one. In the future I'll (possibly) tie race of a destroyed site to the race that founded it, or last reclaimed or captured it, but for now I think it makes sense as is.
You can also sort by name to find it.
But to answer your question, I am planning a general filter for a future release. It's something I would find personally really cool and also really useful for debugging. It won't be exactly as efficient as the name filter in legends mode, but it'll be more extensive (only births > 100, only the dead, only sites that have been reclaimed, etc.)
Post by: FinalSin on August 10, 2010, 08:18:39 am
I don't know how manageable a database that big would be either, but I think it wouldn't be any worse than storing it as XML. Maybe. :P
The other alternative could be to re-engineer part of your code so it becomes a sort of API that could be used to query the XML.
Post by: Ratbert_CP on August 10, 2010, 08:44:29 am
Normalize, normalize, normalize!
Quick, partial, off-the-top-of-my-head schema:
Spoiler (click to show/hide)
Post by: FinalSin on August 10, 2010, 08:54:09 am
Any recommendations for handling events, though?
Post by: Mason11987 on August 10, 2010, 08:56:53 am
Event type would work. I feel like I"m tossing a lot of work out there for my benefit only - anyone else think this might be useful to them?
I don't know how manageable a database that big would be either, but I think it wouldn't be any worse than storing it as XML. Maybe. :P
The other alternative could be to re-engineer part of your code so it becomes a sort of API that could be used to query the XML.
No worries, if I can export to a database of some sort, I'll handle that when I finish some other more requested (and easier) things. I always liked the idea of databases and linking tables together so I think I might refresh myself on my database course in college and do a database design.
*dusts off copy of Visio*
I don't know about manageability either, but we'll see. I don't know about API stuff though.
re: db design
Normalize, normalize, normalize!
Quick, partial, off-the-top-of-my-head schema:Spoiler (click to show/hide)
I like it, definitely a good start. And I know all about normalizing, don't worry there.
I like this guy.
Any recommendations for handling events, though?
I'll probably do an eventtype table. But should I do a table for each type of event? Since they each have different attributes?
Post by: FinalSin on August 10, 2010, 08:58:32 am
Post by: Mason11987 on August 10, 2010, 09:02:45 am
I guess you'll have to. And then - I literally have done no databases since a first-year uni course four years ago - then maybe a master table that uses the event id to identify what kind of type it is?
Yeah, but then I have foreign keys in a column on the eventtypes table that refer to multiple different tables, and that's not right. I guess the simplest, and the correct solution is to just do all events in the same table, and to just leave null the fields a record doesn't use.
Post by: FinalSin on August 10, 2010, 09:21:15 am
Post by: Mason11987 on August 10, 2010, 04:11:50 pm
Yeah, could easily do that. I would almost certainly marry you if any of this happens.
So I've found that it SEEMS like exporting to sqlite is possible, here's some plans.
Spoiler: Thoughts? (click to show/hide)
There is a many-to-many between civs and deities too (which doesn't quite get covered by the deity -> Site -> Civ relationship) that isn't shown.
Specifically I'm not outputting events, because since I can display information in the program without storing events, I think it can be just as useful in the database without a table (by putting stuff like founders in sites and killed by in HF.
Post by: FinalSin on August 10, 2010, 04:18:32 pm
Post by: darius on August 10, 2010, 04:55:35 pm
Post by: Mason11987 on August 10, 2010, 07:22:43 pm
hmm shouldn't the father/mother go to tbl_HistoricalFigs, not leader?
Actually no. there isn't any way of telling which historical figure gave birth to which historical figure (at least not yet). Given the current state of the XML you can only determine parents under a very small set of circumstances. Only if the father was a leader, and he died and his child became the leader afterwards, otherwise it's impossible to know. All of this you can get just from the worldhistory file which only shows information on leaders (kings/queen/princess/law-giver/master/etc.)
Some other glaring omissions (from the XML, and therefore, here) is that there isn't a birth year for historical figures. I think I can go from the first event referring to the HF to be their birth, but I haven't tested this out yet.
Post by: dennislp3 on August 10, 2010, 09:30:22 pm
WorldViewer
---------------------------
Unknown Event type!
<historical_event>
<id>147570</id>
<year>161</year>
<seconds72>-1</seconds72>
<type>plundered site</type>
<attacker_civ_id>781</attacker_civ_id>
<defender_civ_id>797</defender_civ_id>
<site_civ_id>797</site_civ_id>
<site_id>88</site_id>
</historical_event>
Saved to clipboard, please paste to development thread. Thanks!
---------------------------
OK
---------------------------
Dont shoot me im just the messenger =P
Let me know if you need my worldgen files
*EDIT* Also under the world details tab it reads that I have -119210 living HF and 119210 dead ones (total HF count says 0) I assume this is because it did not parse HF (which you already stated will be fixed next release)
Post by: FinalSin on August 11, 2010, 05:03:04 am
Post by: dennislp3 on August 11, 2010, 05:05:20 am
What are you working on anyways?
Post by: Mel_Vixen on August 11, 2010, 06:12:58 am
Yeah, could easily do that. I would almost certainly marry you if any of this happens.
So I've found that it SEEMS like exporting to sqlite is possible, here's some plans.Spoiler: Thoughts? (click to show/hide)
There is a many-to-many between civs and deities too (which doesn't quite get covered by the deity -> Site -> Civ relationship) that isn't shown.
Specifically I'm not outputting events, because since I can display information in the program without storing events, I think it can be just as useful in the database without a table (by putting stuff like founders in sites and killed by in HF.
Errrr a relationship indicator would be helpfull here. I can only guess whats "many to many" "one to one" etc.
Between entity and Site: I guess a site can be counquered Multiple times by multiple entitys (also founded(not so much), destroyed, and reclaimed). This can easely be one table between Site and entity with each entitys/sites Primary key, a date and the type of event.
Post by: FinalSin on August 11, 2010, 11:02:06 am
You would need a stronger argument/suggestion then "more detail into legends"
What are you working on anyways?
I want to use the worlds generated from DF and use it to augment other games - a bit of game design theory, really. Let's say I write a generic set of puzzles in a text-adventure style. Can I then publish that as a game, and say "Hey, make it personal by adding in a Legends printout. I'll reset the entire story so it takes place in this world!"
I'd like to see what I can do with it.
Post by: Mason11987 on August 11, 2010, 02:48:28 pm
Yeah, could easily do that. I would almost certainly marry you if any of this happens.
So I've found that it SEEMS like exporting to sqlite is possible, here's some plans.Spoiler: Thoughts? (click to show/hide)
There is a many-to-many between civs and deities too (which doesn't quite get covered by the deity -> Site -> Civ relationship) that isn't shown.
Specifically I'm not outputting events, because since I can display information in the program without storing events, I think it can be just as useful in the database without a table (by putting stuff like founders in sites and killed by in HF.
Errrr a relationship indicator would be helpfull here. I can only guess whats "many to many" "one to one" etc.
Between entity and Site: I guess a site can be counquered Multiple times by multiple entitys (also founded(not so much), destroyed, and reclaimed). This can easely be one table between Site and entity with each entitys/sites Primary key, a date and the type of event.
I didn't bother doing that yet. But they are all one to many. The lines are actually all supposed to connect from a Primary Key to another table, but I realized I broke that moving stuff around.
It's true that a site can be conquered multiple times. I hadn't taken that into acocunt in the program so I didn't account for that in here yet.
Thanks for the comments, as I bet you could guess, I don't do database design for a living :).
Post by: DalGren on August 11, 2010, 06:28:33 pm
Legends mode/worldgen data is incredibly amusing at times, but the legends viewer in-game surely has its shortcomings. I am glad to see that "relatively useless" data is getting some love, and tools like this will help more people get more from it. And in modding this is valuable to test interactions or what killed a modded race too early (for debug) and such.
I'll make sure to tell anyone interested in Legends about this. Everyone else should do the same! *cracks whip*
There's too much amusing bullshit hidden in those Legends, who knows what kind of goodness the community can end up doing if it's more accessible than it is now.
Post by: Mason11987 on August 12, 2010, 12:16:44 am
This is the best use of VB code I've seen in a long time :P
Legends mode/worldgen data is incredibly amusing at times, but the legends viewer in-game surely has its shortcomings. I am glad to see that "relatively useless" data is getting some love, and tools like this will help more people get more from it. And in modding this is valuable to test interactions or what killed a modded race too early (for debug) and such.
I'll make sure to tell anyone interested in Legends about this. Everyone else should do the same! *cracks whip*
There's too much amusing bullshit hidden in those Legends, who knows what kind of goodness the community can end up doing if it's more accessible than it is now.
Thanks! Solid 4k lines of VB goodness :D. Feel free to offer any suggestions or throw a vote on in DFFD if you think it's worthwhile.
I've really enjoyed genning worlds and testing them out here. I played this game for so long without having any idea about worldgen, I just want to make sure others can get into it without a lot of effort like it took me originally.
V 2.5 (http://dffd.wimbli.com/file.php?id=2637)
@dennislp3
I believe I've fixed the problem with historical figures, please test this.
You also shouldn't get the event popup for plundered site anymore
@Kaelem Gaen
Spoiler: Much better filtering (click to show/hide)
Removed the request for a last year if you're using XML
Cleaned up the screen a little and some minor related fixes (ex: clicking a site/leader/entity/etc. will now show their name in the frame caption above the listbox, which I had meant to do, and did for civs)
Edit - Found a bug that might show up in larger worlds where histfig kills will have blank names. Nothing serious though, I'll fix that soon.
Post by: dennislp3 on August 12, 2010, 04:54:33 am
Post by: Kaelem Gaen on August 12, 2010, 04:56:25 am
Post by: dennislp3 on August 12, 2010, 05:07:40 am
Spoiler (click to show/hide)
Keep up the good work!
Post by: Mason11987 on August 12, 2010, 10:31:50 am
Alright....well...it works better and I like the improvements...except for thatSpoiler (click to show/hide)
Keep up the good work!
ha, alright. Is that how you originally saw it, or was that after you filtered or something?
I'm pretty sure I know what the problem is anyways, I'll fix it today.
V 2.5.1 (http://dffd.wimbli.com/file.php?id=2637)
For people with large xml with particularly large number of historical figures (probably 60k or more), they had many historical figures without a name. This is fixed here.
Post by: dennislp3 on August 12, 2010, 11:25:28 am
*EDIT*
Ok awesome...now the names show up just fine it seems...my only question is where are the "sort by" options for HF? were they never there? I don't remember but I know you can sort every other section by things like length/age/pop etc.
I am genning with a 487 MB XML file (as opposed to my usual 7 or 800 + MB ones =P)
Post by: Mason11987 on August 12, 2010, 11:40:08 am
I didn't touch the filters. I could select names on the left (as you can see the one selected is about 6 or 7 names down or something) but the names did not display....in the descriptions only year and location were shown so hopefully it is related to the update you just posted...let me test it for ya
*EDIT*
Ok awesome...now the names show up just fine it seems...my only question is where are the "sort by" options for HF? were they never there? I don't remember but I know you can sort every other section by things like length/age/pop etc.
I am genning with a 487 MB XML file (as opposed to my usual 7 or 800 + MB ones =P)
Yeah, the list can fit at max 30k things. Turns out my method for sorting items (selection) is terrible at sorting 30k items :). Really, it's painfully slow.
If you put filters in that drop the number far enough (5k) the options will show up. Before I put in options that let you do ridiculously long sorts, I have to improve how sorting is done. Problem is I can't seem to figure out a good way to implement quicksort or mergesort (http://www.sorting-algorithms.com/) yet. But it's been on the list.
Does the 700MB one work?
Post by: dennislp3 on August 12, 2010, 01:32:45 pm
Post by: Thoth on August 17, 2010, 12:07:01 pm
Spoiler (click to show/hide)
Post by: TomiTapio on August 18, 2010, 01:50:20 pm
Oh, those no-name culled figures? You can turn off the culling of unimportant people, in the worldgen.
[CULL_HISTORICAL_FIGURES:0]
Post by: Knight Otu on August 23, 2010, 07:54:01 am
Doren, deity: metals, jewels; in The Curled Dikes
Doren, deity: mountains; in the Wall of Funerals
Doren, deity: minerals; in The Glad Lanterns
And the program uses the last one only for all three. As far as I see, the distinction can't be made by originating civilization, since one force can be worshiped by several civilizations, and it can't be made by name plus portfolio, since they might still share the same portfolio after all.
Do you want the files in question?
Post by: Roses on September 12, 2010, 12:59:13 am
Seems to parse fine up until it gets to my game start year, then a raft of those, is Genesis 3.03, 775mb xml, no idea what else might be useful to you, hope thats at least slightly helpful. Otherwise marvellous proggie, i may ofc have done something dumb, it's not beyond me, but afaik i chose the files in the right order, and is fine right up until it gets to the year i started.Spoiler (click to show/hide)
Don't know if this is still being developed or what, but I encounter the same problem when trying to use the XML to go past my game start year. In addition to the above message I also get
Spoiler (click to show/hide)
Post by: ouroborus on September 12, 2010, 03:07:30 am
Post by: Mason11987 on September 15, 2010, 11:30:26 pm
Post by: Mason11987 on September 16, 2010, 11:58:52 am
Also, there is now a lot of stuff in the historical_event_collections and sections (thought all empty so far) in the entity_populations sections. This should be really cool to work on, an example of something neat in the XML now that wasn't before:
Spoiler: One of hte historical event collections (click to show/hide)
And a few responses:
Seems to parse fine up until it gets to my game start year, then a raft of those, is Genesis 3.03, 775mb xml, no idea what else might be useful to you, hope thats at least slightly helpful. Otherwise marvellous proggie, i may ofc have done something dumb, it's not beyond me, but afaik i chose the files in the right order, and is fine right up until it gets to the year i started.Spoiler (click to show/hide)
Interesting, I hadn't seen that event. Could you maybe explore legends mode and give me a screenshot if you can see what this event actually means? Normally I would do that myself but your save looks like it might be too large to transfer. What if you set the end year to later?
Cool tool it is. By any chance could you release a version that looks good on 1920x1080 monitors? That 5 px font is a little hard to read.
Oh, those no-name culled figures? You can turn off the culling of unimportant people, in the worldgen.
[CULL_HISTORICAL_FIGURES:0]
I'll give it a shot, but it's lower in the list :)
Something that's probably not going to be easy to be tracked, but occasionally, gods have the same name, but are of different civilizations and have different portfolios. For example, I have three dwarven deities named Doren:
Doren, deity: metals, jewels; in The Curled Dikes
Doren, deity: mountains; in the Wall of Funerals
Doren, deity: minerals; in The Glad Lanterns
And the program uses the last one only for all three. As far as I see, the distinction can't be made by originating civilization, since one force can be worshiped by several civilizations, and it can't be made by name plus portfolio, since they might still share the same portfolio after all.
Do you want the files in question?
Interesting! I didn't know this was possible. I think I'll try to identify deities/forces by their name + details + civ, I seriously doubt there could be two DIFFERENT deities of the same areas with the same name worshipped by the same civ. Thanks for pointing this out.
Seems to parse fine up until it gets to my game start year, then a raft of those, is Genesis 3.03, 775mb xml, no idea what else might be useful to you, hope thats at least slightly helpful. Otherwise marvellous proggie, i may ofc have done something dumb, it's not beyond me, but afaik i chose the files in the right order, and is fine right up until it gets to the year i started.Spoiler (click to show/hide)
Don't know if this is still being developed or what, but I encounter the same problem when trying to use the XML to go past my game start year. In addition to the above message I also getI also seemed to get an Unknown error for almost every masterpiece item my site ever made. Clicked through it maybe three or four hundred times.Spoiler (click to show/hide)
I've never seen that event either. If you could send me your save that might be helpful. Thanks!
Seems to me that a search feature could be added for finding potential embark sites. I've always thought that the in-game search feature could be at least faster and cover more variables. (Although I guess some things would not be search-able this way as they don't show up in the exportable data; e.g.: volcanoes.)
The problem is that I can't really get a lot out of the map. Because it shows by blocks, which in-game are 16x16 areas, I can't tell you anything about a 4x4 area through this. In fact almost nothing in the finder I could help with, besides being near caves, or other civs.
Post by: Knight Otu on September 16, 2010, 01:13:38 pm
Interesting, I hadn't seen that event. Could you maybe explore legends mode and give me a screenshot if you can see what this event actually means? Normally I would do that myself but your save looks like it might be too large to transfer. What if you set the end year to later?That does seem like it should be the arrival of a merchant caravan at a player fortress. I've seen those engraved before, so it makes sense that it would be an exportable event to.
Interesting! I didn't know this was possible. I think I'll try to identify deities/forces by their name + details + civ, I seriously doubt there could be two DIFFERENT deities of the same areas with the same name worshipped by the same civ. Thanks for pointing this out.Well, as I said, the same force can be worshiped by different civs if they start in the same area that the force permeates, so it should only be necessary for deities.
Post by: Mason11987 on September 16, 2010, 01:34:29 pm
Interesting, I hadn't seen that event. Could you maybe explore legends mode and give me a screenshot if you can see what this event actually means? Normally I would do that myself but your save looks like it might be too large to transfer. What if you set the end year to later?That does seem like it should be the arrival of a merchant caravan at a player fortress. I've seen those engraved before, so it makes sense that it would be an exportable event to.Interesting! I didn't know this was possible. I think I'll try to identify deities/forces by their name + details + civ, I seriously doubt there could be two DIFFERENT deities of the same areas with the same name worshipped by the same civ. Thanks for pointing this out.Well, as I said, the same force can be worshiped by different civs if they start in the same area that the force permeates, so it should only be necessary for deities.
Agreed, I just hadn't seen it, and when I asked Toady to list all the possible events he didn't list that one (or I missed it).
It's possible I never tried to do this with a world that I've played for a while, so that could definitely be it too.
I treat forces almost the same as deities, so I'll put them in both. Also, I think two civs have worshiped the same deity for me.
I set up a thread in modding so people can work together to figure out the historical event collections (new addition to the XML!), for anyone interested here is the link (http://www.bay12forums.com/smf/index.php?topic=66228.0)
Post by: theqmann on September 20, 2010, 10:39:34 pm
Post by: yarr on October 06, 2010, 02:20:12 am
Post by: helf on October 06, 2010, 11:26:45 am
Post by: slink on October 09, 2010, 02:34:37 pm
Win 7 Pro on quad-core computer: Program vanishes from application list. It has to be killed from the process screen.
WinXP Pro SP2 Virtual Machine on quad-core computer: Program hangs and the process associated with the application is shown as using 99% of the CPU. It can be killed from the application list.
WinXP Pro SP3 on single-core computer: Program hangs and the process associated with the application is shown as using about 14% of the CPU. It can be killed from the application list.
In all instances it is being run with administrator rights.
Post by: Victuz on October 18, 2010, 08:26:07 am
I thought it might be because of the fact that I'm using .16 so I checked it in .14 and .13 but still the same thing happens. In the end I get a parserlog with only one name in it and the thing crashes. Happens both when I'm using XML and when I'm not.
Running Windows XP as an administrator. Any way to get around it? or is it just not going to work for me? :P
Post by: Sowelu on October 18, 2010, 07:59:07 pm
Post by: dennislp3 on October 20, 2010, 07:27:53 pm
From what I probably-mistakenly gathered, it works on .12 and was never updated to .13. So perhaps that is possible or perhaps I am wrong.
Thats exactly the case
Post by: yarr on October 21, 2010, 06:15:48 am
Post by: Fieari on October 21, 2010, 10:57:13 am
Getting the same crash everyone else is reporting.
Post by: Victuz on October 23, 2010, 02:01:22 pm
I personally just got back to it since it's the most reliable version so far. Working cities in adventure mode, working tools, stonesense and all that.
Now all I need is .12 genesis mod and THIS to have fun ;).
Post by: anallyst on November 19, 2010, 07:24:26 am