Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  

Author Topic: Export goes over the edge.  (Read 706 times)


  • Bay Watcher
    • View Profile
Export goes over the edge.
« on: August 04, 2018, 02:54:54 pm »

Forgive this little missive (and if it's TL;DR; then you are oerfectly free to not R). It isn't what I would call a bug, but it is my attempt to document a little personal battle against the particular configuration of ones and zeros I find myself using. If it's of interest to others, I would be gratified, but it seems no great burden uponmthe world if it is of no interest and yet I have committed its own bits and bytes to the forum in one interminably forgettable post that quickly moves off of any first-page display you care to think of.

The point being... I discovered a problem with my worldgen, on my old 32-bit system that I still use for DF. I'm fairly sure it's hitting the 32-bit hard limit for memory (including virtual RAM), but I'm not entirely convinced. But that's a first-order working theory.

It took a few days to narrow down (several overnight worldgens, and a couple of while-I-was-out overday ones) but I took the offending version (slightly customised, but only in geological matters) of the LARGE ISLAND worldgen template and put into it the precise seeds noted in the gamelog from when I first tried to worldgen a new land - within which I spotted some geogrqphy that I'm still rather keen to try to explore/build upon (in parallel copies of the save). The point being that I've had identical setups except for the enforced END_YEAR.

I used to have no trouble at all with the 1050 default upon the largest world footprints (and increased inter-cavern geologies), but added complexity of the world has clearly added to the demands called upon to enumerate the end product (and in this instance I'm specifically talking of a 44.12, but it was true with 44.02 and even in prior generations to the 44 release). I have tended to get a semi-round Year of 750 going ok without problems (maybe a message that virtual memory is being expanded for me, especially if I have other intensive things running like GIMP) except for having to leave that machine alone for some time and get on with other things. But this time I got an application crash at some point after using the "p" to export the world details. Within the DF directory the regionN-00750-01-01-world_map.bmp would appear, zero bytes in size and then after a wait I'm prompted to send details of the crash off (which I decline; and it wouldn't help, anyway, as the machine is standalone). The world_gen_param and._history and _sites_and_pops text files never get created. At no point do I actually get prompted about virtual memory resizing.

By a bit of trial and error (rerunning for years above and below the point of failure, taking roughly half the difference and seeing if that's a "too high" or "not high enough to error", then repeating within this new established range) I discovered year 402 gave me a reportable save (which I seem to be able to, embark on, from the briefest of tests) and a year 403 report request crashes.

Sigmificantly, year 403 can be generated to and accepted without crash (I have yet to try to embark/adventure upon it) but I haven't yet tested the upper bound of a) No virtual memory resize notification¹ and b) No similarly irrecoverable crash². My current theory is that it (whatever 'it' really is) will not happen much prior to an ending-year of 806. At its simplest, the crash-on-report could happen when the genned world data objects are copied whole (to instantly double the memory needed) in order to operate upon the copy to pare it down to the report info (for which the nul bitmap has been opened for writing as.a standard precaution to rule out read-only file system or other such failures, but never gets to the point of being written to).

However, that would be simplistic. Most likely a report-sized (including map-sized) chunk of memory is to be reserved, ratjer than an identical copy. Also more years generate more hiatorical events. Leastways, this aspect of the mystery remains untested by me. Ditto much testing of otherly-seeded versions of the same custom LARGE ISLAND, non-customised (similarly/otherly-seeded) LARGE ISLANDs and (apart from my Region1 save, a "CREATE WORLD NOW = 433333" creation that did not cause any such ripples and is currently be Adventured around, betwixt other demands) anything not a LARGE ISLAND in world size.

I did rerun to Year 403 with a particularly inefficient Perl script (width-first and greedy searching of a dataset phase space, a quick'n'dirty hack towards looking for a solution (unrelated to DF) that demands more memory than it deserves and will spin up the CPU fan on its own) to see if a concurrently resource-greedy process would push DF backmover the edge again, but it did not. A tribute to whatever form of virtualised resource compartmentalising is practiced by this system well into its second decade of utility, I'm sure. And why I still stick with it for these little things

Anyway, apart from not considering a buf, a brief foray onto Mantis hasn't revealed (relevent!) bug reports such that have succumbed to my inept and uncomprehensive attempts at discovery. As I don't consider it 'fixable' nor relevent to pretty much everyone else out there (who do not reserve a particlarly old workhorse/mule of a machine for DF pleasures, as other devices take up other (more) necessary tasks of the day) I'm not inclined to further chase/create this particular line of enquiry.

Yes I could post the WORLDGEN template for others to try under their own circumstances, but with the logical separation of machine I'd have to revist it, copy the necessary onto USB and then post it up..? And I haven't yet really looked (in either play-mode) at the features on the worldgen-following map that caused me to spend so much of my DF-playing machine's time on repeated worldgennings. It's entirely possible that there isn't even that much of interest to the player (over and above any starting world sourced from any other combination of seeds and other worldgen params). That is something I hope to find out, in various extended idle moments over the next week or two.

And so, apart from the below feetnete, that's all I have to say. If you're still reading then go get yourself a cookie (or a strangely green and definitely unsweetened smoothie, according to your current dietry pursuits) as reward. And then, probably, forget all about these mad/maddening ramblings and resume living the rest of your life.

¹ I might need the whole machine restarting each time, but not entitely sure if that's sufficient and/or necessary to make each iteration an equally fair test, depends upon memory handling/reverting practices with the OS - and I haven't previously had very much reason to establish what behaviours it follows.

² Initial thoughts were that tye OS-level memory resize process, during which I am warned that some things asked by the program may not have happened, was the (later) cause of the crash as the twrget of an unfortunately nul-pointer is given an attempted peek or poke. Though, as it appears that I avoided the resizing message (so far as I can tell), I've now put that idea low on the list of possibilities.


  • Bay Watcher
    • View Profile
Re: Export goes over the edge.
« Reply #1 on: August 07, 2018, 02:59:05 pm »

What is the problem? World gen crashes/inability to reproduce seeds should go on the mantis bug tracker.

It's a known problem that world-gen isn't 100% reproducable but it's not an easy thing to fix.


  • Bay Watcher
    • View Profile
Re: Export goes over the edge.
« Reply #2 on: August 07, 2018, 04:13:46 pm »

If it is the error I think it is, I'm not going to bet it fixed. This can act best as a note for others to see. IMO.

And, as far as all tests show, everything does seem reproducible (between reboots, same boot-up after a prior fallout, etc, on the same machine setup with varying degrees of resources currently in use; and I have the exact same event/figure totals for the same end-date, etc). The only obstacle is the time taken to get to the point (where accepting a region-gen is possible, whether or not it can be exported without crashing out) and I mostly only use the export to get a handy map (which I think only changes by settlement distribution/quality, by the time I get to it (amendments of which I can add myself when I'm tracking my way and my side-quests in Advwnture Mode, say).

If you want to check if it reproduces on machines other than the ones I find it convenient to use, I can supply details (maybe tomorrow), but I don't see quite so much necessity to escalate to official levels. Not for my pig-headed sake alone.
« Last Edit: August 07, 2018, 04:15:28 pm by Starver »


  • Bay Watcher
    • View Profile
Re: Export goes over the edge.
« Reply #3 on: August 08, 2018, 07:04:16 am »

Code: [Select]

Just FYI.
Title comes from having been a copy of a "Large Island (Custom)" of 450 years but the seeds edited in. Which explains why this copy is down at 150 year limit (the only change I ever made, between runs, being END_YEAR) as it's the point at which I last saved the change before hunting back up in date-length (not bothering to F6/whatever each time).

Which reminds, me... From way back when.
I must give that a go. See if it still works, see if it fails at the same point. I can even afford to leave it running several days while I'm out, automating the whole process so I don't need to be back at each stage/can ignore it completely.


  • Bay Watcher
    • View Profile
Re: Export goes over the edge.
« Reply #4 on: August 09, 2018, 12:12:04 pm »

Have you considered monitoring DF's memory usage? That should tell you if it gets close to the 32-bit memory limit (4 GB, except on Windows where it's 2 GB but can be increased). There was a bug report on Mantis about 32-bit DF running out of memory (maybe this one]]this one) but it was resolved when 0.43.05 came out. There's really not a lot that can be done if 32-bit DF crashes because you turn up world size/history length settings without upgrading to 64-bit DF.
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.


  • Bay Watcher
    • View Profile
Re: Export goes over the edge.
« Reply #5 on: August 09, 2018, 01:35:33 pm »

The irony is that it is actually a turned down size/history. I used to regularly run to the default 1050 year history for Large-typed worldgens, and if I messed at all with the strata (in case it matters for intrinsic worldgen-size, though I think it only takes effect upon localised procedural generation of features) I would give 20-high "LEVELS_ABOVE_"s.

Spoiler (click to show/hide)

As for directly monitoring memory usage, I never really sat with the Process/Performance tabs open, to be stared at, or keeping perfmon running. I've only really used them before when the entire system is wont to fall over. Which doesn't happen here (the worst I get is the already slightly fragile OpenOffice windows, if open, sometimes going a bit refresh-tardy and focus-lagged when alt-tabbing back and forth between them and anything else hogging resources, but I probably should have switched them to a later and more hardy LibreOffice version by now, anyway). Adding that sort of my thing to things to try if I do devote more time to the problem, though. Mostly, so far, I've just been peripherally aware of CPU usage by fan-speed. DF is the only program (besides momentarily for some of my perhaps excessively forumlae-involved spreadsheets and some GIMP filters/effects, when invoked to act) that tends to send the fan on one box whirring to a degree noticable from the other side of the room.

I'm also maybe soon in the position of getting a spare Win7 licence (obviously I've skipped Vista, not touching 8 with a barge-pole, while 10 I'll leave for machines it's pre-installed upon and frankly I'm unlikely to go and get one of them any time soon, while I don't need to) on one or other of my, boxen currently dedicated to something else. Not top-end, but would be my first personal 64bit Windows system that I have that I'm not actually mostly using dual-booted up onto a Linux environment (and doing things other than DF on). But I'm not rushing this just for the sake of DF, especially as I don't yet know exactly which choice of hardware I'm best handing on down to myself for reallocation to such 'leisure' use. Until then, I'm accepting my limitations in being steadfastly Luddite with my perfectly working hardware increasingly lagging behind the base demands upon it. This was supposed to be more a record of something of maybe transient notability than anything more, but I appreciate your interest.