Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[TranceProgHouse]

I was looking around to see if anyone has pushed the limits of the capabilities of world generation in terms of map size.  Mainly, I am looking for a way to edit the world size maximum and related parameters beyond their limits in the quickest way possible.  I should note that the world gen text file does not "force" those changes in the DF program itself.

I went into advanced world gen and was able to edit the world size maximum of 257x257 to 513x513 with CheatEngine and I believe it did finish one generation, but I just jumped into the next try of 1025 and beyond which of course did not work.  I do still need to go back and test to see if 513x513 really did fully work, since I only scrolled around a bit and didn't check stability at all.

Before I decided to try again though, I wanted to see if there was anyone else doing something similar or has advice on how to make this trial and error a little easier.  I probably have to ask this is in an area where people specialize in memory address location/editing, since I have to find the "new" memory addresses each time I run the program, for any given parameter that is necessary to finish the worldgen like "# of sub regions".  The approach is "try", CRASH, "try again" where each time means finding the addresses again, so it's the most time consuming part.

I was thinking there was a faster way to make this change given the number of mod utilities that have been created or people who know a decent approach to finding and saving a constant path to the variables that have to change once, rather than each time the program runs.  While I did gloss over some details on what I think is the approach, someone who knows can probably direct me more specifically to save some serious head aches, which granted I probably deserve for attempting this.

I should add that even though those are just a few variables mentioned to change, there are many other parameters that would need to potentially be extended beyond their maximum to adequately fill the world.

Most of the motivation to do this was when I figured out what I thought were the approximate "units" of the "world".  I kept it simple on the basis that a single tile can contain a giant sperm whale, took the cubed root of it's volume of 200m^3 or 5.848m and figured that if the total volume of the tile is at least (6^3)m then it could fit anything in the game, granted in a nonsense cube shaped way.  Of course this doesn't make sense on a constant unit basis, but it could be a fuzzy logic thing where the units are undefined and instead it could be a range of units, but that's the "sort of reasonable" maximum.  Anyway, the point is: why not generate an "earth sized map"?  Right?

Assuming I didn't already screw up the math and you can be "ok" with the concept, I can calculate the map "length" of an "earth sized world":  1 world tile = 768 land tiles.  Taking the square root of the surface area of earth divided by 6*768 gives me 4901.35, the number of land tiles in one direction, so a map size of 4097x4097 would be close enough to be an "earth" sized world, I think.  The random answer to a question asking about world generation I found mentioned the map size should use an integer sequence given from a(n) = 2^n + 1 ; not sure if that's needed, but 4097 would work.

Now, I don't really except this to work, but I figured if I got 513x513 to work, or at least so I thought (still have to double check), then maybe I can get the next step up to work.  Given how many crashes I expect, if someone can help me save time on changing those values faster, I'd appreciate it.

Alternatively, someone may be able to prove why it is mathematically impossible to generate a world this size based on some software/hardware limit and maybe save me time in a more direct way, granted with some disappointment.  As far as I can tell though, the worst case is just ending up with a 100 gigabyte world ... which yeah ... that's fine.

[PatrikLundell]

The standard tool for anything regarding DF modding (apart from RAW file modding) is gui/gm-editor. It allows you to modify any field the DF community has mapped, and I'd be very surprised if the X/Y dimensions aren't among those.

The size of an in-game tile is 2*2*2.5 meters, but real world physics are ignored, so a giant sperm whale can fit in there (in fact, you can have an unlimited number of them in a single tile: one "standing" and the other ones "lying on the floor"). Thus, a world tile is 1536 *1536 meters (48 * 48 in-game tiles in each Mid Level Tiles, and 16 * 16 of those in each world tile). If we use the ballpark estimate of having a world tile being 1 Nautical Mile wide (at 20% or so off), you'd need 60 * 360 = 21600 of those to circumscribe the Earth. Assuming the game doesn't break, you'd need a rather significant hard disk farm to store all that, as well as a gigantic amount of memory that would still probably have to swap to disk to generate the world (and an ocean of time in which to do it).

It can be noted that my numbers seem to be similar to yours when the different in-game tile size is accounted for.

I can mention that I recently generated a standard max sized world with all sliders except megabeasts cranked up to max to get some size numbers, and DF used over 6 GB with that world.

[Fleeting Frames]

2.8m* going by falling speed. Though the height rarely matters.

Also I've done abnormally small worlds with gui/gm-editor before, you need to change the world size params to initialize the values iirc, but otherwise that is possible (though crashes sometimes). Issue is that reshaping removes your pre-set values....

[TranceProgHouse]

Ok, I will read into using the GUI/GM editor for this.  Yeah, I tried to justify unit size because I'm probably really asking for trouble trying to get to 21600 when I barely got past the default value.

[feelotraveller]

Three quick thoughts:

1) If not doing so already use an advanced world generation and set [END_YEAR:2] (the hard-coded minimum) to speed up world generation significantly by avoiding the need for history to progress much.  This will let you look at the possibility of physical world generation much more easily without the time needed to process history.  Beware though that increasing history length will also significantly raise memory requirements mentioned below, so secondary testing with proper history length is likely required later.

2) One limitation is FPS in the game proper since the world is activated and all sorts of events are processed in the background.  Larger world -> more world events to track/process.  Note that at least in dwarf mode FPS is a progressive drag on fortresses and is often the reason players abandon longer running forts.  See http://dwarffortresswiki.org/index.php/DF2014:Maximizing_framerate for some tips.

3)  The hard limit you are likely to run into during world generation is the amount of RAM on your machine. I believe that the whole world needs to be contained at once in ram during generation.  (Happy for someone to correct me if this is wrong.)  If you intend to play on the map generated then my understanding is that the most important limitation will be RAM-latency. DF, for all intensive purposes, is single-threaded.  There is a bit more on this at http://dwarffortresswiki.org/index.php/DF2014:System_requirements.  Also search the forum for the recurring topic of the best hardware for running DF for all sorts of opinions and perspectives. 

[PatrikLundell]

No, RAM is not a hard limitation, as virtual memory uses the hard disk for memory (that's what the swap file is used for). However, disk access is orders of magnitudes slower than physical memory, so things start to run at a glacial speed once the swapped out part of the memory is used to any extent (There's also a phenomenon called "thrashing" which is when something is swapped out to disk to get something from the disk, only to swap the swapped out part back again as it's needed, after which the part that's then swapped out is swapped in again... Program design can affect how likely this is, and I'd suspect DF is rather vulnerable).

[TranceProgHouse]

Crashes during world gen greater than 257x257 tend to happen during the lake generation/drainage portion, I've had it complete the hundreds of passes only to freeze in that moment and the program will usually eventually close, so I can't even interrupt it to view what was created, though there are times I believe I have closed it myself, so I could probably let it run over night.  I probably haven't done this enough times to be sure.  I'm trying to remember, but I feel like I may have interrupted a 1025 attempt, but the world was mostly mountains and I was unable to get it to work again, but maybe I'm thinking of one of the 513 attempts.  All of my attempts over 1025 failed.

As far as I know when I last glanced at performance stats, I haven't run out of memory yet.  I have a pretty high end consumer system for modern gaming (i9 9900k CPU) and no component appears to be maxed or failing during world gen, the program itself just hangs and that's only beyond the size limits, so I figured I was just "breaking" the generation algorithms and could figure out which variables to change to what values might allow it to get to the next step.  For fortress mode , I don't know when I'm going to experience gameFPS death, but I have had some moments at around 125 dwarves where during a Civ Alert it can chug hard briefly - I can manage this with burrows and I'm sure other ways, but I haven't quite dug into that yet, oh, pun intended.  I'm sort of new to this game, but not games like it, I just haven known of it for awhile and tried Adventure mode back awhile back since I've played old school roguelikes, so my experience on different topics is kind of all over the place.  I have a STEM background, do a lot of gaming (ha), but very little programming, so this probably limits my ability quite a bit, the whole memory editing and trying to figure out how to trace a route to easily edit what I need to consistently was going to take me awhile to figure out, if at all.

Cancellation spam will straight up drop me to single digits, but that was easily fixed by temporarily disabling the announcements.  My single thread speed and cache is up there, but I'm pretty sure my experience won't be much different from most users overall after seeing gameFPS drop under certain conditions and zooming out will drop FPS to single digits as well, no reason to play like that of course, just an observation.  I don't think this a problem that you can just throw expensive hardware at, I just happen to be an enthusiast when it comes to gaming, though this was a rare move for me since in the past I went with more cost efficient hardware.  Pretty sure my dreams of setting an absurdly high population cap will be crushed long before being even close to reached, if I even make it there of course.  I mostly understand the rules and limits of the game and what's out there, but not totally and I can still easily mess up even though when it comes to games like this I am often careful and deliberate.

I have tried cancelling history mid generation, but yeah, the last times that I tried, it just doesn't get there.  I haven't really had the motivation to try again yet, I still need to look into how I would do this with the GUI/GM because I really am not savvy when it comes to scripting, or at least it takes me awhile to dig though.  Quick example was I was really interested in Dwarf Therapist's scripting, but couldn't find a reference library and it seemed like it was designed for someone with an intuitive knowledge of it, so I set it aside for now since I assumed I was just missing something that would be otherwise obvious to someone familiar.  Specifically I wanted to call all the dwarves who had picks in their inventory, and it looks like you can do it, but maybe I'm over estimating what that program can do, I don't know.  Pretty sure I just needed the item codes and then figure out the command structure to write to do that, but yeah, I'm already getting into the part where I need to look a lot of stuff up and trial and error.

Edit: I should also add, although I noticed the oversized world gen would often hang before steps completed, if those steps completed at all, but there were times where they did not hang at all and actually ran quite quickly, up to where it would crash of course.  I suppose I should make sure to try those seeds multiple times to be sure, but I think that might be a key indicator that the program likes the values and then work back and forth from those values depending on results.

Edit 2: I just did another quick look though of this https://dfhack.readthedocs.io/en/stable/docs/Plugins.html#map-modification , if I am in the right area, went through that page's side bar and some of the definitions listed from other sections, and I'm just not seeing world size.  Maybe I just need to keep reading up on it.

[Fleeting Frames]

Regarding your crashes, there might be another reason for this. See this map.

The mountain wasn't originally there; it was initially painted to be all triangles. It didn't manage to generate even a legends-only world with that, despite 676ish biomes being way below 65k limit. So you might need a psv map with larger biomes.

I'm not certain how to get psv and larger world size to coexist, though.

[PatrikLundell]

This is the documentation for the Lua API https://dfhack.readthedocs.io/en/latest/docs/Lua API.htm. Apart from that, you basically need to familiarize yourself with the DF overall data structure (as mapped by DFHack), and the details for the area you want to mess with.
Locating all the dorfs that have picks in their inventories is certainly possible, as well as getting only those that are members of your fortress (as opposed to visitors).
Apart from gui/gm-editor (which is the string you need to type from the console window to invoke it, usually with a parameter to say what it should display, unless you want the currently selected thing in DF), the DFHack mapped DF data structures are mapped out in XML files that are generated into C headers (for plugins).
"gui/gm-editor df.global.world" gives you most of the DF data structure (there are some others, but the bulk is in tree beneath this one).

[feelotraveller]

There are a lot of possible sources for crashes, such as exceeding the memory limit on the number of sites or living figures, but crashing at the point of generating rivers/lakes makes me think it could be related to the erosion cycle.

In terms of finding the source of crashes (and maximum possible world size for your system) I suggest paring everything back (other than of course world size).  Go very small for numbers of things like number of civilizations, megabeasts, etc., make sure to cull unimportant historical figures, set variations for rainfall, temperature and so on very low (or even zero?), turn erosion cycle count down to zero, and set subregion max to 5000.  With two year history physical maps should generate pretty quickly. 

Once you find max size of physical map which generates reliably start adding to variations or increasing numbers incrementally until shit breaks or you end up with what you want.  When shit breaks make a choice about how important having certain features is versus having the super large map.  (This was my point about gameplay, not that it would be impossible - although pushing far enough that could happen - but that the quality of life of the game decreases as it slows down, and making the map larger contributes to that slowdown.  Some people won't play under 50fps, others are happy with 5fps, but at 0-1fps weird stuff starts to happen and forts become pretty much unplayable rather than just a pain to play.)

Looking forward to hearing how far you were able to push it though. 

[TranceProgHouse]

Sounds good all, I'll set aside a chunk of time at some point to start looking into Lua_API.  From a brief review I think I see how it interacts with on screen elements to an extent, but I'll need to mess around with it to figure it out for sure.  I definitely saw x and y coords in some of the Git pages under world code, but that could be a lot of things.

Decreasing variance makes sense to me, I saw in some generation attempts a map similar to what Fleeting Frames showed happening in the early stages where land features were generated in stratified layers horizontally (as compared to the diagonal layers you see in the map linked). The failures are definitely happening during the erosion cycles, so I'll see what can be done once I figure out a way to avoid doing the manual memory editing, because that is just a nightmare.  The number of times I am going to need to attempt to generate world maps is just too many for it to be worth going through that memory edit process, which is easily 5 to 10 times longer than the actual number changes and running the generation, assuming the generation proceeds of course - but I am much more ok with it hanging waiting for something to happen than slowly and tediously trying to get there.

Being busy with work and wanting to play the actual game sort of comes first, haha, but this is definitely something I'd like to mess with.

[Fleeting Frames]

To clarify, I told the world to be all triangles, and added the mountain to the 257x257 world only when that failed to generate. Merging some triangles instead also generated, so I think it might be linked to some pop numbers or something exceeding allowed numbers.

In your case, I'd just note down :lua <datastructure> = valuetochange to for each value so you can just copypaste what you need to change without going into gui/gm-editor. Here's a reply I provided to another user in PM when they asked for lua scripting.

Spoiler (click to show/hide)

Yes.

Here's a guide on lua's homepage: http://www.lua.org/pil/contents.html - don't need to learn everything before starting

Here's a wiki: http://lua-users.org/wiki/ - it has not just base thing explanations but also code to copy

Here's dfhack's lua api, which documents few general things but mostly dfhack's modules: https://dfhack.readthedocs.io/en/stable/docs/Lua%20API.html

I also learned by looking at existing lua scripts and modifying them. It's important to note that not every lua script is in base dfhack download - I myself have about five dozen extra lua script files.

Furthermore, you don't always need a file; can run lua commands in dfhack with :lua for singleline command, or lua to enter lua interpreter. Many times, my commands start with :lua ~ to print out what comes after ~.


You need to know how to access a given piece of df data. For this, gui/gm-editior is also immensely useful tool; there's been recently graphical variant in RAW explorer http://www.bay12forums.com/smf/index.php?topic=174239.0

By default, gui/gm-editor runs on whatever you have selected. Few useful aliases:

gui/gm-editor df.global - access DF's global variables such as world, current tick and so on

gui/gm-editor unit/building/item - access the selected thing with dfhack.gui.getSelectedUnit/Building/Item().

gui/gm-editor <lua command> - runs the lua command, and accesses the result with gui/gm-editor. A comman example of mine would be gui/gm-editor dfhack.maps.getTileblock(pos2xyz(df.global.cursor))


i.e. a practical example: you want to set an unit to be thirsty with a hotkey. From wiki, this means they need to have waited 25k ticks, but how do you set it?

1. gui/gm-editor unit

write down :lua dfhack.gui.getSelectedUnit()

2. look where the thirst is stored.

ah, there's thirst timer in counters2.

write down

:lua dfhack.gui.getSelectedUnit().counters2.thirst_timer

3. Now we want to set it to 25k

:lua dfhack.gui.getSelectedUnit().counters2.thirst_timer = 25000

4. It works, but what about keybinding? Well, dfhack.init has example keybindings. Lets say we want to use "t" key to make unit thirsty. Easy enough:

keybinding add T ":lua dfhack.gui.getSelectedUnit().counters2.thirst_timer = 25000"

5. But wait! This will run when designate trees for chopping and such. How about only doing it when viewing unit's wounds?

Welll, keybinding, run by itself, gives help screen and your current context (also obtainable by dfhack.gui.getFocusString()).

Code: [Select]

[DFHack]# keybinding
Usage:
  keybinding list <key>
  keybinding clear <key>[@context]...
  keybinding set <key>[@context] "cmdline" "cmdline"...
  keybinding add <key>[@context] "cmdline" "cmdline"...
Later adds, and earlier items within one command have priority.
Supported keys: [Ctrl-][Alt-][Shift-](A-Z, 0-9, F1-F12, or Enter).
Context may be used to limit the scope of the binding, by
requiring the current context to have a certain prefix.
Current UI context is: dwarfmode/ViewUnits/Some/Wounds

(Here you can also double-check that there's no bindings for T)

Final form would be then

keybinding add T@dwarfmode/ViewUnits/Some/Wounds ":lua dfhack.gui.getSelectedUnit().counters2.thirst_timer = 25000"

6. How do you use that? Well, you paste it into  console.

For running always on startup, you could place it into dfhack.init, onLoad.init, or onMapLoad.init

For running from lua sript, you'd wrap the command in dfhack.run_command("")

7. Finally, you'd always want to test it whether it works. No need to compile, you can hotswap-and-insert lua scripts into running DF session:

||

Good luck! If you want some of my examples, you can look at the threads I've posted in Utilities & 3rd party tools subforum, or in my wiki page.

[exdeath]

I made some tests at this thread.
http://www.bay12forums.com/smf/index.php?topic=159216.15

The way I discovered later to speed up the procress, is to search for the value at cheat engine while the map is generating it, then you quit the map generation, start to generate a map with another setting with different map size and search for this new value and then quit the map generation, you do this until the game find the values. This will allow you to edit the ram value that tell what is the map size values the map generation will be using, instead of changing the map size of the setting the map generator will pick to use, this allow you to quickly change between advanced generation settings without having to redo all the process again. And because you dont need to keep changing the advanced generation settings map size value, you wont need to redo all your settings again.

According to putnam this is possible to be easily done with DFhack, but I stopped doing those map size tests and so I never tried doing it with dfhack.



Talking about df map with earth size:

If you use walking speed and assume df tried to be sort of realistic, the df human speed is 1 tile per 9 seconds.
According to this site ( https://www.reference.com/world-view/average-human-walking-speed-8df1ec5d0858683e) walking speed is  3.1 mph, or 1.39 meters per second or 12.51 meters per 9 seconds.
So a tile could be considered 12.51, obviously if you measure tile by other means, things start to break apart.

Dividing earth area by previous found tile size gives (5.1007×10^8 km)/12.51 meters = 40770000000, so a  40770000000 * 40770000000 tiles (game tiles not world tiles) would be an earth map.
With the formula  (x*16*16*16*48=40770000000) you find that the amount world tiles would be 207367 * 207367. Converting it to the formula where amount of world tiles must be (2^x)+1, X must be integer and result must be an odd number, the result is a 262145 * 262145 map needed to have entire earth.
Also, the biggest default df map  is a 257 world, using the formula (2^x)+1=257, you find the value of X to be 8, if you appply the same formula to 262145, the value of X become 18, so assuming df add extra sizes 513, 1025[....] it would need to add 10 more extra bigger sizes to get the size of 262145 needed to have world size accordng to previous formula.

[TranceProgHouse]

Thanks for the thread link and info on DF Hack, I've set this idea aside for quite awhile and might pick it back up soon - though, if the world gen presets aren't mapped out, I'd have to use the method you are referring to.