Bay 12 Games Forum

Dwarf Fortress => DF Modding => Topic started by: Anticheese on February 16, 2012, 02:25:11 am

Title: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Anticheese on February 16, 2012, 02:25:11 am
With the new version of Dwarf Fortress adding even more depth to the world, it makes sense to take every advantage you can to help the game run even smoother.

To this end, we've activated the LAA - Large Address Aware - flag for Dwarf Fortress, which allows 32-bit systems to use up to 3GB of RAM, and 64-bit systems to use up to 4GB. You may make Skyrim jokes.

You can find the file here (http://dffd.wimbli.com/file.php?id=5528)

You can find the original thread at http://forums.nothingtolost.com/dicestation/viewtopic.php?f=6&t=62 (http://forums.nothingtolost.com/dicestation/viewtopic.php?f=6&t=62). A link is included to the utility that was used to produce this mod.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Girlinhat on February 16, 2012, 02:27:32 am
I'm not the most intimate with computer workings, but isn't the main issue with DF processing speed, not really the memory size?  Does this change have any worthwhile effect on speed?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Plasson on February 16, 2012, 03:17:56 am
the process uses ~850k with the regular, and this here SDL exe.  don't seem to have any effect.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Llamageddon on February 16, 2012, 08:20:09 am
I'm not the most intimate with computer workings, but isn't the main issue with DF processing speed, not really the memory size?  Does this change have any worthwhile effect on speed?

I believe many people are having trouble with worldgen crashing on large worlds or particularly long histories as they RAM usage of the program is exceeding 2GB. This is the default upper limit for a normal windows program I think, hence the crash. The file in the OP flags the EXE so windows "knows"(?) the program can use 4GB of RAM (3GB in XP) and won't crash when the normal upper limit of 2GB is reached. I'm pretty certain it should have practically no impact on the actual speed of worldgen as that is pretty much exclusively tied to your CPU.

Skyrim had a similar problem when people started installing loads of custom high texture packs etc, hence the Skyrim reference in the OP. My knowledge of the specifics is a bit ropey too so take the above explanation with a pinch of salt. I do find it hilarious that the new DF with it's super high fidelity graphics  ;) is having the same problem as Skyrim though. All hail Toady, Bethesda needed mods added to achieve this.

Edit: Kind of missed the point there didn't I. I don't want to contradict the author but I kind of agree that Dwarf Fortress game speed is not very RAM dependent as far as I know. I assume it should help with large embarks/adventure mode area transitions and things though, basically anything where the game is loading lots of information to and from memory rather than simulating tantrums/combat etc.

Edit 2: Hope I haven't made a fool of myself sounding like an expert when I'm not. Anyway, just wanted to say thank you very much for posting this Anticheese, it is just what I was looking for.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: EmperorJon on February 16, 2012, 08:34:27 am
World gen will probably be effected, indeed.

I've said this many times, and I know Toady doesn't particularly agree, but it'd be perfectly happy to see him take a year or two out to rebuild the game multithreaded... :p
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Girlinhat on February 16, 2012, 08:52:00 am
So, to clarify, this change will help curb some woldgen crashes for very large, old worlds?  That's much more relevant and probably should have been made very clear.  As it stands, the actual purpose of this project wasn't made clear, only that it had been done.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: MagmaSolutionsInc on February 16, 2012, 05:25:37 pm
Much love here - thx  :D
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Slax on February 16, 2012, 05:37:54 pm
I do this for just about every game these days. Generally just speeds things up, it feels like. Placebo or not. :P
Quick and easy. (http://www.techpowerup.com/forums/showthread.php?t=112556)
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Rafal99 on February 16, 2012, 06:06:15 pm
Awesome, thanks!
I wasn't aware such thing is possible.

Edit: I just compared the exes and noticed they only differ at one byte. This is the most awesome single byte I have seen.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Maklak on February 16, 2012, 06:15:11 pm
So, to clarify, this change will help curb some woldgen crashes for very large, old worlds?  That's much more relevant and probably should have been made very clear.  As it stands, the actual purpose of this project wasn't made clear, only that it had been done.

But there was an answer: "It has been asked whether this will speed up DF, the answer is generally no. The point of this is to prevent the memory from reaching crash point during worldgen or late game when there is so much crap for the game to remember."
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: bombzero on February 16, 2012, 07:14:08 pm
World gen will probably be effected, indeed.

I've said this many times, and I know Toady doesn't particularly agree, but it'd be perfectly happy to see him take a year or two out to rebuild the game multithreaded... :p

how would you multi-thread DF? the reason modern games CAN be multi-threaded is that there are fewer things that need to be placed on the different cores, all of which are not too interdependent.
in DF however almost every single process is interdependent and thus fore is hard to separate in separate cores.

not trying to be an ass, just explaining the technical implications of what you think.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Jay on February 16, 2012, 08:11:32 pm
Quite.  From a technical standpoint, there are no two actions that can run without interacting with each other, other than the game itself / the graphics, which are already on their own threads.
Thread collision (when multiple threads that can potentially interact with each other inevitably modify the same thing at the same time -- like kids who don't know how to share a toy) rectification creates a much larger problem than the multithreading would solve.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: telamon on February 16, 2012, 08:17:40 pm
thanks for this, DF kept crashing whenever i did world gens with more than 300 years of history. how much RAM should i be expecting it to use though? i'm doing this on a laptop which tops out at about 2GB. probably not enough for that much history, eh?

EDIT: and would it be possible to just cache the world's history to hard drive as it generates? ie don't hold it all in RAM until it's done, write it piece by piece.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: jwiechers on February 16, 2012, 08:26:49 pm
Quite.  From a technical standpoint, there are no two actions that can run without interacting with each other, other than the game itself / the graphics, which are already on their own threads.
Thread collision (when multiple threads that can potentially interact with each other inevitably modify the same thing at the same time -- like kids who don't know how to share a toy) rectification creates a much larger problem than the multithreading would solve.

Nah, interdependence is indeed a major issue with multithreading, but it isn't as bad as you suggest, it just requires careful handling and checking, which is one of the reasons why rewriting something to be *efficiently* multithreadable is next to impossible, but newer software can generally be made to utilize cores very effectively even with huge interdependencies.

As for Large Address Aware: Good call, I routinely do that with Dwarf Fortress and other memory hogging software, too. Helps quite a bit and prevents crashes up to 12x12 embarks for me, sometimes even slightly larger ones. Also allows 2000+ years of history in large worlds.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Eagleon on February 16, 2012, 09:44:08 pm
I've needed this for a while. Unfortunately my main computer has a paltry 2 gb (albeit on a streamlined lighter version of XP, so for most applications it's no issue), but using my partner's computer I successfully genned a world with maxed out vamp/werebeast/etc. to over 100000 211907(edit) historical figures(!) and no limit on sites or population after worldgen, and have successfully loaded it and saved in adventure mode with no issue on my own computer. It had 3 ages of legends. Should be interesting to explore.

I'm surprised this hasn't been a bigger issue. It's been very easy for me to run into this limit during worldgen, particularly with some of the larger mods and the crazier settings I've put into worldgen. Still haven't tried bumping up megabeast percentages for that 'epic struggle' feel, but it only reached about 2.7 gb in memory during worldgen as far as I saw so it shouldn't be an issue. I'd love for Toady to make DF handle overload conditions a bit better, even if it resorted to using virtual memory sometimes. Anything's better than crashing after an extended worldgen.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Maklak on February 17, 2012, 04:32:06 am
DF could be partly multithreaded with fork-join or task pool methods. Each frame is does some calculations: temperature, weather, fluid movements, pathfinding, creature movements, and so on. Some of these can be partitioned, done in parallel, and finished before moving to the next phase. I cannot tell anything about effort / speedup tough. When I played with multithreaded matrix computations the speedup I got was small.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Kalithor123 on February 17, 2012, 07:42:26 am
I tested a similiar program some time ago:


http://www.bay12forums.com/smf/index.php?topic=88414.msg2420067#msg2420067


And there was a patch program for fallout as well.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: dirty foot on February 19, 2012, 02:06:23 am
I am not experiencing any issues with RAM during world gen, as previously predicted. The issue seems entirely processor-related. At any given time during world gen, I am max on one of my cores.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Jay on February 19, 2012, 11:26:41 am
I am not experiencing any issues with RAM during world gen, as previously predicted. The issue seems entirely processor-related. At any given time during world gen, I am max on one of my cores.
That's not what the problem with RAM is.

Everything has to get stored in the RAM.  If you have a large world or a long history, especially with the new information the .34 releases store, you can run into the 2 GB virtual limit quite easily, which crashes you out.  Hence, the large address aware flag to allow it to use all of the addresses its 32-bit executable can; 4 GB.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Moogie on February 20, 2012, 08:14:33 pm
I was experiencing reliably worldgen crashes on worlds with uncapped civ/sites at as little as 150 years of history. RAM usage was hitting the 2GB limit on my 8GB, x64 system.

I patched my exe using the ever-trusty ntcore.com memory patcher, and now worldgen successfully completes. It used about 2.1GB to finally get through Finalizing Sites without pooping itself.

For anyone suffering worldgen crashes, this is the answer you may be looking for. Patch your exe, and enjoy!
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Inarius on May 24, 2012, 10:27:43 am
Does this version works with the 0.34.10 ?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Jeoshua on May 24, 2012, 11:05:22 am
No, but the methodology will work to create one for 34.10.  I've done it, myself, and it works fine.

No, I won't upload it .  It's that simple of a hack.

Go download CFF Explorer and search google for "Enable LAA" for instructions.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: BradUffner on May 24, 2012, 10:22:55 pm
DF could be partly multithreaded with fork-join or task pool methods. Each frame is does some calculations: temperature, weather, fluid movements, pathfinding, creature movements, and so on. Some of these can be partitioned, done in parallel, and finished before moving to the next phase. I cannot tell anything about effort / speedup tough. When I played with multithreaded matrix computations the speedup I got was small.

I think one of the best gains would be to split the computations of things like temperature in to map quadrants (or halves, or whatever).  Basically the map gets split in to the same number of pieces as the number of cores, and each core updates it's "slice". Then there is a quick reconciliation along the seams to join them together.  Each core is working on different parts of the same task instead of completely separate tasks.  That's one of the things that miltithreading would shine at.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Argelle on June 16, 2012, 10:18:14 am
Will the LAA be of help on win 7 32 bits system with 4 Gb ? In fortress mode ?
Inside are 2 x 2 Gb RAM
then the task manager annonce 3325 Mo total physical memory
1115 Mo cache*
available* 2279 Mo
free* 1174 Mo
I know DF to go up to 1.5 Gb, so non use at all ?

Info : after DF, sound sense, therapist & ssense kicked in:
926 Mo cache
available 929 Mo
free 3 Mo

* how these 3 add up I can figure it out, but 3325 versus 4000 is due to the graphic card memory I think.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Tierre on June 18, 2012, 07:39:03 am
So the question - LAA helps 32bit win XP or win 7. Will it help x64 versions of OS?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: King Mir on June 18, 2012, 09:19:47 am
It won't help if you're not actually running into the 2GB limit. If you are, then it will help on all systems.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Tierre on June 18, 2012, 09:51:41 am
Well my DF actually crashes on saving:( But saves are made before crash, thanks god:) But i've seen it crash 4 times in the last month while saving - i guess it is then it runs into 2gb limit.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: King Mir on June 18, 2012, 11:07:15 am
There's no real penalty for applying the patch. If it fixes the problem, great. If it doesn't, you can now have larger embarks, which is good too.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: vault13dweller on January 06, 2013, 06:49:17 pm
Any chance this will be updated to work with the latest version of Dwarf Fortress that Masterwork runs on? (34.11)
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Putnam on January 06, 2013, 06:57:38 pm
It doesn't need to be.

Also, Masterwork very rarely does not run on the absolute latest version of DF. Only when the newest version is, well, new--very new, I might add. Mods are usually very easy to update. .28 to .31 was probably the biggest change for modders.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: CaptainArchmage on January 06, 2013, 07:49:50 pm
DF could be partly multithreaded with fork-join or task pool methods. Each frame is does some calculations: temperature, weather, fluid movements, pathfinding, creature movements, and so on. Some of these can be partitioned, done in parallel, and finished before moving to the next phase. I cannot tell anything about effort / speedup tough. When I played with multithreaded matrix computations the speedup I got was small.

I've noticed that the Dwarf Fortress decompilation thing that's going on has made a considerable amount of progress. I'm wondering whether it might be possible to produce a hack to allow multithreading to at least a small degree? I am aware the amount of effort may be just too big for people to handle.

Same question goes for 64bit.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: vault13dweller on January 06, 2013, 11:08:57 pm
But when I use either of the Large Address Aware .exe's with Masterwork Dwarf Fortress, the game says that the index file is missing even though it's clearly there.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Putnam on January 06, 2013, 11:42:59 pm
Which index file?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: vault13dweller on January 07, 2013, 12:57:47 am
Which index file?

The "Main Index File"

http://i49.tinypic.com/20tm8ag.png
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Putnam on January 07, 2013, 01:08:27 am
Give a screenshot of the entire folder dwarf fortress's EXE is in.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Telgin on January 07, 2013, 01:37:16 am
DF could be partly multithreaded with fork-join or task pool methods. Each frame is does some calculations: temperature, weather, fluid movements, pathfinding, creature movements, and so on. Some of these can be partitioned, done in parallel, and finished before moving to the next phase. I cannot tell anything about effort / speedup tough. When I played with multithreaded matrix computations the speedup I got was small.

I've noticed that the Dwarf Fortress decompilation thing that's going on has made a considerable amount of progress. I'm wondering whether it might be possible to produce a hack to allow multithreading to at least a small degree? I am aware the amount of effort may be just too big for people to handle.

Same question goes for 64bit.

64-bit might sort of be doable, but would still be an enormous undertaking.  Theoretically you could disassemble DF and replace any memory accessing instructions to use 64-bit addressing instead of 32-bit, and modify it so that pointer calculations use 8-byte pointers instead of 4-byte.  Then you'd need to modify any allocations to use 8-byte pointers, and any code that accessed any data structures would need to account for increased pointer sizes...  okay, probably not practical at all.  Unless you were able to decompile DF into higher level code that could recompile it in 64-bit, but you'd probably run into big problems with any libraries which I imagine is why Toady hasn't done this already.

Multithreading would probably be out of the question even if you could decompile it.  It's almost certainly doable, but by the time you figured out how to get it working with any speedup the next version of DF would likely be on the horizon and invalidate all of your work.  Race conditions are not a thing to be trifled with when debugging multithreaded code, and that's with proper, not decompiled, code.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Ravendarksky on August 13, 2013, 04:54:04 am
Any plans to update this for the latest version/the new version when it comes out?

Also if there is a copy of this posted somewhere else for the latest version I couldn't find it :(
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Lucelle on August 13, 2013, 11:38:15 am
Would there be any reason to do this if you have 1GB RAM but 5GB virtual memory?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Putnam on August 13, 2013, 01:02:20 pm
Would there be any reason to do this if you have 1GB RAM but 5GB virtual memory?

No.

Also

Quote
1GB RAM
holy shit
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: narhiril on August 14, 2013, 03:42:16 pm
Spoiler (click to show/hide)
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: whitehorn799 on August 09, 2014, 11:38:57 pm
Is there any chance to get this updated for .40?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Putnam on August 09, 2014, 11:45:19 pm
Shouldn't have to be, I'm pretty sure.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: scamtank on August 10, 2014, 02:04:07 am
Yeah, it's not like this whole LLA hooplah was invented for the pleasure of Dwarf Fortress dorks. It's a switch you flip in the executable to make it do a thing different and it's so tricky to do manually that someone saw fit to write a small app to streamline the whole thing.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: whitehorn799 on August 10, 2014, 06:02:08 am
Thanks for the fast reply.

I tried to run this, but I got the same error as the poster a few posts back. Bad init file.

Here's a screen of my directory. https://www.dropbox.com/s/9sdjyh1qc5j05lo/error.jpg (https://www.dropbox.com/s/9sdjyh1qc5j05lo/error.jpg)

Any help would be appreciated. I have 16gbs of RAM, being able to use some more of it would be awesome.

edit: The "data" folder is at the top. I couldn't get it all to fit in one shot, so that's the only file/folder not displayed.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Telgin on August 10, 2014, 12:16:15 pm
I seem to recall Toady stating somewhere that an experimental 64-bit build was in the cards soon.  That was on the same level as finishing plants and trees and adding job priorities, so it hopefully wouldn't be a year or two away in a major release.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: i2amroy on August 11, 2014, 12:16:56 am
I seem to recall Toady stating somewhere that an experimental 64-bit build was in the cards soon.  That was on the same level as finishing plants and trees and adding job priorities, so it hopefully wouldn't be a year or two away in a major release.
Yeah, he mentioned it in the last monthly summary update IIRC.

Also I'd just like to point this out for anyone who is unaware, but this isn't needed on Mac or Linux versions of DF, since they have never suffered from the 2 GB limit that microsoft imposed long ago, and as such will use more than 2 GB just fine with the normal version.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: whitehorn799 on August 11, 2014, 02:47:00 am
Thank you all for the information. I appreciate all the help!
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Maklak on August 12, 2014, 12:09:27 pm
IIRC, Toady One said that 64 bit version is one of a list of things he considers doing soon, not that he is doing it now or even that he is committed to doing it.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: LordBaal on December 23, 2014, 02:36:16 pm
Does this still works with the current version?
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: milo christiansen on December 23, 2014, 02:55:40 pm
This will (AFAIK) work with ANY 32 bit Windows program, so yes.
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: nuget102 on December 23, 2014, 08:43:33 pm
If anyone is having trouble with the download (that is now super old) you guys can try this instead: http://www.techpowerup.com/forums/threads/large-address-aware.112556/
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: Korgoth on August 18, 2019, 06:44:57 pm
hello! thnak you for your effort on this! do you know how can I use or of it is possible to use this on the Old genesis mod made by Deon and continued by TomiTapio?
Because a just dropped both files, the SDL and the Legacy exe on the folder, both in seperate ocasions, and they dont work :(
Title: Re: Dwarf Fortress - Large Address Aware (Or, let DF use up to 4GB of RAM!)
Post by: ArKFallen on August 31, 2019, 10:28:44 pm
hello! thnak you for your effort on this! do you know how can I use or of it is possible to use this on the Old genesis mod made by Deon and continued by TomiTapio?
Because a just dropped both files, the SDL and the Legacy exe on the folder, both in seperate ocasions, and they dont work :(
That download is for DF v34.04, OldGenesis is now in DF v44.12 which has major changes in the code.
I suggest you use the link from the post above yours
http://www.techpowerup.com/forums/threads/large-address-aware.112556/
to get the program that made the download in the first place. Grab the first or second attachment depending on whether your computer has .NET Framework v4.5(then grab the second) or not(get the first). Use the program as directed.

However successfully making DF use more RAM probably won't help with slowdown because most of it is caused by CPU limitations.