Bay 12 Games Forum

Yeah. I can't really guess how the data is managed, and that aspect is still way out of anything I really know anything about. Other than CTD. I don't like CTDs. I know that much.

One change I thought of would be to have a separate 'display' fork from the running code. Simply turn the 'do stuff' DF process into a server that puts state changes into a queue along with reading from a command queue (things like 'build' or 'designate' that force a halt to the running 'do stuff' thread).

Then you would have a separate thread who's mission is simply to display state changes, interpret 'change display' requests (level up/ down, move display in a compass direction, alter display size). Even if your 'do stuff' thread had a frame-rate of 10 FPS, your display thread (running on its own core) would cheerfully display at 50 FPS, and if you gave it a 'build/ designate/ pause' command, it would receive the command right away and be able to throw a HALT signal to the 'do stuff' process. The processes could run separate cores and with totally separate memory. They would have a common I/O queue to communicate state changes and command. Both processes would still have to be aware of what memory they have to work with, so as to not CTD on the memory boundary.

Just ideas. I really don't know what the hell I'm talking about. Other than... it bites hitting a CTD on 2 GB of memory on a 9 GB machine with plenty of open real memory space.

You are correct. I dunno. I think I'll just keep enjoying the game as best I know I can and let other people sort it out. I'll just go on record as thinking that CTD's by memory ought to go away, one way or another.

It sounds as if the only concern with a 16 x 16 embark, were it possible, is FPS death. Ever faster hardware will keep pushing FPS death back, as well as playing so as to keep the mobile count down. FPS death 'of a sort' is also a problem for Civilization, including Civ V: you have to wait for the other Civs to 'take their turn'. So at some point you decide to cut back on the number of Civs and size of your world to make playing the game tolerable.

But if you have a $2000 'game machine' or a cheap used Sun Blackbox emulating Windows, it ain't braggin' if you can swing it.

Civ V does not bother attempting to stop you from playing 'max size world + max size civs'. It certainly doesn't CTD when you attempt to do so. It simply has you wait while the many other Civs are plotting your death behind your back. Some sort of 'test mode' in DF wouldn't hurt.

"Run System Test" could be a new menu option. The system test returns a recommended 'max setting' and offers to write that max setting into your system's config file. The system test simply has a stock fortress and it spends a minute running that fortress from a known start point. A group of dwarves are committed to battle, foes are ready to destroy them. Other dwarves have tasks assigned to them, and the test runs for, say, 500 frames and then ends. The system can then figure out a 'score' for your system, and based on that score suggest you can run a 16 x 16 embark (after ensuring you have the memory to support it, and DF won't just CTD on you) or it may suggest you are pushing it if you do more than a 4 x 4 embark.

That would be a means to provide the game player a tool to evaluate how best to plan their DF gameplay experience without the CTD being a part of that.

Who would want to have a 16 x 16 embark? Well... if I have the hardware and the memory to support it, why not? Well, FPS death. Yeah? Well... what tools do I have to evaluate how big I can make it without FPS death in an established fortress? Crash-To-Desktop. Oh yeah, if you get a CTD, your fortress is 'likely' too big. K'thanks.

CTD is NOT a reasonable means to suggest you are biting off more than you can chew on an embark. A 'standardized test run process returning useful info' would be far more useful than CTD. Hrm... 'standardized test run package' would be another suggestion I ought to post...

There is really only one reasonable solution to the problem: release a 64 bit version.

Not so good a solution without prep work, in my opinion.

• If you have a 64-bit OS with only 4 GB including virtual memory, and DF wants 4.1 GB (well way before that because of the OS, but by way of example...) if you have Crash-To-Desktop (CTD) now, you'll have it then too. Memory handling so as to avoid CTD is required, or a 64-bit release would be agony *anyway*.
• Toady either has to abandon the 32-bit 'flavor', or support *two* 'flavors' on each platform (Linux, Windows). It would be best to have him focus on making DF better, since simply fixing CTD would allow you to nearly double memory usage ability anyway and stay 32-bit
• Multi-threading would be more useful than a 64-bit compile, in my opinion. Right now, DF is locked to a single core. If DF could properly handle the nightmare of forking multiple threads, my OS may use up to 8 CPU cores. You can have display and interface on one core so even if your framerate drops below 5 to 1 frame per second, you can still have 50 fps graphics display and have the game respond. How fast you choose to display would not need to impact how fast the 'do stuff' frame process runs, as it would be on a different core.
• I've been thinking of how DF could be bound to a database such as MySQL instead of all internal memory. Suppose you no longer had to choose how much embark space... the size of the world would be how much space you permitted that world to occupy on your hard drive. How large your fort was within that would would be a function of 'how much land you can grab AND HOLD. No more 'embark size' issues. You may find having a smaller size fort is easier to manage while keeping the frame-rate where you tolerate it. Not to mention your ability to deal with HFS, sieges, and so on. I've not quite fleshed out the notion, but breaking DF out of memory to a disk based database stored on the drive might take DF up to the next level the same way breaking DF out of 2D took DF up a notch. I'm still fleshing out ideas for a proposal to suggest on that one.

I don't think a 64-bit compile is a 'quick/ easy' thing to do. I would not mind having a 64-bit compile, I can run it. But a 64-bit compile that still CTDs, or takes away from the Toady One's focus and passion for DF would be undesirable, in my opinion.

As far as the 2 GB limit for DF, that is from http://www.bay12games.com/dwarves/mantisbt/view.php?id=136 but I initially noticed that the crash-to-desktop would happen around 1.9 GB as reported by TaskManager. I happened to have it up at the time. The only real way to get a good look at a potential site is to embark, then kill the DF process. According to the link above, that's a 32-bit 'feature' of the compiler.

Commercial games tend to recognize their memory limits. Even if they are 32-bit games, there are ways to 'cheat', but CTD's are a huge no-no and much effort is put into quashing those. Even if DF were released as a 64-bit compile, memory allocation still has to be aware of what limits it has to stay within them and avoid CTD's.

Having DFHack & StoneSense wedged into the same memory space outside of DF's control is *NOT* going to help matters. Toady really has zero control over StoneSense busting the memory limit through the SDL.DLL hack. The long-term roadmap needs a better plan.

I'll respond privately about the system spec stuff, at least as best as I can scare it up from the info I have. It's around somewhere, I just don't keep my spec sheet handy.

Working on a real life DF beard. DAMNIT I meant to put away DF 2010 for a while, at least for the next DF release and maybe a fix to the CTD on memory touching ~ 2 GB memory. Can't get the game out of my skull... I'll have to set DF up to play again. Damnit.

http://reviews.cnet.com/4504-4_7-0.html?id=33703094&tag=compare might be my machine, or at least in the same line. Same case anyway. Note 9 GB (installed) / 24 GB (max) - DDR3 SDRAM on the web page. But I'd rather not have system spec issues in this thread cloud the larger issue I intended to address:

In 10 years if you don't have 64 GB on your machine, it isn't a serious game machine, and people will be able to play DF on their smartphones. Will DF *still* not be able to tell if a machine has only 1 GB of memory and safely limit memory requests to fit within that 1 GB without CTD? If DF *STILL* has to fit within just shy of 2 GB, will it be able to do so without CTD? When people wish to play DF on their 64 GB game machine in 10 years, why not allow DF to take advantage of it?

As a quick exercise, make a 256 x 256 world using a 16 by 16 font. That creates a 65536 by 65536 map. Now enlarge that by the 48 x 48 of each embark tile, and you have a 3145728 by 3145728 image. Drop your 8 x 8 embark map on the appropriate spot on that image, and you can get an idea of the amount of rich detail the world *potentially* has on just one Z level. That's a LOT of data, and it blew my mind when I first went through the trouble of overlaying my embark map on top of a proper scaled world map image.

And my above data may be wrong, and AGAIN I please don't let's get lost on the details of overlaying an embark image on top of a world image. Let's just say Toady has done a fantastic job of putting in very rich *POTENTIAL* detail within DF, and he managed to cram all that within a program that must fit within less than 2 GB.

It STILL CTD's if it touches that 2 GB limit. The CTD must die. Breaking DF out of the 2 GB limit is even better. Suppose DF supported the larger memory, and suppose it dropped your 8 x 8 embark in the middle of a 16 x 16 'embark' area it kept tract of? Suppose you could see armies form outside your 8 x 8 embark area? Suppose you had neighbors to negotiate with just on the other side of your embark area? What if you dug outside your embark area *and got caught*, in which case you could start a war with your neighbors? Again, I don't want to get side-tracked with the merits of such proposals. Without fixing memory so 'no more CTDs on hitting the 2 GB limit', nothing like that can seriously even be on the radar.

I'm just saying, DF has a HUGE amount of very rich textured detail that it packs away inside that 2 GB of single-core running memory. Once memory handling is fixed, Toady could examine things like having a separate core thread continue running 'legend' history so events continue, including battles and such, and having that outside world occasionally spill into the working fortress. But none of those ideas fly if the CTD isn't fixed and memory can't break beyond the < 2 GB.