Bay 12 Games Forum

Speeding up the processor cores in general will benefit DF, but I'm betting it's memory bound.  Almost everything is memory bound.  Many modern processor optimizations are finding ways to do useful work while waiting for the memory system to return data you're asking for.  As to how DF could be reworked to improve this... heaven alone knows.  Cache coherency is going to be a problem, and there's probably a point where it goes kablooie when one of the game's many internal memory tables gets too large to fit nicely into the processor cache.

Yeah, there are several levels of cache, and an application can potentially be bound by any one of them, or the speed of the memory itself (and the motherboard chipset, which needs to support the high speed memory). I suspect DF is bound by the latter. It depends on the locality of sequential memory access; if DF tends to jump all over it's memory, or access blocks at a time.

But I do think that there would be some benefit from multithreading, despite the heavy memory access time dependence. Enough to justify considering making it multithreaded as a performance priority. 

Not so good a solution without prep work, in my opinion.

• If you have a 64-bit OS with only 4 GB including virtual memory, and DF wants 4.1 GB (well way before that because of the OS, but by way of example...) if you have Crash-To-Desktop (CTD) now, you'll have it then too. Memory handling so as to avoid CTD is required, or a 64-bit release would be agony *anyway*.

I'm not aware of any 64 bit system that so strangely limits virtual memory to 4GB. Switching to 64 bit will mean the CTD will occur much later. 8TB later.

• Toady either has to abandon the 32-bit 'flavor', or support *two* 'flavors' on each platform (Linux, Windows). It would be best to have him focus on making DF better, since simply fixing CTD would allow you to nearly double memory usage ability anyway and stay 32-bit

I agree that supporting multiple versions does open up complications. But it's likely that the program would not need to be very different when compiled in 64 bit instead of 32 bit mode. It may even be as trivial as recompiling it under different setting.

• Multi-threading would be more useful than a 64-bit compile, in my opinion. Right now, DF is locked to a single core. If DF could properly handle the nightmare of forking multiple threads, my OS may use up to 8 CPU cores. You can have display and interface on one core so even if your framerate drops below 5 to 1 frame per second, you can still have 50 fps graphics display and have the game respond. How fast you choose to display would not need to impact how fast the 'do stuff' frame process runs, as it would be on a different core.

I agree but proper multi-threading is much harder to do than a 64 bit compile. It requires changes in the logical flow of the program, and in managing memory access. I expect a 64 bit compiler sooner than multi-threading support, because a 64 bit compile is much easier to do.

• I've been thinking of how DF could be bound to a database such as MySQL instead of all internal memory. Suppose you no longer had to choose how much embark space... the size of the world would be how much space you permitted that world to occupy on your hard drive. How large your fort was within that would would be a function of 'how much land you can grab AND HOLD. No more 'embark size' issues. You may find having a smaller size fort is easier to manage while keeping the frame-rate where you tolerate it. Not to mention your ability to deal with HFS, sieges, and so on. I've not quite fleshed out the notion, but breaking DF out of memory to a disk based database stored on the drive might take DF up to the next level the same way breaking DF out of 2D took DF up a notch. I'm still fleshing out ideas for a proposal to suggest on that one.

You post this after you said that Toady should be focusing on multithreading? It's a sound idea, but it's harder than a 64 bit compile. Perhaps not as hard as hard as multi-threading. I'm also not sure that the fps bottleneck is CPU bound. If it's memory access time bound, then switching to a database would make the problem worse.

I don't think a 64-bit compile is a 'quick/ easy' thing to do. I would not mind having a 64-bit compile, I can run it. But a 64-bit compile that still CTDs, or takes away from the Toady One's focus and passion for DF would be undesirable, in my opinion.

I don't know about quick and easy, but it's quicker and easier than any alternatives. CTD is not a problem if it only occurs at 8TB.

Quote from: King Mir on December 06, 2011, 04:31:31 pm

2GB is an OS limit on single 32bit applications. 64bit apps have a larger limit.

Never heard of that 2GB limit, before.  I had to read up on it.  I know many BIOs set a physical limit on RAM that a controller cannot exceed, but that's different with ea chipset.   I did hear that Vista accessed RAM in a more efficient manner, it was one of its selling points.  But I can't remember what that was called.  Here is a nice link to limits.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx

DF accesses no virtual memory?  Bummer.  *sigh*  I guess there is a good reason for it, although I can't understand why.  Most RTS games access virtual memory, as a result of there continously expanding design needs, demand for it.

Knutor

The link is correct, but you're misunderstanding it. DF does use virtual memory as needed (in fact it doesn't have much choice to). But the OS limits the (virtual) address space for a single 32 bit app to 2GB. Strictly a 32 bit address in x86 can point to a larger address space, which is why a 32 bit OS can benefit from 4GB RAM.

thisisjimmy is right, but the clincher is the OS limit, not the pointer limit, since the OS limit is smaller. But the reason for the OS limit has to do with the address space limits.

2GB is an OS limit on single 32bit applications. 64bit apps have a larger limit.

Did you guys like totally not even read what I just said?

Anyway, tile swaping IS a dumb idea. One tile can have 7 units of ANY kind of liquid at once. Dont you understand. This means that the tiles dont have to "swap", the water and oil can just mix on a tile basis. They wont create a new liquid but they will mix in one tile. the 7/7 water will go over the 1/7 tiles of oil and the oil will be on the top.

I was responding to harborpirate alternative of having mixing work like a block level cellular automaton.

Quote from: King Mir on November 22, 2011, 04:44:37 pm

Given how pressure works in DF, it wouldn't be easy to make water push other liquids. So you may as well do it right and layer them.

The effect of pushing occurs as a natural outcome of the scenario I described. Nothing special has to occur other than the one look behind/attempt to combine step.

I'm not so sure that layering alleviates all the problems of pressure calculations in the presence of multiple liquids. You'd at least have to calculate the sum of the total depth stored in the new data structure prior to making every pressure calculation.

Its really hard to say just how painful each of these options would be without seeing the details of the code; and we know that isn't going to happen.

That said, I have to agree that the biggest issue with tile swapping is the one you mentioned a couple of posts ago, where depth doesn't match and tiles can't be combined; like a 1/7 oil tile moving around a reservoir full of 7/7 water. It would just look strange.

But how would it know to attempt to combine? A simple swap would involve only two tiles, the 7/7 and the 1/7. That's simple. Checking that there are additional 1/7 tiles around so that the liquids could balance out at 4/7+3/7+2/7 would not be as trivial.

And we can estimate how hard it is to code something. We don't know about the state of Toady's code, but we can estimate how hard it would be to code from scratch. Also, we know how water currently behaves. It's unlikely Toady coded it is a strange way, so we can guess at the current code pretty well too.

Re: layering
I agree that that's the right way to do it, but that does add it's own mixing challenges. For instance: if shallow water and oil mix then the two can just spread out independently. However, if you have two adjacent tiles with no air gap you would still want the oil to spread out.

Quote from: King Mir on November 21, 2011, 05:56:29 pm

Tri-color tiles are impossible[...]

I just have to say: Currently.

Not that I'm particularly calling for it, but it might represent a solution to all kinds of things.

It's not a matter of dwarf fortress. There isn't any software library that's able to display multicolored letters without graphics for letters. Anywhere.

A Tri-color font would require a someone to do a major rework of how fonts work, and provide terminal support for it on every popular platform. No one's going to bother with that. Ever. It's useless. People who need multicolored font use graphics.

Given how pressure works in DF, it wouldn't be easy to make water push other liquids. So you may as well do it right and layer them.