Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Ringmaster]

I've got a 1050 year old pocket world with very low everything apart from history. A grand total of 1000 hist. figs. with 929 dead. Alongside 45672 events.

Edit: This is a strange world.

Spoiler (click to show/hide)

[dirty foot]

Well, he is a math wizard*, so I could see him figuring out a few clever shortcuts. But there's only so much you can do. It could offload rarely-updated parts of history off to the hard drive, so long histories would work in theory, but they'd be even slower than they are now.

*Arithmancer, technically.

Actually, there is a lot you can do, but it's all a matter of how much data you are willing to sacrifice, and how clever you can be.

Compare a .bmp to a .jpg to a .png. 

All of these are image formats, and bitmaps just the raw data represented exactly and faithfully.  Jpgs, meanwhile, are willing to lose a decent amount of data in exchange for cutting the amount of actual data that needs to be stored before you can extrapolate out the data you want.  You lose some resolution, but you can get much more image in the same amount of space.  A png, however, is even more clever, and manages to be lossless with its data at the same time that it expands on the functionality of the original format.

PNGs only came about because many people spent a very long time thinking about how to refine the art of storing art, however.

If I were to give you a word problem to solve where I told you a train was traveling a 240 mile journey at 60 miles per hour, you would simply divide 240 by 60 and get the answer.  If a typical game is told to do the same thing, however, it would start a minute-by-minute pushing of a train object along the grid until it hit its destination, and return the value of how many iterations it had to push.

It's a matter of how you look at the math problem from the first place. 

The trick is to find ways to properly abstract your math problems into a higher level of math that the computer can still run.

Aside from "tricks" to use, is it not possible to program the game for 64-bit versions and use the increased RAM available to avoid having to cull history? I'm guessing it's not an easy thing to program though, unfortunately.

It's times like this that I wished I knew how to program.

[NW_Kohaku]

Aside from "tricks" to use, is it not possible to program the game for 64-bit versions and use the increased RAM available to avoid having to cull history? I'm guessing it's not an easy thing to program though, unfortunately.

It's times like this that I wished I knew how to program.

"Just throw more RAM at the problem" doesn't really help anything but the crashes from going over 2 gigs of RAM.

Besides, if you have geometric complexity growth, then doubling your RAM only lets you go one iteration further, anyway.

What you need to do is change the slope of the complexity curve.

You need to create more JPG-like (or better yet, PNG-like) means of handling information in general - this game tracks far too many items that are simple duplicates of each other that clutter up the gameworld.

If you have a warehouse full of limestone stones, then every single iteration of the game clock will have the game check each and every one of those stones for if their status has changed (such as if temperature has increased)... but you can look at that and tell immediately that the status of that warehouse has not changed in a long, long time, and there are no reactions for that limestone to undergo... but the game checks each and every individual item for a reaction every single game tick, anyway.

Wouldn't it be faster to just check that area for some sort of condition change, and then assume that if the items in that area hadn't been changing for the past some-odd frames, that you don't need to bother checking the individual items, and skip over the whole check? 

This sort of thing gets done with water already, at least, although it's somewhat buggy - water that fills up an area 7/7 and doesn't move will be considered "settled", and the game doesn't even check for movement on those tiles of water, except to occasionally check to see if someone has breached a wall holding that water back.

[dirty foot]

Aside from "tricks" to use, is it not possible to program the game for 64-bit versions and use the increased RAM available to avoid having to cull history? I'm guessing it's not an easy thing to program though, unfortunately.

It's times like this that I wished I knew how to program.

"Just throw more RAM at the problem" doesn't really help anything but the crashes from going over 2 gigs of RAM.

Besides, if you have geometric complexity growth, then doubling your RAM only lets you go one iteration further, anyway.

What you need to do is change the slope of the complexity curve.

You need to create more JPG-like (or better yet, PNG-like) means of handling information in general - this game tracks far too many items that are simple duplicates of each other that clutter up the gameworld.

If you have a warehouse full of limestone stones, then every single iteration of the game clock will have the game check each and every one of those stones for if their status has changed (such as if temperature has increased)... but you can look at that and tell immediately that the status of that warehouse has not changed in a long, long time, and there are no reactions for that limestone to undergo... but the game checks each and every individual item for a reaction every single game tick, anyway.

Wouldn't it be faster to just check that area for some sort of condition change, and then assume that if the items in that area hadn't been changing for the past some-odd frames, that you don't need to bother checking the individual items, and skip over the whole check? 

This sort of thing gets done with water already, at least, although it's somewhat buggy - water that fills up an area 7/7 and doesn't move will be considered "settled", and the game doesn't even check for movement on those tiles of water, except to occasionally check to see if someone has breached a wall holding that water back.

I understand that throwing more RAM at the problem is only a bandage, and no true treatment of the issue. However, this game is only going to get more complicated/in-depth, and I can tell (even without extensive programming knowledge) that there's a hard cap on how much memory can be saved with programming optimization.

I looked into it a little more since I last said something, and it does seem like an insane task to reprogram the game for 64-bit. That said, wouldn't it be more appropriate to start rewriting the code now instead of waiting until later when there's more content. As it stands, 64-bit looks like a significant step in memory usage (last I heard from a programmer friend of mine) in that it virtually eliminates the RAM cap usage for computers. I'd assume that this kind of reprogramming would likely never have to occur again beyond the shift to 64-bit systems. I mean, we're hitting a cap at 2-3-4 gigs of RAM, and things are crashing from that; assuming another 20 years of content addition, we're not going to get too far without an increase in the RAM cap. Trimming or culling historical data or "needless" status checks kind of robs the beauty of how in-depth the game is. Resisting the use of a MUCH higher RAM cap seems entirely against the very premise of Toady's vision, which to me looks like a need for totality and realism.

I know Toady wants to do things his own way and I respect that far more than I can ever convey in words, but logically, his stubborness will only limit his vision in the long-run. Honestly though, I'll love the game regardless.

[NW_Kohaku]

I mean, we're hitting a cap at 2-3-4 gigs of RAM, and things are crashing from that; assuming another 20 years of content addition, we're not going to get too far without an increase in the RAM cap. Trimming or culling historical data or "needless" status checks kind of robs the beauty of how in-depth the game is. Resisting the use of a MUCH higher RAM cap seems entirely against the very premise of Toady's vision, which to me looks like a need for totality and realism.

Actually, no, that's missing the point of what I was stating - if you find ways to store data that is oftentimes repetitive in a different format, you can significantly reduce the amount of actual memory usage required without any actual loss in data.

It simply takes a different format for the way in which the data is formatted, but if you find ways to omit the need to repeat data and omit the need to make needless repetitive checks, you can optimize the game to run faster and leaner without any loss in the detail that the game possesses.

[dirty foot]

I mean, we're hitting a cap at 2-3-4 gigs of RAM, and things are crashing from that; assuming another 20 years of content addition, we're not going to get too far without an increase in the RAM cap. Trimming or culling historical data or "needless" status checks kind of robs the beauty of how in-depth the game is. Resisting the use of a MUCH higher RAM cap seems entirely against the very premise of Toady's vision, which to me looks like a need for totality and realism.

Actually, no, that's missing the point of what I was stating - if you find ways to store data that is oftentimes repetitive in a different format, you can significantly reduce the amount of actual memory usage required without any actual loss in data.

It simply takes a different format for the way in which the data is formatted, but if you find ways to omit the need to repeat data and omit the need to make needless repetitive checks, you can optimize the game to run faster and leaner without any loss in the detail that the game possesses.

Okay, I can see where I misunderstood you, and I apologize for jumping to an incorrect conclusion.

However, I still maintain that there should be a shift to 64-bit operation. I know it's not likely to happen, but it's a nice thought.

[dirty foot]

Also, if my system processes are any indication of the slowdown issue, then CPU speed is much more important than anything else. I'm on year 230 and redlining right now on my crappy little 3.7 gig CPU, with much more RAM left to play with. My estimates put me at finishing around 8 hours from now, I don't know how comfortable I am at running one of my processors so hard for that long.

[Nagidal]

Long history large worlds won't generate because DF crashes after reaching its memory limit, which is 2GB for a 32-bit application.

Can't there be a 64-bit version of the game? It could address all the available memory. I would not mind waiting days for a large world with very long history to generate.

[malvado]

I'd say if Toady allowed the community actually to migrate the game from 32bit to 64bit for him it could probably be done rather fast (or not) , sure there would be some problems if people started making  their own versions out of the source code. I've seen allready several cases where people have found problems in the game and fixed them by modifying the source code through the binaries fixes.

[zanchito]

Not happening. Toady doesn't even want to open the GUI for development, a full recompile would need to essentially make DF open source (you need to recompile the whole thing). It'd be far more beneficial to remake the data tracking to a system like Kohaku proposed than going 64bit.

The 64bit change would require an additional release and 64 bit versions of the game libraries, some of them might just not be 64-bit ready at all, there's nothing Toady can do about that. If he's not doing anything funny with the code, then the recompile would be trivial. If he is doing funny things, it would probably be a nightmare.

[dirty foot]

Not happening. Toady doesn't even want to open the GUI for development, a full recompile would need to essentially make DF open source (you need to recompile the whole thing). It'd be far more beneficial to remake the data tracking to a system like Kohaku proposed than going 64bit.

The 64bit change would require an additional release and 64 bit versions of the game libraries, some of them might just not be 64-bit ready at all, there's nothing Toady can do about that. If he's not doing anything funny with the code, then the recompile would be trivial. If he is doing funny things, it would probably be a nightmare.

Eventually, that memory limit is going to be hit again, no matter how many tricks you use to keep its usage down. Putting off a switch to 64-bit is only going to make the move much more difficult when there are increasing amounts of content. When I bought my wife's computer last year, I was hard-pressed to find one that wasn't 64-bit. So basically, we're going to be sitting on this technology but incapable of using it for a game that would most benefit.

[zanchito]

Everybody should have moved to 64bit computing 5 years ago, if you ask me.

[NW_Kohaku]

Everybody should have moved to 64bit computing 5 years ago, if you ask me.

Yes, well, there are a lot of things Toady "should have done", but refuses to do.  There is no multi-threading, no redesigned interface, no API so others can make a third-party interface, no severely needed optimization of pathfinding, optimization of fluid motion, etc. etc. etc.

And don't even start trying to talk about adding multiplayer.  There isn't a single-player game's forum I go to for the past couple years that doesn't have at least one nitwit a week talking about how much the game needs to be completely rewritten so he can play it multiplayer.

[dirty foot]

Everybody should have moved to 64bit computing 5 years ago, if you ask me.

Yes, well, there are a lot of things Toady "should have done", but refuses to do.  There is no multi-threading, no redesigned interface, no API so others can make a third-party interface, no severely needed optimization of pathfinding, optimization of fluid motion, etc. etc. etc.

And don't even start trying to talk about adding multiplayer.  There isn't a single-player game's forum I go to for the past couple years that doesn't have at least one nitwit a week talking about how much the game needs to be completely rewritten so he can play it multiplayer.

This one always bothered me. Not every game needs to be multiplayer. The only reason why game companies make nearly everything multiplayer now is so they can push out more copies of the game by forcing it to only play properly through their server.

[Caldfir]

Well, the strange things is that while worldgen can eat >2Gb of memory while running, the final save of that world is only ~150Mb.  To me this signals that some judicious pruning of what gets stored could easily solve a lot of problems.  You can talk all day about 64-bit addressing, but the gains you get from writing more memory-efficient code almost always vastly outstrip what hardware support tweaks can do. 

 In any case, I very much hope some worldgen optimization occurs soon.  It feels to me like a sub-1000-year worldgen is somehow unsatisfactory - for one thing, dragons aren't going to be full-grown, and for another, worldgen has almost always, for the history of DF, generated to the year 1050.  Since DF2010 this has become substantially harder to reach as more data has gotten dumped into world generation. 

Like anything, it will take time though.  There are a lot of features/design-elements that it would be nice for DF to implement/adopt, but the fact is that each one would take time to implement.  I'm sure Toady wants to do a large number of things, but everything takes time to do.