So I'm building a game (aka, another project that will probably never see the light of day). This is a rant more for my own benefit of getting my thoughts in order than for actual advice.
Right now I've got my own basic 3D engine (using C++, openGL), rendering a terrain made of blocks like minecraft.
But this is a static terrain, such as in DF mode. There is no dynamic loading/unloading when you move through the terrain.
However, I'd like this terrain to:
1. Extend into a horizon. Mountains should be visible in the distance.
2. Generate new terrain as soon as one reaches the edge of the world.
3. NON-2D: most games have a terrain which is really just a crumpled 2d blanket of vertexes (Wurm online). This terrain is truly 3D.
4. Editable: Mining/digging should be possible.
5. Practically Infinite.
6. Render in a decent amount of time.
And last but not least:
7. Multiple parts of the terrain can be in memory at the same time (multiplaying), and terrain should not "take a lot of space" because it'll be sent over teh internets.
Current design:A terraintile is 1x1x1m. A Terrain is a virtual "parent", containing 7x7x7 of those blocks. Then there is another one above that, containing 7x7x7 of those blocks, and then
turtles Terrains like that all the way down. When enlargement is necessary, take the topmost "world" block, and make another parent block around that. This increases world size exponentially. By having a -3, -2, -1, 0, 1, 2, 3 xyz system per block, absolute positioning doesn't change for the already existing world and it's objects, since for the existing world, only zeroes are added. All xyz locations in the game are relative to a parent, and those relative to their parent, etc. There's no real absolutes, except for the topmost block of Terrain which is 0,0,0. All Terrains are linked to eachother both hierachically (sp?), so top-down and bottom-up, and laterally, every block knows its direct (NESW and Up and Down) neighbours. This datastructure is a form of tree I have not been able to find examples of, and I probably implemented it quite inefficient.
Terrain is saved in SQLite. For small amounts it's ok, but I don't know what the performance will be when handling large amounts of blocks.
That is the static part, and due to the exponential nature, easily runs out of memory.
Now the dynamic part:If a "Terrain" is totally empty (as in air), it won't have to render at all, or even be loaded. If all of the children-terrain of any block are empty, those children don't need to be loaded either. Same if the terrain is totally filled with rock, and never touches air. That limits the number of "Terrains" that need to be loaded severely. But the exponential nature is also way too much. You don't need to dig inifinitely deep or fly infinitely high. A network of "top"-level Terrains, each large enough to hold at least 1/2 of a players' view, all linked in a NSEW-fashion, should be enough.
Then there's objects. Now objects are linked to a Terrain-block, which is fine and dandy for calculating their vertices relative to the viewport. But if the terrain is loaded dynamically, and sometimes not even loaded at all because it's "empty", where's the object then? Registering the object to parent-terrains is a pain, because it increases the number of lists where the object resides.
Then there's rendering. Switching textures during rendering is inefficient, so I want to draw all the gneiss first, then all the soil, then all the chalk, then all the clay. To do this, terrain must be loaded into a list-per-texture. When terrain is unloaded, it needs to be removed from that list. Which necessitates either searching through the list for every block again (slow!), or having an inverse pointer to the list-element per terrain, which removes the list element when destructing the terrain. Having lists like that, and removing the terrain incorrectly will result in invalid pointers and certain DOOM.
So now I'm stuck.1. I can't even write down the design in my head in an intelligible way, which is a sure sign my head can't wrap itself around the design all at once, which means I don't yet truly understand it. (Life lesson for everyone: If someone cannot explain something in clear language, he doesn't fully understand it himself).
2. The Loading and Saving Terrain is heavily dependent on the structure-design, so needs to be rewritten and rethought later. Saving and loading to HD is slow, and needs to be efficient. Also, I don't want terabytes of unnecessary terraindata.
3. The rendering is dependent on the structure-design. I want the rendering to be fully relative. The camera has a location within a terrain, and from that terrain we're going to loop through all the neighbours and check if there's anything to render there.
4. I am not a computer scientist. The type of structure needed, and the internal linking of the structure needs to be lean and mean, especially on the rendering-side and save/load side. But what is the fastest way?
5. Objects and players need to move freely throughout the world. Having no absolute xyz system, there is more extensive administration as to what is where. Moving objects need to re-register and unregister with several terrain-blocks. Or I do it differently. How so I do this fast?
tl;dr: Gief me the C++ code!!1! 
PS: Yes I know I'm in way over my head. Part of the reason of diving in too deep is because I like being there.
