Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[arclance]

Stress test : a 16x16 embark dump.

Code: [Select]

INFO:fgt.mapdata:loaded 768x768x206 1854M

failed.

Code: [Select]

OpenGL.error.GLError: GLError(
err = 1285,
description = b'out of memory',
baseOperation = glDrawArrays,
cArguments = (GL_POINTS, 0, 4816)
)



DF ate 3.6G of memory while embarking. SSD sure helps with swap. Gzipped dump file is 52M.

I need an upgrade.

What is most depressing is that this same stress test worked before. Might had less z-levels, but still.

Let me try it I have 8GB of ram and a 2GB graphics card, I want to see how much slower it runs.

[lxnt]

http://dffd.wimbli.com/file.php?id=6454

[arclance]

It works though it's using 3.50GiB of ram (and very little gpu ram) the gfps is still 60!

Code: [Select]

./fgt -dfdir /media/Linux_Data/Dwarf_Fortress/Vanilla_Games/df_34_10_linux/df_linux ../16x16x206.dump raw/fakefloors
INFO:fgt.sdl_init:SDL_GL_DOUBLEBUFFER requested 1 got 1
INFO:fgt.sdl_init:SDL_GL_CONTEXT_MAJOR_VERSION requested 3 got 3
INFO:fgt.sdl_init:SDL_GL_CONTEXT_MINOR_VERSION requested 2 got 2
INFO:fgt.sdl_init:SDL_GL_CONTEXT_PROFILE_MASK requested 1 got 1
INFO:fgt.sdl_init:SDL_GL_CONTEXT_FLAGS requested 2 got 2
INFO:fgt.raws._parse_raws:yamnomnom done.
INFO:fgt.raws._parse_raws:init.txt done.
INFO:fgt.raws._parse_raws:df material templates done.
INFO:fgt.raws._parse_raws:df materials done.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/RampTop no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/Void no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/GlowingBarrier no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/Campfire no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/Fire no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/MurkyPool no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/OpenSpace no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/Chasm no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/Waterfall no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/MagmaFlow no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/EeriePit no cels defined.
INFO:fgt.raws.RawsCart.compile:nonmat/NONEMAT/GlowingFloor no cels defined.
DEBUG:fgt.raws.InflateFrameseq:len(keyframes) = 5
INFO:fgt.raws._parse_raws:compile done, 13088 code units.
INFO:fgt.raws._parse_raws:Pageman(): 2 pages 512 tiles, font 352K; findex 4K; surface: rgba_surface(size=2048x44, SDL_PIXELFORMAT_ABGR8888, do_free=True)
INFO:fgt.raws._assemble_blitcode:13088 code units emitted, maxframes=24 codedepth=24
INFO:fgt.raws._assemble_blitcode:tileflags: 699x1x1; 2K
INFO:fgt.raws._assemble_blitcode:dispatch: 957x699x1; 2613K
INFO:fgt.raws._assemble_blitcode:blitcode: 115x115x24; 4959K
WARNING:fgt.CArrray:65536 extra bytes
INFO:fgt.mapdata:loaded 768x768x206 1854M
/home/arclance/.fonts/ubuntu-font-family-0.80/UbuntuMono-B.ttf
INFO:fgt.renderer.init:GridVAO(size=Size2(w=166, h=72) num=11952)
INFO:fgt.renderer.init:Size2(w=1280, h=800)
I takes a good 10-20s to load the dump.
I don't know if something is wrong with the dump or if fgtestbed can't handle a map this large but the tileset seems to only be applied to water.

[lxnt]

It works though it's using 3.50GiB of ram (and very little gpu ram) the gfps is still 60!

Wow. The driver must me mapping system memory in rather than copying. Entire dump is used as a single texture.

I takes a good 10-20s to load the dump.
I don't know if something is wrong with the dump or if fgtestbed can't handle a map this large but the tileset seems to only be applied to water.

Hm. Grey are the hidden tiles, that is, not revealed. Water was revealed obviously, since it's ponds at the surface. Green is the fill color, meaning nothing was drawn there. At zeddown=4 lots of it is to be expected since I deliberately chose a mountanious area.

Anything else not being drawn might have something to do with unusually high number of materials present - 957, where typical number is about 450. However it should've been able to handle at least 1024.

Framerate not dropping might be due to nothing being actually drawn.

Well.  I'm going to try integrating this into libgraphics now (so that it draws current view is a separate window).
I'm quite lost on how to submit building data for drawing, maybe that way I'll stumble on some ideas.

[arclance]

It works though it's using 3.50GiB of ram (and very little gpu ram) the gfps is still 60!

Wow. The driver must me mapping system memory in rather than copying. Entire dump is used as a single texture.

It's only using about half my ram so it did not bog down my computer.
It was more than I expected but not the most a program has ever used either (the record is about 12GiB).
It uses less gpu ram than using opengl to play HD videos with mplayer.

I takes a good 10-20s to load the dump.
I don't know if something is wrong with the dump or if fgtestbed can't handle a map this large but the tileset seems to only be applied to water.

Hm. Grey are the hidden tiles, that is, not revealed. Water was revealed obviously, since it's ponds at the surface. Green is the fill color, meaning nothing was drawn there. At zeddown=4 lots of it is to be expected since I deliberately chose a mountanious area.

Anything else not being drawn might have something to do with unusually high number of materials present - 957, where typical number is about 450. However it should've been able to handle at least 1024.

Framerate not dropping might be due to nothing being actually drawn.

Well.  I'm going to try integrating this into libgraphics now (so that it draws current view is a separate window).
I'm quite lost on how to submit building data for drawing, maybe that way I'll stumble on some ideas.

I was sure it was the surface because there is a brook about 10z down from that screenshot.
I recognized the gl "nothing here" green color, is it possible to make that color respect these settings in d_init.txt?

Code: [Select]

[SKY:178:3:0:0]
[CHASM:250:0:0:1]
I went up until the whole screen was green and everything was "unrevealed grey" or "water blue".
Going down is the same but just "unrevealed grey" at the bottom.
Everything but the water is drawn "unrevealed grey" even the tops of the mountains.

For buildings are you able to tell where they are (what tiles they take up)?
If you are you could just read their raws to determine what each tile should be.
Getting material based colors right would be more difficult though.

Have you looked at how buildings work in stonesense?
I know they get building locations but I don't know how they chose what color their sprites are shaded.

[lxnt]

I recognized the gl "nothing here" green color, is it possible to make that color respect these settings in d_init.txt?

Code: [Select]

[SKY:178:3:0:0]
[CHASM:250:0:0:1]


no problem.

For buildings are you able to tell where they are (what tiles they take up)?
If you are you could just read their raws to determine what each tile should be.
Getting material based colors right would be more difficult though.

Have you looked at how buildings work in stonesense?
I know they get building locations but I don't know how they chose what color their sprites are shaded.

Yes, I know all that df-structures project, and, by extension, DFHack and Stonesense can know about structures. I also do parse DF raws, since that's where the material colors and default tile graphics come from.

Problem is, each building tile is (at least) 44 bits of state not counting position, and this is plainly too much to use a 1 to 1.5 -level lookup, as is done with map tiles. On the other hand I don't want multiple rendering passes if I can help it, and certainly not an unbounded number of render passes per frame, like it would be if I draw each building separately. I want to upload something once per frame, and then tell the GPU to draw it all, like it's done now with the map.

[arclance]

Yes, I know all that df-structures project, and, by extension, DFHack and Stonesense can know about structures. I also do parse DF raws, since that's where the material colors and default tile graphics come from.

Problem is, each building tile is (at least) 44 bits of state not counting position, and this is plainly too much to use a 1 to 1.5 -level lookup, as is done with map tiles. On the other hand I don't want multiple rendering passes if I can help it, and certainly not an unbounded number of render passes per frame, like it would be if I draw each building separately. I want to upload something once per frame, and then tell the GPU to draw it all, like it's done now with the map.

You could use an array that defines what to send to the gpu.
Start by filling the array using the map data.
Then update the cells in the array for buildings (or anything else).
Last send the final array to a function that sends data to the gpu as defined in the array.

This way no matter how many changes are made you only draw once per update.

I haven't used any python arrays of this size so I don't know if this would be fast enough though.

[lxnt]

Thanks for suggestion.

As for the arrays, fgtestbed doesn't use python ones, and I went away from using numpy ones, since their use as OpenGL textures was problematic and I needed to get as close to C as practical to ease later integration. See the CArray class in py3sdl2.py.

[arclance]

^ The CArray class looks interesting but I don't have the time to think through how it works right now.
I do think that using the python "array" (lists of lists of lists of .....) would have been too slow for this anyway.

What dimensionality did your arrays end up being?
I could see them being 4D or 5D depending on how you structured the data.

[lxnt]

^ The CArray class looks interesting but I don't have the time to think through how it works right now.

It's a bytearray posing as a 3D array of structs (binary data, see struct module).

What dimensionality did your arrays end up being?
I could see them being 4D or 5D depending on how you structured the data.

Since current hardware can't handle more that 3D textures natively, this is what I ended up with.

It's 3D for the map data, uploaded as texture, which holds, basically, the  (material, tiletype) tuple.

(material, tiletype) are coordinates into 2d dispatch texture, which holds (cx,cy) coords into blitcode texture (plus animation frame count in unused bits), current frame number is supplied externally. Aniframe count is used to loop animations properly.

(cx, cy, frame_no) are taken as coordinates into 3D blitcode texture.  The blitcode texel holds (mode|celindex, unused, fg, bg), where celindex is an index into texture album index  - 2D texture holding tile graphic size and location in the album - enabling non-uniform size of tile graphics - with which texture coordinates into the actual graphics texture (2D) are calculated.

[King Mir]

Buildings can be transparent (the floor tile shows through), so that means an additional layer of data, not a replacement.

One way to do it is by adding 3 additional fields to the map data: building (which building), building coordinate (ie which of the nine tiles of a workshop to use), and building material. You'll also need a feild for if it's walled or floored eventually. And for that matter, smoothed or engraved. You can probably figure out which directional walls to use on the GPU. When you do, remember that doors effect wall textures, so the building location must be part of the texture to compute the wall direction.

Another way to do it would be to have a list of buildings with coordinates. This is a sparse array. The GPU can merge it with the dense array of map data or render it as a second pass. The 3 fields are the same. In this case the building coordinate is not needed. You can then transform the list of buildings into a list of tiles on the GPU. The other fields stay on the map data.

Which you use probably depends on how the information is most efficiently obtained from the application. You probably want to offload as much onto the GPU as you can.

Also remember that you can use two textures if 1 texture has more than 32 bits of data per location. I don't know if that's your problem, but you mentioned 44 bits of building data being an issue.

[lxnt]

Buildings can be transparent (the floor tile shows through), so that means an additional layer of data, not a replacement.

Certainly. Question is how to format this data so that work is offloaded to the GPU as much as possible, but without bogging it down.
I absolutely want to avoid walking around structures on the CPU determining how  a door is drawn in open vs closed state.

One way to do it is by adding 3 additional fields to the map data: building (which building), building coordinate (ie which of the nine tiles of a workshop to use), and building material.

That's 10 bits for material, 5 bits for tile number (trade depot and siege workshop are 5x5), and 7 bits for building type. Note how this lacks open/closed spinning/stopped, build stage, orientation for pumps/waterwheels, etc state. Which add up to 44 bits by my estimation. Now that I got those 44 bits onto the gpu, what do I do with them? Somehow they must ultimately be mapped to graphics, and this is my main problem.

(not to mention bridges, stockpiles, civzones and rooms, their arbitrary dimensions is an entirely separate problem I haven't even thought about)

Another way to do it would be to have a list of buildings with coordinates. This is a sparse array. The GPU can merge it with the dense array of map data or render it as a second pass. The 3 fields are the same.

That would result in the GPU having to walk this list for each tile drawn. Not a very big deal in itself, but I expect to draw 4-6 z-levels,  fake floors under trees and such require up to 8 texture lookups, and yet undetermined algorithm to apply stencils to tile borders will eat who knows how much gpu ticks. So brute-force methods don't cut it.

The building list would have to be sortied or indexed the list on the CPU,  each game engine frame. But what sort of index? It has to be suitable for craetures and items too.

Also remember that you can use two textures if 1 texture has more than 32 bits of data per location. I don't know if that's your problem, but you mentioned 44 bits of building data being an issue.

44 bits are an issue if you try to use them as an index into some table. Otherwise, GL3.0 textures can hold 32bits per channel, that is quite enough.

[Vattic]

One way to do it is by adding 3 additional fields to the map data: building (which building), building coordinate (ie which of the nine tiles of a workshop to use), and building material.

That's 10 bits for material, 5 bits for tile number (trade depot and siege workshop are 5x5), and 7 bits for building type. Note how this lacks open/closed spinning/stopped, build stage, orientation for pumps/waterwheels, etc state. Which add up to 44 bits by my estimation. Now that I got those 44 bits onto the gpu, what do I do with them? Somehow they must ultimately be mapped to graphics, and this is my main problem.

(not to mention bridges, stockpiles, civzones and rooms, their arbitrary dimensions is an entirely separate problem I haven't even thought about)

Also worth remembering that mods can include their own workshops of up to 31x31.

[King Mir]

One way to do it is by adding 3 additional fields to the map data: building (which building), building coordinate (ie which of the nine tiles of a workshop to use), and building material.

That's 10 bits for material, 5 bits for tile number (trade depot and siege workshop are 5x5), and 7 bits for building type. Note how this lacks open/closed spinning/stopped, build stage, orientation for pumps/waterwheels, etc state. Which add up to 44 bits by my estimation. Now that I got those 44 bits onto the gpu, what do I do with them? Somehow they must ultimately be mapped to graphics, and this is my main problem.

(not to mention bridges, stockpiles, civzones and rooms, their arbitrary dimensions is an entirely separate problem I haven't even thought about)

So as I understand it, you need to have the building lookup as 1 step, but the look up index can only be 32 bits. So as a solution, you'd have to separate the material data, which determines the color, from the other data that determines the tile shape. Look up the shape on one texture, and the colors on another texture. You might want a separate texture for each of the color involved (foreground and background for building and floor), so that you can have full color materials. Then overlay or otherwise merge the color and shape info.

Another way to do it would be to have a list of buildings with coordinates. This is a sparse array. The GPU can merge it with the dense array of map data or render it as a second pass. The 3 fields are the same.

That would result in the GPU having to walk this list for each tile drawn. Not a very big deal in itself, but I expect to draw 4-6 z-levels,  fake floors under trees and such require up to 8 texture lookups, and yet undetermined algorithm to apply stencils to tile borders will eat who knows how much gpu ticks. So brute-force methods don't cut it.

The building list would have to be sortied or indexed the list on the CPU,  each game engine frame. But what sort of index? It has to be suitable for craetures and items too.

No, it's all one sparse array, not several z levels of textures. So it's implemented as vertex data instead of a texture. So you have each building location as a vertex, the the data identifying that vertex contain the building data. That gets transformed into vertexes for each tile, so each workshop is transformed from 1 vertex to 9. Then the tile gets mapped to the actual graphic. All in parallel.

[lxnt]

Also worth remembering that mods can include their own workshops of up to 31x31.

Aw, nice. Now it's 10 bits for the tile number. Thanks for the heads-up.

So as I understand it, you need to have the building lookup as 1 step, but the look up index can only be 32 bits.

No, I don't exactly require single-step lookups, in fact splitting them into multiple steps help reduce needed memory for the indices.

Deciding how exactly to split, so that CPU has an easy time generating the data, GPU isn't bogged down, it all fits into the graphic card's memory,  and doesn't need to be updated too frequently is the hard part.

No, it's all one sparse array, not several z levels of textures. So it's implemented as vertex data instead of a texture. So you have each building location as a vertex, the the data identifying that vertex contain the building data. That gets transformed into vertexes for each tile, so each workshop is transformed from 1 vertex to 9. Then the tile gets mapped to the actual graphic. All in parallel.

This might be the way for all the "transients" - things that aren't on 99% of map tiles. Thanks for the idea.