Bay 12 Games Forum

1. make debug
2. debug builds are automatically done through travis ci; I also use them by default when developing. however, I haven't had the time yet to test 28.0.

Yeah, make debug. Whatever.

The question was - is the release build known to be broken?

Is everyone here running debug builds? Because I get bitten by

QSpiAccessible::accessibleEvent not handled:  "2"  obj:  QMessageBox(0x1553bd0) ""
FIXME: handle dialog start.

every time I rebuild the therapist and forget the -f Makefile.Debug part.

System's xubuntu 14.04.1

Such a big changes like abandoning cp437 charset completely don't look realistic to me.

It is not about abandoning anything. It is about gradually extracting the world->cp437 mapping from the core game to where it can be moved forward easier.

In fact, preserving the original look is one of the major goals, PRINT_MODE:TEXT - linux pure text interface - is to stay forever.

So this thread can basically be summarized as "Have the game supply a complete cache of externally readable information every frame with coordinates and render-relevant stuff like tiletype and material. Then render from that only with plugins, one of which is the default ASCII graphics"  Yes?

Yes. I call it a buffer, not a cache, and

However, I don't think you're going to see that dramatic of quality increases unless you provide things like
*multiple or all z levels of info, not just what vanilla would show
*all the items in a tile, not just, for example, the one creature "on top" at the moment that vanilla would show
etc.

And doing that seems likely to cause game slowing.

the buffer isn't supposed to be a complete and accurate snapshot of the world. It's just a frame. Two-three z-levels, only the top item out of the heap, etc. The effort is aimed at removing the 256 cel bottleneck and decoupling graphics code updates from releases, on windows, not making a monster of an interface that'll require top hardware to just not drop below 60 FPS on an embark.

Recently I noticed that there would be another release of the DF after all.

Also, it so happened, FDw Taffer contacted me some time ago about work that have been done on DF graphics.

All of the above makes me to post a reedited take on the the problem (it's in ReST format):

The Tile Reuse Problem And Proposed Solution.
=============================================


The problem:
------------

    - Dwarf Fortress interface is by design limited to 256 characters of 16 colors (cp437 / "ascii" + "ansi" colors).
   
    - Dwarf Fortress has orders of magnitude more information than can be unambigously expressed by that.
   
    - Relaxing this limitation is a lot of work, while there isn't even an understanding what that is to achieve.
   
    - Graphics code is old, rotten and Toady undestandably has no desire to touch it. Even if the code was in perfect
      condition, it is not his area of interest or expertise.
   
    - Graphics code can not be updated for Windows releases without making yet another full release of the game.
      Coupled with above this means that the graphics code just continues to rot.

Thus the problem is split in two parts:
---------------------------------------

    - Graphics code is old, unmaintainable and too tightly coupled with the rest of the game code.
   
    - Graphics code can not express most of the information available (since it gets the tiles, items, parts of interface
      already mapped to cp437 and baked into a single layer).

The solution also cames in two parts:
-------------------------------------

    - An easy to understand and work with, maintainable graphics subsystem, where Toady isn't required to understand
      or mess with its internals, while third parties can maintain it without any access to the core game code.
     
    - An extensible, easy modded graphics subsystem that can express most, if not all, of the information
      available without Toady having to put any work into besides a modest initial investment.
      Which also fully supports anything up from and including the original cp437 interface.
     


The first part
--------------

of the solution is pluggable renderer bound to the core game only by a strict and stable API+ABI.
Feasibility of that was demostrated in the `rendumper project <https://github.com/lxnt/rendumper>`__

The second part
---------------

of the solution was prototyped in the `fgtestbed project <https://github.com/lxnt/fgtestbed>`__

As the latter attempt has shown, graphics mods require quite a lot of information from the core game.

Designing a stable interface that is still capable of exposing enough in-game information is the current focus.

Such interface has not been prototyped because initial fgtestbed work focused on using df-structures/dfhack code,
which provides direct access to the internal game structures. Alas, this is quite complicated and unusable for
extended periods of time after each major release because changes in the data structures have to be reverse-engineered.

The existing interface
----------------------

as implemented in the rendumper prototype receives three layers of information for the entire game
screen each frame. Those are:

    - Basic cp437 rendering of the interface,
    - Creature overlay, which holds creature "texture" indices,
    - If enabled, a truetype overlay, which is a list of text strings and their screen coordinates, to be TTF-rendered on top of everything else.

The renderer knows which creature texture indices correspond to what graphics because creature graphics loading is also done
via renderer-side code (as opposed to game-side).

As you can see, core game code converts tile types and materials, as set in the raw files, into cp437 characters and ANSI color codes
before submitting all that to the renderer. This is where the tile reuse problem originates.

To get rid of it, the renderer is instead to be supplied with:

    - Original tile type and material ids.
    - A mapping from tile type and material ids into actual graphics to render

To this end, the fgtestbed prototype renderer first parses all the raw files the game has. Then it parses another set of raw files,
that describe the hardcoded mappings for stuff like constructed walls/floors ("constructions"), raw tiles (undigged walls), items and buildings.
Those files also include overrides, so that the hardcoded mappings become moddable.

This information lets it know how to render raw map data, which, generally speaking, consists of triplets of "tile_type, material_type, material_index",
plus information about water/magma levels, any construction, zone or building present in the tile (and their material),
dust/steam/cloud in the tile (and their material), any contaminants (and their material).

On top of that go items (and their material, hehe), and, finally, creatures. Oh, I forgot about projectiles. And game interface (selections,
burrows, cart routes I haven't even thought about.

Long story short,
-----------------

the solution of the tile reuse problem is to move the hardcoded mapping of all the stuff in the game from the core game code
to some kind of "raw" or resource files. And let the pluggable renderer parse and map them.

As any big project, this has to be done in many small steps.
============================================================

The first step
--------------

is to adopt the pluggable renderer architecture, as in the rendumper. This does not require any changes in the core game code.

The second step
---------------

would be to enable basic map modding. To this end:

    - Another layer is to be added to the information already supplied to the renderer, that is - (tile_type, mat_type, mat_index).
      Here tile_type is a "derived" tile type, that is, if a wall is constructed somewhere, then the tile_type is one of the wall,
      not the underlying floor or ramp.
    - Raw material information is to be exported from the game to the renderer in such a way that (mat_type, mat_index) tuple is
      easily mapped to the information from the parsed raws. Keeping raws parsing for the graphics data in the renderer allows more
      modding. If that's not acceptable (eg, worldgen-time generated materials begin to matter much), some interface to the core game
      raws database becomes necessary.
     
      Ideally, rendering info (cp437 codepoint, color) will be purged from the
      item definition raws to some kind of rendering definition raws (as is done in the fgtestbed yaml raws) (which, ostensibly, can be inlined into the item raws, but clearly
      separated and overridable)

The third step
--------------

    - Yet another layer with item representations. It would suffice to do one item per tile, but again with item_type and material.
    - The designation bitfield, whose definition would have to be fixed eternally. That would allow for nice presentation of liquid levels,
      designations, etc.


The fourth step
---------------

tbd

I just dropped by to see how this is going, since SDL2 came out.

It did, but there isn't much sense in reacting to that until SDL2 gets into major distributives. And that may take some SDL2 point releases to happen.

TODO list: Wow, looks complicated.
> show fps+averaged times on an overlay. maybe do a graph ala eve online
Well, it would be somewhat useful to have a "DwarfMark" - a Dwarf Fortress save that has 200+ Dwarves, lots of items and some liquid flows, then unpause it on different computers, look at average uncapped FPS over a minute and have a benchmark for which one seems to be the fastest.

200 dwarves, say 50K items ... it'll just drown any signal from the renderer change in the noise. At those numbers the renderer just doesn't matter. Any fort above 100 dwarves is going to suffer, period. Maybe if Toady implements emigration out of the fort, that would help keeping the population down to something sensible. Barring that, we're stuck - mastery of the dwarf management is just another way to lose, and not particularly fun at that.

As I said somewhere above, slow rendering can't really kill the FPS, it just makes the inevitable quicker. Thus the focus on cross-platformness and modding/distributed development here lately.

Besides, I did a test on how fast one can kill FPS. Draining an aquifer into a cavern does that quite reliably, no need for hundreds of dwarves, just a suitable map. Watch the water cover 2-3 embark places of cavern and the game wents below 1 fps, then just crashes.

Bad thing is, optimizing liquid model would require quite a change in the map internal representation, and this is just not going to happen.

BTW, this thread should be moved to Modding/Tilesets and Graphics. It may get more visibility that way.

It might, but that isn't up to me.

PS.

For some reason my attempt to carve a mountain hall barely worthy of its name - out of a cavern section - measly 55x110x5 tiles - is also being met with random crashes. Although this might be due to bugs in the dwarf therapist.

The ansiFormat and defaultAnsiFormat xml attributes. What were they intended for? The current code doesn't seem to use them.

He may be open to having somebody else work on the graphics code. It's worth sending him an e-mail, if you haven't already. If memory serves he's expressed confusion over the graphics code several times now, and if the status quo is to wait for Baughn to reappear, then perhaps he'd be similarly open to having yourself look at things, given that you've already demonstrated the desire and ability to do so.

Well, the existing graphics code shows its age, so to say, and it wasn't exactly pretty to begin with.

I think best time for any discussion of doing something about graphics would be after this year's release is out and most egregious
 bugs are fixed in the point releases. Say, somewhere about the next Future of the Fortress question submission time after the release goes out.

Until that time, I'll be engraving polishing.

EDIT: Also did anybody contact Toady about this. I saw a comment by him recently (possibly the June report) about not being able to implement something because he did not know how the graphics system worked. Could be that all this awesome work might be something he would like to pick up?

I think you refer to this:

Quote from: arkhometha

Toady, how hard do you think it is to implement the fourth thing the in eternal suggestion voting (Full graphics support) and how high is it in your priority list?

Quote from: CLA

Will we be able to use graphic sheets for plants like we can with creatures at some point?

As far as I know, implementing things like item or map tiles won't work without multiple texture atlases, or something, and I don't know how to do that in the new graphics code.

The most straightforward way would be for the game code to draw the map using internal tile types, plus supply some kind of material color, or code, and maybe have a default translation table from tile types to font/tileset characters somewhere in the raws,
or give it somehow to the graphics code. Then leave actual translation to the renderer plugin, which will be free to load additional graphics and whatnot.

Currently it does that conversion from tile types to characters internally, which results in the limits that can only be overcome with dfhack methods, and that would be slow and cumbersome.

In fact, there's a prototype of just such renderer I wrote before starting work on this modular stuff. Link's in the signature.

The hardest part I think would be persuading Toady that he doesn't need to do everything himself, and that he won't lose any control that way.

The whole idea with pluggable renderers is to separate game from graphics, so that Toady can be sure that no matter what he changes in the game, he won't need to touch graphics side, and existing graphics code will continue to work, and on the other hand, that graphics code can be updated without any need for Toady's attention, much less for recompiles or, Armok forbid, any modifications to the game code.