Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - Skeggox

Pages: [1] 2 3 4
1
DF General Discussion / Re: Obsidian: A new DF 3D visualizer toolset
« on: August 03, 2010, 12:59:13 pm »
I'm almost over the worst bump of finalling at work, which will soon mean more free time for Obsidian and more updates. Just be patient - it has not been forgotten.

2
Thanks for the compliment Tack :)

Alas, it's a crazy ass time at work. Averaging about 65 hours a week, with super-extra overtime on weekends. Deadlines are deadlines, but at least things will quiet down just before we ship. Just a few more weeks...

Just to reassure - I haven't forgotten about Obsidian. I have, however, been struggling with the weirdest Lua error lately. Anyone familiar with Lua and binding to C, can you maybe cogitate over the following and offer some suggestions as to what might be the culprit?

<technical>
I'm generating Lua bindings where I start by registering metatables for each C++ class. I generate a C-function for __index, which contains a generated switch statement that compare a hash of the incoming string key with all of the members and methods defined in the class. It will return either the value of a member, or a C-function wrapper for a method. A similar C-function is generated for __tostring (prints the C++ class name) and __newindex (to assign values to members). These class metatables are added to the registry by using luaL_newmetatable with the fully qualified class name. Whenever I need to push such a C++ object into Lua, I create a light userdata set to the object pointer, get the registered metatable using luaL_getmetatable, and set it on the lightuserdata.

So far, so good - all according to examples from the Programming In Lua online book. Except... for some reason most lightuserdata will act as if the most recently retrieved metatable is set on them. So, given class Fortress, class Map and class Cell, each with a __tostring that prints the class type, I get the following:
> fortress = ReadFromDF()
> print(fortress)
Cell::Fortress
> map = fortress.Map
> print(map, fortress)
Cell::Map Cell::Fortress

Here comes the funky part:
> cell = map.Get(0, 0, 0)
> print(cell, map, fortress)
Cell::Cell Cell::Cell Cell::Fortress

See? The root object stays the same (it is actually a full userdata), but the light userdata references (cell and map) are set to the latest metatable retrieved.

Trying to debug this with commandline gdb in Windows is not helping. I've been wholly spoilt by Visual Studio's commercial debugger these last few years, and my brain is just too tired to learn gdb properly at this point. Anyone know of any good free mingw32-friendly debuggers (or good GUI frontend for gdb) on Windows, or do I need to whip out the Archlinux and try out KGdb etc?
</technical>

P.S. Don't even mention Visual C++ Express - yes, its free, but the free debugger smokes crundle butt-lint and the compiler can easily be curb-stomped by a furry wambler suffering from gastrointestinal stress.

3
I think with digging invaders we should use the material system to keep things in check.

With soil and rocks having different hardness, you could require invaders to be armed with hard-enough tools to be able to actually dig through it. So, kobolds with copper tools might only be able to dig through soil, but bronze-armed goblins can take on softer stone (like sandstone). Humans with their iron picks can manage hard rock, like granite. Elves will have to stand there and and look pansy, unless they descend below the kobold level and dig away with wooden training picks :P Creatures will have similar abilities, based on the material hardness of their claws. Groundhogs could maybe manage soil, but you'd need a forgotten beast at least to tackle obsidian.

Constructions could either add or take away from the strength of a natural stone, I'm still undecided on that. Smoothing might also make a difference, potentially.


4
Progress Update

Alas, I have little progress to update today. I'm finalling at work for the next 8 weeks (we release in mid October), so I'm looking at 10-14 hour days. I only get about an hour each day to spend on Obsidian, and it is hard to get flowing in such a short time.

So during the week I'll be tweaking and cleaning up existing code, and on weekends I'll be tackling big stuff, like the terrain generator, etc.

That doesn't mean I haven't made any progress - in fact, I've got the Cell engine to properly load and save from DF 31.0.08 to XML using the latest DFHack library.

I've uploaded a cell file for my current Nurembourg-style castle project: http://dffd.wimbli.com/file.php?id=2573

I'm currently packaging the cell engine as a stand-alone Lua module, for anyone that wants to play with it whilst I continue to work on Obsidian.

5
It'd be nice if Obsidian includes mac compatibility, but I guess it's not a first priority goal, if one at all :)

I'm using CMake, and all external libraries are open source and Linux-based, so there should theoretically be no reason to not have both a Mac and Linux version as well. Once I'm in a position to start releasing, I'll look for Linux and Mac volunteers that know their CMake and programming to get those two platforms up and running.

To speak the truth, I'm very likely to switch over to full-time Linux development of Obsidian soon. I'm getting my fill of Windoze at work, and although Mingw32 tries, it just ain't the same as a good Archlinux system.

I haven't tried Mac development yet (but my lady has a Powerbook and I can always sneak it away from her whilst she is reading and try my hand at it), but AFAIK Mac OS X is basically a Unix system, so the "porting distance" between Linux and Mac OS X should be less than between Linux and Windoze.

In that case I'll be needing volunteers to keep the Windows/Mingw32 platform maintained :)

(And good luck to them - I already have at least 10 packages that needs to be custom installed in windows: libxstl, libxml2, libiconv, libzip1, OSG, OpenThreads, DevIL, Lua5.1, log4cplus and freeglut)

6
To those worried about polycount, remember that there are maaany ways to work around it.

Firstly, you use LOD models - the LOD stands for Level Of Detail. Simply put, you take a model of a barrel with high detail (6k polys) and create successively lower detail versions, up to a certain point (say, 100 polys). The engine then chooses which model to use, based on camera distance. So if you are zoomed up close an personal to a food stockpile, the high-detail versions will be used for graphicy goodness, and in an outside scenery shot the very low count barrels will be used.

Secondly, modern GPU's provide the ability to do instancing - that is where you load a set of meshes and textures (i.e. a model) into the graphics card's memory, and then you load up a list of positions, scales and orientations. The GPU will then use the same model, but just render it X number of times using hardware. This cuts the time spent transferring data between main memory and graphics card, which is often the speed limiting factor.

Lastly, there is a technique called imposters - where you render the model to an offscreen surface to get a 2D sprite (a transparent thumbnail picture), which you then 'stamp' wherever the model appears far in the distance. So the cost of rendering the model then becomes drawing a single quad (two squares). That is the technique VF uses for the tree models - you can see that after a certain distance the trees' appearance change somewhat, looking more 'flat' than usual. The Total War series of games relied heavily on the imposter system as well - otherwise it would be impossible to render 5k+ troops. It explains the spritey look of the soldiers in the early games from that series.

AFAIK OSG supports all three techniques mentioned above, so I am really not worried about performance at this point.

Remember that a big part of Obsidian is its planned ability to export to common raytracing renderers (like Povray, Blender etc.). So we'll eventually need a high-detail raytrace and lower-detail real-time version of each module (and texture?).

7
Progress Report

The save part of the Cell engine is now finished to alpha quality - it can serialize out any data structure defined in the cell definition file to XML. It implements cons hashing of instances, with a simple run length encoding of the actual map cell array. This optimizes the in-memory and on-disk representation for size, since I want Obsidian to handle very large megaprojects without borging the computer's memory. This trades off against speed a bit, but currently the cell engine takes about 12 seconds to save a deep 2x2 fortress, with DF suspended only for the first 4-5. That seems reasonable so far for an unoptimized debug build...

As a nice side effect, the XML file is also pretty darn compact now. A 2x2 embark with 180 z-levels is saved as just over 300kb of text XML. I might not even need a compressed binary serializer... argh, who am I kidding? This is DF - someone is probably already itching to throw a 15x15 embark with 500 z-levels into the front of the Cell Engine just to watch the pretty flames and shrapnel coming out of the back of it.  :P

Ahem. The cell engine currently reads from DF and saves to file the following:
1 ) Tile types
2 ) Materials
3 ) Global Features (DF only has Underworld for now, AFAIK)
4 ) Local Features (Adamantium tubes, pits, stairways, river sources/sinks, etc.)
5 ) Veins
6 ) Stone Layers
7 ) Constructions
8 ) Settlement info*

Items and critters will eventually be added as well - but they are lower priority for now.

Next up: With the engine settling down to a somewhat stable architecture I'm going to finish the Lua bindings - so that you can query/manipulate all fortress data from inside of a Lua script.

After that I'll be grabbing the latest DFHack and updating the Cell engine to use it, i.e. get some support for 31.06+.

Once all that is done I will release a beta Lua module of the Cell Engine for anyone that wants to dump their fortresses to XML and play with them a bit in Lua. ETA is expected to be 2-3 weeks (we're entering finalling, so my days are going to be hectic for a bit)

Skeggox

* The code is written, but my version of DFHack doesn't implement it so I can't test it properly yet

8
I chose the Git because it is a VCS I haven't tried yet, and it was next on my list of things-to-learn (being my first distributed VCS) . I've whipped Subversion already, but Mercurial is even less well known then Git at this point, at least to me.

9
Ah, okay, I see.

Hrmm. Right - I think I'll use http://dffd.wimbli.com/ for binary releases of the tool and source art packs.

Art candidates (models, textures, etc., both source and compiled) will be hosted by the contributor until they are accepted into an art pack, in which case they will be packaged and published into the above-mentioned DF repository.

Links to art candidates will be posted on the wiki Art page that I've just set up (http://df.magmawiki.com/index.php/Utility:Obsidian/Art).

I'll use GitHub for source code only. I'll take a look at creating a repository when I get some time.

10
Hrmm. Can we use github to store the source art, as well as the art assets then? Even if they get large? What are the file size and bandwidth limits for a project on GitHub?

If that is workable, what would you need from me to setup a repository for Obsidian on github?

I'm using Git locally (I'm fairly newb with it - been using Perforce for a while so the learning curve is interesting).
Eventually I can push the source with history to github once I'm ready to release it. Until then we can use it to push binary releases and models, textures, logos, etc.

11
Alas, not yet.

I'm spending practically all of my spare time on just getting Obsidian to a beta release state.

I'm hoping someone will volunteer to set up and run such a central repository?

There is a bare-bone wiki landing page for Obsidian (http://df.magmawiki.com/index.php/Utility:Obsidian) that would make an excellent starting point. Just add some pages for models, textures, etc. and then we can use the discussion pages to coordinate what goes on the lists.


12
Any help is more than welcome, Crunch!

Obsidian's philosophy is that absolutely anyone can contribute anything.

To that end I've chosen to use OpenSceneGraph to render the geometry and models. That means there is a decent composer called OSGEdit(http://osgedit.sourceforge.net/) that you can use to import models and play around with. If you can make it in OSGEdit, Obsidian will render it.

Currently, opsneakie and Djohaal are our modellers/render experts, and GFXiNXS our artist. Contact them (cc me) and lets organize what models and textures we'll need for starting out.


13
Sweet.

I'll have to leave that as a stretch goal for later, though. I know Linux can easily set the backgrounds for terminal console windows, but I'm not so sure Windows can. If Window's can't do it easily, I'll have to implement a custom graphical console, which is a luxury feature atm.

But I'm all for having sweet eyecandy in Obsidian. It makes a big difference in feel and polish, compared to a simple text-only tool.

So hang on to the source for that image...

14
My portrait image is the actual icon for Obsidian, kindly provided by Janus.

Reducing the magma disk down to 64 and less will lose too much detail, I think.

Currently, Obsidian consists of a terminal console, a rendering window and a loading splash screen.

What we could do is use GFXiNS' submissions as the splash screen and credits background, and then use the others as the background for the rendering window and terminal console.

That depends on how colored text will look superimposed on the disk or sphere...

This will all be user configurable through the config.lua file, of course :)

15
Unlike VF and other previous visualizers, I'm not going to spend any dev time in making my own rendering system etc.
I'm using an off-the-shelf scene manager and rendering engine called OpenSceneGraph instead.

So, if OpenSceneGraph can render it, you will see it in Obsidian. OSG comes with a nice scene composer called OSGEdit, so anyone that wants to help with models, textures, shaders etc. can use that composer.

On other news - the save/load part of the Cell engine had a setback, but I'm on top of it again. I tried to write out one of my megaprojects from 40d#, 6x6 embark with 50 z-levels. The XML save file came out at just under 200MB. Erm. Even compressing it only brought it down to 80MB.

I had to go back to the drawing board and rethink things a bit...

This weekend I re-wrote the save/load code to use cons hashing. This should reduce memory and file size by a factor of probably around 100, once I beat the last bugs into submission.

So, yeah, there is definitely progress :)



Pages: [1] 2 3 4