Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 2 [3] 4 5 ... 8

Author Topic: DFHack script: showbiomes  (Read 25411 times)

ragundo

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #30 on: October 01, 2016, 02:52:34 pm »

Hi
I don't know if the biome data gets tweaked after you choose the embark area. I've never seen a 0 in the biome(x)(y) array.

After you embark, I'm 100% sure that my code generate the correct biome type.
Check this comparison between my plugin and the export from DF in Legends mode
http://i.imgur.com/OEv7EhK.png

Also you should generate all the maps in Legends mode to check them with your generated ones, like the drainage one.

PD: I've sucessfully generated all the legends mode exported maps by code using my plugin, but, again, I think IRC is a better way of discussing coding issues.

Greetings
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #31 on: October 01, 2016, 04:16:05 pm »

@ragundo: As far as I've seen so far, there has been nothing wrong with your code, only in my confused understanding of it, so if I was to operate on the embark map level it would have been done some time ago. The issue is that I'm operating one level further down, i.e. inside a fortress where each embark level tile is further divided into 48 * 48 tiles, and there seems to be another level of indirection there, so one delta matrix is needed to go from the 2 m tiles inside the embark to reach the embark level where your matrix is needed to reach the world tile to use.
Thus, the 0 never happens for your matrix, but it does for the one further down, which, doesn't have a 9 in the other end, and the lower level matrix seems to use a different interpretation as well. Definitely confusing...
Again, your code has been very valuable. Without it we'd still be trying to get empirical measurement data to work, and those measurements were too crude as it only showed biome, not individual rainfall, etc. values. In addition, the samples needed to gather that would be huge. Without your matrix I'd never have figured out neither that a neighboring reference tile is used, nor how to reach it. Your matrix provided a starting point for trying to understand the confusing "biome" number patterns as well. The only outstanding issue with your code is my failure to understand what the OFPOS and the bit manipulation does, and so have replaced it with a test versus 200 which the code sort of seems to be performing.

Anyway, I've made a new version, 1.6, which I haven't yet tested further than running it on my 4 random test embarks and seen that it looks reasonable. I've yet to actually test whether the biomes displayed are actually the ones present, though.
Logged

Fleeting Frames

  • Bay Watcher
  • Spooky cart at distance
    • View Profile
Re: DFHack script: showbiomes
« Reply #32 on: October 01, 2016, 05:22:33 pm »

Something off?:

When messing with cold shears, found some swimming banded knifefish in my water that didn't freeze (River/lake vermin fishes).("Hm, filling an ocean/lake biome shear with artificial pool could perhaps be useful for safe FPS-efficient fishing variety and useful application of this script." (inconclusive test))
DLed from dropbox, checked 1 1x1 embark, saw several temperate and artic oceans in the sky.



None of them turned out to be oceans upon inspection with farm plots, though. All the same usual 5.

In happier news though, it gets the surface of at least 1 of the 1x1s correct.
« Last Edit: October 01, 2016, 09:22:15 pm by Fleeting Frames »
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #33 on: October 02, 2016, 02:58:06 am »

My current plan (valid until it hits reality) is to check out whether the surfaces of my test embarks are correctly identified, and, if so, test a few more. If they're not correct I'll try to figure out why, of course.

The step after that would be to investigate the huge number of oddities reported in the sky. The criterion used to detect ocean is simple: the elevation value is < 100. Logically, something higher up shouldn't have a lower elevation value, of course, but logic and assigned values may not necessarily be the same thing. A positive case is that ocean shore is correctly identified as ocean while the adjacent grassland (or whatever it was) on the same z level is not ocean, so in that case different elevation values for tiles on the same Z level is good.

This also points towards a potential need for an elevation value display.
Logged

ragundo

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #34 on: October 02, 2016, 03:21:13 am »

The issue is that I'm operating one level further down, i.e. inside a fortress where each embark level tile is further divided into 48 * 48 tiles, and there seems to be another level of indirection there, so one delta matrix is needed to go from the 2 m tiles inside the embark to reach the embark level where your matrix is needed to reach the world tile to use.
Thus, the 0 never happens for your matrix, but it does for the one further down, which, doesn't have a 9 in the other end, and the lower level matrix seems to use a different interpretation as well. Definitely confusing...
Well, for larger embarks, using "embark" biome map that allows you to quickly categorize fortress tiles inside, outside or in the border of a biome. The problem is in the border area between 2 biomes, where a embark tile expand to 48 fortress tiles, and you need to know  to what biome a fortress tile belongs to. For smaller ones it will be tough.

EDIT : I'll check probe plugin logic to see how it determines the biome type. I remember located the sub that wrote the biome descriptions to the screen in IDA long tome ago.
If you don't solve this problem, I'll try to research a little in IDA about this stuff.
Good luck
« Last Edit: October 02, 2016, 03:53:37 am by ragundo »
Logged

feelotraveller

  • Bay Watcher
  • (y-sqrt{|x|})^2+x^2=1
    • View Profile
Re: DFHack script: showbiomes
« Reply #35 on: October 02, 2016, 03:43:30 am »

Looking good on the 3x3 embark I have been using.  All land biomes are correctly identified.   :) 

'h' now shows the non-sheared higher z-levels as being shrubland (rainfall/drainage) which makes sense and includes a tropical/temperate division 2/3 of the way down the embark as a straight line across the embark tile boundary.  Not conclusive yet but it's looking like success on this front.  (Not sure if it's generalisable but the clear demarcation could provide the most exact answer to the latitude question yet given but thats sort of off-topic...)

However... (hopefully this is not as big as a however as last time) the river and murky pools are all showing as completely temperate.  Almost certain that this is wrong (for the river at least) as all the creatures I have seen (playing this embark -slowly- 1st caravan imminent) are drawn from the pool available for Tropical Lakes (I've always understood this to provide the creatures for rivers).  If the pools follow the land biomes I would have expected them to be a mixed bag, but then I would have expected the river to be mixed as well but it seems like the latter might not happen.  The river on my embarks flows in through 2 embark tiles of temperate grassland and out through one e-tile of tropical moist broadleaf forest.

Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #36 on: October 02, 2016, 05:50:14 am »

A straight line along an embark tile boundary is a sign of either an error in the script or odd data in the world. The world generation apparently "bleed" biomes to generate undulating "blob" boundaries. At higher elevations it seems the "sky" biome (whatever that is) takes over except for an area around where the ground pokes up into it. Why the ground biome isn't used all the way up to the top of the sky is an unanswered question, though. It might have to do with cloud and wind behavior.

Neither rivers nor pools are actually handled by ragundo's code, because these features do not appear on the pre embark level, so I've hacked those by looking for appropriate tile types (murky pool, riverX feature, etc) and have then invoked ragundo's logic for lake tropicality determination for those tiles. It is very possible this doesn't hold up to closer examination, so it's a point that should be investigated. Similarly, ragundo's code for lake detection doesn't work on the fortress level, because it checks an "is lake" flag on the corresponding embark tile, which isn't what we want (every fortress tile within that embark tile would be marked as lake). It might be that the logic used to determine tropicality "home" tile could be used here as well, though, but for the time being any water that's neither ocean, pool or river/brook/stream is labeled as lake, including any artificial pools on the surface.

Anyway, my test embarks identify biomes correctly on the surface, and I've looked at a couple of others as well without detecting any issues.
I've started looking into shears, in particular the ocean ones, and those I've looked at have a region id that points to an ocean region, as well as having an elevation below 100 (I've added two new views, one for the elevation and one for the broad biome (which labels each region as either Desert, Glacier, Hills [whatever that is], Jungle, Mountain, Ocean, Swamp, Steppe, and Tundra [I've shifted the order to the alphabetic designation order I use in my view]). Taiga would thus fall under Jungle (there is another enumeration with the same elements in the same order but with slightly different names: Jungle matches the more logical Forest).

Edit:
Pushed out version 1.7. Two new views and updated help text, but no bugs fixed.

Edit 2:
- Lake detection was bugged, and does now seem to be fixed. Also switched the characters around so the "smaller" one is temperate ('<'). This should also affect brooks and pools.
- Changed all polarities to use a definitive tropicality test vs 200 rather than odd code that doesn't seem to work.
- Looked at shears, and the biomes they claim to be match their parameters along all dimensions I've looked at, but what you can grow on them does not, but it does not necessarily match with what's on the ground below either (although in the cases I'm looking at it does match one of the biomes at the ground). I looked at vegetation as well, but it matched rainfall as usual.

@ragundo: The scripts relies heavily on the Probe code, which I've used as the basis for the logic. However, Probe doesn't actually determine the biome type, it just prints the lower level delta value. Unless some new major thing crops up it looks like the current script logic is working and what remains would be bugs and boundary cases. However, the reality has a tendency to strike you from behind...
Someone might want to update Probe to provide more info (such as the biome) at some time, of course.

Edit 3: Interesting find. I've dug through the C code to figure out why the provided operation fails to get data while Probe does find it, and the answer is actually that Probe just doesn't perform the same check as the other path does, so it happily uses a reference that's way out of bounds (legal range is 0 - 8, while values can be things like 100). Thus, it may provide data for something, but not the tile probed...

Edit 4: Using what looks like the original operation's C code implementation's recovery mechanism, to return the embark tile inserted on offset overflow, I end up with with biomes that look like the ones present at the surface level, but on closer examination they sometimes actually display some parameters that differ, Thus, it may be a case of shearing in from a neighboring tile, but not with the silly data previously displayed.

Edit 5: Uploaded 1.8. Further testing on an embark with a silly checkerboard pattern above the embarks showed a match; tropical swamp above the embark (with none at it) showed papyrus, temperate grasslands above showed artichokes (this biome existed on ground level as well) and an extra dry broadleaf showed buckwheat.
« Last Edit: October 02, 2016, 02:52:40 pm by PatrikLundell »
Logged

Fleeting Frames

  • Bay Watcher
  • Spooky cart at distance
    • View Profile
Re: DFHack script: showbiomes
« Reply #37 on: October 02, 2016, 04:01:39 pm »

Looking at current legend in 1.8, it seems the differences between hilly and flat biomes are removed from list. Also, there's a typo - it is Mangrove Swamp, not Mangrove Marsh (embark is still detected correctly for marsh/swamp biomes).

As for shears, did a test on the third of those embarks...It showed joyous tropical grasslands where there was actually several different terrifying airspaces, testing one of which revealed different dry broadleaf than present at embark z. Is this what you meant by detection failure?

(Granted, checking is kinda burdened by the fact that evilness and biome not always matching up - ran into another example of this with joyous dry broadleaf earlier. Region-pops list-all even showed unicorn, though didn't ran enough tests to confirm lack of fun shrublands.)

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #38 on: October 03, 2016, 02:13:22 am »

Yes, since ragundo's code identifies the official set of biomes that's what's used (so the brackish swamps and marshes are gone as well). Since I just tossed out the original logic and replaced it with ragundo's, that's what it ended up as. It would be possible to add it back in with a little additional processing, but unless there's a desire for it I won't bother. I'm fixing marsh -> swamp of course. There's also a need to update the help text to better match the current state/understanding.

I don't quite follow you regarding the shear testing. Do you say the joyous grassland was displayed by the script for the shears or for the ground? Also, when saying there were actually several terrifying airs paces, how did you verify that? My investigations have led me to the conclusion that Probe is actually buggy, so the info shown for detected shears is a result of using corrupt indices (which is what led to the script reporting ocean shears that were not, in fact, ocean).

Detection failure happens when the biome offset is not outside of the legal range (which is what's currently used for detection), but still pointing at a biome not present at the surface. Such biomes should all be "reasonable", i.e. they should all be biomes present in at least one neighboring world tile. However, all biomes reported by the script should now be "reasonable", unless my understanding of the logic is incorrect. I intend to try to play with the detection logic a bit. If I understand things correctly, there can be at most 9 different biomes in an embark, so I'll try to see if I can just store them in a list as I detect new ones and then mark which ones are present on the ground and which ones are sheared in.

Currently I think the script is more accurate than Probe for detected shears, and the only way to check biomes is your farm crop range detection, with no independent means to detect savagery or evilness (it might be possible to use some DFHackery to create soil and wait to see what herbs and saplings appear [glumprong, feather-tree, sliver barb, whip vine, highwood, and possibly good/evil "grasses" [Sun berries won't show up as there won't be any pools], but that would take a fair bit of time).

@feelotraveler: Do you still get a straight biome line with the latest script version? If so, I'd like to take a look at it to try to figure out what's wrong. If the world was generated without rejections the world parameters plus a description of where to embark is sufficient, but if there were rejections I probably would need the save.
Logged

feelotraveller

  • Bay Watcher
  • (y-sqrt{|x|})^2+x^2=1
    • View Profile
Re: DFHack script: showbiomes
« Reply #39 on: October 03, 2016, 04:30:57 am »

PatrikLundell is satisfied after improving scripting recently.   ;)

Yes 1.8 shows the same thing for me.  Perhaps I wasn't clear but the straight line demarcation appears in the air above the embark all the way up.  (There is also about 5z 'pillar' of the surface biome extending above the central embark tile, was thinking that this might be an artefact of the embark wagon dropping in?)  The demarcation is not generalisable across latitude so presumably it is a coincidence of latitude and temperature present on this embark...  Oh, and the tropical river still shows as temperate: if memory serves 'river' biomes are just special, flowing, cases of lake biomes, but maybe there is more to it?  Anyway its not being detected right.  (I tried rudimentary testing for other rivers last night but gave up after 3 embarks in a row where rivers had no wildlife...)  Here's an image 1z above the surface:



Pretty sure there were no rejections.  This should work for you:

Spoiler (click to show/hide)

Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #40 on: October 03, 2016, 05:10:09 am »

Thanks feelotraveller,

I think the surface biome "poking up" through the "air" biome is normal, since I've seen it in many places. However, the tropicality split of the "air" biome looks odd, so I definitely should take a look at that.
Since you've got the river on the embark as well, I'll get a chance to look at that too.

PatrikLundell is annoyed at watching a script that makes rude gestures ;)

Edit: It's easier to calculate things correctly if the data is supplied correctly... The tropicality calculation for pools and rivers was fed nil rather than a boolean (and LUA is perfectly happy with that). Thus, that should be fixed in the next version. The erroneous shrubland designation as tropical above the line remains to identify. I double checked that the same set of crops can be grown both above and below the line, so it's definitely a script error.
(And it's good to hear that dorfs sometimes can find meals with stuff they like in it. Now, if they only did so routinely...)
« Last Edit: October 03, 2016, 05:50:25 am by PatrikLundell »
Logged

Fleeting Frames

  • Bay Watcher
  • Spooky cart at distance
    • View Profile
Re: DFHack script: showbiomes
« Reply #41 on: October 03, 2016, 08:04:24 am »

There's probably no need for hillyness distinction, since anybody that understands what it means well enough understands to check drainage or hilly biomes.

How did I verify that: Yeah, probe and farm plotting for one of them.

Probe, being bugged, can be considered unreliable of course, but nevertheless found dry broadleaf where tropical grassland was reported

Didn't consider using tiletypes(only muddying, and wasn't sure this would get aboveground crops - might be worth a test), but it might be worth a check in the weird case of mismatch of biome and evilness.

@feelotraveller airshears having straight lines occurs normally. Snow and water can be easily used to demonstrate the air being divided into 16x16 chunks per embark square.

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #42 on: October 03, 2016, 09:01:05 am »

Probe is bugged only for detected shears, and my script should give the same result as Probe elsewhere. If the script doesn't report the same thing as Probe outside of detected shears there's a bug in the script, since it should use the same logic everywhere else.
If we find such a case outside of a reported shear I'd like to investigate it.

Probe doesn't report the biome, so the only sure way to nail the biome is farm plotting, but if farm plotting results in something that differs from what the script reports it's of interest to investigate. Dry broadleaf rather than grassland is definitely something that should be checked, as the only thing that I know of that might be iffy in the underlying logic is temperate/tropical (the rest should thus be bugs in the implementation of the logic).

I use tile type to detect where the surface is, as well as where pools/river/brooks are. In addition to that, there's a check for magma, but I haven't embarked on any volcano yet. Apart from that, I don't use the tile type for biome determination.

The tropical/temperate line in feelotraveler's image is an issue at the moment, since farm plotting shows both parts to be temperate. The upper part uses the embark world map tiles as its references and the calculations come up with Tropical for that, while the lower part uses the tile to the south (closer to the pole on the southern hemisphere), and comes up with Temperate. While I found a bug in the double pole calculations (LUA doesn't truncate on 'integer' division, which C does) it didn't change the results. Changing the tropical/temperate definitive line by 1 changed all of it to temperate, but it also changed the ground tropical environments (including a shrubland) to temperate (both farm plotted as tropical). I have to put that one in the "too hard problems to return to later" folder.

Edit: I've pushed out version 1.9. Minor bug fixes. Detects magma, for instance. Also printed a line for each biome detected in the DFHack window. feelotraveler's biome division line remains unresolved.
« Last Edit: October 03, 2016, 02:46:23 pm by PatrikLundell »
Logged

feelotraveller

  • Bay Watcher
  • (y-sqrt{|x|})^2+x^2=1
    • View Profile
Re: DFHack script: showbiomes
« Reply #43 on: October 04, 2016, 02:56:20 am »

Thanks Patrik.

The river now shows correctly.  I'll have a look at a few others in different worldgens/embarks over the next day or so.  (OT: #$%^& game bugged out on me... and I didn't have a save for close to a season - oh, well caravan has gone, time for a new game   ;) - cue next worldgen).

The division line does seem to be a weird case.  The main script does not detect a shear in that embark but the new (and very handy I might add) information in the DFHack window says 'shear only'.  The other shears I have seen (thus far) are detected as shears and tend to occur - in my limited experience - geomorphically like 'certain things' on the other end of the map.
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack script: showbiomes
« Reply #44 on: October 04, 2016, 03:22:18 am »

Thanks feelotraveller,
I need more error cases to investigate to get further.

The info for the new output is produced like this:
- For each of the 16*16 tiles on the embark, iterate over the (0, 0) tile of that 16*16 block from the top of the embark to the surface and check the biome against a list of detected biomes. If not present in the list, add it.
- When generating the surface biome map, check each tile's biome against the list. If found, mark is as detected at ground level, and if not found, add it and mark it as detected at ground level.
- At the end of the surface map generation, print the list results. Thus, this shear detection does not use the corrupted data detection method, but rather collects the biomes present and marks the surface ones.

I fail to decipher what you mean by "geomorphically like 'certain things' on the other end of the map". Many of the corrupt data shears I've seen have been in the first block column, but I haven't seen any pattern to it.

The air biome line indicates air biomes follow different rules (unknown by me at least) from the surface ones. On the surface, a biome is centered at a single embark tile and can then bleed into any or all of the surrounding 8 tiles, but no further. In contrast, the air, when having no shears, is a uniform layer of the same biome, and all "home tile"references are 0, which means the tile to the NW (I've never seen anything but 0 for the air). I tested the theory that their "home" for tropicality determination purposes was the first world map tile listed for the region the biome belongs to, but in the division line case that would still result in tropical, and, in fact, the only other tile in that list would yield tropical as well, so that logic doesn't hold.
Logged
Pages: 1 2 [3] 4 5 ... 8