Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 10 11 [12] 13 14 ... 24

Author Topic: DFHack plugin embark-assistant  (Read 93710 times)

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #165 on: December 18, 2019, 10:11:48 am »

I just double checked and noticed that the value of the difference/discontinuity always is 4352 (which is 17 * 256 itself - that seems significant somehow), at least while iterating through the feature shell and for a world with a given size of 33x33 - for worlds with other dimensions the constant is likely to have another value.
Okay, tested it for a 257x257 world: The constant offset is 61696 - which is 241 * 256. Where 241 is (257 - 16) - the total dimension of the world minus the width/height of a feature shell...
Does that help?
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #166 on: December 18, 2019, 11:15:49 am »

I'm still on the drawing board, but have a first draft for the key generation (haven't run it yet, and have yet to start tackling the reverse direction, but it should be possible to decompose the sum into its 3 parts, and from there into the original parameters).
Basically the "new" logic is:
- Sum up all the indices in the feature shell rows prior to the current row.
- Sum up all the indices in the feature shells prior to the current one in the current row, taking feature shell processing direction into account, as well as whether it's a complete shell or a last, reduced size, one (and if going right to left the first shell is narrower than the rest).
- sum up all the indices in the currently processed feature shell, accounting for all the complications.

The key should be the sum of the three parts.

Unfortunately the observations don't help much, as the logic was fundamentally flawed in that it tried to generate indices based on the row length while inside a feature shell, which is plain wrong.

Edit: OK, I've finally managed to get a new version of the key/indices generation logic. Again, I haven't tested it against the real code, so it may still use an incorrect algorithm, although I'm reasonably confident it matches what I think it is.

Spoiler (click to show/hide)

P.S. The script contains a number of constant declarations within the functions. These constants ought to be declared outside of them.
« Last Edit: December 19, 2019, 02:07:40 pm by PatrikLundell »
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #167 on: December 21, 2019, 06:48:54 pm »

Thank you for the new version of the script - I'll integrate it as soon as I have finished some more query-tests:
I've thrown together a very basic but working query-plan system and successfully found a volcano in a aquifer zone within a small world, with the same matches as the existing matcher.
It's a very small test case but if everything scales as I think it should it will be as fast as I hoped. Now for the assiduity/drudgery of adding all missing queries related to all the already implemented indices.
And adding all missing indices - which might get interesting. And then adding all those queries...  :o
During this step I'll also test with a big world and less special criteria to see how it behaves performance wise. Probably I'll run the same test described above with a big world tomorrow, just to be sure.
« Last Edit: December 21, 2019, 06:51:50 pm by RedDwarfStepper »
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #168 on: December 22, 2019, 04:58:27 pm »

Just a quick update: Works just as fine when searching for a volcano in an aquifer zone within a 257x257 world.
Happy holidays - if those apply to you :)
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #169 on: December 23, 2019, 04:54:02 pm »

Merry Christmas to you too (I'm allergic to Political Correctness).

Good to hear the sanity check worked as intended.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #170 on: January 03, 2020, 05:38:28 pm »

Happy New Year!
Also: Good news everyone!

I just ran a test case for a big world with
evilness = neutral (# of embark tiles: 16.758.830),
metal_1 = silver (# of embark tiles: 16.423.370) and
metal_2 = copper (# of embark tiles: 15.411.174).
This is pretty close to the worst case possible in my test world (total # of embark tiles: 16.908.544). First I was a little shocked - it took more than 15 minutes to complete.
But then I realized that many embark candidates got checked more than once. After adding a check for that it just took about 50 seconds.
That is not as fast as I originally had hoped but still much faster than the current solution during a subsequent search for such broad terms.
For more limited searches it is a lot faster - the speed is proportional to the criteria with the fewest hits/manifestations in the world.
Next up: Integration and testing of the revised key generation.

Oh and: I think I found an edge-case one-off error in the original matching. When comparing the matches produced by it with those found by the new code there sometimes was 1 additional row/column that seemed correct on visual inspection. It seems to be related to embarks that would contain embark tiles, that have i/k = 0 or 15.
Following an example (at x:168 y:122 i:1 to 6 k:10) within the region here http://dffd.bay12games.com/file.php?id=14584
Searching for a 6x6 embark with aquifer: present and min_magma: Volcano
Spoiler (click to show/hide)
All affected embark tiles in that world
Spoiler (click to show/hide)

Or am I mistaken?
« Last Edit: January 03, 2020, 06:01:06 pm by RedDwarfStepper »
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #171 on: January 04, 2020, 06:28:11 am »

Happy New Year to you too!

I think you've seen the unfortunate effect of incursion handling. The first search excludes the edge tiles because they don't have the required info from the neighbor tiles in the next world tile available. The second pass is capable of making use of the info as it's been generated at that time.
However, if your storage works out as well as it sounds like, it should be possible to perform the job in this sequence:
1. Scan the world tiles, marking potential world tiles.
2. Scan all MLTs and produce preliminary MLT results.
3. Apply incursion effects to all MLTs and produce definite results.

For subsequent scans the steps would be:
1. Scan world tile data to mark potential world tiles.
2. Match MLTs. It may well be that this step can't produce any useful progress indication because the matching isn't performed in sequence, if I understand it correctly. However, it might still result in a "random" change of candidate world tiles into confirmed world tiles, with a final wipe of those that didn't get any matches.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #172 on: January 05, 2020, 06:41:45 pm »

Ah, I see - yes, that is what's happening.

I've integrated the revised creation of ids/keys and they are completely continuous, thanks again for that!
This also seems to speed up the indexing by 20% to 25% (compared to my naive id generation).
Now I'll have to find a way to balance to need for index optimization (which creates run-containers and shrinks the indices) during indexing with the time it takes: Letting it run with unoptimized indices is faster but eats up more memory. Optimizing frequently makes it slower but reduced the max memory footprint. I might get away with a optimization after every feature shell and only process those indices that actually changed since the last optimization - there are some experiments needed here.

To make a qualified statement about the possible steps for the final integration of the index-query-structure (that also needs to be more iterative/update-tick-friendly instead of ignoring the allotted delta-time) I'll have to better understand the incursion mechanic - I'll hopefully find the time during the next days to describe my current understanding/mental image to you to see if it is correct.
« Last Edit: January 05, 2020, 06:50:36 pm by RedDwarfStepper »
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #173 on: January 12, 2020, 05:47:16 pm »

Ok, I took another look at the incursion comment here https://github.com/DFHack/df-structures/blob/4388fbfb8f51be41777406c6e7c518f738c195c7/df.world-data.xml#L376
and also in the code of matcher (process_embark_incursion_mid_level_tile, process_embark_incursion) and survey (process_embark_incursion_mid_level_tile, survey_mid_level_tile + translate_corner) and it left me smiling confused.

So let us play a game of hot/cold or true/false.
I'll postulate some ideas/theories of mine (some of which might contradict others, as alternatives to real questions) on how I think incursions work and you initially only confirm or reject those claims. In a second step we can go into the details. Also I'll update this post to add more claims so that all of them are in one place and also mark the cold/false ones as such.
Also feel free to post your own claims :)
  • Every world tile can "receive" incursions to its corner and edge mid level/embark tiles from its up to 8 neighbors
  • Every embark/mid level tile can "receive" incursions from its up to 8 neighbors, even if they reside in another world tile
  • If the world tiles "lives" at the edge of the world (x=0/world-width || y=0/world-height) no incursion will be received from outside the map
  • Besides biomes almost all other characteristics can be incurred, there are some exceptions though (e.g. rivers, magma...)
  • The receiving world tile/mid level tile "decides"/"knows" from which mid level tile it receives those characteristics
  • The giving world tile/mid level tile "decides"/"knows" to which mid level tile it gives those characteristics
  • The incursion control data structure elements "belong" to the north and west sides of the tiles respectively, as well as the NW corner of them.
  • Every one of the 256 embark/mid level tiles of a world tile is eligible to "receive" incursions (please compare to 1. and 2.)
« Last Edit: January 13, 2020, 01:02:21 pm by RedDwarfStepper »
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #174 on: January 12, 2020, 06:19:56 pm »

1. Correct.
2. Correct.
3. Correct. But the blasted data still indicates that should happen, i.e. it isn't sanitized to only indicate the legal cases, so is has to be checked before being applied, and we have to go through recovery processes to determine what should happen then.
4. Correct.
5/6. Basically correct. The data structure specifies in which direction an incursion should go, so e.g. at the middle of the north side of a tile, there is either an incursion into the tile above, or an incursion from the tile above into the tile (with the exception of the edges of the world, of course, where there's always always a straight edge). Similarly for the corners: one of the 4 tiles incurs into the other 3. Thus, the logic couples the tiles together in dependencies for each edge/corner.
The data for incursions reside in its own structure beside the others.
7. The incursion control data structure elements "belong" to the north and west sides of the tiles respectively, as well as the NW corner of them.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #175 on: January 13, 2020, 01:02:42 pm »

Ok, I updated the post by adding 7.
And to be sure there is no misconception on my side I added 8. as differentiation to 1. and 2.
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #176 on: January 13, 2020, 04:59:07 pm »

8: Correct. The restrictions are in that you can't receive from non existent tiles (i.e. outside of the map), every "receiver" is matched by a "giver" in the corresponding neighboring tile (corners has one "giver" and 3 "receivers"), and there is no "neutral" choice: there is an intrusion in one direction or the other.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #177 on: January 17, 2020, 06:00:07 pm »

Ok, .... that is a lot of incursions :o which really might be the killer for this whole endeavor... As I originally thought incursions only happen at the edges/corners of world tiles.
We'll see.

I had another look at https://github.com/DFHack/df-structures/blob/4388fbfb8f51be41777406c6e7c518f738c195c7/df.world-data.xml#L376
and in connection with 8. I'm wondering: Is it sufficient to process the "giver", knowing where its data is going or is it necessary also to look at the receiving side as well?
Or to put it differently: Does the "giver" have enough data to determine where and what is passed on to the 1/3 "receivers"? And are the following rules
Quote from: PatrikLundell
There are also certain rules forcing ocean/lake biomes to lose edges to mountains, and both of them to anything else, no matter what the original array value is.
mentioned in the comment on the world-data structure already "encoded" in the data or do we have process them ourselves?

If processing the "giver" is sufficient all internal and all outgoing incursions of a world tile can be processed when the world tile is being visited during the survey phase.
All incoming incursions from neighbors could be processed during the first and sole visit of those neighboring world tiles during the survey phase.
And embark matching could be delayed until the world tile and all of its neighbors have been surveyed:
After surveying a world tile one could increment a surveyed_neighbors_counter of every neighbor and when it reaches 3/5/8 (depending on the position of the neighbor => world corner, world edge or "inside") all relevant data is present so that matching of that world tile can be processed. That would result in a short delay of match processing for most world tiles, only those living at the edge of their feature shell might have to wait till the next row of features shells is being surveyed.

If we need to look at both, "giver" and "receiver" ah - I'll have to let this one simmer a little longer, good ideas take some time and looking the other way.
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #178 on: January 18, 2020, 01:20:44 pm »

The "giver" has all the info on what's given, so if all the info is available, it's possible to do the incursion processing when needed, rather than up front.

Unfortunately, DF does not adjust the incursion data to match the incursion direction that actually takes effect in the exception cases, (world edges, lakes, oceans, mountains), so they have to be calculated (and in the case of corners you'll basically have to look at all 4 tiles for that).

You can't process all incursions in a single pass, because neighboring world tiles will have to be processed one before the other, and both will typically produce incursions into the other one, unless the processing allows incursions to be introduced before the "real" data for an MLT is introduced. Also, you can't just merge things together as you go, as that would cause incursion info into a tile to be propagated into further tiles as part of the "real" data for that tile. Essentially you have to keep the "real" data separated from the compound info until the "real" data is no longer needed for any further incursions. This can be done with temporary structures for the "real" data that's discarded when all the merging has been done, for instance.
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #179 on: January 19, 2020, 04:03:11 am »

A new DFHack version has been released. This also means all changes made to this plugin since the last release are released as well.
I'll try to remember what has been done over this period...
Improvements:
- Incursions. This is the big change. And "incursion" is a piece of a neighboring tile's biome that juts into the current tile, allowing for a single Mid Level Tile to host more than one biome, with its attendant features (aquifer, evilness, etc.). Thus, a single MLT can have a partial aquifer, both Evil and Good alignment parts, multiple biomes, etc. The detection logic makes use of incursions where it's deemed to make sense, so e.g. mineral presence does not make use of incursions, as the amount of the minerals would be rather small, and probably disappointing.
Incursion handling has resulted in some unfortunate compromises, however, as the logic can't apply them properly to the MLT's at the borders of world tiles due to mutual processing dependencies, so the first search does not match any embarks including tiles from world tile borders: a second search is required to catch those.
- Waterfall detection searches now allow for searches for falls of given number of Z levels, rather than just presence.
- Evil weather has been added to the pre embark information.
- Added an optional "fileresult" parameter for crude external harness automated matching support. If the parameter is present on the command line when the plugin is activated the search profile stored to file is read and a search using those parameters is started and repeated once to account for incursions as well. Thereafter the number of world tiles containing matches is written to the file <DF>/data/init/embark_assistant_fileresult.txt. This file is deleted before the first search is started, which allows for an external harness to detect the removal of the file, indicating the start of the search, and the creation of the file, indicating its end.
Note that the author cannot provide any assistance when it comes to harnesses due to a lack of knowledge.
- The world tile movement logic has been improved, speeding up subsequent searches substantially.

Bugs fixed:
- A bug caused a crash on worlds without generated metals. This has been fixed (as well as the implementation doing what it was supposed to do).
- A bug caused mineral matching to fail to cut off at the magma sea (a flaw that DF's integrated search has as well). This has been fixed.
- A bug causing about half of the river tiles to be missed has been fixed. This also affected waterfall detection.
- A bug, or rather failure to understand and take the weird way DF implements rivers into account caused a fair number of river tiles to be missed (also affecting waterfalls). This has been fixed.

I generally avoid double posting, preferring to adding "Edit" sections to the existing post, but the contents of this post doesn't have much in common with the previous one.
Logged
Pages: 1 ... 10 11 [12] 13 14 ... 24