Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 15 16 [17] 18 19 ... 21

Author Topic: DFHack plugin embark-assistant  (Read 54173 times)

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #240 on: August 05, 2020, 04:07:14 pm »

Yes, that's the info my image was based on.
The one coordinate the neighbors share has to be the same, yes, with the other coordinate shared with the appropriate neighbor in the other dimension. As you can see, the split_* matrices only has a single value for each tile, reflecting this sharing.

If my memory is correct, Toady mentioned the name of the algorithm used to create the incursion boundaries, but I don't remember the name and don't think I've looked it up to understand it either. I believe this mentioning was done during the last year or so, but don't know which forum it was in, although I think it was written.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #241 on: August 05, 2020, 04:51:40 pm »

I wrote the split_* of an world tile into a file (see below) and I have to confess: I can't see the sharing in those values, not really -
here it says
for split_x:
Quote
for horizontal edges, x=min y=max
So there are two values for every split (for example 15,36 which means in the case of a split_x that the split starts at x = 15 and ends at x = 36, yes?) and I'd expect, that (15,36) would be repeated in the next line again - wait - ah.
I think I get it: The splits "lie" between the tiles, starting left/above the first tile and ending after the last tile right/bellow, so two tiles share the split between them.
Looking at the data between to world tiles I see the same splits at the shared edge, so there is some redundancy - which is good I might not have understood it otherwise :D

So taking a look at split_x data below at (0,1) => 16,33
This split is shared between the tile at 0,0 and 0,1, yes?
And the split at (0,0) => 15,36 is shared between the embark tile 0,15 in the world tile above the current tile and the local embark tile at 0,0, right?

split_x:
Spoiler (click to show/hide)
split_y:
Spoiler (click to show/hide)
« Last Edit: August 05, 2020, 04:53:34 pm by RedDwarfStepper »
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #242 on: August 06, 2020, 03:25:57 am »

Yes. The 17:th value should be a copy of the first one for the next world tile, and so should be redundant (but remove the need to get the data from the next world tile).

As far as I understand it (which might not be correct, which is an important caveat), the x, y values don't say much about where the border is crossed, but rather team up with the other values to form 4 points within each tile. Each of those points then get four paths from them, one for the nearest horizontal edge, one to the nearest vertical edge, and one to each of the two neighboring anchor points, with the paths generated through an algorithm. I think the algorithm can actually result in lines that go "inwards", so the anchor point is further away from the corner on one axis than the point at which the edge is crossed, and I believe I've seen incursions split into several small sections because the line effectively hit the edge.


                   1              2               3              4
   012345678901234567890123456789012345678901234567
 0 000000000000011111111111111111111111111122222222
 1 000000000000001111111111111111111111111112222222
 2 000000000000111111111111111111111111122222222222
 3 00000000000*111111111111111111111112222222222222
 4 0000000004444444444111111111111444*2222222222222
 5 ...

0 = Incursion from NW, 1 = Incursion from N, 2 = Incursion from NE, 4 = Self, * = Anchor point.

The first * would be formed from split_x [self][self].x and split_y [self][self].x and the second one from split_x [self][self].y and split_y [self + 1][self].x. In the "image" above I've come up with two (short) paths to the northern edge from the two northern anchor points, and some hint of the other paths.
Also note that it would be possible to make these paths regardless of whether they're used on not (the latter would be the case for an incursion going outwards).
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #243 on: August 08, 2020, 05:17:25 pm »

The first * would be formed from split_x [self][self].x and split_y [self][self].x and the second one from split_x [self][self].y and split_y [self + 1][self].x.
So if I got that correctly this would mean, that
the third * would be
split_x [self.x][self.y + 1].x / split_y [self.x][self.y].y
and the fourth * would be:
split_x [self.x][self.y + 1].y / split_y [self.x + 1][self.y].y

Correct?

This might help me to check for plausibility on how south-east corner incursions at the edge of the world behave - if I find a proper candidate that does not have an incoming incursion on the edge connected to the relevant corner. Otherwise it's hard to tell which incursion really brought the aquifer/clay/sand.
« Last Edit: August 08, 2020, 05:26:10 pm by RedDwarfStepper »
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #244 on: August 09, 2020, 01:06:58 am »

Yes, that's how I believe it works.
Logged

A_Curious_Cat

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #245 on: September 16, 2020, 06:14:16 pm »

Hey, I've got a quick question here:

I seem to remember that when I first installed DFHack for this version (and embark-assistant along with it), that the number after the text "Matching World Tiles:" in embark-assistant was yellow while embark-assistant was still searching (but hadn't found anything) and then turned either green (if it had found something) or red (if it had searched the whole map and found nothing).

Then when I upgraded to a newer version of DFHack (and embark-assistant) it seems to have been changed so that the number stays yellow no matter what.

I'd be interested in knowing the reason(s) behind this change.

Thanks in advance!

Edit:

Found something else interesting while doing a quick test to ensure that it really did leave the number yellow after having found matching sites:

It looks like embark-assistant searches most of the map using 4x4 squares of world tiles, then when it gets to 4 world tiles from the bottom, it switches to using 3x3 squares, and then for the very bottom row switches to 1x1 squares.

Interesting...
« Last Edit: September 16, 2020, 06:50:48 pm by A_Curious_Cat »
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #246 on: September 17, 2020, 01:30:47 am »

The expansion to take incursions into consideration made the distinction between a tentative match and a full match somewhat iffy (full processing of a world tile relies on the surrounding world tiles being processed, which means either doing two passes automatically to catch everything, or let the user do the second pass, which is the current implementation).

The processing of world tiles processes one "feature shell" (16*16 world tile blocks) at a time to minimize DF's feature shell loading, one feature shell row after another, with alternating rows going towards the right and to the left, and within the feature shell the processing is one column at a time, alternating between down and up. At the right edge the feature shell is 1 * 16 world tiles, at the bottom it's 16 * 1, and at the lower right corner it's 1 * 1. The reason for the convoluted movement process is to minimize the generation of region_details info by moving a single world tile at a time (the process to move to a different tile is done by simulated key strokes sent to DF, as the author is not aware of any other way to get DF to shift its focus to a specific tile), so each world tile should be visited exactly once.
The progress is then displayed as callbacks to the display functionality once every so often to provide the user with some feedback during the lengthy process.
Logged

A_Curious_Cat

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #247 on: September 17, 2020, 06:48:55 am »

The expansion to take incursions into consideration made the distinction between a tentative match and a full match somewhat iffy (full processing of a world tile relies on the surrounding world tiles being processed, which means either doing two passes automatically to catch everything, or let the user do the second pass, which is the current implementation).

What is meant by "or let the user do the second pass"?  Does this mean I should run embark-assistant's find function twice?

The progress is then displayed as callbacks to the display functionality once every so often to provide the user with some feedback during the lengthy process.

I'm still wondering why (even after the process is done) the number still stays yellow instead of turning red or green like it did in the previous version.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #248 on: September 17, 2020, 10:43:46 am »

The expansion to take incursions into consideration made the distinction between a tentative match and a full match somewhat iffy (full processing of a world tile relies on the surrounding world tiles being processed, which means either doing two passes automatically to catch everything, or let the user do the second pass, which is the current implementation).

What is meant by "or let the user do the second pass"?  Does this mean I should run embark-assistant's find function twice?
If you are looking for absolutely all matches than yes, that is what you need to do.
After the first pass it is possible that some matches have not been found. These possible matches are at the outer edge of world tiles and depend on the incursions from other world tiles, which are only available in subsequent passes after the first one.

I'm currently working on a variant of the existing search routine that finds all matches in the first pass, but it's not done yet -

@PatrikLundell: but i'm pretty close now, the iterative search (one world tile at a time as soon as all necessary data has been collected) during the survey phase works, the additional work that needs to be done to get all incursion data during the first pass has been moved to another thread where possible, which results in a 2 minute speed up => ~10 minutes for a 257x257 world.
I postponed any research regarding south-east corner incursions at the world edge till after all features are there - for growing world sizes this case becomes less and less important compared to the total number of embark tiles...
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #249 on: September 17, 2020, 11:36:14 am »

There is a bunch of help pages in the plugin. Those should describe most of the functionality. In order to get the fully correct search result you should run your first search twice. After that the incursion processing dependent info has been collected and cached, so any further searches yield the fully correct result (the second search does not have to be a copy of the first one, but can be something else).

The reason the results after the first pass are yellow is that they're not definite.

@RedDwarfStepper: Looking forward to see what you've come up with. I agree looking into a special edge case is better done after the bulk has been done (as long as that edge case doesn't have the potential to overturn the whole thing, which isn't the case here [as it's also a literal edge case]).
Logged

A_Curious_Cat

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #250 on: September 17, 2020, 12:16:41 pm »

Code: [Select]
Loading bindings from data/init/interface.txt
New window size: 800x300
Font size: 10x12
Resizing grid to 80x25
Resizing font to 10x12

Resetting textures
DFHack is ready. Have a nice day!
DFHack version 0.47.04-r2 (release) on x86
Type in '?' or 'help' for general help, 'ls' to see all commands.
matcher::find: Preliminarily matching World Tiles: 246
matcher::find: Preliminarily matching World Tiles: 0

Note: last line should be in red.

This is the output from my latest run of DF, DFHack, and embark-assistant.  I started by genning a world.  I then loaded that world and ran embark-assistant's find function on it.  As you can see from above, it initially reported 256 preliminary matching world tiles.  Unfortunately, it failed to find any matches.  Then I ran the search a second time.  This time it immediately returned with a red message saying that there were 0 preliminary matching world tiles.  From what I can tell, it does not appear that any second search even ever took place.  Embark-assistant seems to have returned from the second search without ever conducting it.
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #251 on: September 17, 2020, 12:57:16 pm »

A speculation, without looking at the code, is that the original logic that aborts search on 0 possible matches might remain, while it should perform the second search (but not further ones) to finish the job. There's code in there to check tiles known not to have any matches if the second run hasn't been done yet, but the original assessment might cut that short in the zero match case.

If that guess is correct, it's debatable whether the text should be yellow or red, because the tools would then "stupidly" go away for a long time to search even if it's already known there are no matches (when it works as it should).

Edit: Yes, the code skips out of both the first and the second survey when the preliminary matching rules out the presence of any candidates (although there aren't that many realistic searches that would fail before the first one).
Thinking it over again, I believe it should skip out as quickly as possible (i.e. current behavior), as it would allow the user to refine the search for the "real" search process, rather than wasting time building up search info.
As far as I can see from the code, the second line in A_Curious_Cat's case should indeed be returned in red, as it's printed with "out.printerr", which does print in red (as opposed to the "normal" case which uses "out.print" which doesn't change the color). Trying to replicate the problem fails for me, as all the cases of 0 matching world tiles come out in red, so it might be an issue with the DFHack terminal (I've had various issues with the Windows one in the past, in particular with getting color to behave correctly).
« Last Edit: September 18, 2020, 02:59:17 am by PatrikLundell »
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #252 on: November 10, 2020, 06:42:26 pm »

Ok, I think I'm on the final stretch
And a browser crash ate the previous, half-finished version of this post :)

Anyway on my second screen is an ugly, multicolored spreadsheet, that lists all ~40ish criteria that need to be implemented in this version of the plugin (I'm still working with 0.44.12-r2, planning on adding all new criteria once I've got all the old ones working).
Some 20 have a cyan background which means they are implemented and have passed a first series of tests.
4 are a (not really) beautiful magenta, telling me they need to be tested more thoroughly*.
The rest are yellow, they still need to be implemented.
To verify the results of the new search I use the results of the existing code and diff them with the new ones.

*Of course, the first time I wrote this I just had done some more tests that brought up some interesting questions:
- Is it by design or by organic/historic growth of the code, that incursions don't count to the fulfillment of the "all" option of savagery, evilness and aquifer ("all" became "light" or "heavy" here)? And in extension also the "absent", "not all" and "partial".
In matcher::embark_match there is a block of tests concerning savagery, evilness (line 617, line 630 and line 638) and aquifer (line 663) before the incursion handling is done -
if the embark tile doesn't contain the "resource" natively there is an early exit with a negative result. In matcher::process_embark_incursion there are the equivalent checks (line 95, line 111 and line 145) but those can't be reached because of the early exits in the calling method (=> process_embark_incursion_mid_level_tile => process_embark_incursion).
Please tell me this is not by design, I'd really hate to need to change the index code to adapt to this. :D
- The second question is more of a finding, which I wanted to run by you to make sure I'm not missing something:
I have a small world I use for tests (find the save here), and searching just for sand:present results in less world tile matches the second time around, where one would expect more results. Looking into it I found two reasons/flavors of the same problem:
1. All *_count fields of region_tile_datum get initially set in survey::high_level_world_survey (line 807  + line 864) depending on the values of geo_summary.
This represents potential matches, e.g. a world tile might contain sand (...) based on the layers it contains.
Then the *_count fields get reset in survey::survey_mid_level_tile (line 1284 + line 1300) based on the actual data of embark tiles.
For all world tiles that can potentially contain sand (/any other feature that can be incurred) but actually have no embark tiles that natively contain sand this leads to less matches in all subsequent searches after the first one as they are no longer considered a preliminary result in matcher::world_tile_match (line 1671)
For this case there is an "easy" fix, one could add something along the lines of
if (result.sand_found && region_tile_datum.sand_count == 0) {
   region_tile_datum.sand_count++;
}
in line 1263.
Which read like this: If there are any sand (/...) incursions but no native occurrences then the world tile still contains sand (/...).
2. If a world tile doesn't even potentially contain sand (/...) natively than the above fix does not work, it will never be considered a preliminary match and thus never be included in the matching.
In those cases there would have to happen a "lookaround", as the world tile could get incursions from neighbornig world tiles. This of course can only happen in subsequent runs after the first, otherwise the data is not available.

I haven't looked into this "lookaround" any further, not having a lot of time and being unsure how big a change it would be. But not having the original results to compare the matches of the new search against is unsatisfactory :)
I might have a go on all non-incurred features first and come back later to this...

But most importantly: Can you reproduce my findings with the provided save? And if so: Whats your take on them?
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #253 on: November 11, 2020, 05:23:15 am »

-Incursions vs savagery, evilness, and aquifers: Those are conditions that ought to take incursions into full consideration.
  - As far as I can see, lines 630 and 638 are reached only on absence together with the demand for All which looks correct to me?
  - Similarly, if a Light aquifer is demanded (rather than e.g. At_Most_Light) but isn't present, I believe the criterion not to be met?
  - The checks on incursions ought to check what the incursions contain. If the check failed because the native tile didn't meet the All criterion you shouldn't get here, but if it did, you should fail if the incursion doesn't meet the criterion, even if the native tile did.
  Thus, it looks correct to me, but you've found bugs it took me some time to recognize in the past...
  So, as per the request, I tell you that the design ought to take incursions into full consideration for savagery, evilness, and aquifers, and I think it does ;)

- Sand (which is, of course, only a representative of similar things):
  - The initial sand tile count can be too high because erosion can't be taken into account at that stage, but this value shouldn't carry into the results of the first match, as the first match should evaluate every tile unless it finds that it should be impossible to get any match at all (i.e. immediately reporting 0 matches), and the matches reported should be the verified correct ones (i.e. not counting the ones where incursion processing has to be done at the second match).
  - I believe you're correct in that logic to propagate sand provided by incursions from neighboring world tiles is missing, causing subsequent matches to fail to find some of them.
(I wonder if this is the problem A_Curious_Cat had but failed to reproduce?).
- I think this issue means the second search has to be complete as well, i.e. the logic that causes all world tiles to be surveyed on the first pass ought to be extended to also include the second pass. If the first pass cached the native results (i.e. your work), the second pass should be able to be performed using cached data only.
I'm not sure where the code making sure incursion effects are propagated to the world tile summary should go. It would require a bit of work to determine that.
- I haven't looked at your save, but I believe the above (based on your findings, of course) shows there is a logical hole that ought to be capable of reporting sites on the first pass that are then failed to be found on the second. At the same time, the second pass should be able to find sites that were not included in the first one because they hadn't been fully evaluated (which is intended). Logically, there should be cases where the first number ends up higher than the second one because of the bug.
Logged

RedDwarfStepper

  • Bay Watcher
    • View Profile
Re: DFHack plugin embark-assistant
« Reply #254 on: November 11, 2020, 06:14:12 pm »

tl;dr:
I started to write a reply and had to take a break, which allowed me to change my perspective.
I think we might have different definitions of "all".
At least the code and you saying
...
  - As far as I can see, lines 630 and 638 are reached only on absence together with the demand for All which looks correct to me? 
  - The checks on incursions ought to check what the incursions contain. If the check failed because the native tile didn't meet the All criterion you shouldn't get here, but if it did, you should fail if the incursion doesn't meet the criterion, even if the native tile did.
  Thus, it looks correct to me, but you've found bugs it took me some time to recognize in the past...
  So, as per the request, I tell you that the design ought to take incursions into full consideration for savagery, evilness, and aquifers, and I think it does ;)
make me think that.
I also think both definitions are valid, it might come down to taste and pragmatism, memory requirements for data storage might also be a small factor...

Your definition requires all native embark tile data and all incoming incursions to match the specified characteristic (e.g. high savagery), being exclusive, no other characteristics (e.g. low and medium savagery) in any form allowed.
This produces fewer total matches with embark areas that have either no incoming incursions or are situated deeper inside of pure fields of the required characteristic (e.g. untamed jungle biome all around the embark area).

My definition of "all" is less strict: As long as every embark tile in the embark area has at least one correct match (either native or by incursion) that is sufficient.
This will produce more total matches, some of which are less deep inside of absolute characteristic fields e.g. at the edge of a zone where other characteristics with outgoing incursions (= incoming into the embark area) can be found.
To retain the functionality of your definition I could add a "all exclusive" parameter or even distinctions "all_complete", "all_at_least_native" - the latter would make additional indices necessary as the native data would need to be stored separately from the incursion data - which I'm not a big fan of.
And by using "absent" on the other characteristics one can force the matching to behave just as strict as with your definition, while being able to distinguish more detailed e.g. evil:all + good:absent allows for neutral spots by the less strict definition.
There is also a rather small argument to be made about "all" and "absent" being more symmetrical if “all” is less strict defined – see below in the last paragraph of the original reply.

Could you live with my definition of "all" - at least for the time being?

Original reply:
Okay, lets take a broader look - first the definition of "all" also in comparison to the other requirements:
> All as a parameter means every embark tile has to have this value.
My interpretation of this is as follows: All embark tiles that are part of the currently examined embark area (e.g. 4 x 4 = 16 tiles) must contain the specified characteristic, either natively, via incursion or both to satisfy the criteria.
> Present means at least one embark tile has to have this value.
At least one embark tile must contain the characteristic, either natively, via incursion or both to satisfy the criteria (=> savagery_found[LEVEL]=true; embark_match / process_embark_incursion )
> Absent means the feature mustn't exist in any of the embark tiles.
No embark tile can have the criteria, neither native nor via incursion.

Looking in the code the way "present" and "absent" work are pretty clear. For the first one single hit (natively or incursion) is enough to satisfy the whole criteria resulting in a match, for the second one single hit (natively or incursion) is enough to prevent a match.
But looking at how "all" works, there is a difference: The hit must be native at least and if there are any incursions each and every one of those must have the correct characteristic as well.
Lets have a small example:
We have set the filter for High Savagery:all, the dimensions to x:2 + y:1 => 2 embark tiles.
Currently an embark area is being processed.
The first embark tile has native High Savagery and passes this condition.
All incursions going into the first embark tile also have High Savagery, and thus pass this condition.
The second embark tile also has native High Savagery and passes the condition in embark_match.
All incursions but one have High Savagery, one has Medium Savagery. This results in the embark area not being a match.

That makes "all" more like an "exclusive all", as it enforces that there can't be anything else but the specified characteristic.
Which is unusual as no other criteria defines requirements for other characteristics at least in the "old" version I'm currently working on, no sure about the two different variants of aquifers and forests in the current version...
This is also less intuitive as there is an explicit way to force this exclusivity: Using "absent" on all characteristics that should not be present.

The new code currently handles "all" as I understood it originally: All embark tiles must provide at least one hit for the requested criteria, either native or via incursion, but apart from that it is okay for them to have other characteristics either natively or via incursion as well.
This is symmetrical with "absent": a single hit for an embark tile is enough to (dis)qualify that embark tile, one missed embark tile disqualifies the whole embark area
Also this leads to more matches - which can be nice or not, really depends, but that can be controlled via "absent".
Logged
Pages: 1 ... 15 16 [17] 18 19 ... 21