One of the purposes of this thread is to pick up suggestions for improvements...
I haven't played DF before 0.40.X, so I have little idea of what they looked like. I don't know off hand how to locate volcanoes, but it ought to be doable. Magma pipes can probably be located, but I don't know how precise the info on depth is. A guess would be that the info is what the highest cavern reached is, rather than an actual depth in Z levels.
When it comes to search profile saving, I'm considering an active implementation, i.e. save and load actions (or export/import, or whatever names seem suitable given available command keys, etc.). The data would be saved into a file with a fixed name, so if you want multiple profiles you'd have to juggle that by renaming the files.
This is a really nice utility. I don't suppose there is a way to add volcanos/magma within a certain range of surface into it? I miss being able to search on criteria with volcanos from the older versions of DF. Particularly I recall finding surface or near surface magma more frequently with that.I don't remember that ever being a vanilla feature - what version was that?
This is a really nice utility. I don't suppose there is a way to add volcanos/magma within a certain range of surface into it? I miss being able to search on criteria with volcanos from the older versions of DF. Particularly I recall finding surface or near surface magma more frequently with that.I don't remember that ever being a vanilla feature - what version was that?
Metals 1 N/A
Economics 1 N/A
Minerals 1 N/A
(
Metals 1 IRON
Metals 1 N/A
) OR (
Metals 2 N/A
)
Economics 1 N/A
Minerals 1 N/A
(
Metals 1 IRON
Metals 1 N/A
) OR (
Metals 2 COPPER
Metals 2 N/A
) OR (
Metals 3 N/A
)
Economics 1 N/A
Minerals 1 N/A
(
Metals 1 IRON
Metals 1 N/A
) OR (
Metals 2 COPPER
Metals 2 TIN
Metals 2 N/A
) OR (
Metals 3 N/A
)
Economics 1 N/A
Minerals 1 N/A
I don't think it would be easy to get those alternatives in.
The Matching World Tiles should be that, i.e. the number of world tiles that have at least one match in them. The number starts high as the first pass counts all tiles that are not incapable of having a match (e.g. requiring clay, and there's no clay in any of the mid level tiles). The detailed pass then removes the ones that do not have any match in them.I think my confusion comes from the mismatching terminology. You're using "world tile" to refer to a tile on the middle map, and yes, that's the one where I see initial yellow Xs and final green Xs, and the number of those Xs does indeed match what's reported by the plugin on the right. The thing is, DF itself labels the middle map "Region" and the right map "World", while your plugin refers to the middle map as "World". So just changing the text to be "Region Tiles" would avoid the confusion. (I get that in the DF mod dev world there are various conventions for referring to world/region/local/fortmode tile zoom levels, but from the user perspective all they know is what labels are shown on-screen, so it's weird when those don't agree with eachother.)
Unless there's a bug, each matching world tile should be marked on the middle map (the one labeled "Region" by DF), first with a yellow "X" after the tentative pass, and then a green one when the final check has been made (assuming there was a match: otherwise the marker is removed, of course). If those markers are missing, there's a bug somewhere.
The reason there are no indications on the World map (the rightmost one) is that I haven't been able to replicate the logic DF uses to squash multiple tiles into one for display: I've come fairly close, but that's not good enough.From my very limited experimentation it seemed to me that it just depends on world size. In a "small" world there seem to be 2 region tiles per world tile; "medium" has 3, "large" has 5. I think. But I guess maybe they're not evenly divisible? I guess that'd be easy enough to test by just counting the dimensions of each map for each world size. Anyway, yeah, it's that right-hand world map that would be great to get Xs on; right now in a medium or large world if your search turns up just a few results, it takes awhile to hunt around and find them on the region map.
size region world ratio distribution
------ ------ ----- ----- ------------
Pocket 17 17 1 1,1,...,1
Smaller 33 33 1 1,1,...,1
Small 65 32 2.03 2,2,...,3
Medium 129 43 3 3,3,...,3
Large 257 51 5.04 5,5,...,7
worldX = MIN(regionX / (regionW / worldW), worldW - 1)realTiles = "number of real top-level tiles in the world in this dimension, i.e. 17, 33, 65, 129 or 257 for standard worlds"
regionTiles = "number of displayed (center) region map tiles in this dimension; dependent on window size"
normalWorldScale = "ratio of displayed (right) world map tiles to (center) region map tiles for all columns/rows except the last" = CEILING((realTiles - 1) / regionTiles
worldTiles = "number of displayed (right) world map tiles in this dimension" = MIN(regionTiles, CEILING(realTiles / normalWorldScale - 0.5))
finalWorldScale = "ratio of displayed (right) world map tiles to (center) region map tiles for only the last column/row" = realTiles - (worldTiles - 1) * normalWorldScale
regionCoord = "offset to a position in the (center) region map, from 0 to realTiles - 1"
worldCoord = "offset to a position in the (right) world map, from 0 to worldTiles - 1" = MIN(worldTiles - 1, FLOOR(regionCoord / normalWorldScale))
What does Toady call region tiles? It was something amusing, but I forget.Mid Level Tiles, if you use the most common interpretation of "region tile". That's the one I use since then, as it ought to be unambiguous. The most amusing term is probably "feature shell" though.
Edit: Hmpf! When testing I found DF behaving weirdly with a 129 Y dimension world: it changed the scale factor at 19 and 22 available lines as expected, but weirdly it only uses 18/21, leaving the World map one tile shorter than the Region one (and a lot of World Tiles mapped onto the tiles of the last row).
normalWorldScale = CEILING((realTiles - 1) / regionTiles = CEILING((129 - 1) / 22) = CEILING(5.82) = 6
worldTiles = MIN(regionTiles, CEILING(realTiles / normalWorldScale - 0.5)) = MIN(22, CEILING(129 / 6 - 0.5)) = MIN(22, CEILING(21.5 - 0.5)) = MIN(22, CEILING(21)) = 21
normalWorldScale = (realTiles - 1 + regionTiles - 1) / regionTiles
worldTiles = MIN(regionTiles, ((realTiles + ((normalWorldScale - 1) / 2)) / normalWorldScale))
The most amusing term is probably "feature shell" though.That was it! :P
Would it be possible to include neighboring races in the finder, i.e. "Elves/Humans/Goblins/Tower: NA/War/Yes/No", or is that information not easily available during the search?It's possible, but not trivial. It would require its own search pass (although it would have to run only once and it's a world tile level search), because it depends on distances from civs, which means both figuring out what the distances are, and checking for impassable terrain (including the passability resulting from a player fortress [and I don't know if other pass-through sites can be placed on "impassable" tiles apart from probably dwarven fortresses on mountains, and I'm not completely sure what sites ARE pass-through]). If I was to do it, I'd probably implement a fairly crummy flood fill check for each site of each civ, which wouldn't be particularly efficient. Another issue is that I don't know how DF decides which civ gets to represent each race, which would be required to implement a relations check (presumably war is determined by the mother civ being at war with them [which ties into the dwarven civ selection], but the logic would still have to determine if that civ is considered the closest one).
Searching does indeed take a long time. This is because it scanning a lot of data, plus that this data has to be loaded by DF itself. It can be noted that DF's own searches also take quite some time.
It should be possible to leave the search indications in the state they were when a search was cancelled, but I'd have to investigate what happens if you were to move the focus to world tiles that have not yet been searched. I suspect additional logic would be needed, and I wouldn't be surprised if you could get incorrect indications in such areas because they were based on data that hadn't actually been produced, in particular if the very first search is aborted.
Gotcha I will give it another go then.Another thing to try is to search for something that's very common, such a e.g. just the presence of a river in the form of a brook or larger. That should match most of the tiles, and thus should make it easy to see if (and how) it works in general. If you do that on a small world it doesn't take a long time...
Fuel: I have to look at it before answering, as such a functionality would require means to figure out how to determine if a mineral can be used as fuel (as the logic shouldn't be dependent on what's in the vanilla raw set).
An alternative solution to the fuel thing would be the ability to select multiple items at once in each category of economic stone (and why not other stones too), and when this is done the search criteria will be that at least one of them is present. This could be a fair bit more difficult to implement depending on how the search is implemented though, I wager.Instead of searching for a particular value, each geo biome element would have to be matched against a list, which should be about the same cost as matching another element (i.e. a list of two alternatives would cost about the same as two separate mandatory ones). The trickier part to implement that would probably be to change the list logic to allow for multiple selections while still being usable, somehow (as usual: it's typically the UI that causes the most trouble. For instance, the UI can't show what's selected, as there's room for only a single entry, so it might have to show something like "<Multiple alternatives selected>" instead.).
haven't tried, but can you add "trees" and "other vegetation" with many different values aswell?I don't understand the question...
it's just that in the vanilla site finder you cannot select anything about trees at all.As I said, biomes imply their general tree densities and the plugin also provides region type as a search criterion. This means that mountain present (doesn't matter if you select biome or region type, as there's a 1:1 mapping for this one) + region type "jungle" present would find you every place where forest (of any kind, as that is what "jungle" actually maps to) is present together with mountain within an embark (while also matching the other criteria, of course).
so in certain worlds, i would look for a site with no trees and certain other properties in vein, but the sitefinder cannot help much with it yet.
in others i want a steep mountainside next to a dense jungle and look at 100 sites just to find that there is no place on the whole map, but a sitefinder with trees min/max would be a great help with that.
I need plugin for embark directly on underworld spire.just edit d_init to show stuff not on "FINDER" but on "ALWAYS"
after the "mousequery edge" bug on 44.12 is resolved, i tried it out, loved the way it is now, but the game is still too prone to crashing.From this, it sounds like both versions are unstable for you, so I'm not seeing a clear reason to make a build for 0.44.09. Let me know if I'm misunderstanding. I also haven't seen a clear consensus on 0.44.09 being more stable than 0.44.12 (and why .09 and not .11?), but maybe I missed something.
so, could you make a version with the new features, but for 0.44.09-r1 ?
I need plugin for embark directly on underworld spire.Not totally sure what the issue is here, but two options:
It's not impossible to put together a DFHack release for an old build of DF. We've considered that before, but only for major versions (e.g. 0.34.11) and have never actually done it. It'd be a bit annoying to test, since saves aren't forward-compatible.well, the 44.09-r1 is stable for me and the mousequery bug was fixable by using a different fork of dfhack or twbt (i don't remember right now). i can't find the necessary files online anywhere, which is why i repacked that fixed/patched version as a backup for myself.after the "mousequery edge" bug on 44.12 is resolved, i tried it out, loved the way it is now, but the game is still too prone to crashing.From this, it sounds like both versions are unstable for you, so I'm not seeing a clear reason to make a build for 0.44.09. Let me know if I'm misunderstanding. I also haven't seen a clear consensus on 0.44.09 being more stable than 0.44.12 (and why .09 and not .11?), but maybe I missed something.
so, could you make a version with the new features, but for 0.44.09-r1 ?
For comparison: the weapon trap crash of 0.43.05 was something widespread enough that we would have considered supporting 0.43.04 too. However, that was infeasible due to the 64-bit changes in 0.43.05.
I feel like there was some longstanding bug in 0.34.11 too, but DFHack for 0.34.11 also supported 0.34.10 anyway since they were binary-compatible.
Have you read the help screens? Unless I've made a grave mistake, it should state that a match for the embark rectangle is indicated by the single tile up in the top left corner or the rectangle, with the other tiles in the rectangle indicating the rectangles with those tiles as their corners.
Have you read the help screens? Unless I've made a grave mistake, it should state that a match for the embark rectangle is indicated by the single tile up in the top left corner or the rectangle, with the other tiles in the rectangle indicating the rectangles with those tiles as their corners.so i'd have ti place the 3x3 with its top-left tile on the flashing tile?
The reason for this is that there is a need to indicate a match for any kind of rectangle, with a clear indication of where the matching rectangle is. You'd also note that with a 3*3 rectangle the two lowest rows and the two rightmost columns never show any matches (because that would indicate a match in non existent mid level tiles outside the area).
Note that there are search conditions that require multiple embark tiles with different conditions in different tiles, such as e.g. partial aquifers, so a "matching tiles" logic doesn't work.
after the "mousequery edge" bug on 44.12 is resolved, i tried it out, loved the way it is now, but the game is still too prone to crashing.Two notes:
so, could you make a version with the new features, but for 0.44.09-r1 ?
START:dwarfort -gen 99 RANDOM PROFILE
Launch dfhack
Open region99
dfhack-run search-script
unless result matches 'found 0 tiles' exit
dfhack-run die
rm -rf data/save/region1
run this workflow again
matcher::find: Preliminarily matching World Tiles: 16307and searching takes 9 minutes, 25 seconds to finish.matcher::find: Preliminarily matching World Tiles: 10421and takes 9 minutes and 3 seconds.matcher::find: Preliminarily matching World Tiles: 1037and only very few world tiles get highlighted with the black "X" on yellow ground. This plugin add ability to see buildings if I embark on site?Yes or no, depending on what you mean...
I just want embark directly on underwold spire.This plugin add ability to see buildings if I embark on site?Yes or no, depending on what you mean...
The smallest overlay allows you to see the presence of sites (caves, shrines, etc.) but not the buildings within those sites (display of caves can be enabled by advanced world gen, but the others are not displayed by DF). Thus, if you want to embark on a vault with the aim of digging down to it you can, but the search function does not provide any option to search for them: you're restricted to scour world tiles visually.
The easiest way to find a kind of site you're looking for is probably to write a script that identifies the world tiles where those are, and then use Embark Assistant to see where within that world tile the site is. The main purpose of the display is rather the opposite, i.e. avoiding accidentally embarking on a feature you don't want to mess with.
Candy spires are not sites, and thus not shown on the embark overlay. However, the plugin allows you to search for embarks with a given minimum number of spires, and the results will match only such sites.I want embark on dark fortress with candy spire in embark area.
There are 64 spires in each world tile, and they're randomly distributed within 2*2 mid level tile blocks aligned with the edges. Thus, a 2*2 embark flush to any corner will have exactly one spire, which randomly appears in one of the mid level tiles. If you're not aligned with the 2*2 grid the number is random, and a 3*3 embark will have 1-4 spires (it cannot align completely, so the number is random within that range.
The Region Manipulator tool allows you to shift spires within their 2*2 grids so you can increase (or decrease) the number of spires, e.g. 4 spires in a 2*2 non aligned embark (this also happens naturally, of course, but if you want it to match other search criteria as well it might not be present naturally.).
It's largely a matter of which situation you want to optimize for:
a) Minimize the time it takes to search a full set of potentially matching world tiles; or
b) Minimize the time it takes to search a sparse set of potentially matching world tiles.
Now, if the cost to a) of achieving b) is small, it may be worth aiming for b).
Everything is not retrieved for every world tile. If the top level check rules out a match the tile is stepped into and out of after the first scan, without any processing. On the first one the data that's collated on the world tile level has to be collected, and that basically means collecting everything.Spires is not visible. How make them visible?
The real memory killer here is the geo biome derived info, as each mid level tile has one boolean vector for metals, one for economic materials, and one for any non organic material, and I think those vectors are 5-600 long.
Another thing to consider is that the memory usage shouldn't approach 2 GB, as that would crash the 32 bit DFHack.
However, stepping onto a world tile takes time, as DF has to generate all the mid level tile information for the all tiles in that world tile, so skipping 10 tiles can save some time, and even more time can be saved if you can skip loading a feature shell (through a diagonal movement instead of one horizontal and one vertical in either order).
It can only be good to have more eyes looking at the code and trying to improve it.
@DerMeister: If you search for candy spires and then look at tiles with dark fortresses (they're clearly visible, so you don't have to search for them), you can select embarks that have spires. Since dark fortresses tend to take up a whole world tile, there should be 64 spires the search can locate. You can also use Region Manipulator to get the spires displayed as an overlay, and thus clearly visible.
Note, however, that dark fortresses tend to be full of goblins, which not only means there's a risk of rather rapid extermination of the embark team, but also an extreme slowdown due to the excessive number of units at the site. Also you'd have to use DFHack's Embark Anywhere to be allowed to embark on top of a settlement.
@DerMeister: As I said, spires are not displayed by this plugin. Instead, you search for matching embarks and embark in one of the matches.And how make spires visible?
The alternative, as mentioned as well, is to use the Region Manipulator, which does provide a display of spires: http://www.bay12forums.com/smf/index.php?topic=164136.msg7454345#msg7454345 (http://www.bay12forums.com/smf/index.php?topic=164136.msg7454345#msg7454345).
The script has a fair number of help screens. Read those. Hint: it's one of the overlays available through feature manipulation.@DerMeister: As I said, spires are not displayed by this plugin. Instead, you search for matching embarks and embark in one of the matches.And how make spires visible?
The alternative, as mentioned as well, is to use the Region Manipulator, which does provide a display of spires: http://www.bay12forums.com/smf/index.php?topic=164136.msg7454345#msg7454345 (http://www.bay12forums.com/smf/index.php?topic=164136.msg7454345#msg7454345).
I think this thread is probably the best place to discuss issues regarding the plugin: I don't think there's a huge crowd of people that would find lots of time being lost reading posts irrelevant to them, and there is at least some chance something posted might be of use to someone else. If there are protests about that assessment I'd switch to a separate thread in this forum.
Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.No, and it won't be. Doing that would require digging up a corresponding version of DFHack, modify that, and release such a version.
for me 44.09 is the most stable version. i was able to play 3 ingame years without a crash.Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.No, and it won't be. Doing that would require digging up a corresponding version of DFHack, modify that, and release such a version.
This, in turn, would require ensuring all the DF structures relied on being available in the old version (or added to it, after verifying they indeed were the same back then), etc.
I have no objections to you trying to perform that feat, but to get a release you'd likely have to do all the groundwork as well as persuading the DFHack maintainers to release your updated old DFHack version (and I'd secure a promise before starting to ensure the work won't be in vain). Of course, you can do the work "privately" by downloading a base version of DFHack to your computer and do the work there for your private use.
I'd recommend 0.40.24 if you're looking for stability, but I started with DF at 0.40.0X (6, 7?), so I've never seen the old version.
Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.No, and it won't be. Doing that would require digging up a corresponding version of DFHack, modify that, and release such a version.
This, in turn, would require ensuring all the DF structures relied on being available in the old version (or added to it, after verifying they indeed were the same back then), etc.
I have no objections to you trying to perform that feat, but to get a release you'd likely have to do all the groundwork as well as persuading the DFHack maintainers to release your updated old DFHack version (and I'd secure a promise before starting to ensure the work won't be in vain). Of course, you can do the work "privately" by downloading a base version of DFHack to your computer and do the work there for your private use.
I'd recommend 0.40.24 if you're looking for stability, but I started with DF at 0.40.0X (6, 7?), so I've never seen the old version.
However, I'm unable to see how you can compress what you mentioned into 200 MB, as my quick calculation results in: ... about 560 MBAh, that is the nice thing about Roaring Bitmap: It's an inverted index, so you only add those entries to the index which for example have a river or coal, which results in a lot less total entries than 257*257*256 for every attribute/aspect
Nevertheless, this might be a useful approach, although the speed impact will have to be investigatedAbsolutely, if it is even a little slower during the survey or the match phase it won't make any sense to pursue this approach any further.
Note, though, that there are currently 3 vectors over the inorganicsI already stored them separately... :D
Incursion processing is indeed complicatedYes, I see that now. I had a look at your comments here
Also, caching search result should probably be a auto-off toggle.Yes, I also thought that might be a good idea - but for the big (257*257) maps which would benefit the most the users probably have a beefy system with enough ram. Just loading the map needs more than 1 GB ram already.
OK, so if I understand it correctly ...You are absolutely correct, this would be some kind of database-table or -index design.
It can be noted that this implementation strategy change would force the use of storage of all criteria in memory, and great care has to be taken to ensure the memory usage is kept in check: I do not agree with the assumption that a full size world equates the 64 bit version of DF and hence "sufficient" amounts of memory, as you could create a normal sized embark in a full size world before DF provided a 64 bit implementation, and nothing fundamental has happened since then. The main reason to keep away from large worlds in the FPS drain caused by world activities. If a full world uses 1 GB (I haven't checked), that would mean this plugin would be required to stay well below 1 GB to work with the 32 bit Windows DF version.Again you are right - assuming that everybody has a lot of ram was a misconception.
I made the crude measurements (task manager) asked for:Thank you for taking the time to get these numbers.
Old New
DF only 1076.8 1023.1
EA started 1115.2 1579.0
Max 1139.8 16XX.X
Done 1127.5 1592.3
Time 5:50 26:50
I had a quick look at the changes, which were a lot lighter than I would have expected when the word "refactoring" is used.Hehe, I'm just starting to pick up speed - I don't want to break something right away. But we'll get there
I'm a bit surprised that the code would get smaller, though, as I had expected the compiler to be smart enough to factor out common parts.Yeah, me too. Actually changing variable names doesn't make a difference, but it seems function call (like .at()) aren't optimized away with the default project settings.
I have two style comments, though:Done and done.
- As opposed to the C crowd, I don't have a pathological aversion to typing, so I would have used readable identifiers to a much larger extent than the TLA/FLAC usage. (TLA = Three Letter Acronym, FLAC = Four Letter ACronym, which is not a standard one).
- I find camel case to be extremely ugly and instead separate words using underscores (or, occasionally, use Germanic [and DF!] word mashing [and those languages somehow do just fine without camels]).
:D Hehe, it always comes to the point of breaking something eventually and then taking a step back and going slower again and succeeding.I had a quick look at the changes, which were a lot lighter than I would have expected when the word "refactoring" is used.Hehe, I'm just starting to pick up speed - I don't want to break something right away. But we'll get there
tile->region_type[i][k] = world_data->regions[tile->biome_index[mlt->at(i).at(k).biome_offset]]->type;
i.e. "biome" -> "biome_index".I made the crude measurements (task manager) asked for:
Old New
DF only 1076.8 1023.1
EA started 1115.2 1579.0
Max 1139.8 16XX.X
Done 1127.5 1592.3
Time 5:50 26:50
I'd missed to add exportmap to own_scripts, and tweakmap wasn't the latest version in either place (fixed now).Thanks!
Edit:Ah, that sounds more like it - but damn - your system/CPU must be quite a little bit beefier than mine.
Finally got Release working (it help to try to run the right DF copy...). As you suggested, it showed a dramatically improved performance at 7:30, with the high and final memory usage a 1601.7 MB.
One complication is that DF uses a data structure with "feature shells", i.e. 16*16 world tile blocks, and loading those take a significant amount of time (about half of the total search time), which has caused me to ensure the world is processed one feature shell at a timeAfter removing all of real logic it still took around 8 minutes to iterate over the whole large map - instead of the 12 minutes with all the logic.
Taking care of the real world takes precedence over the virtual one...Very true, especially if the real world does not give you a choice about that :)
I'm just starting to look at what you're at, and probably misunderstand half of it, so the comments should be taken with that in mind.
- defs.h; inorganics: There are more than 256 of those, so they won't fit in unit8_t (assuming that's what's supposed to be stored).That seems true for my test world (but might even have been wrong there) but this was a put there to try improve performance where vector<bool> seemed slow - which they are in the debug build as some optimizations only are made for release builds. Has been removed.
- index.h: The plural of "index" is actually "indices".Changed - hehe, in my mother tongue it would have been me to point this out :)
- index.h: I don't see the point of creating maps of the names of various inorganics, as you can easily extract those from the DF structures. If it's some kind of backward indexing I'd just store the index of the corresponding inorganic (in the DF structure).I'm lazy and wanted to be able to look the name up during debugging and have it ready when writing the "report" at the end of the survey/index phase.
- index.h: As discussed earlier, it would make sense to merge the three (four?) inorganics vectors of maps into one, but it makes sense trying to get things to work first before trying to optimize them.Exactly my thought - the fourth vector (inorganics) will replace the three others as soon as I'm sure that I'm not missing anything.
- Index::setup: It seems you make the assumption all worlds have x and y dimensions that are the same. That doesn't hold (mine are 17 * 129, for instance).Oh boy - that one is - I never even knew that is possible!
- Index::createKey (and setup, maxKeyValue): Does the range matter? If it doesn't I'd just make the key out of the lower 12 bits of x and y, plus the lower 4 bits of i and k, with a sanity check to ensure the world dimensions don't exceed the limits (which they can't in vanilla, and it seems from another thread they can't approach those values). The next point indicates the value range does matter, though.The range of the key values by itself does not really matter - apart from the fact that there is an upper limit (MAX(uint32_t)) - but the value is currently used to reserve memory for the vector keys_in_order (of addition that is), which in turn is used to help test performance. Roaring works best if the keys come in strict ascending order. It still works fine of they are "random", but in that case it performs better if it gets a little help now and then by calling runOptimize and shrinkToFit. That is what you stumbled upon in your next comment.
- Index::add: It seems this operation relies on the order in which the keys are added, and that order does not work well with either DF's feature shells or an efficient processing of world tiles without feature shells (unless there's some way to move directly from the last world tile of a row to the first world tile of the next one). A comment explaining what's special with 511 would be useful: I can't figure it out.As said above if the keys don't come in strictly ascending order helping Roaring by calling runOptimize and shrinkToFit improves the memory consumption during the survey/index phase.
x=0,y=0 => x++ x=15,y=0
>----->------>----->---->----->---->---->----
↓ y + 1
x=0,y=1 <= x-- x=15,y=1
-----<------<-----<----<-----<----<----<----<
↓ y + 1
x=0,y=2 => x++ x=15,y=2
>----->------>----->---->----->---->---->----
....
So every other/odd row is being processed "backwards", in descending x order, right?...
Also of importance is that the movement pattern is complicated enough as it is (it took me a fair while to get it to work correctly).
...
- Index::add: I'd change the code checking for duplicate keys to a one liner (which I usually avoid) and comment outthe whole line when not used, rather than just the output. The compiler will have to perform the function call unless it can somehow deduce it doesn't have side effects even if there's nothing to do when true.Again that's just an debugging artifact that helped me to get a better understanding of what happened - so yeah that can & will be removed.
- Index::add: Is there a reason for the omission of code inside the candy check? The later storage of level?Yes exactly, the empty block is already removed in my current working copy.
- Index::add: I'm not sure what you're after with the (explicitly redundant) checks for various layer materials (not sure if it covers veins). If activated, it would exclude gems, for instance, and regardless, if someone has hacked in something, I don't see why it should be filtered out (And I just realized someone hacking in steel as a layer would get it filtered out if all inorganics were merged, but the issue would still be isolated to metals, and it would be possible to handle metals only separately).You mean
world->raws.inorganics[mineralIndex]->flags.is_set(df::inorganic_flags::SEDIMENTARY) ||
world->raws.inorganics[mineralIndex]->flags.is_set(df::inorganic_flags::IGNEOUS_EXTRUSIVE) ||
world->raws.inorganics[mineralIndex]->flags.is_set(df::inorganic_flags::IGNEOUS_INTRUSIVE) ||
world->raws.inorganics[mineralIndex]->flags.is_set(df::inorganic_flags::METAMORPHIC) ||
world->raws.inorganics[mineralIndex]->flags.is_set(df::inorganic_flags::SOIL)
?I've been thinking, and believe it's a mistake to try to massage the current inorganics presence storage. All you really NEED is two bytes to store the first and last layer of the geo biome present for each MLT (in addition to the index of the geo biome itself, which is currently stored), as all the rest of the info is available from the geo biome itself (And since the two values are in the range 0-15, you can actually store both of them in a single byte). To get the first/last layers we could cut away a fair bit of the code from the modified Prospector code of the MLT processing (the removed code would still be needed elsewhere to extract the actual inorganics, though).
With that basic information stored, you can either process the geo biome each time you need the data, or you can try to pre process the geo biomes to speed up the information extraction.
- The layers potentially worn away by erosion are all soil layers, and DF never seems to generate more than 4 of those. It's possible to hack the geo biome to get more soil layers and/or deeper soil, and DF can erode up to 10 Z levels, if I remember correctly. The suggestion below doesn't actually make any use of this info, though.
- DF doesn't use more than 16 layers of the geo biome even if hacking has added more (DF stretches the last one to fill the gap to the magma sea if needed).
This means that one possible approach would be to make a bit array for each layer of each geo biome and then merge the ones you have in each MLT with OR operations (16 layers * 33 byte bit array * X geo biomes). Even if DF doesn't croak at a silly max size PSV world with a checkerboard layout (forcing each world tile to get its own geo biome), you'd still not use more than 35 MB to store the info in a more convenient format than the geo biomes themselves.
- 12 + 12 + 4 + 4 = 32, so that would result in a 32 bit key.That seems easy enough ;D I got 32 as a result too when adding those values, but wanted to be sure that i understood correctly.
... I think it would be possible to get the key generation code to mimic it, i.e. to get keys generated in sequential order, but it would probably take a number of attempts to debug it (on the other hand testing would be simple: just indicate a failure if the next key isn't the previous one plus one). This is probably a case where the effort should be on the coder (i.e. you) rather than on the user...That's a fascinating idea! A key iterator, that generates the key values analogue to the movement pattern of the iterator of the world tiles - hm, never would have thought about that one.
- When it comes to the checks for flags, you've failed to take the two first lines in the block into your copy.I did, didn't I... I fixed that. But the duplicated index contents (galena and lead ...) happened when I deactivated the filter altogether. No filtering would probably be true for a merged inorganics vector as well. So that's something to consider when doing that.
- River size: ...Yes, you are right - I'm currently very code-centric and was talking about region_tile_datum and the level of processing of those tiles. I'm aware of survey_rivers in survey.cpp and was wondering if the river size or something analogue is also available on the level of embark tiles (mid_level_tile) in the structures of details->rivers_horizontal/vertical. Or if the river_size of a region_tile_datum automatically is the same for all related mid_level_tile that have a river and they inherit it from their "parent" region_tile_datum?
Oh, wait a minute: does the question about "region" refer to the world tile, rather than the regions in the DF structures? If it does, I'd suggest a change of terminology to avoid confusion.
- The current data structure has one level of data collected/summarized at the world tile level to allow for an early weeding out of world tiles that cannot have matching embarks. Info that is common for all MLTs ought to be stored there, if it's missing from there. Thus, I think the level you're looking for exists already.Again, you mean region_tile_datum, don't you? If they can be addressed/found/iterated fast and easily during the matching/query phase then yes they would be the right place. Actually I'm not sure anymore why I thought there might be the need for a new structure on the same level as region_tile_datum - perhaps it will come back to me...
- I'd be hesitant about saving matches to disk for later use for a number of reasons:That are really some tasty worms you got there!
- I don't know whether storing everything in indices (with the attendant lookup) is going to be faster than accessing pre processed geo biome info using first/last layer as the key pair. I do know it's going to require more memory, but I don't know the answer to the crucial question of whether it's going to require too much memory. The key gain with either approach is that you'd have to scan the world only once.Me neither, and yes, that key gain is what I'm after - perhaps those to approaches can be combined - again an possible optimization for when it works, that's pretty much what I wanted to say.
- Incursion handling with the index approach ought to be handled such that the relevant incursion info is integrated into the info for the MLT, i.e. it should be possible to store that the MLT has evil, neutral, AND good in it, for instance, that it has a partial aquifer coverage, and that it contains biome X, Y and Z. As far as I understand this isn't hard to do: all it would require is a few adjustments, but most of it seems to be ready for that (If I understand the indexing correctly, there's nothing blocking the current structure from adding the key for an MLT in more than one of the evilness indices, for instance).Storing the same key in different indices is absolutely no problem, also a query (e.g. find a MLT with a evil biome) would be fine with it, as it does not care about the fact, that the same key is in more than one complementary index. Moving all incursion information into the MLT would be the most elegant way for sure. But right now/at the beginning I could live with a solution that handled incursions as a special case, that gets processed later on when the neighboring MLT that belongs to the adjacent region_tile_datum is being surveyed - that's how I imagine it anyway - does that match the current way of it is done? (hehe, starting with the questions about incursion already).
There are also certain rules forcing ocean/lake biomes to lose edges to mountains, and both of them to anything else, no matter what the original array value is.mentioned in the comment on the world-data structure already "encoded" in the data or do we have process them ourselves?
A new DFHack version has been released. This also means all changes made to this plugin since the last release are released as well.Splendid!
I'll try to remember what has been done over this period...
Thanks for the crash report.
It took a bit of time to find out what was wrong, but it is indeed a bug.
The line should beCode: [Select]tile->region_type[i][k] = world_data->regions[tile->biome_index[mlt->at(i).at(k).biome_offset]]->type;i.e. "biome" -> "biome_index".
The reason it crashes only on pocket worlds is that the number of regions in those worlds tend to be fewer than the number of biome types, i.e. the index passed in was the biome type, but was interpreted as the index of the region.
The bug should screw up the reasonably (I think) uncommon searches for region types when it doesn't cause a crash.
There are also certain rules forcing ocean/lake biomes to lose edges to mountains, and both of them to anything else, no matter what the original array value is.If this includes the biome and all the features I would have to retain all this info for later, so it could be undone.
Also, you can't just merge things together as you go, as that would cause incursion info into a tile to be propagated into further tiles as part of the "real" data for that tile.Currently my design keeps the real source data (MLTs) separate from the index data (pure sink) during the survey phase - so any further propagation should not happen, but I'm wondering, how could further propagation happen if always 2 tiles (with the exception of corners) are being linked as you said here
every "receiver" is matched by a "giver" in the corresponding neighboring tile (corners has one "giver" and 3 "receivers"), and there is no "neutral" choice: there is an intrusion in one direction or the other.Sorry if I should have understood/interpreted that wrong, which is quite likely. That would be just more evidence I need to see those rules and the data in the context of program execution.
# matcher::embark_match
// tldr; does the actual matching for the current MLT and delegates the incursion processing for the outer edges of the embark rectangle as there is no need to process the inner part of the embark rectangle
// starting ~ line 773 the both north corners (NW+NE) and the north edge of every northern embark MLT of the embark area, then the south corners (SW+SE) and the south edge of every southern embark MLT are being processed
// starting ~ line 888 both west corners (NW+SW) and the west edge of every western embark MLT and then both east corners (NE+SE) and the east edge of every eastern embark MLT of the embark area are being processed
// the second loop respects all already processed corners, so no duplicate processing
# survey::translate_corner/survey::translate_ns_edge/survey::translate_ew_edge
// those 3 methods apply the rules described in world_region_details for edges.biome_corner/edges.biome_x/edges.biome_y to find the origin direction of the incursion to the current MLT corner/edge
# matcher::process_embark_incursion_mid_level_tile
// locates the correct incursion origin MLT + related RTD/world tile with the passed origin direction for further processing
# matcher::process_embark_incursion
// actually processes the matching with the incursion data with the correct MLT and RTD/world tile
biome_x biome_y should be used at the following lines: for i, interaction in ipairs (df.global.world.interaction_instances.all) do
if interaction.region_index == region [Surface] then
Weather_Page.Interaction_Index = interaction.interaction_id
break
end
end
So, once you've found the region, you'll have to iterate through the list of world coordinates in the df.global.world.world_data.regions [interaction.interaction_index].
Going from an embark tile, you'd find the df.global.world.world_data.region_map [x]:_displace (y).region (or possibly region_id, I don't have the XML in front of me) and then iterate through the interaction_instances to find a match (or not, of course) for that id.
Since we haven't yet reached the stage where DFHack is released as such, but nightly builds are made available, they're announced now (but if you don't build DFHack yourself you may have to wait another night for a build...)."nightly" doesn't actually mean that - builds are usually finished within an hour, unless there are a lot of builds running.
Nope, not this time. For some reason Toady decided to make the structures 16*17 and 17*16, with actual data in the extra row/column. That's fine as it saves you from going to the next world tile to get the data, except that the corner, 16/16, is missing, so for that one you still need the that world tile.Hm, I can see that split_x and split_y are [16][17]/[17][16]
int8_t biome_x[16][16]; /*!< 0=Reference is N, 1=Reference is current tile (adopted by S edge to the N) */
int8_t biome_y[16][16]; /*!< 0=Reference is W, 1=Reference is current tile (Adopted by E edge to the W) */Am I missing something?
For the western/northern edges the incursion info exists within the world, but there's special handling for the cases where the incursions are specified to come from the non existing side, which really affects corners only (for edges it's just a flip).By "flip" do you mean wrapping around to the other/opposite end of the world so e.g. southern tiles (y=world-height - 1, k=15) check for incursions against y=0, k=0?
A "flip" is just reverse the incursion direction, i.e. returning 4.Ok, understood the handling of edges at the world borders.
Edit: I'm unsure how to interpret the comment about the "uncovered" cases. Does it mean your handling has trouble, or does it mean there's an issue in the released version? I don't get any errors when I try it along the world edges."manifested" would have been the better word.
if (own_edge) {
// the edge belongs to the currently processed tile thus its counterpart is the north tile (k - 1)
effective_edge = world_data->region_details[0]->edges.biome_x[i][k];
// region_type_of actually properly handles the case that we need information from the world tile north (y - 1) and returns df::world_region_type::Lake in case of y < 0, which prevents incursion processing
south_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k);
north_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k - 1);
}
else {
// the edge belongs to the tile south of the currently processed tile thus its counterpart is the south tile (k + 1)
// here the case that we need information from the next world tile south is being handled properly
if (k < 15) {
effective_edge = world_data->region_details[0]->edges.biome_x[i][k + 1];
}
else {
// outside of the world - so no incursion
if (y + 1 == world_data->world_height) {
return 4;
}
// just the next world tile to the south
effective_edge = survey_results->at(x).at(y + 1).northern_row_biome_x[i];
}
north_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k);
south_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k + 1);
}
The above line (reading cached data)effective_edge = survey_results->at(x).at(y + 1).northern_row_biome_x[i];resulted in an illegal access error without handling the case y + 1 == world_data->world_height were before it was justeffective_edge = world_data->region_details[0]->edges.biome_x[i][k + 1];which read invalid data but did not crash
adjust_coordinates(&effective_x, &effective_y, &effective_i, &effective_k);instead ofadjust_coordinates(&effective_x, &effective_y, &effective_i, &effective_i);?Ouch! I tried using load-save to load the region from the command line but it looks like it only works for worlds that have already been embarked upon. I did a quick search of the DFHack documentation but I couldn't find anything that would let me load a new world from the command line. Am I missing something?It's possible, but due to the asynchronous nature of loading a world, it's complicated - probably more complicated than the load-save script. I'm not aware of an existing script that supports this at the moment.
Edit:
"./dfhack +embark-assistant fileresult" doesn't seem to work either.
Does anyone know how to get from the linux command line to the "embark site selection phase" automatically?
Also, the documentation for load-save has an improper example.
The example says "./dfhack +load-game region1" when it should say "./dfhack +load-save region1".
...I know it works for the northern and the western world edge.
Long question short:
Is it possible to process corner incursion at the edge of the world with only two candidates and if so for which world-edges is it possible?
x == world_width - 1 && i == 15 && corner_location == 8
and southern world edge y == world_height - 1 && y == 15 && corner_location == 8
with only the two easterni == 15 && k = [0-14] and k + 1
or southern i = [0-14] and i + 1 && k == 15
corners that are available?x == 14
y == 16
start_y + finder->y_dim - 1 == 15the last one translates to k == 15x < 16 && y == 16 && start_y + finder->y_dim - 1 == 15The reason for that overriding section is precisely that there is no data to read, and fallback values have to be used (and the fallbacks are what I've seen DF produce when I've looked at these edges).?
That's clearly wrong, as you point out.You give me too much credit for this one.
However, it seems that the code should have compared the region types and swapped the results around where appropriate, so a straight "return X" seems to be hasty.
Let me paraphrase one key takeaway: The offset/division of the dimension/edge (horizonal - x/vertical - y) that is shared with the neighboring tile is used in both cases either if the incursion is incoming or if the incursion is outgoing to the neighboring tile.
* 10 40 *
11 13
38 41
* 12 36 *
for horizontal edges, x=min y=maxSo there are two values for every split (for example 15,36 which means in the case of a split_x that the split starts at x = 15 and ends at x = 36, yes?) and I'd expect, that (15,36) would be repeated in the next line again - wait - ah.
The first * would be formed from split_x [self][self].x and split_y [self][self].x and the second one from split_x [self][self].y and split_y [self + 1][self].x.So if I got that correctly this would mean, that
The expansion to take incursions into consideration made the distinction between a tentative match and a full match somewhat iffy (full processing of a world tile relies on the surrounding world tiles being processed, which means either doing two passes automatically to catch everything, or let the user do the second pass, which is the current implementation).
The progress is then displayed as callbacks to the display functionality once every so often to provide the user with some feedback during the lengthy process.
If you are looking for absolutely all matches than yes, that is what you need to do.The expansion to take incursions into consideration made the distinction between a tentative match and a full match somewhat iffy (full processing of a world tile relies on the surrounding world tiles being processed, which means either doing two passes automatically to catch everything, or let the user do the second pass, which is the current implementation).
What is meant by "or let the user do the second pass"? Does this mean I should run embark-assistant's find function twice?
Loading bindings from data/init/interface.txt
New window size: 800x300
Font size: 10x12
Resizing grid to 80x25
Resizing font to 10x12
Resetting textures
DFHack is ready. Have a nice day!
DFHack version 0.47.04-r2 (release) on x86
Type in '?' or 'help' for general help, 'ls' to see all commands.
matcher::find: Preliminarily matching World Tiles: 246
matcher::find: Preliminarily matching World Tiles: 0
...make me think that.
- As far as I can see, lines 630 and 638 are reached only on absence together with the demand for All which looks correct to me?
- The checks on incursions ought to check what the incursions contain. If the check failed because the native tile didn't meet the All criterion you shouldn't get here, but if it did, you should fail if the incursion doesn't meet the criterion, even if the native tile did.
Thus, it looks correct to me, but you've found bugs it took me some time to recognize in the past...
So, as per the request, I tell you that the design ought to take incursions into full consideration for savagery, evilness, and aquifers, and I think it does ;)
(I wonder if this is the problem A_Curious_Cat had but failed to reproduce?).
I'm not a fan of your definition of All as "present in all tiles", rather than being the reverse of Absent.
... but it's not a user friendly and intuitive way of setting the criteria, which is why All was introduced.
... (...but would also be the cause of bug reports).
...
I can't come up with a scenario where I'd ever somehow would want a feature present in all tiles but being satisfied with it being provided by a tiny incursion somewhere in it: If a small incursion would be sufficient, it would probably be enough to have it anywhere in the embark, rather than demand it in every tile.
I can make a case for demanding every tile to meet the criterion natively while allowing for incursions to differ as it might result in a smaller set of matches when you want the dominant part of your embark to match the criterion, but incursions vary a lot in size, so that's more an illusion of control that actual control.
In my view Absent and All are symmetric: Any presence invalidates Absent, and any absence invalidates All.
With your caching, I'd probably aim for a three step process:Hehe, that's pretty much exactly what I've implemented, have a look here (https://github.com/bseiller/dfhack/tree/embark-assistant-index/plugins/embark-assistant)
- Process all MLTs in all world tiles.
- Process all incursions into all MLTs. However, it might be that it may be better to keep incursion processing for the interior MLTs in the first stage, as is done currently.
- Perform matching against all the fully processed MLTs. This step would be the only one needed for matching attempts after the first one.
Edit: My approach to deal with the incorrect top level data won't work: You need access to all the surrounding MLTs to process incursions, and attempts to process incursions as they become available would require the storage of both the native MLT results and the partial summary. Back to the drawing board...Yeah, I had some fun with that situation as well - it added quite some complexity to the code. Also I store all 64 outer/edge mid_level_tile/mid_level_tile_basic (<= contains only data relevant for incursions) of a world tile inside region_tile_datum => north & south row +west & east column. The internal/intra world tile incursions are being processed parallel to the native data, the cross world tile incursions delayed until all neighboring world tiles have being processed - using counters and such to track the progress...
However, it's not caused by missing incursion info as the whole 4*4 embark area in the lower right corner of the (5, 3) tile has temperate grassland as the native biome in each of the tiles, and even a single one ought to propagate it to the world tile level.Hm, curious, yet all the additional matches seem to be located at one of the edges of the world tile, either x == 0 or y == 12 (or more general y == 16 - y_dimension)
I uploaded the save (http://dffd.bay12games.com/file.php?id=14584) and profiles I use for getting an idea how much memory it would take to "index" everything for a large world.has at least one example (major river + stream) of this at x:166, y:61; starting from i:0, k:13 going to i:4, k:13.
&survey_results->at(fetch_x).at(fetch_y).*_row[i]
&survey_results->at(fetch_x).at(fetch_y).*_column[k]it potentially uses the wrong mlt to check for incursions.embark_assist::survey::region_type_ofin survey::translate_corner here (https://github.com/PatrikLundell/dfhack/blob/9b53222932a831ae024d7b48763bf0889dcdeef9/plugins/embark-assistant/survey.cpp#L1759) and probably here (https://github.com/PatrikLundell/dfhack/blob/9b53222932a831ae024d7b48763bf0889dcdeef9/plugins/embark-assistant/survey.cpp#L1708) and here (https://github.com/PatrikLundell/dfhack/blob/9b53222932a831ae024d7b48763bf0889dcdeef9/plugins/embark-assistant/survey.cpp#L1708) as well should be using effective_x and effective_y like thisembark_assist::survey::region_type_of(survey_results, effective_x, effective_y, ...divisor = (64 / steps * lat);to divisor = (64.0f / steps * lat);| range | # | ∑ # min/max |
| min <= 0 | 27908 | |
| min > 0 | 48355 | 76.263 |
| max<= 0 | 10583 | |
| max > 0 | 64243 | 74.826 |
| range | # | ∑ # min/max |
| min <= 0 | 97967 | |
| min > 0 | 188173 | 286.140 |
| max<= 0 | 32768 | |
| max > 0 | 248539 | 281.370 |
| range | # | ∑ # min/max |
| min <= 0 | 2243169 | |
| min > 0 | 14758891 | 17.002.060 |
| max<= 0 | 2243169 | |
| max > 0 | 14758891 | 17.002.060 |
| range | # | ∑ # min/max |
| min <= 0 | 1980576 | |
| min > 0 | 15013574 | 16.994.150 |
| max<= 0 | 1980576 | |
| max > 0 | 15013574 | 16.994.150 |
| range | # | ∑ # min/max |
| min <= 0 | 7843764 | |
| min > 0 | 9387251 | 17.231.015 |
| max<= 0 | 2243169 | |
| max > 0 | 14758891 | 17.002.060 |
Why embark assistant don't differentiating of lairs? Mound and burrow are different in game code.Are you talking about on the local map when embarking? What DFHack version are you using? 0.47.04-r4 (released yesterday) adds support for this (PR (https://github.com/DFHack/dfhack/pull/1713), changelog (https://docs.dfhack.org/en/0.47.04-r4/docs/NEWS.html#misc-improvements))
Thanks! I'll update my dfhack soon.Why embark assistant don't differentiating of lairs? Mound and burrow are different in game code.Are you talking about on the local map when embarking? What DFHack version are you using? 0.47.04-r4 (released yesterday) adds support for this (PR (https://github.com/DFHack/dfhack/pull/1713), changelog (https://docs.dfhack.org/en/0.47.04-r4/docs/NEWS.html#misc-improvements))
Max temperature is the "official" temperature of the world tile (i.e. the value in the world tile data), while the min temperature is the max temperature - the maximum variation (which lasts for a single tick in some cases), so the max temperature variation is completely dependent on the official world tile temperature, which, in its turn, is dependent on latitude, elevation, and the temperature parameters for world gen.Yes, I saw how min temperature was derived from max temperature and that in turn was taken from region_map_entry and transformed.
| range | # | ∑ # min/max |
| min <= 0 | 7506116 | |
| min > 0 | 9729380 | 17.235.496 |
| max<= 0 | 2243169 | |
| max > 0 | 14758891 | 17.002.060 |
I assume you've seen floating point values of the type 1.99998 when the mathematically correct value was 2, but truncating it to integer would nevertheless result in 1...Actually I saw a natural value of 0.5 for divisor which was rounded up (ceil) to a 1 after being multiplied with 0.75 here (https://github.com/PatrikLundell/dfhack/blob/0b7ab90d3d8987acac62971db8f42ba55f52567d/plugins/embark-assistant/survey.cpp#L451) resulting in
max_temperatur - 1!tile->reanimating_possible && !tile->thralling_possiblewould need to be changed to!tile->reanimating_possible != !tile->thralling_possibleor easier to readtile->reanimating_possible != tile->thralling_possible
My statement on the unsuitability to mix floating point math with integer math was intended to be general, not referring to this specific code. Floating point results end up near the mathematically correct result (most of the time: there are exceptions), but can end up both above, spot on, and below. Applying truncation or rounding up to such values can give the wrong value, where "proper" rounding might work.Ah, okay, now I see. Yes, of course, you are absolutely right.
The intention of the "Any" value for reanimation is "at least one of them", so it's good your results reflects that.Good, so I'll leave it as is.
If I'd intended to say exactly one of them I'd probably have called it "Either".I wanted to be sure, as I experienced first hand - also with my own code in multiple projects - how software grows and changes gradually until its current internal logic no longer fits the original designations of variables, classes or display labels that were named under the impression of their initial creation - a process often imperceptible for the one perpetrating it, kind of a temporary syndrome similar to mental blindness from staring to long on to something, the last days have been full of that at work actually. :)
I don't think there would be any reason for a player to want one or the other type of reanimation but exclude both of them, while I can see that players may want to ensure a challenge by requiring at least one of them.As I only accidentally have encountered reanimation and thralling - which ended badly quit fast - I really wouldn't dare to guess what players that are actively looking for it might want :D
Assuming you're referring to https://roaringbitmap.org/You're correct - I'm using their Amalgamation/Unity Build (https://github.com/lemire/CRoaringUnityBuild) which made the integration easy.
I'm not completely sure what your changes do at the moment, mainly because I'm not particularly familiar with the plugin internals, so PatrikLundell would be a better person to make an assessment in that regard.If desired I could do a little write-up with the core concepts behind my additions - putting it alongside the code as a little readme might help others that look into the plugin in the future.
I can do some basic testing on Linux (or more complicated testing with instructions).That would be much appreciated!
It can be noted that others have touched the Embark Assistant code without me being involved in the past, such as when the biome determination code was factored out, so my branch isn't the master.Ok, that is something I never knew or noticed.
If I understand it correctly, the first PR won't have any Roaring Bitmap code in it (local variables rather than .at() functions everywhere, and similar things), and so can be started immediately, while the one(s) relying on it will have to have it in place to be referenced, and so have such a PR accepted before that work can be done.You are correct in your understanding that the first few PRs won't have any references to Roaring in them at all - but at least the first 2 or 3 won't be a classic refactoring either, but smallish changes that should improve memory consumption or performance.
Well, actually, develop is the "master" for DFHack development (with "master" basically being the latest release, and thus the stable version). Thus, basing your fork off of "develop" would be the way to go (that way the rest of the code would be up to date as well). Once your fork's pull request has been merged, you can refresh your fork with the latest from develop (which shouldn't contain any changes to the Embark Assistant code, but may refresh other things), update it with the next set of changes, and issue a new pull request.Mostly see the first paragraph about me having a faulty base assumption and not verifying properly early on which is the leading repository.
Actually, the above is a bit of a simplification, as there are three, not two versions involved:
...
In retrospect...Just to make sure I understood correctly - as it is late: You think the total effort is larger doing it the way we do it now? That is by "delayed" PRs which have to be reviewed and then merged which might produce merge conflicts?
Not quite. ...Yes, I see. Had I known what I know now and found a easier and faster way to do it while getting the caching out there to the users, that would have been good.
I'm no fan of git: it's a shotgun with a default aim at your feet, a very sensitive trigger, no safety catch, and a confusing and unhelpful tangle of obscure commands and switches.It is indeed a very mighty weapon, not for the faint of heart. I consciously try to tread lightly and sometimes create branches just to test a chain of commands. In the end all question have an answer somewhere out there, but sometimes I don't know what the question is :D
world_data->region_map[adjusted.x][adjusted.y] here (https://github.com/DFHack/dfhack/blob/84eaf0414870826fb6a925794cf1a39c881f80d6/plugins/embark-assistant/survey.cpp#L559)and here (https://github.com/DFHack/dfhack/blob/84eaf0414870826fb6a925794cf1a39c881f80d6/plugins/embark-assistant/survey.cpp#L560)as well?Yes, you're correct. It should be "adjusted." there as like in all of the surrounding code, so yes please, take care of that one.Will do.
Personally I find it easier to discuss these issues here.Fine by me.
world_tile pointer it also uses x and y instead of "adjusted."The is_brook flag check has to be for the current world tile, as that property is independent of the biome, and so cares only for which world tile the tile resides in.Ah - that was what I meant when I wondered if there is a reason for the difference.
The third reference is to biome related code, and so needs to use adjusted coordinates.
I'd rather follow lethosor's advice and use uint8_t instead of char.
Any advice would be greatly appreciated.Hm, that sounds ominous.
Hi RedDwarfStepper and PatrikLundell...Thanks for that Deuslinks!
However, the Call Stack tab shows an address in SDLreal.dll, with the rest of the call stack entries pointing to DF itself, and the fact that it refers to SDLreal.dll is odd, since that ought to be DF's original version of SDL.dll, rather than the DFHack one, as if DF was running with the original DLL but somehow managed to call DFHack ones anyway (I tried to start DF with DFHack disabled, and couldn't reach the Embark Assistant, as expected).
I suspect we need it captured with symbols.I'll try and create a debug version (aka "RelWithDebInfo") of the master-branch and upload it.
00007FFDC856E82F lock dec dword ptr [rbx+8]Searching for "lock dec dword ptr" brings up hits for "semaphore", "Multiprocessor Protection" and "atomics" - this all to me points to code that handles multi-threading or thread-safety - which kind of makes sense for the real SDL code, as I think I recall that it is the only part of DF that is multi-threaded.
This operation looks like it could point to an invalid location "00007FFDC856E82F lock dec dword ptr [rbx+8]" if rbx contained garbage.
I'll try and create a debug version (aka "RelWithDebInfo") of the master-branch and upload it.=> here we go https://dffd.bay12games.com/file.php?id=15559
@lethosor: This is what the call stack looks like.
> SDLreal.dll!00007ffdc856e804() Unknown
Dwarf Fortress.exe!00007ff775ad9a6d() Unknown
Dwarf Fortress.exe!00007ff775ada0a7() Unknown
Dwarf Fortress.exe!00007ff775ada580() Unknown
Dwarf Fortress.exe!00007ff775ada87d() Unknown
Dwarf Fortress.exe!00007ff775adb062() Unknown
Dwarf Fortress.exe!00007ff77645cf8e() Unknown
Dwarf Fortress.exe!00007ff77645cd25() Unknown
Dwarf Fortress.exe!00007ff77645c1fa() Unknown
[External Code]
The cpu is very new had the whole pc built last month it is a AMD Ryzen 5 5600X Six Core CPU (3.7GHz-4.6GHz/35MB CACHE/AM4).hehe, thought so - oh I really hope I'm right.
Also no.worries things going on in life take precedence.Thanks for that. Also believe me I was beyond worrying at that moment more asleep than awake :D but giving a heads up so no one waits needlessly was the easy and right thing to do.
I tried to look at the crash dumps and it looks like all of them crash along the same call chain (which I guess is good in that it seems to be the same one all the time), but I still don't get anything more out of it. I hope RedDwarfStepper might be able to use the symbol files produced when the DLLs were produced to make the dumps somewhat more intelligible, though.
ntdll.dll!NtWaitForSingleObject () Unknown
KERNELBASE.dll!WaitForSingleObjectEx () Unknown
SDLreal.dll!00007ffdcbcee804() Unknown
> SDL.dll!SDL_SemWait(void * sem) Line 707 C++
Dwarf Fortress.exe!00007ff775ad9a6d() Unknown
Dwarf Fortress.exe!00007ff775ada0a7() Unknown
Dwarf Fortress.exe!00007ff775ada580() Unknown
Dwarf Fortress.exe!00007ff775ada87d() Unknown
Dwarf Fortress.exe!00007ff775adb062() Unknown
Dwarf Fortress.exe!00007ff77645cf8e() Unknown
Dwarf Fortress.exe!00007ff77645cd25() Unknown
Dwarf Fortress.exe!00007ff77645c1fa() Unknown
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
> embark-assistant.plug.dll!std::distance<std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<enum df::enums::interface_key::interface_key> > > >(std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<enum df::enums::interface_key::interface_key> > > _First, std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<enum df::enums::interface_key::interface_key> > > _Last) Line 1126 C++
embark-assistant.plug.dll!embark_assist::overlay::ViewscreenOverlay::interpose_fn_feed(std::set<enum df::enums::interface_key::interface_key,std::less<enum df::enums::interface_key::interface_key>,std::allocator<enum df::enums::interface_key::interface_key> > * input) Line 104 C++
SDL.dll!df::viewscreen::feed_key(df::enums::interface_key::interface_key key) Line 5 C++
embark-assistant.plug.dll!embark_assist::matcher::find(embark_assist::defs::match_iterators * iterator, std::vector<embark_assist::defs::geo_datum,std::allocator<embark_assist::defs::geo_datum> > * geo_summary, std::vector<std::vector<embark_assist::defs::region_tile_datum,std::allocator<embark_assist::defs::region_tile_datum> >,std::allocator<std::vector<embark_assist::defs::region_tile_datum,std::allocator<embark_assist::defs::region_tile_datum> > > > * survey_results, std::vector<std::vector<embark_assist::defs::matches,std::allocator<embark_assist::defs::matches> >,std::allocator<std::vector<embark_assist::defs::matches,std::allocator<embark_assist::defs::matches> > > > * match_results) Line 2967 C++
embark-assistant.plug.dll!embark_assist::main::match() Line 90 C++
embark-assistant.plug.dll!embark_assist::overlay::ViewscreenOverlay::interpose_fn_render() Line 244 C++
Dwarf Fortress.exe!00007ff775d4fef8() Unknown
Dwarf Fortress.exe!00007ff775ad9e22() Unknown
Dwarf Fortress.exe!00007ff775adadd9() Unknown
SDLreal.dll!00007ffdcbcee471() Unknown
SDLreal.dll!00007ffdcbcee855() Unknown
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
ntdll.dll!NtWaitForSingleObject ()does not sound so bad/crashy.input->count(df::interface_key::SETUP_LOCAL_Y_UP) ||https://github.com/DFHack/dfhack/blob/2fc5fbacb5bdf7183d61045a4786d58571081eec/plugins/embark-assistant/overlay.cpp#L104template<class _InIt> inline
_Iter_diff_t<_InIt>
distance(_InIt _First, _InIt _Last)
{ // return distance between iterators
return (_Distance1(_First, _Last, _Iter_cat_t<_InIt>()));
}via in xtree size_type count(const key_type& _Keyval) const
{ // count all elements that match _Keyval
_Paircc _Ans = equal_range(_Keyval);
return (_STD distance(_Ans.first, _Ans.second));
}SDL.dll!DFHack::Core::getHotkeyCmdin another worker thread (see below) those two might interact in a bad way - just throwing out wild theories ... ntdll.dll!NtWaitForSingleObject () Unknown
mswsock.dll!SockWaitForSingleObject() Unknown
mswsock.dll!WSPAccept() Unknown
ws2_32.dll!WSAAccept() Unknown
ws2_32.dll!accept () Unknown
> SDL.dll!CPassiveSocket::Accept() Line 244 C++
SDL.dll!`anonymous namespace'::ServerMainImpl::threadFn(std::promise<bool> promise, int port) Line 475 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > >::_Execute<0,1,2>(std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> & _Tup, std::integer_sequence<unsigned __int64,0,1,2> __formal) Line 241 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > >::_Run(std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > > * _Ln) Line 247 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > >::_Go() Line 233 C++
SDL.dll!std::_Pad::_Call_func(void * _Data) Line 210 C++
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
2.: ntdll.dll!NtDeviceIoControlFile () Unknown
KERNELBASE.dll!ConsoleCallServerGeneric() Unknown
KERNELBASE.dll!GetConsoleInput() Unknown
KERNELBASE.dll!ReadConsoleInputA () Unknown
> SDL.dll!DFHack::Private::prompt_loop(tthread::recursive_mutex * lock, DFHack::CommandHistory & history) Line 288 C++
SDL.dll!DFHack::Private::lineedit(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & prompt, std::basic_string<char,std::char_traits<char>,std::allocator<char> > & output, tthread::recursive_mutex * lock, DFHack::CommandHistory & ch) Line 386 C++
SDL.dll!DFHack::Console::lineedit(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & prompt, std::basic_string<char,std::char_traits<char>,std::allocator<char> > & output, DFHack::CommandHistory & ch) Line 598 C++
SDL.dll!fIOthread(void * iodata) Line 1494 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Run(std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void *),void *>,std::default_delete<std::tuple<void (__cdecl*)(void *),void *> > > > * _Ln) Line 247 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Go() Line 233 C++
SDL.dll!std::_Pad::_Call_func(void * _Data) Line 210 C++
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
3.: ntdll.dll!NtWaitForAlertByThreadId () Unknown
ntdll.dll!RtlSleepConditionVariableSRW() Unknown
KERNELBASE.dll!SleepConditionVariableSRW () Unknown
[Inline Frame] msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait_for(Concurrency::details::stl_critical_section_interface *) Line 216 C++
msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait(Concurrency::details::stl_critical_section_interface * lock) Line 210 C++
> msvcp140.dll!do_wait(_Cnd_internal_imp_t * cond, _Mtx_internal_imp_t * mtx, const xtime * target) Line 77 C++
SDL.dll!std::condition_variable::wait(std::unique_lock<std::mutex> & _Lck) Line 565 C++
SDL.dll!DFHack::Core::getHotkeyCmd(bool & keep_going) Line 1913 C++
SDL.dll!fHKthread(void * iodata) Line 234 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Run(std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void *),void *>,std::default_delete<std::tuple<void (__cdecl*)(void *),void *> > > > * _Ln) Line 247 C++
SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Go() Line 233 C++
SDL.dll!std::_Pad::_Call_func(void * _Data) Line 210 C++
ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
Thanks for that. Unfortunately, I fail to any immediate culprit.You're welcome. And: Yeah, me neither.
I have no idea how sets are implemented, but it doesn't make sense that checking a set should be capable of blowing up if you're actually using the correct set.Well, if there is concurrent access to the set and at least one thread modifies the set while another thread reads it (=> count) this can lead to an inconsistent state that leads to undefined behaviour which can result in a crash.
I can't say this feels like much progress...Well with results of the modified dlls we learned that it is no recent change that causes the problem.
Assuming the program blows up in the EA thread (do we know it's there, rather than the thread being at that location when things blow up elsewhere?) it looks like the data received is somehow corrupt, which is uncomfortable as it's provided by DF rather than our code (I'd rather get egg on my face from screwing up than being able to say it wasn't my fault when the former means I can actually fix it, but the latter doesn't).Thinking about it again, we really can't be sure that this is exactly what blows up. We only know for sure that EA is at least one part of the problem: If our user starts a search with EA it crashes. But that does not mean that no other plugin is contributing to this crash. It could very well be a combined effort of two or more actors... :)
- Would more crash dumps with the debug-dlls help? Perhaps we're missing something?In general, I would say yes. You currently have a crash dump with several active threads, and it can be difficult to know where the crash actually originated sometimes, despite what the dump might say. It's likely that the crash occurred in the embark-assistant thread, given that using embark-assistant triggers the crash, but it would be easier to rule out unrelated threads with a handful of other crash dumps (say, 5 maybe).
PPS:
One more thing I just found: The current pressed key is CURSOR_UPLEFT_FAST, coming from here
https://github.com/DFHack/dfhack/blob/1c32783dd2628f22c5355b84e49e5c7357b52cb8/plugins/embark-assistant/matcher.cpp#L2966
or to be exact here
https://github.com/DFHack/dfhack/blob/1c32783dd2628f22c5355b84e49e5c7357b52cb8/plugins/embark-assistant/matcher.cpp#L2967
so the end of a loop - hm.
I did get one call that was initiated by embark-tools (I think the key issued was A102, but I may be misremembering)For what it's worth, this is normal when other plugins are enabled and hook into the same screen - they'll end up calling each others' hooks, and unless they specify a priority, there's not a fixed order to these calls across DF runs (although they should usually occur in the same order in a single DF session, unless plugins are loaded/unloaded).
Evil weather, when present: BR = Blood Rain, TS = Temporary Syndrome
PS = Permanent Syndrome, Re = Reanimating, and Th = Thralling. Incursions.