Bay 12 Games Forum

Assuming the program blows up in the EA thread (do we know it's there, rather than the thread being at that location when things blow up elsewhere?) it looks like the data received is somehow corrupt, which is uncomfortable as it's provided by DF rather than our code (I'd rather get egg on my face from screwing up than being able to say it wasn't my fault when the former means I can actually fix it, but the latter doesn't).

Thinking about it again, we really can't be sure that this is exactly what blows up. We only know for sure that EA is at least one part of the problem: If our user starts a search with EA it crashes. But that does not mean that no other plugin is contributing to this crash. It could very well be a combined effort of two or more actors...
So to make sure about that I have one more test for Deuslinks: remove all other plugins and try to provoke the crash.
Since that is a quite tedious test I'd like him to test something else before that. Another hunch...

@Deuslinks:
Could you please start a new session like always, best with a big world, load EA if not already done and manually move the cursor to the top left corner of the region view and only then open the view for the search criteria via 'f', change the criteria and then 'f' again?
Does it still crash right away? If so could you please upload the resulting DMP-file?
If is crashes but at the end of the search - so after some time instead of right away - please upload the resulting DMP-files and have a try with dll uploaded here.
How does that one behave? If it crashes as well:
We (read: you) need to temporarily "deactivate" all other plugins that might interfere - this is more involved than the other tests, if you feel up to it you should have some time at your hands and best create a copy of your LNP/PeridexisErrant's Starter Pack so you don't destroy your regular setup.
Okay, here we go:
- Please move all plugin-dlls with the exception of embark-assistant.plug.dll from the plugins folder to a separate folder, e.g. "deactivated_plugins" or just create a copy of the "Dwarf Fortress 0.47.05\hack\plugins\" folder, rename the copy to "deactivated_plugins" and then remove the dlls from the original folder
- The only remaining dll-file in the plugins folder should be the "embark-assistant.plug.dll" with a size of around 653 KB, that you downloaded from here.
Do what you do and never mind the warning about the missing plugins... Crash? => Please upload the dump and don't mind my sobbing

The main thread is a follows:
ntdll.dll!NtWaitForSingleObject() Unknown
  KERNELBASE.dll!WaitForSingleObjectEx() Unknown
  SDLreal.dll!00007ffdcbcee804() Unknown
> SDL.dll!SDL_SemWait(void * sem) Line 707 C++
  Dwarf Fortress.exe!00007ff775ad9a6d() Unknown
  Dwarf Fortress.exe!00007ff775ada0a7() Unknown
  Dwarf Fortress.exe!00007ff775ada580() Unknown
  Dwarf Fortress.exe!00007ff775ada87d() Unknown
  Dwarf Fortress.exe!00007ff775adb062() Unknown
  Dwarf Fortress.exe!00007ff77645cf8e() Unknown
  Dwarf Fortress.exe!00007ff77645cd25() Unknown
  Dwarf Fortress.exe!00007ff77645c1fa() Unknown
  kernel32.dll!BaseThreadInitThunk() Unknown
  ntdll.dll!RtlUserThreadStart() Unknown

the worker thread concerning embark-assistant:
> embark-assistant.plug.dll!std::distance<std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<enum df::enums::interface_key::interface_key> > > >(std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<enum df::enums::interface_key::interface_key> > > _First, std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<enum df::enums::interface_key::interface_key> > > _Last) Line 1126 C++
  embark-assistant.plug.dll!embark_assist::overlay::ViewscreenOverlay::interpose_fn_feed(std::set<enum df::enums::interface_key::interface_key,std::less<enum df::enums::interface_key::interface_key>,std::allocator<enum df::enums::interface_key::interface_key> > * input) Line 104 C++
  SDL.dll!df::viewscreen::feed_key(df::enums::interface_key::interface_key key) Line 5 C++
  embark-assistant.plug.dll!embark_assist::matcher::find(embark_assist::defs::match_iterators * iterator, std::vector<embark_assist::defs::geo_datum,std::allocator<embark_assist::defs::geo_datum> > * geo_summary, std::vector<std::vector<embark_assist::defs::region_tile_datum,std::allocator<embark_assist::defs::region_tile_datum> >,std::allocator<std::vector<embark_assist::defs::region_tile_datum,std::allocator<embark_assist::defs::region_tile_datum> > > > * survey_results, std::vector<std::vector<embark_assist::defs::matches,std::allocator<embark_assist::defs::matches> >,std::allocator<std::vector<embark_assist::defs::matches,std::allocator<embark_assist::defs::matches> > > > * match_results) Line 2967 C++
  embark-assistant.plug.dll!embark_assist::main::match() Line 90 C++
  embark-assistant.plug.dll!embark_assist::overlay::ViewscreenOverlay::interpose_fn_render() Line 244 C++
  Dwarf Fortress.exe!00007ff775d4fef8() Unknown
  Dwarf Fortress.exe!00007ff775ad9e22() Unknown
  Dwarf Fortress.exe!00007ff775adadd9() Unknown
  SDLreal.dll!00007ffdcbcee471() Unknown
  SDLreal.dll!00007ffdcbcee855() Unknown
  ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
  kernel32.dll!BaseThreadInitThunk() Unknown
  ntdll.dll!RtlUserThreadStart() Unknown

Really can't say anything concerning the main thread, apart from
ntdll.dll!NtWaitForSingleObject()does not sound so bad/crashy.
But the ea worker thread, hm, the last real source code line 104 corresponds to this
input->count(df::interface_key::SETUP_LOCAL_Y_UP) ||https://github.com/DFHack/dfhack/blob/2fc5fbacb5bdf7183d61045a4786d58571081eec/plugins/embark-assistant/overlay.cpp#L104
and ends up in xutility:
template<class _InIt> inline
_Iter_diff_t<_InIt>
distance(_InIt _First, _InIt _Last)
{ // return distance between iterators
return (_Distance1(_First, _Last, _Iter_cat_t<_InIt>()));
}via in xtree
size_type count(const key_type& _Keyval) const
{ // count all elements that match _Keyval
_Paircc _Ans = equal_range(_Keyval);
return (_STD distance(_Ans.first, _Ans.second));
}
Could it be that input gets messed up/invalid? Or that DEFINE_VMETHOD_INTERPOSE misbehaves? Also there is a reference to
SDL.dll!DFHack::Core::getHotkeyCmdin another worker thread (see below) those two might interact in a bad way - just throwing out wild theories ...


There are 3 more worker threads that contains references to dfhack
1.:
ntdll.dll!NtWaitForSingleObject() Unknown
  mswsock.dll!SockWaitForSingleObject() Unknown
  mswsock.dll!WSPAccept() Unknown
  ws2_32.dll!WSAAccept() Unknown
  ws2_32.dll!accept() Unknown
> SDL.dll!CPassiveSocket::Accept() Line 244 C++
  SDL.dll!`anonymous namespace'::ServerMainImpl::threadFn(std::promise<bool> promise, int port) Line 475 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > >::_Execute<0,1,2>(std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> & _Tup, std::integer_sequence<unsigned __int64,0,1,2> __formal) Line 241 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > >::_Run(std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > > * _Ln) Line 247 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int>,std::default_delete<std::tuple<void (__cdecl*)(std::promise<bool>,int),std::promise<bool>,int> > > >::_Go() Line 233 C++
  SDL.dll!std::_Pad::_Call_func(void * _Data) Line 210 C++
  ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
  kernel32.dll!BaseThreadInitThunk() Unknown
  ntdll.dll!RtlUserThreadStart() Unknown
2.:
ntdll.dll!NtDeviceIoControlFile() Unknown
  KERNELBASE.dll!ConsoleCallServerGeneric() Unknown
  KERNELBASE.dll!GetConsoleInput() Unknown
  KERNELBASE.dll!ReadConsoleInputA() Unknown
> SDL.dll!DFHack::Private::prompt_loop(tthread::recursive_mutex * lock, DFHack::CommandHistory & history) Line 288 C++
  SDL.dll!DFHack::Private::lineedit(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & prompt, std::basic_string<char,std::char_traits<char>,std::allocator<char> > & output, tthread::recursive_mutex * lock, DFHack::CommandHistory & ch) Line 386 C++
  SDL.dll!DFHack::Console::lineedit(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & prompt, std::basic_string<char,std::char_traits<char>,std::allocator<char> > & output, DFHack::CommandHistory & ch) Line 598 C++
  SDL.dll!fIOthread(void * iodata) Line 1494 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Run(std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void *),void *>,std::default_delete<std::tuple<void (__cdecl*)(void *),void *> > > > * _Ln) Line 247 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Go() Line 233 C++
  SDL.dll!std::_Pad::_Call_func(void * _Data) Line 210 C++
  ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
  kernel32.dll!BaseThreadInitThunk() Unknown
  ntdll.dll!RtlUserThreadStart() Unknown
3.:
ntdll.dll!NtWaitForAlertByThreadId() Unknown
  ntdll.dll!RtlSleepConditionVariableSRW() Unknown
  KERNELBASE.dll!SleepConditionVariableSRW() Unknown
  [Inline Frame] msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait_for(Concurrency::details::stl_critical_section_interface *) Line 216 C++
  msvcp140.dll!Concurrency::details::stl_condition_variable_win7::wait(Concurrency::details::stl_critical_section_interface * lock) Line 210 C++
> msvcp140.dll!do_wait(_Cnd_internal_imp_t * cond, _Mtx_internal_imp_t * mtx, const xtime * target) Line 77 C++
  SDL.dll!std::condition_variable::wait(std::unique_lock<std::mutex> & _Lck) Line 565 C++
  SDL.dll!DFHack::Core::getHotkeyCmd(bool & keep_going) Line 1913 C++
  SDL.dll!fHKthread(void * iodata) Line 234 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Run(std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void *),void *>,std::default_delete<std::tuple<void (__cdecl*)(void *),void *> > > > * _Ln) Line 247 C++
  SDL.dll!std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64>,std::default_delete<std::tuple<void (__cdecl*)(void * __ptr64),void * __ptr64> > > >::_Go() Line 233 C++
  SDL.dll!std::_Pad::_Call_func(void * _Data) Line 210 C++
  ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>() Unknown
  kernel32.dll!BaseThreadInitThunk() Unknown
  ntdll.dll!RtlUserThreadStart() Unknown


The cpu is very new had the whole pc built last month it is a AMD Ryzen 5 5600X Six Core CPU (3.7GHz-4.6GHz/35MB CACHE/AM4).

hehe, thought so - oh I really hope I'm right.

Also no.worries things going on in life take precedence.

Thanks for that. Also believe me I was beyond worrying at that moment more asleep than awake but giving a heads up so no one waits needlessly was the easy and right thing to do.

Ok, so here we go
https://dffd.bay12games.com/file.php?id=15562
The zip contains 5 adapted versions of the plugin (all build against dfhack 0.47.05-r1), starting with the current version without changes (0._0.47.05_r1_embark-assistant.plug.dll) just to make absolutely super sure it still causes a crash.
If it does not, well, I checked with the release on github (https://github.com/DFHack/dfhack/releases/download/0.47.05-r1/dfhack-0.47.05-r1-Windows-64bit.zip) and the two files are pretty different.... So I'm wondering - but lets not go there for now.
It also contains the unchanged original dll files from the respective release in the folder "_originals", they are there just for reference sake and can be ignored for now.

@Deuslinks: You can just copy the dlls into your plugins directory - you'll see a warning for each one but they won't interfere with the properly named "embark-assistant.plug.dll".
As per PatrikLundell's very good suggestion keep your original "embark-assistant.plug.dll" around by renaming it to "embark-assistant.plug.dll.orig".
Now starting with "0._0.47.05_r1_embark-assistant.plug.dll" create a copy of the file and rename the copy to "embark-assistant.plug.dll" - it might be easier to delete the current "embark-assistant.plug.dll" beforehand depending on your workflow/file handling tools.
Then do your thing
Rinse and repeat with the next version in increasing order if the currently tested crashes.
For the moment we're only interested in the version which reliably does not cause a crash. If even the last version ("4._0.47.04-r4_embark-assistant.plug.dll") causes a crash - well - let me know
Also let me know if there are any other questions.

I'll have to confess I'm really out of my depth here.
My initial hope was that using a debug-version of the plugin would provide more readable call stacks.
But seeing that the last call stack mostly happens in Dwarf Fortress.exe (for which we don't have any sources) itself and not in any code associated with DFhack this probably won't be the case.
Especially if it is really is corrupted memory, which like a booby trap only triggers after being set and left alone.

So I'd like to suggest an additional/alternative approach, one where I can actually contribute something apart from confusion:
- our current assumption is that the error has its cause in the embark-assistant plugin. As we have 1.5/2 data points and the CTD is very likely to happen when a search with embark-assistant is started I think this is a strong hypothesis.
- my/our (implicit) assumption is that it is caused by code that was added recently (version-wise) as we haven't had any such bug reports previously. Not as strong a hypothesis as the previous one but good enough to work with for the moment.
To prove or refute these assumptions we could create a version of the plugin that corresponds to version 0.47.05-beta1 or perhaps better 0.47.04-r5.
If the error is gone then we are sure that something added after that version is the cause - otherwise we have to go further back.
Rinse and repeat until the error is gone or until it gets very unlikely that embark-assistant is the sole cause.
Then we can slowly start readding isolated changes until we get the crash again... so some kind of binary search + divide and conquer on the source code level.
Any thoughts?

PS: I can probably get a first version done tonight if Deuslinks still is game.

I suspect we need it captured with symbols.

I'll try and create a debug version (aka "RelWithDebInfo") of the master-branch and upload it.
It might be so slow though that the crash won't be provoked, the next step would be "ReleaseOptWithDebSymbs" which I cobbled together myself when I needed to debug something in the later search process.

00007FFDC856E82F  lock dec    dword ptr [rbx+8] 
This operation looks like it could point to an invalid location "00007FFDC856E82F  lock dec    dword ptr [rbx+8]" if rbx contained garbage.

Searching for "lock dec dword ptr" brings up hits for "semaphore", "Multiprocessor Protection" and "atomics" - this all to me points to code that handles multi-threading or thread-safety - which kind of makes sense for the real SDL code, as I think I recall that it is the only part of DF that is multi-threaded.
If SDL crashes after embark-assistant somehow corrupted the memory this might be tricky
Anyone knows of any compiler options that activate runtime memory checks that induce a crash exactly at that moment when some invalid operation happens?

Hi RedDwarfStepper and PatrikLundell...

Thanks for that Deuslinks!

@PatrikLundell: If it really is system-dependent we probably won't be able to reproduce the crash with our systems.
But throwing out a wild guess: It could be one of the Microsoft Visual C++ YYYY Redistributables. On my system there are 15(!) different versions installed, ranging from 2005 to 2015 alternating between x86 and x64...
Or as it does not happen every time it might have something to do with how some memory area is (un)initialized, which could depend on which other programs are running or what has happened in the memory of the DF process beforehand.
Still pretty hard to debug.
But if is modification of the starter pack itself or the config of the installation we might catch it with a zipped download of the installation minus the world saves and minus the folder "\LNP\utilities\" (so big!) we have already.
What do you think?

I'd rather follow lethosor's advice and use uint8_t instead of char.

Ah, I wasn't planning on disregarding his advice, I just wanted to see if another approach would result in a more favorable outcome.
And after experimenting around for 30 minutes and reading for 2 hours I learned something new:
- Compilers aren't as smart as we both wish them to be
https://blog.royalsloth.eu/posts/the-compiler-will-optimize-that-away/
- And sometimes you really have to trick them to get them to get it right:
https://travisdowns.github.io/blog/2020/01/20/zero.html

And at last: std::memset is much faster for a std::vector<POD> than std::fill, but std::vector<bool> is a special case as we already knew...:
https://stackoverflow.com/a/1665038/167865

=> Using std::memset on a std::vector<uint8_t> takes a total of 2 seconds, which is a good speedup in my book...

The is_brook flag check has to be for the current world tile, as that property is independent of the biome, and so cares only for which world tile the tile resides in.
The third reference is to biome related code, and so needs to use adjusted coordinates.

Ah - that was what I meant when I wondered if there is a reason for the difference.
So, just to make sure there is no misunderstanding: To retrieve the proper value for the "is_lake" flag the coordinates have to be "biome adjusted", but for the "is_brook" flag that is not true and the none-adjusted coordinates must be used, yes?