Bay 12 Games Forum
Dwarf Fortress => DF General Discussion => Topic started by: last76 on May 05, 2014, 12:51:30 am
-
Dear all, after having played Dwarf Fortress for some time my professional experiences in risk management must be ventilated.
I wonder if any precautions has been taken if any unfortunate events would happen to the developer of Dwarf Fortress?
It would be very unfortunate if the world would lose this piece of art if something happened to the developer which would make him unable to continue the work.
-
Yep, precautions have been taken already. Toady has stated that in the case of his unfortunate demise the source code will be released, on the condition that there was no foul play involved (so no murdering for the source code! ;P). Let me see if I can find a link as well, don't remember where/when exactly he said it.
-
Good to hear. I assume Toady contacted a lawyer for these precautions?
-
Yeah, we've mentioned my untimely demise before, most recently because this town has lots of drunks and meth and some of the stupidest ass people that yell at you from vehicles as you might ever encounter (despite warnings from well-meaning people with misgivings about Texas, nothing ever happened down there, whereas up here in Silverdale I've been yelled at or had crap thrown at me or whatever no fewer than fifteen times while shopping, etc., even though I walked more down there), and the idea was to release everything. I should amend that though -- if there are suspicions of foul play, and the preponderance of the evidence suggests that the death was related to getting the source released, then, well, no source then. Yes, you will be punished for the wrongdoings of others, so pray for my safety! No killing for the source!
This is what I could find, though I'm fairly sure it's been mentioned elsewhere as well. No idea on the specifics of how it's set up though.
-
Thanks. Impressive to remember a post from 2008...
-
Thanks. Impressive to remember a post from 2008...
Hehe, the search engine is our friend! I did remember he had used the wording "foul play" though, which helped somewhat. And it wasn't thaaat long ago I read it, only been here a bit over 2 years ^^
-
He's also set up several forms of physical backups in different locations too, if I'm remembering that quote correctly.
-
... And I heard he also wired plastic explosives to those physical backups which will go off if a single binary byte of data is tampered with without inputting a password that changes hourly to a cryptographic algorithm that Toady has committed down to his dreaming subconscious ...
((lets get a chain of ridiculously improbable precautions the man who thinks of everything has probably cooked up going for teh lolz :P ))
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
My opinion tough.
And I believe it might be a better idea to restart from scratch anyway, than continuing on a decade old code base which nevertheless how awesome the game as idea is, isn't the most efficient in execution in a number of areas, albeit looking into some stuff like the world creation code wouldn't be bad to be able to quickly boot up a clone project to a playable state.
I mean there are a sure a few of us who'd like to code a DF-like from scratch, but alas, all of them do not have the time to.
Anyway, DF has already inspired a whole generation of games, albeit none is similar enough to be considered a replacement candidate.
-
so no murdering for the source code!
Presumably if you were willing to murder him for it, you'd also be willing to simply break into his house or whatever and steal it WITHOUT murdering anybody.
Plus, even if you are a weird psychopath who sees the code as more valuable than a human life, you'd still have to realize that keeping him alive = more / better code in the future that you could steal again, etc. etc.
-
If Toady were to die there would be no further development, as his entire playerbase would be killed, mummified and buried with him in a giant pyramid so that we may serve him in the afterlife.
-
If Toady were to die there would be no further development, as his entire playerbase would be killed, mummified and buried with him in a giant pyramid so that we may serve him in the afterlife.
Yet another lesson why you really should read the End User License Agreement that comes with your software.
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. Dwarf Fortress is legendary; it has a dedicated community including accomplished programmers; Toady has laid out arcs of future development; and the game seems to be popular particularly among the kind of folks who could pick up the baton. If anything, I would expect too many dedicated projects to grow up, fragmenting the community. But after ten years, those would settle into a handful of DF variants which would be in it for the long haul.
-
If Toady were to die there would be no further development, as his entire playerbase would be killed, mummified and buried with him in a giant pyramid so that we may serve him in the afterlife.
Yet another lesson why you really should read the End User License Agreement that comes with your software.
Best response(s) ever.
-
If Toady were to die there would be no further development, as his entire playerbase would be killed, mummified and buried with him in a giant pyramid so that we may serve him in the afterlife.
Yet another lesson why you really should read the End User License Agreement that comes with your software.
Sigged
-
so no murdering for the source code!
Presumably if you were willing to murder him for it, you'd also be willing to simply break into his house or whatever and steal it WITHOUT murdering anybody.
Plus, even if you are a weird psychopath who sees the code as more valuable than a human life, you'd still have to realize that keeping him alive = more / better code in the future that you could steal again, etc. etc.
Had to think about this: http://xkcd.com/538/
-
If Toady were to die there would be no further development, as his entire playerbase would be killed, mummified and buried with him in a giant pyramid so that we may serve him in the afterlife.
Yet another lesson why you really should read the End User License Agreement that comes with your software.
Best response(s) ever.
Indeed. I laughed out loud here.
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. (cut)
History of open-source games shows otherwise. Project like DF needs extremely dedicated person like Toady. No Toady, no project. At most, few years of small development, bugfixes, trying to make some things, sad agony when amount of interested people dries up and later death.
As others said, it makes sense to start new project. There is only one teensy weensy little detail left: find some new Toady-like extremely dedicated programmer. Good luck with that.
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. (cut)
History of open-source games shows otherwise. Project like DF needs extremely dedicated person like Toady. No Toady, no project. At most, few years of small development, bugfixes, trying to make some things, sad agony when amount of interested people dries up and later death.
As others said, it makes sense to start new project. There is only one teensy weensy little detail left: find some new Toady-like extremely dedicated programmer. Good luck with that.
DCSS and liberal crime squad both seem to be doing alright, both adding new features on top of the previously abandoned projects.
-
As others said, it makes sense to start new project. There is only one teensy weensy little detail left: find some new Toady-like extremely dedicated programmer. Good luck with that.
We could just clone toady and in force his soul into the new body via some ancient dark ritual.
We could even use the pyramid for its intended purpose violating the most fundamental laws of nature and man
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. Dwarf Fortress is legendary; it has a dedicated community including accomplished programmers; Toady has laid out arcs of future development; and the game seems to be popular particularly among the kind of folks who could pick up the baton. If anything, I would expect too many dedicated projects to grow up, fragmenting the community. But after ten years, those would settle into a handful of DF variants which would be in it for the long haul.
I imagine that in the event DFHack would simply become DF.
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. Dwarf Fortress is legendary; it has a dedicated community including accomplished programmers; Toady has laid out arcs of future development; and the game seems to be popular particularly among the kind of folks who could pick up the baton. If anything, I would expect too many dedicated projects to grow up, fragmenting the community. But after ten years, those would settle into a handful of DF variants which would be in it for the long haul.
I imagine that in the event DFHack would simply become DF.
I imagine df and dfhack would merge into a huge monster that would be fueled by babies and pupies and souls of the damned. Though bugfixes would be guaranteed.
-
Toady has laid out arcs of future development;
There are a few possibilities what exactly would become and determining which is the most likely is more due to personal experience.
However, one I can tell you for sure. No human will dedicate his or her life to follow the arc laid out by toady. Toadys arc dies with toady, thats for sure.
If someone else wants to dedicate a hugh effort into, you can be sure that will be to fuel his or her own ideas.
-
However, one I can tell you for sure. No human will dedicate his or her life to follow the arc laid out by toady. Toadys arc dies with toady, thats for sure.
I know I would, though probably with different priorities.
-
I don't know about TWENTY YEARS, but I for one would probably be willing to dedicate a good 5 years at least to this game if I were suddenly thrust into the helm.
Mainly because I think that's all it would take. I wouldn't give two hoots about 90% of the stuff in adventure mode, and would treat it only as a convenient sideboard tool for manipulating stuff between (in time and space) fortresses. Such as ferrying things around, or seeding characters to be a member of your fort. Thus, stuff like prison terms and reputations and dialogue and blah blah = totally unnecessary.
And fortress mode is significantly further along already. The arcs there are far more "doable" and IMO less controversial or things that most people would have a difference of opinion on, and could be themselves finished more rapidly.
Also, much of the charm of DF is after all hilarious random unpredictable destruction and violent capriciousness, which by its very nature is at odds with adventurer mode. Either you embrace that, and 99% of the time your adventurer is a bloody smear on the road within half an hour (frustrating / not that fun), or you fix it, and lose all the DF spirit, rendering you with just another not so exciting roguelike. Also not that fun, might as well go play Skyrim. Seems lose-lose to me. Fortress is where it's at.
-
I imagine that in the event DFHack would simply become DF.
And Stonesense. They should be able to fully integrate it as UI before project "DF Open Source" burns out.
-
And Stonesense. They should be able to fully integrate it as UI before project "DF Open Source" burns out.
Can't STAND stonesense. The ASCII graphics are far superior, IMO. They are as beautiful as your imagination. Stonesense is as beuatiful as... I dunno... mediocre mid 90's industrially cranked out SNES games?
It's like taking Moby Dick or Hamlet, and adding LEGO dioramas. Blech.
-
Speaking from watching software projects, most times, in the long run a projects that restart a code base from scratch more often than not pass by the originator. Especially if at has grown beyond a decade.
So if you are willing to dedicate yourself, there is nothing stopping you starting right now!
-
It's like taking Moby Dick or Hamlet, and adding LEGO dioramas. Blech.
Lego dioramas are pretty original, IMO.
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. (cut)
History of open-source games shows otherwise. Project like DF needs extremely dedicated person like Toady. No Toady, no project. At most, few years of small development, bugfixes, trying to make some things, sad agony when amount of interested people dries up and later death.
As others said, it makes sense to start new project. There is only one teensy weensy little detail left: find some new Toady-like extremely dedicated programmer. Good luck with that.
Freespace Open has been going for about a decade I think and has some pretty massive download rates.
-
Anyway, to my experiences in events like tis, the project at large is more likely to die. You cannot expect a coding community to start from zero to full should the event arise, such a thing needs years of caring to build up. Small fixes, chaotic add-ons, yes. But no more a dedicated project.
I respectfully disagree. (cut)
History of open-source games shows otherwise. Project like DF needs extremely dedicated person like Toady. No Toady, no project. At most, few years of small development, bugfixes, trying to make some things, sad agony when amount of interested people dries up and later death.
As others said, it makes sense to start new project. There is only one teensy weensy little detail left: find some new Toady-like extremely dedicated programmer. Good luck with that.
Freespace Open has been going for about a decade I think and has some pretty massive download rates.
Freespace's Engine was finished when it went open-source. If it was released in an unfinished state, with bits of code for features that nobody knew being left around and cluttering up the place, and hadn't gone through extensive bugfixing/optimization before hand, it would have been a lot more difficult for someone to pick it up and start making additions. Not that it would be impossible, but you would need people studying the code extensively.
Fortunately avoiding all of this is rather trivial modding - just open toadyone.txt and add [NO_AGE], [NO_EMOTIONS], [NODRINK] and [NOEAT]. Memorialize everyone on the map to avoid ghosts then atom-smash them. Turn off caravans, immigrants and invaders, and seal the fortress off from wildlife. Toady can now continue working until you experience save corruption.
-
It's like taking Moby Dick or Hamlet, and adding LEGO dioramas. Blech.
Lego dioramas are pretty original, IMO.
It works for the Bible. (http://www.thebricktestament.com/home.html)
-
I think I remember seeing that somewhere.
-
so no murdering for the source code!
Presumably if you were willing to murder him for it, you'd also be willing to simply break into his house or whatever and steal it WITHOUT murdering anybody.
Plus, even if you are a weird psychopath who sees the code as more valuable than a human life, you'd still have to realize that keeping him alive = more / better code in the future that you could steal again, etc. etc.
Unless they were one of those crazy people out for the infamy. So clearly we need to have contingency plans that would ensure that all motivations are foiled and all murder attempts are nonexistent. Someone wants the code? Throw code into magma. Someone wants the fame? On the same day, nuke Justin Bieber and say North Korea did it. Any other motivations?
I suppose our greatest threat will be the reprogrammed terminators sent from the future by the human resistance to try and stop Toady from creating DorfNet. It's all in vain however, and I for one welcome my graphicless overseers. Cave in traps should deal with them.
-
so no murdering for the source code!
Presumably if you were willing to murder him for it, you'd also be willing to simply break into his house or whatever and steal it WITHOUT murdering anybody.
Plus, even if you are a weird psychopath who sees the code as more valuable than a human life, you'd still have to realize that keeping him alive = more / better code in the future that you could steal again, etc. etc.
Unless they were one of those crazy people out for the infamy. So clearly we need to have contingency plans that would ensure that all motivations are foiled and all murder attempts are nonexistent. Someone wants the code? Throw code into magma. Someone wants the fame? On the same day, nuke Justin Bieber and say North Korea did it. Any other motivations?
I suppose our greatest threat will be the reprogrammed terminators sent from the future by the human resistance to try and stop Toady from creating DorfNet. It's all in vain however, and I for one welcome my graphicless overseers. Cave in traps should deal with them.
No guys.
DF causes the future to play out as "Dorf Matrix", and everyone finds themselves in the highly-boring-unless-you're-a-dwarf-or-adventurer world of DF, being played by Toady, ThreeToe, and the playerbase, who now abuse 14 billion (est. number of people) humans. They're also the future of humanity.
That's right, the future involves spending the rest of your life with a fellow forumite. Partner up now so the good ones don't get taken.
Then again, limiting the future of the world to the crazy people called "us" around here is a highly volatile move.
Oh, and we keep scientists around to mutate us into dwarves.
-
so no murdering for the source code!
Presumably if you were willing to murder him for it, you'd also be willing to simply break into his house or whatever and steal it WITHOUT murdering anybody.
Plus, even if you are a weird psychopath who sees the code as more valuable than a human life, you'd still have to realize that keeping him alive = more / better code in the future that you could steal again, etc. etc.
Unless they were one of those crazy people out for the infamy. So clearly we need to have contingency plans that would ensure that all motivations are foiled and all murder attempts are nonexistent. Someone wants the code? Throw code into magma. Someone wants the fame? On the same day, nuke Justin Bieber and say North Korea did it. Any other motivations?
Some men just want to watch the !!world!!.
-
History of open-source games shows otherwise. Project like DF needs extremely dedicated person like Toady. No Toady, no project. At most, few years of small development, bugfixes, trying to make some things, sad agony when amount of interested people dries up and later death.
Freespace Open has been going for about a decade I think and has some pretty massive download rates.
Amount of downloads and time of staying alive does not say anything about tempo of introducing new features and the like. But okay, let's say Freespace Open counts. So what? One (or few) exception from general rule does not change anything.
In my opinion, DF Open Source will burn out for simple reason: legacy issues. I am not even talking about bugs, presumably bugfixes would be first things that would be taken care of in case of DF going open source (this alone would be for me enough to make whole thing worthwile, even if nothing happened after that).
So what I mean by legacy issues? Toady do not want to do anything involving such newfangled (merely many decades old) concepts like multithreading*. In modern and future world where you have zilion slowish cores in one processor, CPU-heavy program relying on single thread is doomed to fade into irrevelance. DF is alive only thanks to promises of new features and development done by certain extremely dedicated individual. Without Toady, DF cannot stand on its own.
IMSVHO starting from scratch (and using Toady code as guide and start point to develop things like algorithms for world generation etc) would be simpler than rewriting DF code for multithreading. This way in my view holds greatest chance of success. We avoid any legacy issues, old code that would break in unexpected ways and plehtora other issues - but it would still be DF in spirit.
And Stonesense. They should be able to fully integrate it as UI before project "DF Open Source" burns out.
Can't STAND stonesense. The ASCII graphics are far superior, IMO.
I don't think anyone will ever drop ASCII. Stonesense, ASCII, 3D - that would be just another user interface layer. Assuming anyone would get arsed to make decent UI API for Open DF, of course...
* There are other problems, but multithreading one is most jarring.
-
I can't even describe how much you're overstating the importance of multithreading. Better branch prediction is at least twice as important. What could be shoved onto a different thread? Pathfinding? The game would still have to wait for everything else to finish before it did its pathfinding.
-
What could be shoved onto a different thread?
There isn't (or doesn't need to be, rather) any causal contingency within a single tick. As in, you as a player have no right to complain, per se, that in one case, the machine on the left fired first within the same tick, and in another instance, the machine on the right fired first within the same tick, or what have you. Or that monster landed a hit first versus dwarf, or that magma flowed or temperature updated first, etc.
Therefore, within one tick, I think it's entirely reasonable to do pathfinding, temperature, flow, combat ALL as separate threads, why not? MOVEMENT might have to be privileged at the end, so as not to walk inside a wall that was completed in the same tick. But that's about it, I think, and movement is going to take up a very small part of the processing time of a tick.
If you pathfind and then in the same tick, a wall finished being built, so what? As long as movement itself is privileged, you'll just get a cancellation and a repath the next tick, that's all.
Edit: Actually now that I think about it, I don't think multithreading would even have to lose your guarantee about which machine fires first, etc. Since you could still APPLY all the changes in a canonical order. All that matters is that temperature doesn't have to rely on the combat outcomes of that same tick, and it doesn't. So they can be calculated on separate threads.
-
That said, multithreading is still not the be all end all of optimizations, though. It will AT THEORETICAL BEST double or quadruple your framerate, and in reality, of course much less than that. Probably more like 50% boost
Whereas other optimizations probably could have even more of an influence than that. For instance, proper item stacking, and not checking every boulder in the fortress to see which is closest. Which goes hand in hand with not keeping track of so many irrelevant details of each object, or so many pieces of clothing. Also, creatures in closed rooms don't need to keep trying to path out every tick... they can just retry whenever a tile is mined or wall built/torn down, etc. etc.
(Masterwork for instance only has "leather" not "giant blue female diabetic mole leather" and several other optimizations of that sort, and runs 25% faster so they claim than vanilla, even with more content)
Altogether, I'm guessing it would add up to significantly more speed than multithreading, but multithreading would still be a major player in optimization for sure.
-
You were the only one who said anything about machines. I only complained about pathfinding, since that's what's always brought up and that's probably the least feasible one (since state changes elsewhere can all affect pathfinding).
Also, boulder checking is optimized, believe it or not; an individual boulder is checked position and closeness-to-stockpile-and-buildings wise two ticks in a row, then 2 later, then 4 later etc. until each only is checked every 1024 ticks. When a boulder is checked and its state found to be changed, it'll reset back to the beginning. This was in an interview.
I'd say multithreading's best is a 50% boost. It'll probably be less.
-
I only complained about pathfinding, since that's what's always brought up and that's probably the least feasible one (since state changes elsewhere can all affect pathfinding).
It doesn't matter if stuff AFFECTS it. What matters is only if stuff affects it in such a way that impossible events or game crashes result due to the conflict. Which I can't think of any examples of.
Like I pointed out in the previous posts, if for instance a wall gets finished being built in the same tick, then a multithreaded pathfind might still assume that route is open. But that won't crash the game... it will simply cause the dude to run into the wall and then go "huh..." and repath. Just inefficient movement is the worst outcome. And honestly, even that would only happen 0.2% of pathfinds. How often do walls get finished being built in the same tick as your path IN your path? Even if it's in the very next tile, as long as you privilege movement at the end of everything else, you won't get crashes.
And so what? 0.2% of the time a dwarf walks further than it needed to. That is a VERY reasonable price to pay IMO for 50% faster FPS. It's totally in line with the quirky personality of dwarves anyway.
-
Creature blocking is way more a problem than wall building.
-
Again, prioritizing actual applied movement as something calculated after everything else is done during a tick solves any issues of creature blocking. All paths will be correctly and reasonably based on the positions of creatures at the beginning of the tick. All creatures will prone and collide appropriately, and so on.
But as far as I can think of, movement is the only thing that needs to have its own calculation phase.
-
Fortunately avoiding all of this is rather trivial modding - just open toadyone.txt and add [NO_AGE], [NO_EMOTIONS], [NODRINK] and [NOEAT]. Memorialize everyone on the map to avoid ghosts then atom-smash them. Turn off caravans, immigrants and invaders, and seal the fortress off from wildlife. Toady can now continue working until you experience save corruption.
Wait since when did Toady drink? I just assumed he absorbed H2O out of the air through his skin.
-
Honestly as coder, I personally hate classical multi-threaded code. Simply because its a nightmare to debug when you have a race condition, a lock up, or randomly destroyed memory due to a failed mutex. Around the millennium when it was the cool thing to do, I also did some multi-threaded stuff, nowadays I avoid it like hell.
Its correct that many things that take some process load, like path finding, can be computer in parallel with classical multithreading from one tick to the next. However, you'll need one master thread to collect all computations and execute all the changes on the main data tree which then will become a bottle neck. Otherwise if you allow all threads to apply changes on the main data tree, welcome to race condition and mutex nightmare. With mutexing you have the classic problem, make a few, and you'll gain little benefit from multithreading since one thread will hold large pieces of the tree while others wait for it, or go fine grained, and lose a lot of the speed benefits on the load created by mutexes.
I agree that it is impossible to add something like this on an existing code base. If it would have been to be planned from start. Anyway, I agree with putnam. Maybe you get +100% or so speed. But then you'll hit a limit with classical multi-threading, no matter how many cores you go.
I didn't follow CPU-developments of late, but I got the idea that also here, the multi-coreing hits a limit, since unless you're creating something like a VirtualMachine host for lots of VMs, many have learned, you can only take useful advantage with a few cores, while more hardly get anything new. I think we have to accept we're hitting a limit not only on linear speed, but also on parallelism with core in Van-Neumann design.
One thing that might be big is being able to utilize the GPU for DF calculation. I googled a bit, there are people that use GPU for path finding. BTW: Even on supercomputers, most of the top hundret nowadays go vector-based, GPU. There is less and less classical calculating in the upper area.
If I'd had the time to design DF from base, I'd go multiprocess with a database design. Have one main process (the database) that holds the world but does not compute anything, and a few processes for stuff that communicate with the main process via a socket what they'd like to change from tick to tick and getting the changes from the world as answer. Creatures, stuff and areas can be assigned to processes.
-
However, you'll need one master thread to collect all computations and execute all the changes on the main data tree which then will become a bottle neck.
Yes, it has to wait for whatever the slowest thread is. But if made up data:
Pathing = 50% of calculations
Temperature = 40%
Combat = 5%
trajectory and vectors = 5%
Then add on 5% as long to collate all of them in a main thread at the end and apply final movements, even a dual core processor can now still finish the tick in 55% of the total time it did before. Not much more, ever, though.
Although hm... pathfinding doesn't really have to be on one thread, does it? Nor does all of temperature, really, does it? Couldn't you assign 1/4 of all creatures to one thread (or even each creature to its own), and some number of tiles to threads for temperature? If you only have a dual core, that wouldn't add any extra speed in the above example. But if you had quad core, it would now only take 30% as long as the vanilla tick (pathfinding a temp. both sliced and diced optimally, all cores running for 25% of the normal time + 5% for collating at the end). If you had 8 cores, it would take 17.5% (12.5% as long, all 8 cores maxxed out, + 5% collating) as long as a vanilla tick... Ignoring operating system needs for now.
How much of a benefit you can actually see depends on how fine-grained you can make the thread cutoffs. And why couldn't it be potentially as fine-grained as individual creatures for pathfinding, for instance?
In a future with 16 core processors and greater than 16 creatures in a fortress, you might very potentially see 1,000% improvements in FPS!
-
All of the pathfinding is going to be accessing the same map, so you can't actually thread it in that manner or you'd probably get a lockup of some sort... unless you copy the entire map for each thread, which might cause other issues.
-
It doesn't have to make a new copy of the map every tick. Only when the map changes (mining, building/tearing walls, bridges toggling), and even then it only has to update changes to each copy. So it might affect standing RAM usage, but shouldn't affect processor speed in pathfinding to have many copies available for different threads to read.
And a 4x4 embark 50 tiles in z height = about 2 million tiles. Allow for a few relevant states for pathfinding (ramp/not, door/not, open/not, whatever) probably can represent each with 1 byte = pathfinding relevant map copies only take up 2 megabytes each. Times the number of cores on your system, quad core only needs 8 megs of RAM to avoid any read bottlenecks from threads sharing.
-
I think you're gravely underestimating there how slow RAM is.
-
Every creature can have its own core. But you have to gather all the creature decisions into the one main thread to apply them.
The percentages you give, they are already a sum of the basic computation and the costs of applying the data to the tree. You can parallelize the former well but not the later. Especially with temperature, I don't see how one can make much useful computation for temperature without being able to apply it on the data tree right away.
As I wrote, going too fine with threads, basically speaking, threads have to go through semaphores to come together again, like for a tick. Semaphores are not too cheap. Go too fine-grained, and you'll have the semaphores eat up all the benefits you'd get from parallelism before.
(Classical) Multi-threading isn't the golden grail we were told ten years ago. Vector-based multi-threading on the otherhand is awesome in its gains, but very difficult to adept your algorithms for it. Its not well suited for the human brain.
-
I think you're gravely underestimating there how slow RAM is.
Yes, I am at my limits of knowledge here. Is there only one memory bus for all cores? If so, multithreading pathfinding will not help much at all.
And if that is how computers are built today, should we expect that in the near future there MIGHT be a memory bus for each core with redundant connections? Or is there some good physical reason that's infeasible?
I.e. in general, would two threads accessing DIFFERENT memory addresses take twice as long to wait on memory as one thread? Or as long? Or 1.2x or something as long?
-
Honestly, how exactly its working on base level I'm no expert in, I've doing most of my coding on levels where these things are abstraced away, so take it with a grain of salt.
On a classical computer (desktop) there is only one system bus, so when another core wants to use it, it has to wait to get free. Cores have their own caches tough, so if the memory is in the cache, it doesn't have to use the bus. There is also prefetching going on, if the bus is idle, which is a guess-work by the CPU what memory might be used next. etc. There has been a lot of research into that. One core can block the whole bus if it does nothing but truely true random memory access. But in practice, a) access is not random but focused on pages that get cached, b) the core wants to do some calculations before it needs new memory again, which gives another core air to use the bus.
You can't just multiply bus access. A RAM chip only has so much wires.
-
I can't even describe how much you're overstating the importance of multithreading. Better branch prediction is at least twice as important.
I do not overstate importance, I focused on multithreading because it is most obvious and jarring problem. I know branch predictions can do miracles in some cases (http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array). Out of curiosity, where in DF you see potential to improve branch prediction? If I was to guess, probably inventory (these zilion boulders and other things lying around) handling.
With multithreading I was imagining things more streamlined and without nutty ideas. For reasons unknown to me GavJ wants to paraelize everything in main loop - and THAT is bound to create horrible, horrible problems. Nope, forget about that, GavJ. This is nonsense that will randomly crash your game.
So, how to do it?
In main loop for each tick we still do major things separately (DUH), each one in block. For example, as first block would be things by nature impossible to sensibly paraelize, like I dunno, finishing constructions, movement, maintenance of tick and things like that. Next blocks would be things like temp calculations etc, and as last - pathfinding. Multithreading would be used only in one block (of course only if in given block it would make sense at all) at once and main loop would wait in each block for every thread to be finished before going further.
This should be easier and leave state of memory consistent after every block.
Example: If you have say 4 threads assigned to pathfinding and say 100 creatures requesting pathfinding, each thread pathfinds for 25 of them. It will work without any lockups or copy of map for each thread, because whole shebang is in its own block. Pathfinding does not modify map and modifies only certain data for exactly one creature (path to follow) exactly one time in whole block. Map is not modified during pathfinding, as map modifications was already done in previous blocks of current tick.
-
With multithreading I was imagining things more streamlined and without nutty ideas. For reasons unknown to me GavJ wants to paraelize everything in main loop - and THAT is bound to create horrible, horrible problems. Nope, forget about that, GavJ. This is nonsense that will randomly crash your game.
That's a very compelling argument you have there. "This is INSANE, because... because... unstated reasons!"
Can you please describe in more specific detail what sort of "horrible problems" or "random crashes" would occur from calculating, say for example, pathfinding, liquid flow, and temperature completely at the same time if you wanted to? A dwarf might accidentally have a path through 5/7 water for one tick? So what? That happens all the time anyway! When movement is checked later and separately, it will stop anything impossible from resulting.
Of course you can't do *everything* in parallel, and I explicitly said you wouldn't. You always need some number of "blocks" but each block doesn't have to have just one thing. It can have anything and everything in it that doesn't use the same memory addresses or rely on necessary contingencies with each other. Then the next block can have the next set of things that aren't contingently reliant.
And if you get yourself out of the mindset that EVERYTHING has to be "squared off" and fully up to date by the end of the tick, a lot more parallelism becomes possible. For example, who on earth cares whether on one tick, you do liquid flowing for a tile, and then temperature based on its new position, versus temperature based on its old position and then temperature-less flowing afterward? It will catch up in the very next tick, and I don't see how that will meaningfully affect anybody's gameplay.
-
If one threads writes memory while the other reads it, the second reads random garbadge. a lot of care has to be taken that such things cannot happen and are a real pain to debug since you cant reproduce that bug at will, it can quickly become a bug hell where crashes just happen every so often and nobody understands why... then you can ugly stuff like thread A requests exclusive access to block 1, gets it and block 2, which is taken by another, so it waits. thread B requests exclusive access to block 2, gets it, and block 1, and is taken by another, so it waits. Bang. Hang!
Really trust people who have written some multi-threaded stuff, and trust the coders who say, you'll have to structure it very tightly to work at all.
There is some value in doing pathing in parallel. But you just cant do anything. And as we discussed already, if you overload the system bus, you don't get anything worthwhile from more cores.
-
*eyeroll* This is a very straw man-y thread.
Nobody is suggesting you just slap random things into threads without thinking about it at all and with no care or consideration. Obviously that will break your game.
What I'm saying is that most of the intensive calculations in the game don't actually LOGICALLY require contingencies on one another, so there almost certainly is indeed a way to run them in parallel, after of course having thought it through carefully. I'm just saying it could be done with some effort, because there's nothing fundamentally in conflict betwee, say, pathing and temperature such that they logically would need to be run in sequence.
The only things to worry about between those two in terms of logical conflicts are paths that end up moving through newly formed ice or obsidian, or rely on ice that is going to melt, etc. But those are only actually conflicts with MOVEMENT, not pathfinding. A path can go ahead and be defined through a wall no problem, as long as the movement itself is in a protected part of the calculation and checks the tile it's moving into each time it happens to make sure it's still open. Which of course it already does.
Yes, you'd still have to do stuff like make sure the pathing has a cache to work from so it doesn't read the same memory address as temperature is writing an ice wall to, etc. You have to think it through, but I'm saying since there aren't any inherent reasons why they really need to be using precisely the same memory address, a solution is there. Probably a pretty simple one (a map of an entire default embark site would fit into a core's cache easily, even without any compression. With compression [such as "next 15 blocks are impassable" instead of "impassable impassable impassable impassable impassable impassable impassable impassable impassable impassable impassable impassable impassable impassable impassable"], it would take up a tiny space in memory)
-
What I'm saying is that most of the intensive calculations in the game don't actually LOGICALLY require contingencies on one another, so there almost certainly is indeed a way to run them in parallel, after of course having thought it through carefully. I'm just saying it could be done with some effort, because there's nothing fundamentally in conflict betwee, say, pathing and temperature such that they logically would need to be run in sequence.
Well, there's also fire, which is a mite more risky should it fail, especially since I'm not sure if there are any special rules for moving in and out of fires (adventure mode won't let you do it unless you careful-move, but I'm not sure if the AI is doing it in pathfinding or movement).
-
That's just a matter of adding another movement check during the movement phase, not a conflict with pathfinding.
It seems like Toady might not want to do this for dwarf personality reasons, based on how long that's been a "bug" (feature?), but assuming dwarves aren't suppsoed to walk into fire, you just have to check whether there's fire in the next spot or next couple spots in your path, when you are checking movement in its own very short little period of protected time when it checks that near the end of the tick. Same time you check to see if there's a wall there now.
-
GavJ, go ahead and write a larger project using multi-threading. It seems to me you got some theoretical ideas about threading, but never had the "joy" to work through a complexer project utilizing it, and seeing how tons of semaphores a) eat up your coding time, b) eat up your debugging time and c) even eat up cpu time gained first place.
Nobody says there isn't some value gained, but you're definietly overestimating it. Unless you'd go some completely different like SIMD.
Like I told you from bening, which has been rephrased a few times by others. At the end you'll have to have one main thread that collects all the computed data and applies it. It be the big semaphore all stuff will have to go through. This will be a bottle neck. It will be faster than currently, but with 8 cores, definitely not 8 times faster. If you'd leave that main thread away, you'll have no predictable race conditions and chaos. If you have it, you'll have the pathing diamond shape. Which is the worst kind of shape.
I imagine with classical cores, you'll get twice as fast as currently or something like that. Not much more, you'll hit a limit.
This whole thread discussing threading is pointless, if you want to, you're free to write a DF-clone using multithreading. Thats all that will ever get out of it.
The DF code suffers much bigger performance from much more trivial things anyway, like fat dwarves, or the container-temperature flickering and the such. Sometimes I got the idea, it does a lot of binary searches, where hashtables would be applicatable.
-
I have indeed written multithreaded code for a largescale game project (a creatively powerful 3d photoshoppy equivalent of minecraft). The project ended up being scrapped, interestingly (considering the grpahical difference to DF!), due to complete failure of me or my partner to wrap our heads around the abysmal thought-graveyard of humanity that is OpenGL. The multithreading actually worked super well.
Although it was from the beginning, which makes all the difference of course. And I also agree that regardless, DF does have a ton of other much lower hanging optimization fruit first.
Also I did it in Java, which had a bunch of super useful tools for doing things like dynamically adjusting threads on the fly for you. And in general fills in a lot of gaps efficiently for people who don't 100% understand all the engineering level things going on. This very much helped with avoiding unnecessary overhead, for instance. I don't know how good those tools are in other languages or Toady's expertise.
-
Its one of the uppersides of java that its much easier to write proper thread-safe code than in most other languages.
As I've repeating myself several times now, the abysmal thought-graveyard of humanity is though, where the real performance gains lay. Utiziling the GPU even for non-graphical-vector stuff, like temperature or pathing.
-
Of course you can't do *everything* in parallel, and I explicitly said you wouldn't.
My impression is you wanted that on beginning (http://www.bay12forums.com/smf/index.php?topic=138339.msg5259215#msg5259215) (except actual movement for some reason), but later changed goalposts, of course without admitting that maybe making everything in main loop threaded simultaneously isn't best idea in universe.
That's a very compelling argument you have there. "This is INSANE, because... because... unstated reasons!"
It is obvious to everyone here (except you, apparently).
Can you please describe in more specific detail what sort of "horrible problems" or "random crashes" would occur from calculating, say for example, pathfinding, liquid flow, and temperature completely at the same time if you wanted to?
These two things (pathfinding and temperature) theoretically should not be affected - unless some data is shared. Like, you know, dwarves avoiding too hot places. At end is larger explanation of potential problems with that.
And if you get yourself out of the mindset that EVERYTHING has to be "squared off" and fully up to date by the end of the tick, a lot more parallelism becomes possible. For example, who on earth cares whether on one tick, you do liquid flowing for a tile, and then temperature based on its new position, versus temperature based on its old position and then temperature-less flowing afterward? It will catch up in the very next tick
Nonsense. Complete, utter nonsense.
If you do not wait for flow or temperature calculations to end of tick, these calculations start to get out of sync with rest of game - and this is cumulative. Rest of game gets what is essentially unpredictable and randomly changing water/temp values based on something that may happened on this tick, last tick or who knows what, when and where. In other words, it will NOT catch up, as state of flow/temp relies, among other things, on previous state - and that will be incerasingly out of whack.
Next problem - map remodelling. Changes on map will be reflected late and inconsistently (like bridge closing and still letting some magma through like it wasn't here).
Another problem: in same tick, different "blocks" in main loop (or even different creatures/items/whatever) may get different value of water/temperature. Hell, it is possible that during handling of one creature (say temperature calcs for different layers of tissue) temperature will change in middle of calculations. This kind of thing can and will create extremely subtle and obscure buggy behaviour happening in already complicated code. I do not know how temperature calcs are done - if code writes values multiple times in same place during one run, it will get even worse.
I have name for someone that would accept this kind of risk for ANY gain.
and I don't see how that will meaningfully affect anybody's gameplay.
*insert sarcastic laugh*
-
Its quite plausible to have pathing base on the temperatures of the tick before. So pathing can be done while temperature is done in the main thread. This is of course imples to have a copy of the whole dataset tick by tick*. Just add the very first step for dwarves to not set into stuff that burns them instantly.
Personally I'm a hugh fan of immutable datastructures anyway. This makes next to other benefits multi-access much easier. Its sometimes just not the most effective to copy the whole thing just change one value... I for one just accept this possible perfomance loss.
-
These two things (pathfinding and temperature) theoretically should not be affected - unless some data is shared. Like, you know, dwarves avoiding too hot places. At end is larger explanation of potential problems with that.
No it doesn't matter that dwarves want to avoid hot places. Go ahead and pathfind and just don't give a crap whether there is some slim chance that the square will catch on fire that same turn you pathfind. If it does, it's no big deal. Your only penalty is a dwarf walking a little bit further on rare occasions. NOT game crashes. NOT fiery death. As long as the dwarf checks for fire in front of him before he actually TAKES his next step, he can still cancel the now-dangerous path before he actually enters the danger.
Hence my emphasis on movement being the primary thing that needs to be non-parallel, not pathfinding. Planning wher eyou might go in the future does not actually move you anywhere impossible or dangerous. Moving moves you there...
Also, imagine a path that will take, say, roughly 500 ticks to complete. And let's assume that at some point it ends up obstructed. There is only a 1/500 chance that the thing that ends up obstructing that path will begin obstructing it in the exact same tick that the path was planned. Whereas there is a 499/500 chance that it will happen during some other tick, and that even if you calculated them in sequence, you'd still have to rely on MOVEMENT, not pathfinding, to see the dwarf to safety. So you're already obligated to check in front of you each step.
This obsession with thinking for some reason that pathfinding needs super up to date info all the time every time is the source of disagreement here. It doesn't. It can do perfectly fine with 1 or 2 or even 5 tick old information if it has to, almost every time. And the other times, it's just inefficient, not game crashing.
If you do not wait for flow or temperature calculations to end of tick
I apologize if I was unclear. Yes, every type of calculation needs to finish before every tick ends.
What I meant was that, for example, temperature does not NEED to have up to date information as of this exact same tick from liquid flow. Even though liquid flow is a major source of temperature change, you could just use the previous tick's state and be totally fine. Maybe one random guys' crazy dwarven computer won't work anymore, or something. But the other 99.9% of everybody else who doesn't absolutely need ice to melt THIS tick instead of NEXT tick gets a faster game, and that's worth it.
The only reason why, say, flow and temp would need to be done in sequence within a tick would be if temperature MUST be accurate based on fluid flow in that same tick. I see no reason why this is the case. So yes, you still wait for each thing to finish before ending the tick, but you don't need to wait for flow to finish within the tick before doing temp. You can just work off a cache of the last tick's state.
My impression is you wanted that on beginning (except actual movement for some reason), but later changed goalposts, of course without admitting that maybe making everything in main loop threaded simultaneously isn't best idea in universe.
Nope, goalposts still the same. I didn't say everything. I said:
* Pathing
* Temp
* Liquid flow
* Combat
I may be wrong about some of that, but I stand by it for the moment.
Combat is the only one I think is a bit iffy, but I am still pretty sure it would work, as long as it has its own thread to itself, not split up into several. You go based on last tick's positions, and combat resolves before you move this tick (since movement happens at the end), which should be fine.
-
Lol, this thread has been mostly, if not completely, derailed.
-
Lol, this thread has been mostly, if not completely, derailed.
Not only derailed, it was derailed with the assistance of about fifteen impulse ramps.
-
Fortunately avoiding all of this is rather trivial modding - just open toadyone.txt and add [NO_AGE], [NO_EMOTIONS], [NODRINK] and [NOEAT]. Memorialize everyone on the map to avoid ghosts then atom-smash them. Turn off caravans, immigrants and invaders, and seal the fortress off from wildlife. Toady can now continue working until you experience save corruption.
Wait since when did Toady drink? I just assumed he absorbed H2O out of the air through his skin.
It's to prevent him from harming himself by trying to drink or eat.
The last time we heard of him drinking and eating, he had ended up in a bad condition.