Post by: Impaler[WrG] on October 12, 2009, 08:06:55 pm
As you all know Path finding is by far the biggest drag on DF's FPS and the main limiter of Fortress size, indeed the current #4 suggestion on Eternal Suggestions is "Speed Up Pathfinder". Many programmers have suggested ways to improve the speed of these path searches or ways to over-come certain limitation inherent in the current implementation. If a solid system can be demonstrated we hope it will be incorporated into DF, to this end the library will use the BSD license.
I've invited a number of programmers and designers together to work on this task. Our initial team is already large but more are welcome to join (to compensate for those individual unfortunate enough to be hit-by-a-bus)
Puzzlemaker (lead)
Slogo
Draco18s
Silverionmox
Sunken
numerobis
And what project would be complete without a mission statement:
Create a flexible BSD licensed C++ path-finding library specialized for grid based systems like that of Dwarf Fortress and able to efficiently handle rapidly changing map geometry and other challenges such as but not limited too multiple travel domains, multi-tile creatures, variable movement cost terrain and path caching.
The idea here is that the library will be very modular and easy to include into a game, acting almost like a server in which it receives an initial dump of map data from the client, and then during game play the client sends map updates on changes and path requests and the server sends back the needed paths.
Khazad & dfHack would provide a platform for testing, viewing and measuring how effective it is (by use of a path request generator) and allow the programmers to concentrate on pure path finding without worrying about anything else. The first task is to of course do a bunch of design work some of which will take place here, everyone is free to help out on that but please do keep things on topic.
Post by: Puzzlemaker on October 12, 2009, 08:13:31 pm
Post by: Alexhans on October 12, 2009, 08:21:45 pm
Post by: Idles on October 12, 2009, 09:34:13 pm
http://harablog.wordpress.com/2009/01/29/clearance-based-pathfinding/
http://harablog.wordpress.com/2009/02/05/hierarchical-clearance-based-pathfinding/
I was planning to implement his system in a personal project, but alas this semester has a heavy workload. He has done performance analysis on his implementation of the system, and shows it to be much more efficient than straight up A* for long path-lengths. One of the big elements this paper does not cover is the method of updating the pathfinding graph based on changes to the game world, but it looks like a relatively easy problem to solve. The code also does not address extending this 2D system to 3D, but that problem is also not terribly difficult to solve.
The system handles hierarchical pathfinding (finding paths at a high level and low level), multi-tile creatures, and multiple travel domains (walking, swimming, amphibious, flying, etc.). Connected components checking (currently used by DF) can be done relatively quickly by inserting the path start and end nodes into the abstract graph and finding an abstract path between them. This new form of CC checking would even work for flying monsters, unlike the current code!
Definitely worth a read as a starting point for designing the system.
Post by: buman on October 12, 2009, 11:44:15 pm
Quote
acting almost like a server in which it receives an initial dump of map data from the client,
This would allow it to be coded as a parallel thread yes?
Post by: Impaler[WrG] on October 13, 2009, 12:55:34 am
Post by: zwei on October 13, 2009, 01:58:49 am
Simple drawbridge raise or door lock can disconnect and reconnect large parts of world, leading to potentially huge overhead and path recalculations with no benefit, if you are to deliver real improvement, you will need to be able to store several versions of already pathed map cached to avoid pointless recalculations for expected, but random, changes.
Also, since you know how map will look like in n+ seconds (cue taken from dig/build/cut tree designations and dwarf locations), you can have pathed maps (or sections of pathed map) ready to be instantly plugged in when construction/designation is being excecuted and when it is finished.
You can also, for example, precalculate map for next season because you know that tree saplings will become obstacles.
If you work with prediction in separate thread, you can have already updated map when game commints changes and really requests path.
I think you are looking for more df-connected system than you would expect.
---
Also, I hope you did consider batching map changes and not commiting them and recalculation pathing before actual path is requested: Often, several changes can happen at same time.
Post by: Angellus on October 13, 2009, 02:10:48 am
Major challenge you will face are disconnected / reconnected world areas and 'soft' changes.What he said, the smartest move probably is to let the 'server' constantly run to calculate different map possibilities and store them. Maybe even try and program the pathfinder to change the blocked parts? (If bridge is in way, then take route 16)
Simple drawbridge raise or door lock can disconnect and reconnect large parts of world, leading to potentially huge overhead and path recalculations with no benefit, if you are to deliver real improvement, you will need to be able to store several versions of already pathed map cached to avoid pointless recalculations for expected, but random, changes.
Also, since you know how map will look like in n+ seconds (cue taken from dig/build/cut tree designations and dwarf locations), you can have pathed maps (or sections of pathed map) ready to be instantly plugged in when construction/designation is being excecuted and when it is finished.
You can also, for example, precalculate map for next season because you know that tree saplings will become obstacles.
If you work with prediction in separate thread, you can have already updated map when game commints changes and really requests path.
I think you are looking for more df-connected system than you would expect.
---
Also, I hope you did consider batching map changes and not commiting them and recalculation pathing before actual path is requested: Often, several changes can happen at same time.
Post by: Impaler[WrG] on October 13, 2009, 04:12:02 am
Quote
What he said, the smartest move probably is to let the 'server' constantly run to calculate different map possibilities and store them. Maybe even try and program the pathfinder to change the blocked parts? (If bridge is in way, then take route 16)
Constantly creating random paths and caching them sounds like a very wasteful approach, not only would their be very very little likelihood that those random paths would match the actually needed ones but it would eat up cpu cycles constantly (I'm not going to assume multi-threading at this point). I think any kind of useful caching system will be one that caches only paths that have actually been requested by the client AND keep getting requested frequently. Of course their would be a means to adjust cache size and the 'stinginess' of the cache/don't cache logic would respond appropriately.
Quote
Also, I hope you did consider batching map changes and not commiting them and recalculation pathing before actual path is requested: Often, several changes can happen at same time.
This is an interesting idea that should be explored. Were defiantly going to need a system that can receive map changes in groups of tiles (such as the raising of a draw-bridge which affects dozens of tiles simultaneously) so it would seem that saving certain connectivity calculations until a path is requested might indeed save a lot of recalculating.
The high level connectivity zone issue (such as when the only drawbridge over the chasm that divides the whole map in two is raised and we need to know that their are no possible path across) can be solved with a Hierarchical node system like that in the links provided by Idles. Most such systems use only one 'layer' of abstraction above the actual map data but I see no reason not to use many in a tree structure. The leaf nodes would consist of groups of connected tiles and each subsequent node moving towards the root would be a larger and larger interconnected group of nodes, the final set of nodes directly connected to the root would constitute the fully disconnected regions of the map and it would be a simple matter to check if that node matches for any two tiles.
The tree would of course need to be changed to reflect changes in the map but this would be relatively simple because an area that detaches from the rest of the map can just be 'grafted' onto the root and then grafted back when they reconnect. That keeps the modification to the tree as small as possible. I'd need to think further on how some of the other systems like movement domains would work, it might makes sense for them to each have a separate tree or to have separate nodes in the same tree or even shared nodes, though I suspect separate trees are the best solution.
Post by: DaveT on October 13, 2009, 04:25:25 am
Bonus points if every time a group of male dorfs get together at a party they discuss routes and improve on their knowledge of surrounding terrain.
90% lighthearted. 10% because I'm going to implement something like that for a pseudo game that I'm 'developing' (as in I want to be but get very little time to spend on it).
Post by: Impaler[WrG] on October 13, 2009, 04:39:10 am
Getting this right will be key because a game like DF is making huge numbers of checks for potential paths for every one that is actually conducted. I'd say its so important that design should be centered on optimizing this huristic function even to the detriment of full path creation. Some control over the amount of uncertainty in the huristic would also be nice, say either as an accompanying argument on each request or as a setting at initialization. This would allow the client to trade off time and accuracy to for example test a large number of possible paths with low accuracy and then test the three best with higher accuracy for to make a final decision.
Post by: Firnagzen on October 13, 2009, 04:45:46 am
Post by: Draco18s on October 13, 2009, 06:38:19 am
Major challenge you will face are disconnected / reconnected world areas and 'soft' changes.
The way to do this might be to store chunks of the map that become disconnected as the result of a path change (update as needed).
Eg:
Whole map (stored)
#######^###### <-- path to more map
#.............#
#######*###### <-- door
##..........##
##############
When the door closes, you store this
##############
##..........##
##############
As a separate chunk that shows the paths only in that portion of the map, and assumes that it is disconnected from the rest. When the door opens again, you go "ok, not using it, but I'm not throwing it away" and save it for later when the door gets locked again.
Post by: DeathOfRats on October 13, 2009, 06:54:29 am
Of course, something like that is not really suitable for great expanses of open space, but rather for areas where you have lots of small rooms, and it would still require some sort of algorithm to do the pathing, since it's just a data representation.
Post by: Puzzlemaker on October 13, 2009, 07:27:02 am
Post by: Draco18s on October 13, 2009, 07:27:54 am
Ideally, you only store any contiguous region that has creatures in it, though separating out what does and what does not may be a non-trivial problem.
Also, you would have to garbage collect old chunks that haven't been used in "a while" as to keep down the memory footprint (mind, we are attempting to trade CPU for memory here, but there is no sense in using more than we have to). Another non-trivial problem would be finding the "the stupid dorf locked himself in" single-tile chunks and delete them when the dwarf is let out (eg, a wall was built on that particular tile and that area will never be an active set*). If not, it will eventually fall out of the inactive set.
*This here might be a part of a larger issue: do you update the inactive chunks as walls are built? Likely would be trivial in both the programatic sense as well as the computational sense. Just update all chunks with the new wall (each chunk that does not have that location inside it does nothing). If it splits a region, don't worry about it. Assuming a chunk is divided by new walls (making two smaller chunks) don't worry about it, you can worry about splitting it when it becomes active again later--assuming it ever does.
Post by: DeathOfRats on October 13, 2009, 07:37:54 am
I still think that some sort of node-based pathfinding is, in general, a better bet than tile-based, though. Actually, I'd say that tile-based is just a degenerate case of node-based, in which eat tile has a node. Also, reusing previously calculated paths (always a good idea) would be much easier with nodes than with tiles.
@Draco18s: I think I see more or less what you mean, but, I'm not 100% sure how it'd work. How do you path from a location with a dorf to one, say, on the other side of the map (which would, presumably, be in none of the chunks you've stored)? Would that cause the system to suddenly store a good portion of the map in memory?
Post by: Draco18s on October 13, 2009, 07:47:34 am
@Draco18s: I think I see more or less what you mean, but, I'm not 100% sure how it'd work. How do you path from a location with a dorf to one, say, on the other side of the map (which would, presumably, be in none of the chunks you've stored)? Would that cause the system to suddenly store a good portion of the map in memory?
2 things:
1) non-trivial to decide that a chunk is never used (eg. no creatures in it, no path to anything else), but could be possible for smaller chunks.
2) if it's a chunk there are no paths to any other chunks, which is what makes it a chunk. Thefore, attempting a path outside of the chunk, (as previously stated) fails.
As for deciding which chunks are active and inactive, my gut reaction would be to use the connectivity map (the floodfill). Each portion of the map that is one "color" in the floodfill is its own chunk. Hopefully not every time the floodfill is updated do we need to make new chunks (because a wall was built, a tile was mined, etc), as we could hopefully isolate which tile changed status (from walkable to not, and vice versa) and just update that one tile on all chunks (active and inactive). The only reason to make new chunks or to swap out chunks would be if doors, floodgates, bridges, or other such structures changed status. Then isolate which chunks are effected and move them to the inactive set and find smaller inactive chunks which have already been computed that make up the lopped off sections (then perform a new chunk creation for any remaining empty areas).
Post by: smjjames on October 13, 2009, 08:43:41 am
Post by: DeathOfRats on October 13, 2009, 08:46:17 am
I would imagine updating new tiles in one chunk or the other not to be too difficult, as long as the chunks retain location information. Besides, technically you'd only need to consider active chunks (I cannot really think of a case in which an inactive chunk would need to be updated - or why it can't be done at activation time, for that matter).
My main problem with that is that you're going to end up with at least one giant chunk (i.e., the great outdoors), more if you do something silly like mine out a whole level underground (by contrast, that whole level could well be a single node in a graph, saving lots of memory and processing time unless you're pathing inside that particular level). Pathing within those chunks would still be a pretty big problem.
Still, I like the idea of only considering for pathing those areas that can be reached by dorfs (though I'd prefer to be able to include things like flying/swimming creatures, pets and building destroyers, not sure how we could handle that easily), and considering areas without possible connections as separate areas (at least, that would reduce lag from pets and other critters trying to path through closed doors).
Post by: zwei on October 13, 2009, 09:15:14 am
A major problem with that room idea is dynamically defining the rooms. The players can create very strange shapes, which may be hard to keep track of using any algorithm.
You can easily solve this by applying computer vision segmentation algorithms. Efficient ones have only two passes (one over image data, one to complete shapes.). That is 2d of course, but extending algorithm to 3d is trivial and retains same properties (two o(n) passes) and commonly used in medical applications to analyze MRI data.
Post by: DeathOfRats on October 13, 2009, 09:24:18 am
Post by: Slogo on October 13, 2009, 10:55:06 am
It helps with caching the popular routes too as you can cache the routes that connect one region to another.
Post by: Fieari on October 13, 2009, 11:59:40 am
Post by: Silverionmox on October 13, 2009, 01:01:38 pm
A major problem with that room idea is dynamically defining the rooms. The players can create very strange shapes, which may be hard to keep track of using any algorithm.We should piggyback on the player's efforts there. He uses room definitions, burrows and zones, and we should make good use of that. As soon as workshops are defined similarly to bedrooms etc., much of the interior of a fortress will have been defined as rooms - and that's where most of the pathfinding happens. When the military arc gets some more attention, there probably will be first line of defense zones, second line of defense zones etc. that the player can define as part of his fortress strategy, to give orders more easily. Those can be used as well.
Post by: Fieari on October 13, 2009, 01:07:23 pm
Post by: Slogo on October 13, 2009, 01:44:53 pm
Post by: Silverionmox on October 13, 2009, 02:32:31 pm
There's no need for the divisions to be defined along any sort of human friendly mechanism. It's about what the computer can handle the best.If we're using divisions, we might as well use the existing divisions that are supplied ready-made by the player. Those automatically are in the areas where the player wants stuff to happen and pays attention to. If we save on calculations at the cost of reduced accuracy, nobody cares if something in the wilderness doesn't go in a straight line to its goal. Inside of a fortress, that might be annoying if it happens systematically.
Post by: Impaler[WrG] on October 13, 2009, 02:49:13 pm
DeathofRats: Concave/Convex sounds much too difficult, I favor a system of all rectangles as described in this thread
http://www.bay12games.com/forum/index.php?topic=42678.0
Not only are rectangles the most common thing people build in DF their also efficient at covering the unmodified terrain as well and are much simpler to set up and can aid in doing multi-tile creatures as we can be assured that a creature thats longest dimention is shorter then the rects shortest can move through the rectangle.
Silverionmox: I'd be careful with using such data as DF allows zones to be chopped up and built over to the point that they would be useless for pathfinding, and as were already going to need a system that creates nodes without such cues I don't see much advantage using the data. Though here might be something to be gained in being able to request a path at the level of a node which is known to be contiguous with a destination.
I've though a bit more about the Tree structure I described earlier and think it could do quite a lot. Paths could be found by a kind of tree traversal rather then by a traditional method like A*. Think of it as progressively slicing away everything that is not a path from an area know to contain both end points.
To start with the leaf nodes will have connections (edges) amongst each other. These edges determine the next node up the tree as the rule is that all the daughter nodes of a node are connected. The connections amongst the daughter nodes of a node constitute the "INTRA" connections of the node and when they change the membership of the daughter nodes needs to be checked to see if they still connect. Any connection by a daughter node to a non daughter node is an "INTER" connection. Inter-connections between the daughter nodes of two nodes will result in an Intra-connection between the parents, when they change the Intra-connection of the parents is updated. The Inter-connections at each level of the tree cause Intra-connections on the next level up. All connection changes propagate up the tree until the tree is accurate again.
To path through the tree the two leaf nodes containing the endpoints are identified and an upward propagating check is made for a matching parent node is made. If a common 'ancestor' node can be identified below the root node then it means theirs a path and the path must exist within the set of tiles described by that ancestor node. Now starting with common ancestor the tree is descended in successive steps. In each step a group of nodes is searched for a path at Inter-node levels between two nodes known to contain the end points. The resulting set of nodes on the path are then opened and their non-leaf daughter nodes become the search set the next step until all nodes on the path are leaf nodes. Because each search level is through a small number of possible nodes it will be far faster to do a dozen searches through networks of only a dozen nodes each then to search the leaf level nodes directly.
Post by: Idles on October 13, 2009, 03:21:22 pm
To path through the tree the two leaf nodes containing the endpoints are identified and an upward propagating check is made for a matching parent node is made. If a common 'ancestor' node can be identified below the root node then it means theirs a path and the path must exist within the set of tiles described by that ancestor node. Now starting with common ancestor the tree is descended in successive steps. In each step a group of nodes is searched for a path at Inter-node levels between two nodes known to contain the end points. The resulting set of nodes on the path are then opened and their non-leaf daughter nodes become the search set the next step until all nodes on the path are leaf nodes. Because each search level is through a small number of possible nodes it will be far faster to do a dozen searches through networks of only a dozen nodes each then to search the leaf level nodes directly.
So what are you going to use as leaf nodes? Obviously not DF tiles, or you'd be duplicating much of the data inherent in DF's internal tile representation. Are your dynamic rectangles == leaf nodes? If you use dynamic rectangular grouping of tiles, rather than a uniform grid of tile "chunks", you then have to keep track of the spatial location of each rectangle, so that it can ultimately be mapped to DF tiles. This system does, however, allow you to optimize for areas in which pathing is homogeneous, such as large open expanses of sky.
Also, remember that a search performed on our path node hierarchy must return a valid path if a valid path exists within the current pathing engine, so we must be careful when optimizing the connections in the path tree. It is not as important, however, if our computed path is sub-optimal (longer) than the current best-path method.
Post by: Dae on October 13, 2009, 04:35:57 pm
Here is some explanation about the pathfinding algorithm used in Baldur's Gate, amongst others. (http://web.cs.ualberta.ca/~mmueller/ps/hpastar.pdf)
Post by: Impaler[WrG] on October 13, 2009, 04:53:47 pm
Quote
It is not as important, however, if our computed path is sub-optimal (longer) than the current best-path method.
To a degree yes I agree that we do not necessarily have to have the optimal path, a path that's just slightly longer can be adequate. The extra length of any path returned vs the length of the optimal path can be expressed as a percentage. For example if the optimal path is 100 long and the found path is 106 units long that means 6% inefficiency. We should at the very least measure this value and ideally be able to set the acceptable level of inefficiency and get a useful trade off as a result.
Post by: Impaler[WrG] on October 13, 2009, 05:27:42 pm
Post by: Symmetry on October 13, 2009, 05:32:42 pm
One thing that always annoys me in DF is the way rocks for making rock craft are taken from the nearest tile. A better heuristic could help select rocks which are actually closer, reducing the need for lengthy paths. The shorter the path the quicker to calculate after all. I realise this is slightly outside the scope of the project.
The Node tree idea is interesting, but I feel it solves the easiest problem. We can already find paths, and we can already sort-of find paths between nodes in a A* hierarchy without too much difficulty. Improving the speed of this i good but improving the accuracy of the heuristic will give us more benefit. The improvement we should concentrate on relates to how the nodes in the hierarchy are calculated. At what point do we break a single node in two? At what point do we coalesce?
One huge advantage of using a node based hierarchy for DF is that we can split nodes around variable terrain features. eg. Drawbridges. But further it would be possible to split nodes at locked doors or at the water's edge. Then we can treat different nodes as connected or not based on whether the path-finder in question can swim or can break down doors.
Also, and most usefully, it seems like we could split the nodes at points that help increase the accuracy of rough heuristic calculations. If the distance between two nodes is say 10 units across another node, but the heuristic naively uses the size of that in between node and guesses at 100 units, when we finish finding the path and discover how large the error is we can split that node to improve the heuristic in future.
I'm a bit fuzzy about how you handle coalescing, and mainly quickly detecting a coalesce is desirable. Also the exact way you'd split a node with more than a single tile border is fuzzy too.
Post by: Impaler[WrG] on October 13, 2009, 06:37:33 pm
Quote
I'm a bit fuzzy about how you handle coalescing, and mainly quickly detecting a coalesce is desirable. Also the exact way you'd split a node with more than a single tile border is fuzzy too.
That's cause I'm still fuzzy on it myself, as was hinted at earlier I see the rectangular areas as being the leaf nodes. The splitting and merging of these nodes will be necessary as the map changes but exactly how it would be handled I'm not quite sure. Their seems to be a method called "Framed Quadtree" that could be of use here.
As for the heuristic I was wondering if it would be possible to store a great deal of pre-computed heuristic data inside the nodes of the tree structure I'd been describing. A really simple solution is to just store a value in each node that comes from estimating the travel time inside the node, for a leaf node this would be the 1/2 hypotenuse of the areas sides. Larger nodes would just mash together the lower level heuristics and divide again. In essence this would be a rough estimate of the distance across the node and once the common ancestor node (the lowest level node that contains both end points) is found returning its value would at least tell us if a proposed path is very long or very short. That might be a bit better then using Manhattan as the heuristic.
A more sophisticated heuristic would be to actually do a bit of A* pathing and store the results, say for each combination of Inter-2-Inter connections within a node (for example Node A connects to B, C and D so a test would be done for B<>C, C<>D, D<>B across the area of A and stored in A. This would allow any proposed path to be estimated by summing up a large number of small legs and be much more accurate, though at the cost of more memory and cycles spent on updating each time the map changes.
Post by: Sunken on October 14, 2009, 02:35:50 am
You were talking about the subdivision into space, mentioning convex regions and rectangles. I am using a third approach which I call "trivially connected zones". A TCZ is a region where a dorf can always reach any other tile in the TCZ by simply moving stupidly towards that tile - including "sliding around" obstacles in a least-resistance manner, like dumb homing enemies in Chip's Challenge or Nethack. Thus pathing costs within a TCZ are negligible, and TCZs can be potentially much larger than convex areas - a whole outdoor forest with sparse trees, for example, can be one TCZ, where it would be hundreds of rectangles or convex zones. Indoors, a large room with single-tile or two-tile workshop obstacles can likewise be a single TCZ.
Pathing is reduced to an A* search over the gateway tiles connecting TCZs. The paths are not optimal - in fact the theoretical inefficiency is up to 50% - but in practice the inefficiency should be much smaller. I've yet to do extensive statistical tests.
There's an algorithm for computing maximal TCZs from a seed tile. It's not precisely inexpensive, but it's local in nature and so won't be too expensive in upkeep.
Post by: Sunken on October 14, 2009, 02:39:21 am
Post by: DeathOfRats on October 14, 2009, 03:36:25 am
Mind linking us to the TCZ algorithm you mention?
Post by: Sunken on October 14, 2009, 04:55:59 am
As for my implementation, it is rather rough around the edges and I'm still working on it.
But basically, the idea is: Start with assuming all the world belongs to the TCZ. Move outwards from the seed tile, and whenever a tile conflicts (that is, it wouldn't be possible to silly-move from it toward some other tile in the TCZ), forbid either it or the other tile, depending on which is closest to the seed. Then backtrack to reprocess any neighbors of newly forbidden tiles.
Keep it up until the whole world has been processed.
That's the gist of it, though there are many obvious simplifications that can be made.
Post by: DeathOfRats on October 14, 2009, 05:21:51 am
The check could be done locally (say, on a 10x10 tile grid, which would mean a limit on the exponentiality of the thing), but then I don't think it would be too difficult to find a situation in which the silly-move requirement was violated.
Or would it be better to just limit the size of a TCZ to something we can manage? Is there something I am misunderstanding?
Post by: Sunken on October 14, 2009, 05:34:30 am
So it's not exponential - but even if it were, that wouldn't mean "unacceptable" right off - remember this is a one-time calculation, in principle, and its time complexity is much less important than the cost of pathing itself. Besides, the larger the TCZs, the fewer they are.
Anyway, you can bound the time taken by limiting the maximum size of the TCZ from the start, reducing the "whole world" to just a 30x30 grid or something like that. You wouldn't be able to cover the whole forest then, but hey.
Post by: Alexhans on October 14, 2009, 06:53:31 am
Post by: DeathOfRats on October 14, 2009, 07:08:21 am
Incidentally, that's part of my criticism to Sunken's TCZ (an idea which I otherwise like very much). I don't think you can realistically have TCZs (much) bigger than whatever size you're using for the local checking, since it makes it easier for the silly-move requirement to be violated (a little bigger might be possible, but I don't think you could, say, have a TCZ the size of a 2x2 z-level if you're doing 24x24 local checks, even if there is some overlap between the checks - overlapping would cost CPU time, but it would result in more robust segmentation, I'd expect).
Still, and like Sunken says, I don't see why we couldn't limit these TCZs to a size that's manageable locally (say, 12x12 or 24x24), and then use a hierarchical pathing algorithm like the one in the paper Dae linked us to (awesome paper, btw - I looked for a few others in the same vein in the IEEE paper database, and found a couple I want to read, but I didn't find any quite as impressive).
Post by: Sunken on October 14, 2009, 08:33:19 am
How you'd deal with changing terrain is more or less like this:
When a tile changes, find the TCZ(s) it contacts (use an efficient bounding-box algorithm for quick lookup).
Resume TCZ processing for all neighbors of the changed tile (and itself, if it was cleared) and process from there as before.
Finally, check tiles from which TCZs have withdrawn. If any remain uncovered by a zone, re-seed them until coverage is reestablished.
This will (unmodified) not reclaim space as aggressively as restarting the TCZ from scratch, but that may be just as well - maybe that tile will close up again anyway.
Post by: Shades on October 14, 2009, 09:09:09 am
That's part of the problem, and why I'd say it's important for whatever algorithm is chosen to have two characteristics: that segmentation works locally (i.e., I just mined a tile, I don't have to recalculate all the pathing for the whole damn world), and that different areas can be re-zoned (divided or merged). If those two criteria are met, it shouldn't be a problem to deal with terrain modifications
I think you'll be okay with convex zone / TCZ (the TCZ reads to me as basically how you use convex zones in most real life cases at any rate) under this regardless of size.
A new tile mined out is almost always adding to zone or creating a new zone, and building a construction (or growing a tree) is a simple check to see if it can be just ignored (ie not adjacent to another no-move square).
In the worst case, which will happen when players build walls for the most case, you can just recalcuate the TCZ and it's not a slow calculation as it's a single plane (I wouldn't recommend a TCZ that spans z-levels even though it's technically possible with ramps) and you can start from the point of dividing the zone into four with this new block as the corner (it should be trivial to decide which corner based on which adjacent sides it is blocked on)
Post by: Sunken on October 14, 2009, 09:16:53 am
This leaves fliers basically unaddressed by the approach, sadly. They'll have to be considered specially.
Post by: Shades on October 14, 2009, 09:41:14 am
I agree that TCZs should be 2D. They are adjacent to zones on other levels at the points where there are stairways, for purposes of path planning.
This leaves fliers basically unaddressed by the approach, sadly. They'll have to be considered specially.
You can have fliers using zones too, all zone needs to store is adjacent zones and in these cases every square would allow you to pass to an adjacent zone (vertically at least). Not the most efficent system but it would work.
Alternatively you could assume everything above a 'top level' zone is a trivial connection to critters passing through these can move freely, which although means a special case is probably a cleaner solution overall.
Post by: Sunken on October 14, 2009, 10:54:28 am
You need to store more than adjacent zones in the general case. You can't path plan on the zone level without knowing the costs, and the costs depend on exactly which tiles you're entering and leaving the zone from. The hierarchical clearance-based approach accepted a suboptimality in this respect, but that can get pretty bad for large zones.I agree that TCZs should be 2D. They are adjacent to zones on other levels at the points where there are stairways, for purposes of path planning.
This leaves fliers basically unaddressed by the approach, sadly. They'll have to be considered specially.
You can have fliers using zones too, all zone needs to store is adjacent zones and in these cases every square would allow you to pass to an adjacent zone (vertically at least). Not the most efficent system but it would work.
I suggest instead storing gateway tiles between TCZs and path planning between these instead. But that solution would be prohibitive for fliers.
Quote
Alternatively you could assume everything above a 'top level' zone is a trivial connection to critters passing through these can move freely, which although means a special case is probably a cleaner solution overall.That's a possibility, although it's not trivial to carry out in practice.
Post by: Draco18s on October 14, 2009, 12:54:25 pm
You need to store more than adjacent zones in the general case. You can't path plan on the zone level without knowing the costs, and the costs depend on exactly which tiles you're entering and leaving the zone from.I agree that TCZs should be 2D. They are adjacent to zones on other levels at the points where there are stairways, for purposes of path planning.
This leaves fliers basically unaddressed by the approach, sadly. They'll have to be considered specially.
You can have fliers using zones too, all zone needs to store is adjacent zones and in these cases every square would allow you to pass to an adjacent zone (vertically at least). Not the most efficent system but it would work.
Keep in mind guys that we do need to be able to maintain a functional set of traffic rules, such that we can restrict traffic from some areas. Entering a zone on a normal traffic tile, but having to path over a stripe of "restricted" to get to the other side gives you a valid path, but doesn't take into account the restricted nature of a portion of the zone.
Post by: Sunken on October 14, 2009, 12:58:32 pm
Obviously, this makes for trouble if the traffic zones are changed a lot or are badly shaped (dots spread out in the middle of rooms and so on).
An interface change could help remedy this: If the player can see TCZs when setting traffic zones, he can elect to paint entire TCZ with the same cost, which means no TCZ reshuffling is necessary.
Post by: DeathOfRats on October 14, 2009, 01:12:49 pm
EDIT: Forgot to answer to Sunken earlier (oops). What I meant by local checks is that, if you have a large TCZ, you want to limit your checking to the area near the point you're checking, to avoid exponentiality. So you might decide to check only a 24x24 area around the seed point (or around the newly added point, but that runs into heavy redudancy), for example.
If, despite that, you allow the TCZ to grow much larger than the size you limit the checks to, you open yourself to letting in shapes that don't comply with the silly-move requirement.
Post by: Impaler[WrG] on October 14, 2009, 04:33:42 pm
Thus I'm in favor of rectilinear areas, the shortest side of the rectangle can be used to determine the size limit for agents passing through it and one area will be adequate for all agents without knowing what sizes to expect. Like wise the connection between two rects would itself be a rectangle overlapping the two connected zones, the footprint on each side being at least as deep as it is wide. This would avoid the situation ware two zones that can accommodate large agents have a small connection like a 1x1 door. The connection size would be 1 and large agents would not try to path through the area.
Post by: Puzzlemaker on October 14, 2009, 07:42:30 pm
My MSN is death_spear684@hotmail.com
My AIM is Aiko12349
Just, ya know, in case you want to bug me/motivate me.
Post by: Sunken on October 15, 2009, 02:41:02 am
EDIT: Forgot to answer to Sunken earlier (oops). What I meant by local checks is that, if you have a large TCZ, you want to limit your checking to the area near the point you're checking, to avoid exponentiality. So you might decide to check only a 24x24 area around the seed point (or around the newly added point, but that runs into heavy redudancy), for example.
If, despite that, you allow the TCZ to grow much larger than the size you limit the checks to, you open yourself to letting in shapes that don't comply with the silly-move requirement.
...I still don't get it ??? Checking for what? Are we still talking about incremental changes to the map? Or do you mean the initial TCZ creation? Either way there is no exponentiality involved even if you don't limit the area you process (especially with the simplified incremental update I suggested a post or two ago). Or do you mean some other check?
Post by: Sunken on October 15, 2009, 02:48:40 am
The problem I'm foreseeing with TCZ as currently envisioned is that their agent size dependent. A 1x1 agent can go through a forest easily because all the obstacles are 1x1 and it can continually home-in on its destination and just take the diagonals when it runs into an obstacle. But what about larger agents like wagons, two close trees can form an impenetrable barrier. Their would need to be a TCZ for each agent size which will probably be prohibitive.True, I never even thought about multi-tile agents when I thought up the TCZ approach.
But really, won't multi-tile creatures/wagons be rare enough that we could use brute force pathing for them? If a colossus should show up, we just perform a single, specific clearance-based flood fill starting at his current position and then brute A* over this - just like wagons (seem to) work right now?
Remember that the bigger the creature, the fewer its movement options. A four-tile creature can't even move through doors - A* inside a typical fortress should be a reasonably shallow search!
Simply put, I don't really feel multi-tile creatures/wagons are enough of a performance problem in practice to worry about. Especially since there's already a special clearance based system in place.
Post by: Shades on October 15, 2009, 03:01:22 am
If we combined this with a special case for underground routing system that used the standard convex (without sliding) area generation then we will end up with a, most likely, very small graph to traverse that knew where it connected to the outside.
This would also have the benefit of a simple way to make dwarves prefer inside to outside routes by weighting the outside connections highly (should someone wish to do that).
Post by: zwei on October 15, 2009, 03:13:07 am
The problem I'm foreseeing with TCZ as currently envisioned is that their agent size dependent. A 1x1 agent can go through a forest easily because all the obstacles are 1x1 and it can continually home-in on its destination and just take the diagonals when it runs into an obstacle. But what about larger agents like wagons, two close trees can form an impenetrable barrier. Their would need to be a TCZ for each agent size which will probably be prohibitive.True, I never even thought about multi-tile agents when I thought up the TCZ approach.
But really, won't multi-tile creatures/wagons be rare enough that we could use brute force pathing for them? If a colossus should show up, we just perform a single, specific clearance-based flood fill starting at his current position and then brute A* over this - just like wagons (seem to) work right now?
Remember that the bigger the creature, the fewer its movement options. A four-tile creature can't even move through doors - A* inside a typical fortress should be a reasonably shallow search!
Simply put, I don't really feel multi-tile creatures/wagons are enough of a performance problem in practice to worry about. Especially since there's already a special clearance based system in place.
Three wagons on larger map are framerate killers (along with some ten extra critters.), for me it literally means drop of about 20% in fps anytime they appear.
---
However, you can easily create map for any odd-sized (1, 3, 5 ...) creature by expanding all obstacles: you can path them as one tile wide creature (see 'wagon access' map - all trees there count as 3x3 obstacles with tree in center)
Similar apporach can also work for even sized creatures - just grid transformation so that you path it as one-tile item.
This can even work for rectangle shaped creatures. (you create two maps - one for 90 degree rotation) and add then together with OR operator to get basic map of possible creature placement, then add them together with AND to get locations where you want to be able to turn - AND this map with map of access for square or longer side to get map where you need to turn and can, then .... well, it can take some work to figure out fully, but i think it can be done with bit operations.
Post by: Sunken on October 15, 2009, 03:16:36 am
The advantage of TCZs to convexes or rectangles is that they are larger (they will always be at least as large as a convex or a rectangle, since TCZs subsume those concepts). The idea is that they have a greater chance of filling up an entire room, leaving only the doorways or other choke points as interfaces to be considered in the search. A single obstacle in the middle of a room is enough for a convex or rectangular subdivision to split it into several zones, which will have potentially long open-space interfaces, making the branching factor of the search bigger. TCZs are designed to make that less common (though by no means eliminate it!).
Only extensive testing will verify whether this idea actually holds water - especially given the extra cost of creating and maintaining the TCZs (though I'm hoping that will amortize well over time).
Post by: Sunken on October 15, 2009, 03:19:09 am
Three wagons on larger map are framerate killers (along with some ten extra critters.), for me it literally means drop of about 20% in fps anytime they appear.Are you sure it's mostly the wagons and not the critters?
But anyway, I would guess that not much optimizing has gone into the current wagon navigation system yet. With only small improvements it could probably handle most practical cases. Or so I'd think. Again - tests will tell...
Post by: zwei on October 15, 2009, 03:26:35 am
Three wagons on larger map are framerate killers (along with some ten extra critters.), for me it literally means drop of about 20% in fps anytime they appear.Are you sure it's mostly the wagons and not the critters?
But anyway, I would guess that not much optimizing has gone into the current wagon navigation system yet. With only small improvements it could probably handle most practical cases. Or so I'd think. Again - tests will tell...
Sure? No, but strong enough suspicion because lagging stops once they all reach trade depot.
Post by: Shades on October 15, 2009, 03:34:46 am
The advantage of TCZs to convexes or rectangles is that they are larger (they will always be at least as large as a convex or a rectangle, since TCZs subsume those concepts). The idea is that they have a greater chance of filling up an entire room, leaving only the doorways or other choke points as interfaces to be considered in the search. A single obstacle in the middle of a room is enough for a convex or rectangular subdivision to split it into several zones, which will have potentially long open-space interfaces, making the branching factor of the search bigger. TCZs are designed to make that less common (though by no means eliminate it!).
The reason I suggested standard convex shapes rather than TCZ in my last post was exactly for the reason that they won't span so many objects and so end up smaller. In this way we can ignore the fact that we get a suboptimal route because it will only be marginally suboptimal and vastly faster to calculate.
Because we don't need to calculate interface point to interface point, other than when we hit a zone and need to move to the next one, we don't have to worry that the interfaces will be larger stretches.
If we limited the size of TCZ's as someone suggested earlier this would have a similar effect of course, but again we would end up with long interfaces as the TCZ's are stopped prematurely.
With most fortresses the interfaces locations at doors would probably result in small zone, but I wouldn't want to rely on it.
Post by: dreiche2 on October 15, 2009, 04:35:46 am
I myself also started implementing my pathfinder ideas quite some while ago, but then real life took over. What I want to mention though is two things:
1) It could be possible to utilize that the transitions in between z levels, i.e. stairs and ramps, are known beforehand and relatively sparse compared to how many tiles there are in total. That's basically what I tried to implement, but I didn't get very far.
2) :
Discussions about pathfinding ideas come up and die out regularly on this forum. I think for a PathFinder project to be actually fruitful it would be most beneficial if, for a start, someone sits down and implements a basic algorithm testing framework. That is, have a program or library that generates pathing jobs on DF-maps or simplified maps. Different pathfinder algorithms would have access to the map data and would retrieve the pathing jobs. Have some standardized benchmarks to compare the different algorithms' performance. Maybe even have some GUI for easy inspection of the maps, pathing jobs and generated paths.
Having such a testing framework would serve as starting point for people to try out their ideas. I think implementing this framework would be the most useful thing for a start.
Post by: Sunken on October 15, 2009, 05:05:27 am
The reason I suggested standard convex shapes rather than TCZ in my last post was exactly for the reason that they won't span so many objects and so end up smaller. In this way we can ignore the fact that we get a suboptimal route because it will only be marginally suboptimal and vastly faster to calculate.It is indeed a different ball game if you settle on suboptimal paths. I have a feeling that won't work well in practice, because the inefficiency is unbounded in principle - the cost of moving through B from A to C may be 1 and the cost of moving through B from D to E may be 100.
Because we don't need to calculate interface point to interface point, other than when we hit a zone and need to move to the next one, we don't have to worry that the interfaces will be larger stretches.
But I can't prove that it's impracticable, it's just a hunch.
If one does go for full accuracy, and plans with interface tiles, then a key issue is whether the zone subdivision "fits" the environment - whether it finds natural choke points and minimizes the sizes of interfaces. That's where I think the flexibility of TCZs are a strong point. They're not the final answer by any means, of course.
Post by: Shades on October 15, 2009, 05:41:34 am
I have a feeling that won't work well in practice, because the inefficiency is unbounded in principle - the cost of moving through B from A to C may be 1 and the cost of moving through B from D to E may be 100.
But I can't prove that it's impracticable, it's just a hunch.
As long as we ensure a limit on size of zone, which I figured breaking on each object would do, then we should ensure the distance across a zone is small enough that it's only a small percentage of inefficiency. Technically we could get cases where we have 19 zones at an average of 20 squares each or 20 zones at an average of 1 square and so pick the wrong route but this should such a rare case we can probably ignore it.
I would recommend the weight of a zone is based on it's size (and so average cost to cross) as well though, just to help this kind of problem be avoided it.
Further, a common strategy would be to use the suboptimal route to give an initial movement for the critter and then let the perfect route be generated over a number of updates and splice the routes together. Although this is generally to allow user interaction to show an instant response so probably won't help in our case.
Post by: Starver on October 15, 2009, 06:04:02 am
Maybe you guys could solve the animal pathfinding issue vs animal forbidden doors? Unless more than the pathfinding code is involved.(Coming late to this (comparitively), and not yet having read all that is discussed so far, so excuse me if I'm a little behind.)
If you look at the complexity involved in maintaining (frexample) region-connectivity across a whole map for one route-finding algorithm (e.g. the one for non-amphibious land-based creatures), it seems relatively trivial to maintain parallel route-maps for such as aerial, aquatic, magma-surviving, etc, as well as a "pet map" that handles pet-forbidden doors as non-routable[1] (but includes knowledge of transient 'openness' if a non-pet wanders through...).
The above possibilities means multiplication 5-fold of the map (or, rather, 5 maps if better optimisation of each map can be made), In a worst-case !N problem of every-tile-to-every-tile routes being stored it would be an insifnificant increase. For better logN-type situations it would not be as good, of course.
Now, I spent a while last night writing in an idea I was adapting from a vector-based environment that I'd had a long time in
planning (started thinking about it circa the time of original Doom, which dates it), but reading this thread further it looks like some of the key features (e.g. convex zones, built up from merging the leafs of a trinary tree structure where shared border cells can be optimised away inbetween "limiting features", letting each higher level zone governing a convex mass of connectable sub-zones) are really not that innovative, so I'll go back to the drawing board before saying anything about that.
Post by: Shades on October 15, 2009, 06:28:18 am
it seems relatively trivial to maintain parallel route-maps for such as aerial, aquatic, magma-surviving, etc, as well as a "pet map" that handles pet-forbidden doors as non-routable
Trivial yes, but memory hungry if a trivial method is used. As you mention they might be better ways to store the 5 (or more) fold data.
A possibility would be that each zone was either passable or not (ie: no zone could have both a passable and a shorter but non-passable route) so zone calculations would have to generate two zones where there was two adjacent routes, one with was all open, one which was not passable by a subgroup. Then the path finding could just treat non-passable as inf cost if that critter cannot pass that way. Not perfect, but fairly simple.
Post by: Impaler[WrG] on October 15, 2009, 06:36:39 am
Quote
Discussions about pathfinding ideas come up and die out regularly on this forum. I think for a PathFinder project to be actually fruitful it would be most beneficial if, for a start, someone sits down and implements a basic algorithm testing framework. That is, have a program or library that generates pathing jobs on DF-maps or simplified maps. Different pathfinder algorithms would have access to the map data and would retrieve the pathing jobs. Have some standardized benchmarks to compare the different algorithms' performance. Maybe even have some GUI for easy inspection of the maps, pathing jobs and generated paths.
Having such a testing framework would serve as starting point for people to try out their ideas. I think implementing this framework would be the most useful thing for a start.
Excellent point dreiche, a system for benchmarking the whole thing is going to be key, real performance numbers put an end to debates like nothing else can. I suspect we will see multiple versions of the zone creation routines and will need to pick amongst them based on performance testing.
I'd like to speculate on the API, it should be as neutral towards the actual game design as possible, the only thing that should be assumed is a grid system in 2 or 3 dimensions with multiple movement types.
Grid coordinates should be in 3 Signed ints and movement types expressed as bit fields (thus allowing for the combination of multiple types), that gives us something like
void loadMapData(int x, int y, int z, int Movement)
void changeMapData(int x, int y, int z, int Movement)
void buildZones()
loadMapData is used to inject the map data and just stores it without processing it, buildZones would do a full scan an the map and build the initial zones, changeMapData would update a tile with new data and trigger what ever zone changes are appropriate.
The pathfinding work would be done with
Path getPath(int x, int y, int z, int x, int y, int z, int Movement, int size)
int getEstimate(int x, int y, int z, int x, int y, int z, int Movement, int size)
getPath returns some kind of Pathway description object the format of which is still not determined, multiple type could be possible such as a zone level path and a per tile path. getEstimate is the quick heuristic I spoke of earlier, it doesn't do as much work as a full path but it needs to be a LOT faster and be within about 10% of the true path length.
Post by: Starver on October 15, 2009, 06:37:07 am
Thus I'm in favor of rectilinear areas, the shortest side of the rectangle can be used to determine the size limit for agents passing through it and one area will be adequate for all agents without knowing what sizes to expect.I'm probably reading this wrong (still progressing through the thread) or it is commented on later, but, a rectilinear area that is 2x5 would not be a limiter/untravelable area for a wagon wishing to travel across it between the twi 5-wide borders.
The 2-wide borders would be considered non-passable, of course. Though that would be wrong if the corners were not passage-limiting features, and instead just arbitrary points chosen to create a rectangle. Two adjacent but side-slipped 2x5 rectangles (so that they cannot be represented as a single 4x5 rectangular area or larger, due to limiting features carving out 'corners') might allow free passage to a wagon-sized agent along their shared borderline and thus partially across one or other of the 2-wide border.
My homegrown method (which looks like it has been long-presaged, I need to sit down and read up on what's actually been done by others on this) working around convex areas relies upon all corners being limiting features, and thus the borders can be similarly reliably sampled for passage or otherwise for arbitrary-sized units, and the truth table of passage for the convex-area-of-convex-areas (one, or more, levels up) can account for the intricracies of inter-zone travel accordingly.
(Until I get to the end of this thread, I am of course postulating into the wind, I think the above is probably already known, and possibly rejected for some reason that hasn't occured to me.)
Post by: DeathOfRats on October 15, 2009, 06:45:36 am
EDIT: Forgot to answer to Sunken earlier (oops). What I meant by local checks is that, if you have a large TCZ, you want to limit your checking to the area near the point you're checking, to avoid exponentiality. So you might decide to check only a 24x24 area around the seed point (or around the newly added point, but that runs into heavy redudancy), for example.
If, despite that, you allow the TCZ to grow much larger than the size you limit the checks to, you open yourself to letting in shapes that don't comply with the silly-move requirement.
...I still don't get it ??? Checking for what? Are we still talking about incremental changes to the map? Or do you mean the initial TCZ creation? Either way there is no exponentiality involved even if you don't limit the area you process (especially with the simplified incremental update I suggested a post or two ago). Or do you mean some other check?
I was talking about creation, but the incremental changes suffer from the same problem, too. Wouldn't you have to check your addition against all other tiles in the TCZ? Or could you just settle with checking against certain key points, or points close (i.e., local) to the point where you are making the change?
Post by: Impaler[WrG] on October 15, 2009, 07:02:47 am
Quote
I'm probably reading this wrong (still progressing through the thread) or it is commented on later, but, a rectilinear area that is 2x5 would not be a limiter/untravelable area for a wagon wishing to travel across it between the twi 5-wide borders.
Yep hadn't considered the possibility of zones abutting each other in ways that create movable areas still larger then the zones. It's a tricky problem all right.
Post by: Starver on October 15, 2009, 07:44:02 am
What I also meant to mention is that a bit-wise "passable/not-passable" flag was made, that had not been given any form of compressive optimisation (by programmer or compiler) and was inhabiting an entire byte regardless of only being true/false, then a number of different flags could be loaded into the same byte of storage to no extra memory usage (and trivial logic applied to the setting/retrieval of the state). But I still err towards zone-trees optimised towards differing requirements.it seems relatively trivial to maintain parallel route-maps for such as aerial, aquatic, magma-surviving, etc, as well as a "pet map" that handles pet-forbidden doors as non-routable
Trivial yes, but memory hungry if a trivial method is used. As you mention they might be better ways to store the 5 (or more) fold data.
Quote
A possibility would be that each zone was either passable or not (ie: no zone could have both a passable and a shorter but non-passable route) so zone calculations would have to generate two zones where there was two adjacent routes, one with was all open, one which was not passable by a subgroup. Then the path finding could just treat non-passable as inf cost if that critter cannot pass that way. Not perfect, but fairly simple.This reminds me of the vectorised world I have breifly aluded to. Surface-to-surface convex polyhedra, the sides being indexed (at the simplest level) towards the polyhedra and subsequent side of that polyhedra to which it abuts (if any, and not a complete boundary, i.e. a wall), but while normally it would be a 1:1 matching for all phenomena (or with outright exceptions, such as "invisible forcewall" preventing physical movement but allowing vision, and "energy barrier" allowing everything through but certain classes of weapon discharge, "glass" being like the invisible forcewall but able to be "smashed" and revoked by any physical weapon discharge of sufficient power) there were possibilities of making physical transit and optical transit progress into alternate 'target' polyhedra (with equivalent 'receptive' surface), which I suppose one could liken to a "stargate, without the rippling pool effect" (though you could have given it a rippling pool effect as well, remember how old this idea was... I can't remember when Stargate first popped into my consciousness). Alongside other interesting possibilities[1], this meant that the environment could involve 'overlaid' polyhedra where the physical and optical aspects could pass through alternate 'overlaid' dimensions in the same logical space.
So... if we could not establish every zone to be 'one-size fits all' and be able to pack all necessary border-to-border passagability information into each, it would not be too difficult to have transit from one zone (say an internal zone, entirely constrained by the bedrock, and thus pretty much the same to any creature that hadn't a tunelling capability to fall back on) guided by the zone-boundary definition to split so that surface travellers make use of a 'surface friendly' target zone, one of many overlaying the landscape that are interupted by trees and channels and other impassables, while aerial creatures are free to fly into the 'atmospheric'-styled zone, which may still have trees as obstacles at the surface (either zone limiters in Z-spanning convex polyhedra, or singular features to avoid within that) but treats channels as vertical borders into 'deep ditch' zones with aerial access possibilities, or can be ignored and passed over. The 'logical' positions of ground-dwellers in the surface-zoning would be known within the aerial-zoning hierarchy through passing the absolute cartesian coordinates between the two, so allowing appropriate pathing and 'hunting' of ground dwellers by aerial ones (despite inhabitting a different 'zone-space') and a ground-based archer would have no problem targetting a flyer. If the surface-pather was amphibious and passed into an aquatic-styled zone to swim below the surface, this would put them beyond a non-transitable boundary as far as any particicular aerial agent was concerned, if it was hunting and not an aquatic-capable creature (e.g. gannet) able to undertake that transit.
[1] Scaled travel: i.e. transit between polyhedra surfaces of differing areas, essentially altered the avatar's relative size in the environment if he looped back to origin via a different route ('Distorted' travel, also, where the avatar changes relative width and height).
Mono-directionality in some/all of the physical/optical/sonic/etc propogations.
"Movable holes" in the environment, that could be slide around one one or both sides.
Simple implementation of mirrors (back then Duke Nukem 3D hadn't popped up, so I'd not seen it done) by linking the optical 'outward' transit to the same facet of the polygon
Rooms with a rotational symmetry of 0.5, where you have to walk around the central pillar 'twice' to get back to your original position.
Some of the above was inspired by "Toonyverse" physics, I'm sure you can imagine.[/1]
Post by: kurokikaze on October 15, 2009, 07:46:47 am
Post by: Sunken on October 15, 2009, 08:19:58 am
I was talking about creation, but the incremental changes suffer from the same problem, too. Wouldn't you have to check your addition against all other tiles in the TCZ? Or could you just settle with checking against certain key points, or points close (i.e., local) to the point where you are making the change?As I already said, you only need to check a tile against the closest other tile in the TCZ that is in conflict with it. You do this by finding conflicting directions, and processing just those tiles that are closer to the seed and in this direction. It is on the order of the size of the TCZ, but with a very small factor on it.
Post by: Sunken on October 15, 2009, 08:23:31 am
The only extra cost is that you will need a whole new connectivity map for each movement type combination (not just each movement type). However, given that this map runs over zones, not tiles, it should be cheap enough that we can afford it.
And, yeah, fliers. I dunno about those. They're tricky.
Edit: I meant "multi-movement-type creatures" above, not "multi-tile creatures".
Post by: Ampersand on October 15, 2009, 11:24:22 am
The Dwarf Fortress map is not 2 dimensional, or 3 dimensional. It is 5 dimensional. Allow me to explain.
The fifth dimension is the most obvious; the Z-level dimension, the first four are not so obvious.
First we have the recognizable 2D grid that makes up the visible map, but there is an invisible, second 2D grid overlaid on top of that, each square of this consisting of 16X16 squares of the visible grid. I'm not sure how important this grid is in terms of path finding to a particular point, but I do know how tasks are assigned in this grid. With the top left most 16x16 square being square 1, the one below it being square 2, and so on, tasks are ordered first by where in this super grid they are, with a preference for the lower numbers, and then by their position within the subgrid of points within that square, starting from the top left within the subgrid.
So we're dealing with coordinates that look more like (X, Y,{A, B}), Z. The order is important, as the Z coordinate is not relevant in terms of calculating distance, as we all know far too well, Everything with the parentheses is used to determine how far away something is, then the Z is added on afterward to do actual path finding to that point. This is obviously something that needs fixing.
I'm not entirely sure if the A, B coordinates are used in the pathfinding, as opposed to only task ordering, though exploiting their existence may make pathfinding more efficient.
Post by: Slogo on October 15, 2009, 01:04:42 pm
For the Annotated A* portion of the algorithm (ignoring the hierarchy portion of the algorithm) I can see it working like this.
Two types of clearance metrics (transportation types).
One is a main type, in DF's case this would be Ground and Air. The other is a secondary type. For DF I think you'd need Water, Fire, Smoke, Magma, obstacle (all), and obstacle (no pet). Each main type would have a pre-calculated clearance value stored in the map. Each sub-type would be regulated for an on-the-fly clearance calculation. This saves a lot on memory and, as you can see by the sub-types, they're infrequently encountered and/or very dynamic. This makes calculating them on the fly the best option anyways.
Each tile would have to be tagged with a main type and optionally tagged with any relevant subtypes. In DF's case every Ground tile would also be an Air tile but not vice versa. Creatures can only navigate over tiles of the appropriate type (ground or air) and for each tile calculated in the path finding algorithm the subtypes would be checked before placing it on the priority queue used by A*. To check for subtypes for a given square you simply recalculate the clearance for a square up to the creature's size but counting any inappropriate subtypes as a blocking tile.
Handling non square creatures provides an interesting twist to the algorithm.
The clearance based algorithm will actually have to be modified to handle non-square multi-tile creatures and the z-axis. I'd change the algorithm like this:
Each tile has the following stored (all short ints):
- Maximum x axis clearance measured left to right
- Maximum y axis clearance measured top to bottom
- Maximum z-axis clearance measured from low to high
- Maximum square clearance measured left->right and top->bottom
- Minimum z-axis clearance measured across the square above
It sounds like a lot to store but I don't really think it'll be too bad, some values like maximum z-axis clearance could be changed to be calculated on the fly as it may not be checked very often.
The way we can check if an entity can fit in a particular tile would go through these steps:
1. Is creature's x and y sizes < the Maximum square clearance and is the creature's height below the minimum z-axis clearance for this square (all pre-calculated values)? If so creature can fit on tile and end.
2. Starting at the specified tile and moving from left->right based on creatures x size check the following:
- Is Maximum y-size for this tile > creature's y length
- Is maximum z-size for this tile > creature's z height
- Is the maximum x-size for this tile > creature's x length
- Is maximum z-height for this tile > creature's z-height
Essentially you are checking each tile one to make sure that each tile is clear if the creature stands at the rectangular tile given. Since we pre-compute all these values it's a O(n) algorithm where n = # of tiles a creature occupies, but it gets better than that. By using the square properties that are also pre-calculated this algorithm could be even further optimized. For example if we wanted to fit a 3x2 creature in a tile at x,y and our square size was 2 then if the square size of the tile at x,y-1 was 2 our creature could fit in the given location.
Handling updates to the world is relatively cheap.
Yes you have to recalculate clearance values but it's super easy to do and only requires a minimal amount of clearance values to be updated. You only need to recalculate clearance values for tiles to the left (lower x value), 'above' (lower y value), and lower down (lower z-value). All other tiles will be unaffected. The way the algorithm would work is you'd run four loops
1. Would check all tiles in a straight right->left line until an obstacle was hit
2. Would check all tiles in a straight bottom->top line until an obstacle was hit
3. Would check all tiles in a upper->lower line until an obstacle was hit
4. Flood fill of all tiles with a lower x, y, and/or z value (basically all tiles not covered by the loops above). The tricky part here is knowing when you can stop checking tiles, I believe it's possibly to generally only update a minimal number of tiles with a clever algorithm to do this.
This is just a rough outline of how something like this could work, I'll organize and collect my thoughts more as I look into it but I really do think it's the way to go for this sort of problem. It provides a way to efficiently store all the data we need for multi-tile pathfinding while also providing a potentially efficient algorithm.
Also remember that if a multi-threading solution is used performance becomes much less of an issue (though still important obviously). Given that creatures can take >1 frame to make a movement we can potentially have leeway in how long we have to calculate their specific path.
Post by: Impaler[WrG] on October 15, 2009, 03:26:09 pm
Post by: cephalo on October 15, 2009, 03:48:43 pm
For 'soft' blocks, treat them like hard blocks as far as region ID is concerned. Each region could have list of what regions it might be connected to depending on door state.
This way you can avoid failed paths. I believe it's much cheaper to seed fill an area than to search a whole area for a path with the astar algorithm.
Post by: Shades on October 16, 2009, 02:42:05 am
Post by: Sunken on October 16, 2009, 02:45:51 am
In fact I see no reason that the concept of TCV cant be mixed with zones that must be crossed with A*, the high level zone search can be agnostic as to the nature of the zones so long as it can query them for being able to be crossed by various agents.Right, that's an important point. And this modularity should be maintained while developing, so that new approaches for each layer can be tried out and compared.
Post by: Shades on October 16, 2009, 02:56:34 am
Right, that's an important point. And this modularity should be maintained while developing, so that new approaches for each layer can be tried out and compared.
As long as our testing harness is suitably templated then we can just make sure all map operations are reasonably standard operations and so easy to replace with a new system. Maybe it's worth knocking up a quick harness this weekend, what kind of testing environment do we need to run.
A simple grid based on a fake df map of some kind? Ideally with some horrible example paths to attempt? Then the harness can just be run and display times taken at each stage, as well as distances generated (or at least % of optimal path).
Post by: Impaler[WrG] on October 16, 2009, 03:55:34 am
Random Workshop -> Random Workshop
Random stockpile tile -> Random Workshop
Random Workshop -> Random stockpile tile
Random stockpile tile -> Random stockpile
Random Map tile -> Random stockpile tile
Random Map tile -> Random Map tile
(Random Map tile means a tile any ware on the map which is to simulate things like item pickup, tree cutting etc etc)
This would produce paths that are more like those actually coming out of DF and provide a lot of identical repeat path requests (or at least very close to identical) which will be a major factor in determining the efficiency of path caching.
Post by: Starver on October 16, 2009, 06:28:51 am
One is a main type, in DF's case this would be Ground and Air. The other is a secondary type. For DF I think you'd need Water, Fire, Smoke, Magma, obstacle (all), and obstacle (no pet).[...]I forgot to bring in my memory stick with what I wrote up last night, but (if I remember correctly) I had seven different types of basic "passable" listed down for the system I was thinking of:
- Walk - standard movement: i.e. surface spot to surface spot
- Tunnel - for (future) burrowing entities (though might want to make distinction between burrowing (soil) and tunelling (rock (and soil)) entities, or even avoid "burrow" as a term to avoid mixing up with the upcoming user-defined areas thingummy): relates to all underground-to-underground pathing possibilities, and any initial transition from 'open' to subterrainean, if required[1](
- Fly - all controlled aerial manouevers, with or without a surface immediately below (though perhaps flyer/walker creatures could choose the method that best suits them, if there is a 'ground' of some kind to stand on, given their energy levels?)
- Swim(Water) - all movement within water, though surface swimming might want to be decoupled from subsurface swimming
- Swim(Magma) - ditto as water, but separated for obvious reasons
- Climb - for those that can (c.f. Adventure mode single Alt-movement out of a river, but also spiders and other 'wallcrawlers' without limit, could also govern movement across ceiling of multi-Z cavern). Paths for climb-enabled creatures might allow only a certain number of successive climb 'movements' in a path (perhaps a single one), representing the limit to their ability to heave themselves on top of a certain height wall. Large creatures or ones that can jump might be given an allowance in the Raws, and the pathing algorithm would decrement accordingly in searching for the movement
- Jump - relates initially to a deliberate stepping off of edges (applicable only to those that don't fly, of course), but for the sake of compactness includes "fall" (air-only location to air-only location immediately below, unless inertia is a concept, in which case larger and gap-crossing can be weighted in according to ability). Bigger (or "jumping") creatures might have an allowance based on how far they can 'step down', but otherwise a 'jump' is weighted according to relative chance of injury (perhaps literally 'weighted'... drop a mouse down a mineshaft and it might survive when it hits the bottom, drop an elephant and it'll 'splash'). While not strictly a pathing issue, emergency movement away from danger (comparable to the current dodging away from opponents) might force creatures to take action beyond any 'jump+fall' allowance they would have and enter a 'plummet' count, which would of course be heavily injury penalised.
Obviously, the above relates to my own ideas, I'd completely forgotten about pet-specific routing biases (despite mentioning my ideas about it, the other day) and I'm not sure what if smoke features in my idea (save for a visibility bias in pathing, in which case would 'mist' and 'miasma' also be necessary?), and for a moment (until I saw the "(no pet)" version), I wasn't sure if 'obstacle' meant like lone pillars in TCZs or my climb/jump idea.
For creatures with multiple movement opportunities would have weighted tendencies towards one or another method, but would consider shorter 'difficult' paths than longer 'easy' ones, where the shortcut is short enough. e.g. tunnelers, except as mentioned in [1], below, would walk normally but (assuming they had 'knowledge' of a target they want to reach, like breaching a fortification or wanting quick egress from a half-closed trap situation) would be free to tunnel. Scrambling across a barrier might suit a climber who normally walks, flying creatures who could might be forced to perambulate in single-tile-cross-section tunnels for some reason or another, or at least be penanalised for trying to flap their way through when other creatures exist. But I appreciate that a lot of this is no longer strictly pathing, but involves (possible future) game mechanics.
[1] I could imagine some creatures could be purely rock-dwellers leaving no trace of tunnel, and not intruding upon 'open' areas, but the main three possibilities would be: a) rock-morphers 'eating' or otherwise digging through leaves behind 'solid' rubble like snail trails; b) classic tunelling, a la dwarf, with certain (variable? improvable?) chance of leaving/not leaving spoil; c) full on consumption, evaporation or metabolising of all rock, leaving void.
PPS, I don't know if it's since I last let IE update, but the textarea editing box on this machine doesn't scroll like it should/used to, and keeps jumping back up even when I'm cursoring down. Annoying. I'm having to type this in a text editor and CnP it in.
Post by: Slogo on October 16, 2009, 09:12:07 am
Dodging and falling really isn't a concern of path finding either but rather an issue of locomotion. A creature dodges to a specified location or falls down based on gravity and dodging, not some pre-planned path.
As for having preference for certain types of routes, well that's easy! All you need to do is allow for our methods to accept a set of weights for different types/sub-types of tiles. These weights would factor into the A* heuristic causing the optimal path to 'bend' towards prefered methods of movement when possible.
Post by: Sunken on October 16, 2009, 09:34:57 am
Digging creatures probably isn't really a concern of ours. If a creature is digging through rock then I'd imagine their ideal route would be a straight line down the tunnel. I'd expect any path finding algorithm to simply get the location of the next hole to dig and we'd be responsible for moving them to the appropriate dig location.I assume he was talking about tunneling creatures, not just digging straight down. Still, since a tunneler has such great freedom of movement, raw A* will probably be plenty fast for that.
Quote
Dodging and falling really isn't a concern of path finding either but rather an issue of locomotion. A creature dodges to a specified location or falls down based on gravity and dodging, not some pre-planned path.Jumping down a low cliff or hole is very much a valid choice of deliberate pathing in some situations. Some creatures may be impervious to the effects of a fall and it would be silly of them not to use gravity to get to where they want to go.
Quote
As for having preference for certain types of routes, well that's easy! All you need to do is allow for our methods to accept a set of weights for different types/sub-types of tiles. These weights would factor into the A* heuristic causing the optimal path to 'bend' towards prefered methods of movement when possible.You can't factor them into the A* heuristic, only the exact cost.
Post by: Starver on October 16, 2009, 09:49:19 am
Digging creatures probably isn't really a concern of ours. If a creature is digging through rock then I'd imagine their ideal route would be a straight line down the tunnel. I'd expect any path finding algorithm to simply get the location of the next hole to dig and we'd be responsible for moving them to the appropriate dig location.Not right now it isn't, and I keep forgetting that this is a project to see how to handle currently expected pathing, but it should be considered insoar as future sapper/siege-breaking developments. Normally, it would be a straight (or at least appropriately staggered) route, but if we're actively defending against such activities with water/magma-filled cavities, etc, and the sappers know that they need to dig around, it would be good to be able to cater for that.
Quote
Dodging and falling really isn't a concern of path finding either but rather an issue of locomotion. A creature dodges to a specified location or falls down based on gravity and dodging, not some pre-planned path.I mention dodging and falling, but I meant for deliberate jumping off (and climbing up) to feature as a valid pathing possibility. I know I've often made 'causeway' labyrinths (enforced by bridge removal, or just permanently like that) to guide enemies through a carefully controlled killing-fields. Smart enemies could "Alt-Move" off-and-below like an adventurer on a river-bank, to no actual injury, and then Alt-Move onto the next adjacent leg of the causeway (or run around the channelled area, which often leeds more directly to my interior, given I might want to send soldiers there to handle enemies that did fall off). And larger creatures should be able to stride up onto/down off of at least one Z-level of wall/equivalent, so they should be able to exploit that routing as a matter of course, no question.
But as long as we can get air/ground/possibly-amphibious routing sorted in the same algorithm, then the above concerns are probably minor details that can be easily integrated into an improved system, so I'll leave these as just considerations, and get back to the algorithm side, when I get chance.
Post by: Starver on October 16, 2009, 09:54:48 am
I assume he was talking about tunneling creatures, not just digging straight down.Yes, and I think "straight" was straight (and/or stagger-equivalent of straight) in any direction, A->B, was how Slogo meant as well. Lestwise, that's how I took it.
(At least when it isn't 'random-walk' messing about by exploratory enemies without the information needed to head to a specific location. I know that information, or at least the lack of, is beyond the immediate scope of this 'omniscient' pathing process, though...)
Post by: Fieari on October 16, 2009, 11:54:49 am
If we want to stick with 7 movement types, I'd drop "jump". It's a cool idea, but I think it'd get too complicated, especially because if you wanted things to jump different heights you'd need a different movement category for each height.
Post by: chmod on October 16, 2009, 12:24:03 pm
Post by: Slogo on October 16, 2009, 12:44:33 pm
For movement types, I think an important one would have to be "Building Destroyer", for pathing through gates or constructed walls with trolls or armies with battering rams, catapults, etc. Also, Tunneling as a path type is important because while the underground is USUALLY trivially connected, there are gaps in the form of caves, magma, underground rivers, and so forth.
If we want to stick with 7 movement types, I'd drop "jump". It's a cool idea, but I think it'd get too complicated, especially because if you wanted things to jump different heights you'd need a different movement category for each height.
You can use another subtype to handle everything since they're relatively cheap to have. They don't need to drop out anything, you can easily have more than 7. Just add a building destroyer 1 subtype for tiles with floodgates, windows, etc. and allow for building destroyer creatures to pass through. Impassable workshop tiles and other constructions would get a building destroyer 2 subtype. For doors, hatches, and the like they'd be covered by allowing building destroyer creatures to path through obstacle (no pet) and obstacle (all) tiles. Building destroyer creatures, when ready to move, would just check that their next tile doesn't have a construction, if it does destroy it first.
Post by: shadow_slicer on October 16, 2009, 01:24:51 pm
Use a bitmask to denote some required capability or cost class modifier (if capability then cost is infinite) to pass through this tile. Then the library doesn't have to worry at all about what all of it really means. This also would make it trivial to add additional classes if they turn out to be needed.
Code: [Select]
OpenSpace = 0x0001, //flying
Building = 0x0002, //building destroyer
Water = 0x0004, //swimming
Rock = 0x0008, //digging
Magma = 0x0010, //swimming (magma)
Outside = 0x0020, //keep dwarves inside
Sunny = 0x0040, //for vampires
Smelly = 0x0080, //eewww miasma
Post by: Slogo on October 16, 2009, 01:29:33 pm
I'd imagine that all of our tile 'types' and 'subtypes' (if we go that route) would be user configurable so they could add in or remove whatever they want. Some of the... fancier... things to do with the pathing data (like stop to destroy a building) are beyond the scope of path finding, we just need to make sure we can support it.
Post by: Impaler[WrG] on October 16, 2009, 04:54:08 pm
In any even the engine doesn't really need to worry about the nature of movement types other then to know how many bits are needed. It might be a nice idea to have this be flexible in some way so data storage is as efficient as possible. An option (perhaps a compiler time option) for either an 8 or 16 bit field to hold the movement types would be nice.
The far more interesting challenge is going to be maintaining the zone hierarchy and high level connectivity map along side all this movement type data.
Post by: Sunken on October 16, 2009, 06:02:03 pm
I apologize for not reading the whole thread, but has anyone suggested simple machine-learning for pathfinding? I would start simple, as that's all you need in the beginning of a fort. Random movement coupled with reward/punishment for finding good paths. The whole thing would be a bit organic. But you could save a general routine to use with all dwarfs. So it would be like a hive-mind. I've seen some success with things this simple before, although it was just a proof on concept on a simple Cartesian plane.
I'm all for learning and Toady likes the idea too, I've heard, but I think it is just too hard. You need an appropriate representation to learn things in, and that is the most difficult part to set up. What are the abstract features that you are going to feed into the trained system? And how will you get them? Very interesting issues, and the subject of my ever procrastinated Ph.D., and also immensely complicated. Even in a "toy" world.
Post by: Exponent on October 16, 2009, 06:09:17 pm
Digging creatures probably isn't really a concern of ours. If a creature is digging through rock then I'd imagine their ideal route would be a straight line down the tunnel. I'd expect any path finding algorithm to simply get the location of the next hole to dig and we'd be responsible for moving them to the appropriate dig location.I assume he was talking about tunneling creatures, not just digging straight down. Still, since a tunneler has such great freedom of movement, raw A* will probably be plenty fast for that.
Just want to point out one reason why something like A* should be the minimum in digging scenarios, rather than the far more simplistic idea of naïvely making a straight/staggered line: Things like open spaces (such as chasms/pits) or liquids would require pathing around, even for a digger, unless the digger is also a flier or swimmer. And in the future, various densities of materials might be used, such that some diggers can only dig through dirt or soft stone, and not harder stones.
Post by: Fieari on October 16, 2009, 06:14:08 pm
I wouldn't want to use Building Destroyer 1 and 2 though. I'd much prefer to use the new attack physics and just have the movement tag there to specify that a creature will TRY to break through stuff that's been built. The COST of movement can be calculated by the material of the object. I think this should be kept seperate from those who dig through natural materials, because I'm thinking about the way army seiges should work. Some unit types should try to bash through walls, while different ones dig under.
Post by: chmod on October 17, 2009, 01:09:42 am
In my very brief tests of this type of system I actually thought it was quite the opposite. I got very promising results in an afternoon. The simple punishment and reward system isn't really machine learning as much as it is prodding pseudo-randomness. If a generic framework gets set up for testing different pathfinding, I think a learning system should be thrown in, and I'd be happy to contribute.I apologize for not reading the whole thread, but has anyone suggested simple machine-learning for pathfinding? I would start simple, as that's all you need in the beginning of a fort. Random movement coupled with reward/punishment for finding good paths. The whole thing would be a bit organic. But you could save a general routine to use with all dwarfs. So it would be like a hive-mind. I've seen some success with things this simple before, although it was just a proof on concept on a simple Cartesian plane.
I'm all for learning and Toady likes the idea too, I've heard, but I think it is just too hard. You need an appropriate representation to learn things in, and that is the most difficult part to set up. What are the abstract features that you are going to feed into the trained system? And how will you get them? Very interesting issues, and the subject of my ever procrastinated Ph.D., and also immensely complicated. Even in a "toy" world.
Post by: Sunken on October 17, 2009, 01:30:40 am
Post by: Sunken on October 17, 2009, 01:44:05 am
I think no pathfinding scheme will be complete unless it can help deal with this problem as well.
The bottom line is, we won't always be given "go from this tile A to this tile B", but will sometimes be asked to "go from this tile A to any tile B that has these properties P".
I'm sure what we've been talking about could be tailored to deal with this. For example, each TCZ (or convex or rectangular zone) can have a cache containing its stored items. A* would simply search until the topmost open node is any node in a TCZ with the desired items in it, instead of one with the desired goal tile in it.
So it's not a huge conundrum; We'd just do well to have this in the back of our minds.
Post by: Impaler[WrG] on October 17, 2009, 01:58:18 am
Quote
I think a learning system should be thrown in, and I'd be happy to contribute.
I'd love too but I'm afraid of encoring the wrath of the fans of your Dwarf Therapist tool if I pull you away from that wonderful tool. (jk) We have Puzzlemaker and Slogo working on the initial implementation and if anyone else wants to get SVN access talk to them, they will send referrals to me for who should be given access.
Post by: Shades on October 17, 2009, 07:00:56 am
The question is how many bits we want to assign to movement types. Just remember how many map tiles there are, and that each bit can really add up when applied to each and every map tile in a world.
Doesn't matter, just use an std::bitset and we can treat it like an array. Even a 500 x 500 map with 100 z levels would only add 3 megs of map. We don't need to care about types else where in the world only the live game.
Here's something y'all may have thought of but that I hadn't. In the "stockpiling pathfinding" thread (and many others previously) it is pointed out that dwarfs often select startin points and destinations stupidly - getting the "closest" stone from 30 Z-levels away for example.
I think no pathfinding scheme will be complete unless it can help deal with this problem as well.
The bottom line is, we won't always be given "go from this tile A to this tile B", but will sometimes be asked to "go from this tile A to any tile B that has these properties P".
Agreed, it's trivial to write tests that take this into consideration (find me the path to the nearest stone rather than the path to square X) so we might as well. The A* heuristic for this is simple as well although other algorithms might be more complex.
Post by: Kardos on October 17, 2009, 11:56:05 am
1: Any Pathfinding system will need to work on a 16x16 embark area and or an area with extreme z-level variances withen the constraints of CPU and memory usage. Keep in mind that even if someone has 8GB of ram, DF might not neccesarily be built to use more then 1GB.
2: Allowing the system to utilize various pathfinding algorithms should be encouraged. What might work well outdoors might work poorly withen the fortres or in the water. This of course means that the various pathfinding algorithims might need some sort of interation for the purpose of creatures tracking other creatures.
3: Even if an algorithm works on a 2-d level, it has to be able to interact on a 3-d scheme to avoid the cube mechanics currently utilized in DF (climbing 30 z-levels for a piece of stone on a path thats several hundred tiles long when there's another perfectly acceptable piece 31 tiles away on the x/y plane.)
4: Movement types that aren't common now shouldn't be disregarded or thought of as trivial. With progressieve releases, flying, swimming and multi-tile creatures (and objects) will be become more and more common. Any system thats developed should be able to absorb and integrate these types into the pathfinding without massive changes to the code.
5: It might be a good idea to work in parrallel on other objects that require a path such as liquid or gaseos flows, as well as siege mechanics, allowing catapults and ballista to fire in varying degrees of rotation in both the horizontal and verticle plan
Post by: qwertyuiopas on October 17, 2009, 02:59:54 pm
Doesn't matter, just use an std::bitset and we can treat it like an array. Even a 500 x 500 map with 100 z levels would only add 3 megs of map. We don't need to care about types else where in the world only the live game.
I can tell you that you want at LEAST an unsigned byte for each tile, partly because each access of a bit would take a few more precious CPU cycles, and then multiply that by the potential millions per second. It doesn't add much, but preformance is essential. Also, by using a full byte, you add support for conditional blockage at a future date(8 bits per tile).
Also, by ONLY storing a pit per tile, you cannot precompute common paths, you cannot optimize in any way. You reduce the options down to storing data elsewhere or having an implementation no better than the one integrated into DF itself. In fact, probably worse.
30 megs is nothing these days. If it doubled the speed, 100 megs would mean nothing.
Storage space is not important. Speed is. And on the topic of speed, should C++ be used? The overhead of it's enhanced features might be an issue if C can be used just as easily.
Post by: Shades on October 17, 2009, 04:56:57 pm
Also, by ONLY storing a pit per tile, you cannot precompute common paths, you cannot optimize in any way. You reduce the options down to storing data elsewhere or having an implementation no better than the one integrated into DF itself. In fact, probably worse.
std::bitset is not a bit, its a very very fast way of allowing array concept flags for boolean values. Also any structure that supports the array operators could replace it with zero other code changes, so if we found an array of bytes was enough faster to warrant the memory it's easy to do.
And on the topic of speed, should C++ be used? The overhead of it's enhanced features might be an issue if C can be used just as easily.
I'd class this as premature optermisation which is always a mistake, once we have something working then if it's a problem it's easy to refactor away classes and remove any overhead. Compilers are pretty good at avoiding such things these days anyway though, and personally I'd rather work with the features than without because I'm lazy.
(And lazy people are efficient and being inefficient takes more effort)
Post by: Draco18s on October 17, 2009, 05:05:55 pm
Post by: Sunken on October 17, 2009, 07:10:02 pm
Post by: Jifodus on October 17, 2009, 11:00:04 pm
For additional optimizations that would likely be beneficial is to look at path caching and picking inefficient paths using path caching vs finding the optimal path every single time.
Side note, using C++ may lead to inefficient implementations. I'm really looking at the word "classes," because if you're already thinking of multiple classes you've already made it inefficient. And unless you're planning on using template programming to create precompiled optimizations you'll probably do worse using C++.
Post by: Impaler[WrG] on October 18, 2009, 12:50:33 am
Post by: Sunken on October 18, 2009, 07:55:47 am
I highly doubt you'll be able to optimize the algorithms any more than what they currently are now. For the most part, I believe that the only way to optimize pathing is to allow players to be able to place waypoints instead of dreaming up a magical computer algorithm for figuring out rooms/halls or a TCZ that players already do naturally.
Apart from the arrogant tone, I agree with the potential of letting the players help. TCZs can already utilize what is already in place - doors - to let the player influence the shape of zones. Waypoints is one possibility, surely. But would players bother?
Post by: Slogo on October 18, 2009, 10:47:48 am
Quote
5: It might be a good idea to work in parrallel on other objects that require a path such as liquid or gaseos flows, as well as siege mechanics, allowing catapults and ballista to fire in varying degrees of rotation in both the horizontal and verticle plan
Liquids and gases don't path so there's no work to be done there. Toady's algorithm is already pretty slick for that type of calculation other than not allowing stability for partially filled containers of water.
The general path finding can definitely be optimized, I believe Toady is just using an A* algorithm at the moment which is slower than A* using a hierarchy. Even if it can't his path finding is a closed source solution, ours will provide a usable library for other projects and games.
Post by: zwei on October 18, 2009, 12:23:51 pm
First thing you need to have is interface: "Test client" and "Protype". Prototype does not even need to implement anything better than good old flood fill pathing algorithm. Primary goal being to agree on interface and to implement it and to use it ... so that any implementation relevant problems are discovered right away.
So, I guess typical usage will look like this:
Spoiler (click to show/hide)
Post by: Ampersand on October 18, 2009, 12:33:14 pm
Post by: zchris13 on October 18, 2009, 05:11:29 pm
Post by: Silverionmox on October 18, 2009, 05:17:24 pm
They will do it automatically when defining rooms. Especially the new burrows will be useful in that regard (they will also reduce the length of the average path that needs to be found, and that will potentially mean huge FPS gains). Burrows will typically be defined functionally by the player (eg. the farming burrow, the kitchen and pantry burrow, the residential burrow, etc.) and that means most of the movement of a typical dwarf between burrows can be predicted. That opens perspectives.I highly doubt you'll be able to optimize the algorithms any more than what they currently are now. For the most part, I believe that the only way to optimize pathing is to allow players to be able to place waypoints instead of dreaming up a magical computer algorithm for figuring out rooms/halls or a TCZ that players already do naturally.
Apart from the arrogant tone, I agree with the potential of letting the players help. TCZs can already utilize what is already in place - doors - to let the player influence the shape of zones. Waypoints is one possibility, surely. But would players bother?
We'll have to look what the player wants sooner or later, if we want to make something that allows the player to realize his wishes more effectively in the game on the computer. The easiest way to do that is to look at things he'll need to do anyway (defining burrows, rooms, placing doors, ...) Placing a door is usually a big flashing sign: CHOKEPOINT HERE. After that, why wouldn't the player want to tell his dwarves that 'that is the main corridor' and 'that is the main entrance' etc., instead of seeing every hauler queuing through the maintenance tunnel because that way it's one square shorter?
I remembered something, that may point to an obvious inefficiency in the current code. The normal traffic weight for an open space is two. Experiments have demonstrated that reducing the normal weight on the entire map down to one produces a marked benefit in terms of frames per second, as dwarfs manage to determine paths faster.Changing it in the init settings doesn't make a difference apparently (129 dwarves and a fair bit of tunnels).
Post by: Slogo on October 18, 2009, 06:59:25 pm
Objects like doors should naturally be along any hierarchy divisions. Not because there's a door there but because it's a narrow chokepoint giving us a well defined region with limited routes to neighboring regions.
Post by: zchris13 on October 18, 2009, 07:10:13 pm
Post by: Silverionmox on October 19, 2009, 04:27:30 am
Relying on player hints for zones is a bad road to go down. It means that the library we write isn't a 3d tile based path finding algorithm but instead a pathfinding algorithm specifically for dwarf fortress. You can't expect every use of the path finding library to be Dwarf Fortress, even if its requirements are the driving force behind the design.Designing the Universal 3d Based Path Finding Algorithm might be a little too ambitious. If that's the goal, a lot more factors will have to be taken into account and as a result the project will become larger and less focused. That requires more work, more persons involved, more coordination and all that for less tangible and slower improvement in DF. If recruiting happens elsewhere too, fine. If it doesn't, the project will flounder due to overextension. I think it's better to develop a library primarily useful for DF, that's also useful for other projects, rather than a universal library that might also be useful for DF.
Post by: Starver on October 19, 2009, 07:44:38 am
If we want to stick with 7 movement types, I'd drop "jump". It's a cool idea, but I think it'd get too complicated, especially because if you wanted things to jump different heights you'd need a different movement category for each height.I'm not married to 'just' seven types, they were just the seven that occured to me. And another one nicely gives fills 'flag' byte. Not to mention that I quite like the Build Destroyer routing possibility. (Although had I also missed the Pet Passable possibility (being unintentiaonally alliterative, there)?)
As to "jump" (maybe partner with "climb", as jumps should will only ever be level/downwards, and climb only[1] upwards) and the limitations, the pathing could (similar to pathing with horizontal clearances) cull routes from with sequences of climb/jump-type pathing segments greater than the [TAG]-allowed amount for a creature of that size/ability.
Sorry, spent too much time on another project, at the weekend, to bash away at parctical implementations/demonstrations of my own ideas.
[1] Ignoring "climbing across an underhang" being similar to "jumping into/across a gap that happens to have an underhang above it".
Post by: Starver on October 19, 2009, 07:56:00 am
I highly doubt you'll be able to optimize the algorithms any more than what they currently are now. For the most part, I believe that the only way to optimize pathing is to allow players to be able to place waypoints instead of dreaming up a magical computer algorithm for figuring out rooms/halls or a TCZ that players already do naturally.I think there's two different Optimisations at stake here:
- Writing an established algorithm in better code, by one or more measures of 'better' (neater, quicker, less memory intensive, easier to use, whatever)
- Actually using a "better algorithm" that is optimised for what the code is used for. A bit like the DFMap-thingy system optimises BMPs far better (non-lossily) than even PNG, given the expectations of the image being in cells and of limited cell types, or the way that JPEG's lossy compression is 'good enough' for smooth/photographic images even if it causes artefacts if misused/badly tuned to its subject, and gives a smaller footprint.
There's possibly a third, but right now I've still got some more thread-reading to do, so I won't complicate matters.
Post by: Slogo on October 19, 2009, 09:05:53 am
Relying on player hints for zones is a bad road to go down. It means that the library we write isn't a 3d tile based path finding algorithm but instead a pathfinding algorithm specifically for dwarf fortress. You can't expect every use of the path finding library to be Dwarf Fortress, even if its requirements are the driving force behind the design.Designing the Universal 3d Based Path Finding Algorithm might be a little too ambitious. If that's the goal, a lot more factors will have to be taken into account and as a result the project will become larger and less focused. That requires more work, more persons involved, more coordination and all that for less tangible and slower improvement in DF. If recruiting happens elsewhere too, fine. If it doesn't, the project will flounder due to overextension. I think it's better to develop a library primarily useful for DF, that's also useful for other projects, rather than a universal library that might also be useful for DF.
Well obviously that's why we're keeping it focused on Dwarf Fortress, I just don't think we should rely on things ultimately DF specific so the game can keep a broader application to other tile based rogue-ish games.
For example doors, or more generally dynamic open/shut tiles (in DF's case this would also mean floodgates, bridges and hatches) are a good criteria for breaking up regions. It's something that would likely apply to other games. Something like stockpiles, room definitions, zones, or burrows on the other hand are very DF specific and would likely pose problems for applying the game to other applications. Even with DF these are a bad idea, how are you going to optimize pathfinding based on room definitions in Adventure mode?
Post by: Starver on October 19, 2009, 09:28:47 am
how are you going to optimize pathfinding based on room definitions in Adventure mode?
- Actually have useful-to-the-Adventurer room definitions
- ? ? ?
- PROFIT!
Post by: Shades on October 19, 2009, 09:37:58 am
This probably means we will end up with separate routing algorithms for dwarfs in fortress / other things in fortress / adventure mode as the latter two won't need traffic designations.
Post by: Sunken on October 19, 2009, 09:49:15 am
For example, we have the pets-and-doors issue, but in general this may be much more complex: different castes of creature may be allowed through different sets of doors (guards, workers, guests, nobles, the owner of a given room).
Then there's the issue of knowledge of spaces - an invader might not be allowed to path through spaces he doesn't know about.
Then there's traffic limitations - "dwarfs don't go outdoors", "absolutely forbidden zones" and other player-imposed restrictions, possibly applying differently to different agents.
Also, you might envision situation-dependent limitations - disallowing civilian dwarfs from moving through "enemy held territory", automatically maintained (rather than user-specified in detail).
What all this adds up to, I think, is that each path search is in principle unique, and we should have a system that permits arbitrary limitations to be placed on any individual path search. A path request could be as general as:
"Find a way from tile A to any tile with a limestone block; staying indoors at all times; going through no restricted doors except my personal room; walking, or jumping down no more than 1 z-level at a time; staying out of dangerous areas"
It's not as hard as it sounds, in principle: as has been stated already, A* can do all this - we just place custom limitations on which search nodes we are allowed to open. Whatever zone types we choose need to be annotated with the data we use for the limitations, naturally, but that will be obvious once we get there.
The tricky bit is that A* needs a connectivity map to be efficient. We can't maintain a connectivity map for each such custom request so we must find some way of cache the most commonly used ones, and compute an exact or approximate variation of the connectivity map when a special path request needs it. Requests for unusual restrictions are just that - unusual, so maybe we can get by with a flood fill every time for them. The more common ones ("general indoor worker dwarf walking through any unlocked door") will stay cached and updated constantly.
It's not a trivial problem but should be possible to solve, and with big dividends.
Finding good heuristics will be another challenge.
Post by: Shades on October 19, 2009, 10:06:19 am
Some restrictions are obviously more complicated than others, or at least require more game world knowledge. Enemy held territory is open to interpretation for example, further more how would a dwarf starting on a path even know that the last few squares where held by the enemy. (Of course you could argue it's the same way they know a sock is laying there waiting for an owner :) )
We could use a simplistic decorator like pattern to allow fairly arbitrary comparissions, as long as it's all inline and templated it will be a minimal although possibly significant overhead.
I would recommend sticking to the basic bit array for things like magma and water and looking into expanding it if and when it's required. As long as the check functions are isolated and well tested its easy to refactor and the basic system should allow duplication of the DF system without overly complicating the problem.
Post by: Slogo on October 19, 2009, 10:37:09 am
Designations like Indoor/Outdoor could also be solved easily if we use a Type->Subtype allowance flag system like I described earlier. If a creature is designated to only be allowed indoors then they can check the sub tile of any tiles during A* to see if its of a valid type much like they might with water or magma.
Post by: Sunken on October 19, 2009, 10:42:10 am
I would recommend sticking to the basic bit array for things like magma and water and looking into expanding it if and when it's required. As long as the check functions are isolated and well tested its easy to refactor and the basic system should allow duplication of the DF system without overly complicating the problem.
Quite, but it's important to remember not to circumscribe ourselves while coding. No optimizations based on the knowledge that capabilities will be bitmasks of a certain size! No optimizations based on the knowledge that the goal is a single tile!
Post by: Shades on October 19, 2009, 10:45:54 am
Designations like Indoor/Outdoor could also be solved easily if we use a Type->Subtype allowance flag system like I described earlier.
To be honest you only need a single flag for indoor/outdoor as it's one or the other, but I agree it should be in the list of flags.
Blocked - Non Construction (tree/wall etc), Construction (for building destroyers), Indoor, Water, Magma, Pet Passable, Room (of any kind, probably don't need to specify), Restricted Traffic, Low Traffic, High Traffic, Open (for fliers)
Nothing else springs to mind, but it's getting on in the day and my brain is shutting down.
Traffic could be combined but the list I put is all single flags as it is.
Quite, but it's important to remember not to circumscribe ourselves while coding. No optimizations based on the knowledge that capabilities will be bitmasks of a certain size! No optimizations based on the knowledge that the goal is a single tile!
Agreed, I would say that the only 'flags' requirement is that it supports the array operator and the goal is based on some goal state comparison passed into the function (probably a comparator that can be in-lined away during compile)
Post by: Sunken on October 19, 2009, 10:47:31 am
Designations like Indoor/Outdoor could also be solved easily if we use a Type->Subtype allowance flag system like I described earlier. If a creature is designated to only be allowed indoors then they can check the sub tile of any tiles during A* to see if its of a valid type much like they might with water or magma.More importantly, subtyping can make for efficient representation of the connectivity map. For example, a dwarf with access to a certain room is also a dwarf with general dwarf access; thus, if a goal is reachable in the general-access connectivity map he doesn't need to check (or create) a connectivity map that uses his special room.
Post by: Shades on October 19, 2009, 10:51:08 am
Post by: Jifodus on October 19, 2009, 10:52:30 am
I remembered something, that may point to an obvious inefficiency in the current code. The normal traffic weight for an open space is two. Experiments have demonstrated that reducing the normal weight on the entire map down to one produces a marked benefit in terms of frames per second, as dwarfs manage to determine paths faster.This only happened if you manually overrode the weights in your fortress because it would tell DF to ignore the exterior of the fortress until you've fully explored the interior of the fortress. Since for the most part, the interior of the fortress is smaller than the exterior.
By extension, you can influence path finding by placing walls of restricted areas based on if a section of the fortress or for a single room rarely needs to be accessed. In addition, by using low traffic along major hallways will also help.
These optimizations unfortunately don't help when building something, because it appears that it explores the entire map regardless.
Relying on player hints for zones is a bad road to go down. It means that the library we write isn't a 3d tile based path finding algorithm but instead a pathfinding algorithm specifically for dwarf fortress. You can't expect every use of the path finding library to be Dwarf Fortress, even if its requirements are the driving force behind the design.How would adding player hints make it specifically for Dwarf Fortress?
Objects like doors should naturally be along any hierarchy divisions. Not because there's a door there but because it's a narrow chokepoint giving us a well defined region with limited routes to neighboring regions.
It's just a point of interest, the path finding algorithm can use it or ignore it. Most games use waypoint navigation, however, players don't usually see it. I would've suggested using area navigation (which is essentially the same concept as TCZ's), but it would've been pointless since DF uses discreet areas and it would only useful for hallways larger than 3 tiles wide.
The tricky bit is that A* needs a connectivity map to be efficient. We can't maintain a connectivity map for each such custom request so we must find some way of cache the most commonly used ones, and compute an exact or approximate variation of the connectivity map when a special path request needs it. Requests for unusual restrictions are just that - unusual, so maybe we can get by with a flood fill every time for them. The more common ones ("general indoor worker dwarf walking through any unlocked door") will stay cached and updated constantly.Thank you for the clarification. Most of what I could got from all the previous posts was that "We'll optimize the algorithm," which didn't make any sense since A* is already optimal assuming that the heuristic is >= actual cost. I still would like to know though, what is the algorithm are you planning on using to discover areas, since an idea is one thing, but an implementation is something else.
It's not a trivial problem but should be possible to solve, and with big dividends.
Post by: Shades on October 19, 2009, 11:02:00 am
Thank you for the clarification. Most of what I could got from all the previous posts was that "We'll optimize the algorithm," which didn't make any sense since A* is already optimal assuming that the heuristic is >= actual cost. I still would like to know though, what is the algorithm are you planning on using to discover areas, since an idea is one thing, but an implementation is something else.
I was assuming something that acted like a flood fill to find the zones and something that used an A* walker to find the route. However part of the point is to abstract out the algorithms so people can drop in custom ones that might work better for our specialised situation (grid based map for example). If you prefer to think of it as way point navigation then each zone is a way-point and it is connected to all adjacent zones.
Post by: Silverionmox on October 19, 2009, 11:10:45 am
Marking an area as especially relevant for certain actions is something that comes up in a lot of games, I imagine. It's certainly useful.Relying on player hints for zones is a bad road to go down. It means that the library we write isn't a 3d tile based path finding algorithm but instead a pathfinding algorithm specifically for dwarf fortress. You can't expect every use of the path finding library to be Dwarf Fortress, even if its requirements are the driving force behind the design.Designing the Universal 3d Based Path Finding Algorithm might be a little too ambitious. If that's the goal, a lot more factors will have to be taken into account and as a result the project will become larger and less focused. That requires more work, more persons involved, more coordination and all that for less tangible and slower improvement in DF. If recruiting happens elsewhere too, fine. If it doesn't, the project will flounder due to overextension. I think it's better to develop a library primarily useful for DF, that's also useful for other projects, rather than a universal library that might also be useful for DF.
Well obviously that's why we're keeping it focused on Dwarf Fortress, I just don't think we should rely on things ultimately DF specific so the game can keep a broader application to other tile based rogue-ish games.
For example doors, or more generally dynamic open/shut tiles (in DF's case this would also mean floodgates, bridges and hatches) are a good criteria for breaking up regions. It's something that would likely apply to other games. Something like stockpiles, room definitions, zones, or burrows on the other hand are very DF specific and would likely pose problems for applying the game to other applications. Even with DF these are a bad idea, how are you going to optimize pathfinding based on room definitions in Adventure mode?
AI settlements will need to be generated with their own set of room definitions, traffic designations, etc.. That will break save compatibility again, so it's not trivial - but something like that has to happen if the townsfolk ever are not to act like they all suffer from terrible amnesia, don't really know what they're doing there, and just amble around wondering whether they are just a randomly generated npc in a big simulation.
There certainly are improvements to be made in the more general area, but can you knock 80% off A* or another algorithm that is about the best that so many others (computer scientists, roboticians, game designers, mathematicians, etc.) have done so far? It's comparatively easy to optimize the pathfinding of the 90% of the fortress inhabitants that spend their lives mostly crafting, hauling and sleeping so their 90% of cpu usage can be reduced to 10%.
If it works well, other game designers might start to use areas more often in pathfinding.
Post by: Sunken on October 19, 2009, 11:21:35 am
Thank you for the clarification. Most of what I could got from all the previous posts was that "We'll optimize the algorithm," which didn't make any sense since A* is already optimal assuming that the heuristic is >= actual cost. I still would like to know though, what is the algorithm are you planning on using to discover areas, since an idea is one thing, but an implementation is something else.
Well... "A* is optimal" means that A* finds the optimal path - not that it does it in the least possible time!
The time taken is given by the shape of the search space we give to A* (which in our case comes from the zones we use, the limitations imposed by movement types, and the use of a connectivity map), and the heuristic function we use (absolutely crucial), and also by the low-level programmatical implementation (such as which language to use). I'm just claiming the two former are vastly superior targets for optimization efforts than the latter.
Post by: Starver on October 19, 2009, 11:37:38 am
A: Good for the running of the programming, as well as speed-up/resource improvements it add's 'character' to the routes taken (perhaps reflects a little "artificial stupidity" on behalf of the agents)
B: Bad for the player, because she can no longer meta-game the situation (normally she'd expect a particular behaviour and thus account for it, suddenly it's unknowingly complex what might happen)
C: Both/neither/something else?
(An idle bit of postulating, not a call for a formal vote or show of hands... Just thinking out loud. Besides, some people consider the current "route around the mountain to finish the tunnel" situation as an 'artefeact glitch' of the current system, and one which could be solved by methods <foo> or <bar> or <baz>...)
Post by: Sunken on October 19, 2009, 11:50:48 am
I'm counting on that with my TCZ suggestion, because "silly-move" navigation inside TCZs is not optimal. In extremely contrived circumstances it can be as much as 50% inefficient.
That's on the small scale, however. On that scale, one could even get by with having dwarfs use different pathfinding locally depending on whether the player is looking at them or not... if it is visually displeasing. Large-scale pathing should be acceptable.
Post by: Slogo on October 19, 2009, 12:57:42 pm
Quote
How would adding player hints make it specifically for Dwarf Fortress?
Because the idea of sectioning off a large enough % world with designations is something pretty unique to dwarf fortress. A lot of other tile based games, including adventure mode of DF, don't have the luxury of these designations. It's something that would require a fair bit of coding for vague benefits at this point. Since we need to support the 'no hints' case anyways (even in DF) the whole idea is basically committing to a lot of work for unclear at best gains.
Even within DF there are plenty of cases where player hints may be less than ideal or just not workable. What about a general 'barracks' bed room design where you have an open room with many bedroom designations in it? What about a stockpile room that's really 8 different stockpiles? What about non-continuous stockpiles or overlapping rooms? The divisions based on the player hints in these cases are going to be less than ideal or outright ambiguous.
Post by: numerobis on October 19, 2009, 01:23:06 pm
http://idefix.uchicago.edu/~bhudson/src/astar_test.tgz
It reads a file (look at test-input for an example), stores it as a grid, then does 500 random A* searches. It's easy to do 500 random searches using a different algorithm. Ideally we'd have a library of input files, and a library of paths to find in those files.
It is in C++, which is of course very slightly slower than C except when it's much faster (http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=gpp&lang2=gcc&box=1).
User annotations are not my idea of a good time: users will get them wrong, and they've (we've) got more fun things to do than help the pathfinder.
A convex subdivision of above-ground space would be a fairly big boon for flyers. I'm not sure it would be for below-ground space, where much less is convex. We have to make sure our subdivisions are enough bigger than mere grid squares.
We don't have to decompose geometrically though. We have the whole field of graph partitioning to draw on. For example, one subdivision commonly used for clustering in multi-commodity flow problems and the like (i.e. network routing -- which is pretty much what we're doing here, if you think of dwarves as packets): start a seed at a random spot somewhere in the graph. Build a cluster by taking in neighbours of nodes in the cluster until the ratio (number of neighbours) / (number of nodes in the cluster) is some small value. Intuitively, this means you repeat until you hit a chokepoint. Repeat until all nodes have been put into some cluster. Now, create a new graph by saying that clusters are nodes now, and there's an edge between two clusters if in the original graph, there was an edge between a node in one cluster and a node in the other. Repeat until you have a small number of clusters, and you've created a clustering hierarchy. Under various assumptions that I'm fairly sure we satisfy (but which I don't remember so I can't check -- however, grids usually satisfy most of what you'd want to satisfy), clustering is linear or so time and yields a log n depth structure. The size is linear, but depending on how you set the constants, particularly on the lowest-level clustering, is basically tiny. Edge weights (e.g. restricted zones) are easy to include and don't change anything fundamental: instead of number of neighbours / number in cluster, it's sum of edge weights to neighbours / sum of edge weights in cluster.
Given the hierarchy, it's relatively cheap to find an approximately shortest path from s to t: find the least common ancestor of s and t in the hierarchy, and path in that graph. Then, you recursively find a path from s to the next "node" (actually, itself a cluster) and so on. Each cluster is approximately round, so whatever path you find is pretty much a straight-line path through the graph -- that's why it's approximately a shortest path. The path-finding time is near-linear in the length of the path.
It seems non-hard to run this dynamically (maintain the clustering as you add or remove edges and nodes), but we never did work out the details.
There are two types of pathfinding we should be interested in: (1) going from s to t, where s is the dwarf and t is the dwarf's unique bed. (2) going from s to T, where s is the dwarf and T is the set of all booze barrels, of which there are many, strewn about all over the place. What I mentioned above handles (1), but I'm not clear on whether it handles (2) nicely. Note that A* also only handles (1).
Post by: numerobis on October 19, 2009, 01:29:11 pm
Post by: Jifodus on October 19, 2009, 02:07:47 pm
Even within DF there are plenty of cases where player hints may be less than ideal or just not workable. What about a general 'barracks' bed room design where you have an open room with many bedroom designations in it? What about a stockpile room that's really 8 different stockpiles? What about non-continuous stockpiles or overlapping rooms? The divisions based on the player hints in these cases are going to be less than ideal or outright ambiguous.I was thinking more of global scale. What you're thinking of is localized conditions, in a localized situation like you describe path-finding is already near optimal. The path-finding that really slows DF down is at a large scale, when it pointlessly explores winding tunnels.
When you consider small scale areas like path-finding within a room, or even neighboring rooms the cost of pointless exploration is fairly minor. But if you were to attempt to find a path from deep within the fortress to the map edge, it'll take many tries and dead-ends before it makes it to the surface. After it makes it to the surface, the complexity usually goes down so it's more or less a straight path to the edge.
The goal of the waypoint suggestion is to make it so the player just has to put about 10 waypoints down in the main hallways and have path-finding costs decrease substantially. The player shouldn't have to figure out the optimum path between the waypoints or even which waypoints should have a precomputed path to another waypoint. The main goal is to keep the number of hints low and act as highway to different parts of the fortress.
While it's already possible to do something similar with the current traffic designations, it's high maintenance and would require manipulating the weights since the difference between normal traffic and high traffic is only 1.
Post by: Slogo on October 19, 2009, 02:14:44 pm
Even with waypoints the issue still stands: It's DF specific. It has no application outside of a manager type role. You couldn't even use the waypoints in adventure mode for example.
Post by: Jifodus on October 19, 2009, 02:56:43 pm
Even with waypoints the issue still stands: It's DF specific. It has no application outside of a manager type role. You couldn't even use the waypoints in adventure mode for example.Waypoints are a general method for taking a complex topography and creating a simple representative graph. In no way is it limited to a manager type role.
Even if waypoints aren't optimally placed, having even having parts of precomputed paths that will go through major hallways can help with travel between distant parts of the fortress. This is helpful even in adventure mode, the game engine could place waypoints inside each of the buildings in an above ground village. For a fortress/cave that is player/computer generated, several random waypoints may be placed with the same results. If a player wants to place several waypoints to act as more accurate hints to where primary paths should be it is left as an option, but not a requirement.
If you run into one of the precomputed paths during normal path discovery, you can switch to a bidirectional method where you start at the endpoint trying to find a path to the starting point. If the backwards path finding also hits a precomputed path, it's now possible to skip detailed searching between the start and end and use the precomputed paths. After finding a path, it can potentially be used as a new precomputed path or, depending on how much of the path used existing precomputed paths, can be used to determine if a new detailed path is required.
Post by: Slogo on October 19, 2009, 03:25:06 pm
A Hierarchy with caching of movement between regions is essentially the same as a pre-computed path. Instead of A* ing over 100s of tiles you simply A* to the first border of your region, follow the computed paths between regions, then A* to the destination. Even without the precomputed route cached you can at least distribute the pathing work over time using the hierarchy.
Also with a dynamic map regions are easier to maintain than waypoints. It's a lot harder to tell if a waypoint is out of date compared to a region. With regions you know exactly what region needs to be updated based on a map change. With waypoints it's a lot more difficult to tell if a new opening creates a better path or blocks a set path.
Post by: JanusTwoface on October 19, 2009, 04:01:59 pm
A Hierarchy with caching of movement between regions is essentially the same as a pre-computed path. Instead of A* ing over 100s of tiles you simply A* to the first border of your region, follow the computed paths between regions, then A* to the destination. Even without the precomputed route cached you can at least distribute the pathing work over time using the hierarchy.
One potential problem with this is that heavily trafficked paths (specifically in DF, but possible in other games) impose a penalty to movement when there are multiple agents in the same square.
It's not a horrible problem in the case of larger hallways because the agents can have a secondary routine that automatically routes them around each other in problems like that. Like staying to the right when passing (or left as the case may be).
The bigger problem comes from narrow/twisty paths that have similar costs. IMO, a good path finding algorithm should be able to randomly choose the worse path proportionately to how much worse and the average traffic through both to help balance the load along precomputed paths.
On another note, has anyone looked around for any open source projects that do the same kind of thing? It seems unlikely (albeit possible) that the DF community would be the first to come up with such an idea. It's entirely possible that someone else has already done the original work and we can build from there.
Post by: Slogo on October 19, 2009, 04:17:01 pm
If you did want something to handle traffic flow in a more sophisticated way then I'd say that you could store something like a baseCost and trafficCost for tiles (short ints). The base would be static while the trafficCost could respond to traffic dynamically. When a creature travels over the tile the cost of the tile could go up incrementally. Conversely over time, and to keep it simple I'd say every x game time regardless of when the tile was last stepped on, the trafficCost could be decremented. Then during pathfinding actualCost = baseCost + trafficCost * weight where weight < 1 (likely much less).
The other simple and more pronounced way to handle traffic is going to be in how the A* algorithm expands out + any smoothing operations we perform. If you favor an algorithm or smoothing that will expand out to generally run down the middle of a hallway or one side of a hallway then that'll help a lot.
Overall I'm wary to do any system like that because of the overhead it would add to pathfinding. There are probably some better algorithms for dense populations out there but I'd have to research it more.
I also don't think it's any worse than what you see now as the current system will generally route dwarves along the same paths constantly.
Post by: Footkerchief on October 19, 2009, 04:27:35 pm
On another note, has anyone looked around for any open source projects that do the same kind of thing? It seems unlikely (albeit possible) that the DF community would be the first to come up with such an idea. It's entirely possible that someone else has already done the original work and we can build from there.
The much-linked article on HAA* (http://harablog.wordpress.com/2009/02/05/hierarchical-clearance-based-pathfinding/) includes the code that guy used for testing (http://code.google.com/p/ahastar/downloads/list) and a link to what looks like a more generalized library. (http://www.cs.ualberta.ca/~nathanst/hog.html)
Post by: JanusTwoface on October 19, 2009, 04:32:01 pm
Shiny. That gives me even more reason to read the article (it's open in one of my tabs already, but I'm a little behind on going through them).On another note, has anyone looked around for any open source projects that do the same kind of thing? It seems unlikely (albeit possible) that the DF community would be the first to come up with such an idea. It's entirely possible that someone else has already done the original work and we can build from there.
The much-linked article on HAA* (http://harablog.wordpress.com/2009/02/05/hierarchical-clearance-based-pathfinding/) includes the code that guy used for testing (http://code.google.com/p/ahastar/downloads/list) and a link to what looks like a more generalized library. (http://www.cs.ualberta.ca/~nathanst/hog.html)
Post by: Silverionmox on October 19, 2009, 05:13:20 pm
Well, then design the library for other games... but then I wonder what it's doing here? Trying to to serve many masters will end up satisfying none. Both approaches are complementary rather than mutually exclusive anyway.QuoteHow would adding player hints make it specifically for Dwarf Fortress?
Because the idea of sectioning off a large enough % world with designations is something pretty unique to dwarf fortress. A lot of other tile based games, including adventure mode of DF, don't have the luxury of these designations. It's something that would require a fair bit of coding for vague benefits at this point. Since we need to support the 'no hints' case anyways (even in DF) the whole idea is basically committing to a lot of work for unclear at best gains.
Just use any present designations as a starting point. Let the algorithm determine the rest of the terrain sections automatically.. which would have to happen anyway without using the designations. The result will be a little different than full automation, but the sections will resemble the vision of the player more closely.
Even within DF there are plenty of cases where player hints may be less than ideal or just not workable. What about a general 'barracks' bed room design where you have an open room with many bedroom designations in it? What about a stockpile room that's really 8 different stockpiles? What about non-continuous stockpiles or overlapping rooms? The divisions based on the player hints in these cases are going to be less than ideal or outright ambiguous.A barracks/communal bedroom can be defined as a single room.. If the player wants to do the work to make a barracks-like bedroom setup (because of space constraints, roleplaying goals, etc.), why shouldn't the dwarves think that way and try to avoid the personal space of as many dwarves as possible? The same for the 8 different stockpiles in the same room. If the player considers them separate spaces, then the dwarves will too. Non-continuous rooms are invalid as section and should be considered as two parts - trivial to check and split. The overlapping part of the rooms can be considered a separate space, and merged with a nearby one if it happens to be to small to be meaningful. Alternatively, room overlap could be cut out since it doesn't serve a useful function anyway - introducing shared rooms instead, if and when that's necessary. Otherwise ignore them if they don't give clear information on cells. But how is that going to be any worse than making the computer guess what the player wants to happen?
Post by: zchris13 on October 19, 2009, 07:08:38 pm
Let it be able to draw on parameters for each creature. Let you be able to draw on predetermined factors. (such as sociability), or the ratio of the times the dwarf has traveled a path to the number of times it has had to walk around another dwarf.A Hierarchy with caching of movement between regions is essentially the same as a pre-computed path. Instead of A* ing over 100s of tiles you simply A* to the first border of your region, follow the computed paths between regions, then A* to the destination. Even without the precomputed route cached you can at least distribute the pathing work over time using the hierarchy.
One potential problem with this is that heavily trafficked paths (specifically in DF, but possible in other games) impose a penalty to movement when there are multiple agents in the same square.
It's not a horrible problem in the case of larger hallways because the agents can have a secondary routine that automatically routes them around each other in problems like that. Like staying to the right when passing (or left as the case may be).
The bigger problem comes from narrow/twisty paths that have similar costs. IMO, a good path finding algorithm should be able to randomly choose the worse path proportionately to how much worse and the average traffic through both to help balance the load along precomputed paths.
On another note, has anyone looked around for any open source projects that do the same kind of thing? It seems unlikely (albeit possible) that the DF community would be the first to come up with such an idea. It's entirely possible that someone else has already done the original work and we can build from there.
Post by: Slogo on October 19, 2009, 07:25:43 pm
Well, then design the library for other games... but then I wonder what it's doing here? Trying to to serve many masters will end up satisfying none. Both approaches are complementary rather than mutually exclusive anyway.
Just use any present designations as a starting point. Let the algorithm determine the rest of the terrain sections automatically.. which would have to happen anyway without using the designations. The result will be a little different than full automation, but the sections will resemble the vision of the player more closely.
It was written clear as day on the very first post...
Quote
Create a flexible BSD licensed C++ path-finding library specialized for grid based systems like that of Dwarf Fortress and able to efficiently handle rapidly changing map geometry and other challenges such as but not limited too multiple travel domains, multi-tile creatures, variable movement cost terrain and path caching.
That is our goal. Nothing more nothing less. Considering that using designations both backs us into a corner and creates more headaches than performance gains I don't see the value in it.
Post by: Jifodus on October 20, 2009, 12:25:10 am
Now I remember why I was mentally averse to this thread. Good luck with your general library, because you will need it for certain.QuoteCreate a flexible BSD licensed C++ path-finding library specialized for grid based systems like that of Dwarf Fortress and able to efficiently handle rapidly changing map geometry and other challenges such as but not limited too multiple travel domains, multi-tile creatures, variable movement cost terrain and path caching.
That is our goal. Nothing more nothing less. Considering that using designations both backs us into a corner and creates more headaches than performance gains I don't see the value in it.
Post by: Footkerchief on October 20, 2009, 12:43:28 am
Also, my two cents on using user-defined rooms as graph regions: experiment with it later. It doesn't matter yet. I personally don't expect it to pan out, but experimentation is half the purpose of this project, right?
Post by: JanusTwoface on October 20, 2009, 12:46:09 am
Any sane design approach will have both some very general low-level pathfinding code and some DF-specific code, with clear divisions between them.Wait... We play DF. I don't think any one of us is completely sane by any definition.
Post by: Silverionmox on October 20, 2009, 05:05:42 am
And I'm voicing my concerns with that approach in the very same thread, where else? Trying to be useful to an unknown number of hypothetical future games might prove to be just as constricting.Well, then design the library for other games... but then I wonder what it's doing here? Trying to to serve many masters will end up satisfying none. Both approaches are complementary rather than mutually exclusive anyway.
Just use any present designations as a starting point. Let the algorithm determine the rest of the terrain sections automatically.. which would have to happen anyway without using the designations. The result will be a little different than full automation, but the sections will resemble the vision of the player more closely.
It was written clear as day on the very first post...QuoteCreate a flexible BSD licensed C++ path-finding library specialized for grid based systems like that of Dwarf Fortress and able to efficiently handle rapidly changing map geometry and other challenges such as but not limited too multiple travel domains, multi-tile creatures, variable movement cost terrain and path caching.That is our goal. Nothing more nothing less. Considering that using designations both backs us into a corner and creates more headaches than performance gains I don't see the value in it.
As Footkerchief says, there will be room for both general and DF-specific code. There's no reason to exclude DF-specific elements from the start. Even better, those elements can grow into concepts that are universally useful.
Solving general goals from a general viewpoint has been attempted by a lot of people already. If you want to do useful work in that area, the first months will encompass looking for papers, projects and studying what your predecessors did and what their problems were. Otherwise you're just going to replicate that in a laborious way, and that's not useful. If after that enough people are still able/interested to make more improvements, great! I just suspect that the greatest payoff in FPS improvement will come from DF-specific code.
Post by: Sowelu on October 20, 2009, 05:11:29 am
Solving general goals from a general viewpoint has been attempted by a lot of people already. If you want to do useful work in that area, the first months will encompass looking for papers, projects and studying what your predecessors did and what their problems were.
You are my most favorite person EVER. *pastes a gold star onto*
Seriously, people seem to get all attracted to the romantic sides of programming (making a grand useful library), without realizing that solving hard problems is 99% good algorithms and 1% translation into code. Computer science is not coding--it's figuring out the best way to solve problems.
However, I -do- like the idea of a project to create a pathfinding library like this. It's hideously insane to try and do room-based pathfinding for arbitrary rooms...BUT, if you dump months of effort on the problem, you might actually come up with a good solution (and you might end up with a paper of your own to publish!).
Just, people, please realize. Doing this well could be a thesis-worthy project. And an alarming number of thesis papers are on subjects that don't turn out to be useful or productive at all. Go after it from an academic standpoint, NOT a coder standpoint... I would love to see what turns up!
Post by: Impaler[WrG] on October 20, 2009, 06:26:01 am
A library is really no more difficult to program then anything else and we really have no choice, we don't have access to Dwarf Fortrees's code and the best way to make it usable for DF is to make it universally usable with a well designed API. We've already got a slew of planned features that are geared towards DF needs such as multi-tile creatures and path caching. If we find ways to add more DF friendly features that's a plus and we would definitely want to look into it. But people must remember that a lot of "pathing improvements" are all about WHAT things to request paths or estimates too. the long awaited 'Burrows' concept is like this, its going to restrict dwarves to working within a defined area. And It's entirely 'Client side' from the perspective of the pathing engine as it nips expensive path requests in the bud. Keep this concept in mind and ask yourself if a speculative feature is really client or server before saying we need it.
Also this whole thing on using Room data from DF is entirely moot until someone actually finds a way to extract that from DF, in any event we have to make the system work without it first as has been pointed out several times already.
As for research and what kind of initial implementation were going to make we know its going to be a Heirarchial A* implementation as described in the earlier linked examples. I'm all for lifting as much design & code as we (legally) can from that example as well. In fact I'd like to see the code team discuss more of the high level details of implementation plan and move the debate onto the details of that. On a related note I found another nice paper that has a nice method of zoning, it uses a two tier system using a grid to divide the map into square Sectors, the sectors are then sub-divided into one or more continuously connected areas called regions which are tested for connectivity to each other and become the unit for A* pathing as in the previous examples. The Sectors make it easy to translated requested grid coordinates into the start zone for the path. Here's the link, its a pdf slide show.
http://www.cs.ualberta.ca/~nathanst/talks/mmabstraction_talk.pdf
Edit: Found the white-paper for the above slide, much more in depth of course
http://www.cs.ualberta.ca/~nathanst/papers/mmabstraction.pdf
Post by: Silverionmox on October 20, 2009, 07:23:41 am
1. Pure pathfinding, GI
2. DF-specific pathfinding optimizations, very much dependent on what Toady is comfortable with revealing from his code
3. Non-pathfinding tweaks that reduce pathfinding demands.
1 will be the main focus of the coding, 2 is not even an option yet if ever, and 3 are just suggestions as normal.
Any attempts or success in recruiting elsewhere for #1? Two coders will not suffice for the long term, if only for afk issues.
Post by: Starver on October 20, 2009, 09:21:35 am
http://www.cs.ualberta.ca/~nathanst/talks/mmabstraction_talk.pdfInteresting. Read through that. Have also glanced at the white paper link, but not been able to absorb it all in the few minutes I can spend right now.
Is there an answer in the latter (or the former, if I missed it) about how one ensures that "all regions connect to all other regions"? Complex defensive structures could quite easily not just disbar pathing from local location 1 to local location 2 (including through exiting the region and then heading back in) but could quite easily shut half the map off from the other half. Not usually an issue in a game that autogenerates a landscape with isolated pockets of disconnection, but here we'll be dealing with players that, if they aren't shutting whole regions off from each other (vertically, as well has horizontally) merely through defensive play, are full-on MegaProjecting themselves a Great Wall of MountainHomes dividing the map in two or flooding the surrounds with impassable (to most) lava-lakes creating wide expanses of impassability, around little pockets of free but isolated movement.
The visual given seems to suggest a regular region layout, but I could seen the need to shuffle those regional borders around in the DF-map, if you must have somewhere-in-every-region-routes-to-somewhere-in-every-other-region, extend 'tongues' of region, if need be, and in fact do much the opposite of the aim of having everywhere within a regional juristiction trivially connected to each other, and handle the (im)passability between other regions on a level further up the chain. (Yes, so the sub-regional possibly-logically-isolated elements take on that aspect, but with transient connectivities I think it makes more sense to have connected (howsoever transient) elements become themselves a region of a higher order, and these regions connections to similarly ordered regions be referenced upon by the next higher order.)
BTW, not that my own ideas are going in this direction, but could you see a major problem with taking a roughly calculated route, far from optimal (e.g. could easily be /\/\/\/\/\), subdividing it into segments and then assessing the optimal pathing between the centres of each segments as mini challenges (all of them resource-light, even combined) rinsing and repeating for a loop or three until the inefficiences of the route are smoothed out (becoming __________, in this stupidly simple example). On a 100-step-line cycle, 49 quite trivial "two steps at a time" tests highlighting kinks in an unecessarily bendy route (but necessarily and unavoidable bendy around obstacles), and then paying attention to the areas around that route neighbouring that 'betterment' might help. (Of course, the ultimate counter-example is where, from the start point, one has a choice of stepping left to go through one obstacle course or right through another, the exit to each being straight onto the finish-point. The 'shrinking' algorithm would never have chance to 'snap' the route from one side to another, to see if that were more optimal. Or even not an obstacle course at all. Which is a bit of a downer. God job that's not my chosen approach, eh? :))
Post by: Urist McDepravity on October 20, 2009, 11:57:15 am
It should be some kind of test-server with client libraries, or wise versa, but with ability to run all needed testcases on all algo's and precisely measure time/memory usage.
It should feed initial map to the solver, trace results for preprocessing, then feed tests and map altering commands.
We have quite many map-alterers, including some very rapid ones (cave-ins, water floods), so cache-based algo's should easily adapt to changes.
We have some 'flapping' alterers, like doors, which should be special case as well, i believe (caching algo's may decide it worth to cache graphs for both states if, for example, this particular door change its state often. And it could be so, if its linked to the plate and water-based events generator, or just lever which is pulled often).
Having single test framework would let different teams to create different algo's while maintaining compatibility.
Post by: Combatjuan on October 20, 2009, 12:46:50 pm
I do not, unfortunately, have much time available to me in the next two weeks so I certainly can't do much in the way of maintaining and setting up a test framework for all this (at least until that time). Someone posted a suggested API a few pages back that seemed quite reasonable to me.
I am convinced that the best approach to this is to construct a test interface and an API and start coding and let the competing ideas compete on real tests. Let's see which regioning turns out to be most efficient. Let's see if those who want a more general solution can write a fast such solution in a reasonable amount of time and if those who want a specific solution can do so while still making it sufficiently extensible/adaptable. I know of no other way to answer such questions.
And then I'll go and be a hypocrite and add that I don't have time to get a github going or an API spec or anything. And I'll also offer this consideration which I don't think has yet been mentioned:
* I believe that the map is broken up into 48x48xMAX_HEIGHT tile sectors and that these sectors are dynamically loaded (in adventure mode) as one moves over the map. It might make sense to confine our higher level regions to being within a sector since these may need to be culled from the tree much like door/drawbridges (the above-posted Dragon Age pdf article actually took this approach anyway). And relatedly, it also sounds like Today is making some changes to how sky is being handled as well as various underground layers, though I doubt they will affect things significantly.
Post by: Sunken on October 20, 2009, 02:53:21 pm
BTW, not that my own ideas are going in this direction, but could you see a major problem with taking a roughly calculated route, far from optimal (e.g. could easily be /\/\/\/\/\), subdividing it into segments and then assessing the optimal pathing between the centres of each segments as mini challenges (all of them resource-light, even combined) rinsing and repeating for a loop or three until the inefficiences of the route are smoothed out (becoming __________, in this stupidly simple example). On a 100-step-line cycle, 49 quite trivial "two steps at a time" tests highlighting kinks in an unecessarily bendy route (but necessarily and unavoidable bendy around obstacles), and then paying attention to the areas around that route neighbouring that 'betterment' might help. (Of course, the ultimate counter-example is where, from the start point, one has a choice of stepping left to go through one obstacle course or right through another, the exit to each being straight onto the finish-point. The 'shrinking' algorithm would never have chance to 'snap' the route from one side to another, to see if that were more optimal. Or even not an obstacle course at all. Which is a bit of a downer. God job that's not my chosen approach, eh? :))I think you put your finger on it. In formal terms: a simulated annealing approach; sensitive to local minima
Post by: numerobis on October 20, 2009, 04:23:10 pm
Not lots of people discussing why what I posted isn't going to cut it.
Anyway, I updated it, new URL (http://idefix.uchicago.edu/~bhudson/src/pathing.tgz). I know nothing about good places online to put code repos. I'm using mercurial for this.
README:
read.h/.cpp : read in a grid from a file
point.h : x/y/z
grid.h : store the grid. No information except whether squares are passable.
graph.h : Two kinds of graphs: adjacency lists, and a thin wrapper over a grid.
heuristics.h : an example of a heuristic. A particularly simple example.
astar.h : an example of a search algorithm. This one is a particularly bad implementation of A*. Works on any graph, and any heuristic that knows how to handle vertices of that graph.
main.cpp : put it all together. Do 500 random searches of A* on the L1 heuristic and grid graphs. Report "performance" -- the number of times we touched a grid node.
Requires boost. The makefile is set up for the fink package manager on OSX: it passes -I /sw/include. To set it up for macports, use /opt/local/include. For linux, leaving it as-is won't cause problems. For cygwin, I have no idea.
The interface a graph presents is just: (a) given a node, return an iterator to the neighbours of the node. (b) given a node, return the weight of the node (the cost of entering it, from anywhere). We probably want (c) given a pair of nodes, return the weight of that edge, if any edge exists. I didn't implement that, it's easy to add.
The interface a grid presents is just: given a point (x/y/z), return whether that point is in bounds, and passable. We probably also want: given two points, return whether you can go from one to the other (otherwise, we can't represent floors and stairs properly), and who can enter the square, and so on. Plenty to add here; upon changing that, we'd need to change the GridGraph implementation also to use the edge-based passability definition.
The file format is: x/y/z on the first line, saying how big the map is. Then the maps, one per level. Lines started with ";;" are comment lines (since # often means wall). With a grid that supports floors and stairs (rather than just open space or wall), it will be easy to add parsing for < X > versus . versus _. No support in grid or the file format yet for weights for nodes (though there is in the graph structure).
I have no file format yet for searches to do, nor for dynamic updates. The former is easy to deal with. For the latter, I feel that so far, even just the static case is worth experimenting with.
Post by: sir monko on October 21, 2009, 03:18:49 am
but after skimming this thread, here's my opinion:
dreiche2 already mentioned it, and i wholeheartedly agree: the single most important thing would be to provide a testing and benchmarking framework, so a contributor would have a rough clue if his improvements actually work. this may be (a lot!) harder than it sounds; benchmarking is an art of its own. a careful selection of maps, paths and populations would be needed, so solutions wouldn't specialize too much. long paths, short paths, unreachable destinations, ... ideally, this framework should also account for changing environments.
the closer it'd be to the real thing the more accurate, but also less usable. *sigh* benchmarking is hard.
at least the benchmarking framework wouldn't have to be written in C++. use something simpler. java (noooo!), python, javascript, Lua, whatever. it's about the idea, not the implementation. absolute performance doesn't really matter at this point.
i don't even know how pathfinding is done currently; afaik there's A* calculated on-demand, stored in the dwarves memory and followd until it invalidates because the dwarf runs into an obstacle that wasn't there when the path was calculated. if no path can be found the algorithm terminates after a certain number of iterations to avoid flooding and uses the most promising solution found til then.
this approach is actually pretty nice, really. most importantly, it's simple. also, by storing and evaluating-on-the-go it allows for inaccuracy. on the one hand that looks realistic (why should the dwarf know a wall on the other side of the fortress vanished since he was there a week ago?), on the other hand it greatly reduces complexity.
i already described what i'd do about it: caching. just store the already calculated paths and keep them in memory. of course, this adds a bit complexity, but not as much as, say, machine learning (which sounds nice, but i'm afraid it may be a bit too hardcore) or automatic zoning and waypointing or future prediction magic.
how path caching works:
1. a dwarf needs a path between A and B
2. look at the cache if the path is already there
* if not, calculate as usual
* if it is, take it from there, and increment some use-counter
3. follow the path; if it's blocked, mark for garbage collection.
* every X cycles, garbage collect:
* throw out old and invalidated paths
* if memory's sparse, throw out least used paths
the pros/cons/limitations of path caching, from the top of my head:
- ~ trades memory for speed
- + scalable: you can control the cache size. the more memory, the more cache.
- + optional (gracful degradation)
* no memory available for caching? no problem - it then behaves exactly as it does now.
* also, this could be a bit of a limitation, because it may not speed up machines that'd needed it most. though i don't really know; it may very well be that path caching speeds up even those, depending on how it's influenced by virtual memory/page swapping/etc. - + algorithm independent. doesn't care if A* or HHA* or whatever runs in the background. in fact, the current A* implementation should work without much adaption.
- - in the worst case, if no path is used more than once, there would be a slight cache-lookup penalty. this could be reduced by data structures optimized for fast lookup (hashes, trees, ...)
- possibility for partial paths: if one path is a sub-path of another one, use this one
- possible to implement as part of some kind of "pathfinding-manager" that also handles threading/concurrency, alternative algorithms (one for short paths, different one for long paths, ...)
finally, i already implemented path caching in a very simplified manner to see if it's actually possible.
my findings (if i remember correctly):
- caching is possible without adding too much complexity
- in my very simplified environment it worked very well. between 50% and 90% cache hits
- i have no clue if it would work in an environment as complex as DF (my guess: not as good, but still)
back to work now.
Post by: Sunken on October 21, 2009, 03:39:08 am
Post by: Impaler[WrG] on October 21, 2009, 04:15:41 am
Post by: sir monko on October 21, 2009, 05:00:43 am
I doubt if you'd see many cache hits for DF on a tile-by-tile level, but it might be more practical on a higher abstraction level such as rooms or zones. The cached path might be suboptimal in that case, but one could live with that.
one:
the current path is suboptimal too. a* doesn't guarantee the best path, and because it's stored after calculating (instead of recalculated after every single step) it lets dwarves run into paths that became blocked in the meantime too.
of course cache hits would work *even better* on a higher abstraction level, but i fear those abstractions are actually very hard to implement and possibly very costly to manage in an ever-changing map. if someone manages to implement it successfully: congratulations, you have probably solved most of DF performance problems.
but i'm not convinced. acutally, i fear the added complexity to be a major show-stopper.
two:
my test ran with paths between (randomized) points of interest (POI) and arbitrary locations. i thought of POIs as workshops, piles, etc.
i assume that most movements are not between quasi-random destinations (e.g. a hunter wandering around until he finds game) but few highly frequented locations.
Any successful caching system will defiantly need to work at higher levels in the abstraction hierarchy to be successful, if done properly I expect >99% hits at high level abstraction if the cache size is north of a thousand.
depending on the abstraction, 99% hits is not unreasonable. i assume the number "1000" is just speculation.
in my humble opinion, per-tile path caching (with some tricks) and without further abstraction layers seems like the most reasonable approach, as it doesn't add a lot of complexity and i'm sure will increase performance.
summary:
as always, we have rival forces at play:
+ more cache = more hits = less accuracy = higher lookup cost = more memory needed
* more hits are great!
* less accuracy doesn't really matter IMO, as paths don't get old and are garbage collected as soon they're invalidated. in the worst case a dwarf doesn't use a new shortcut for some minutes.
* higher lookup cost: ha, this is what benchmarking would tell us. to be honest, i don't think lookup would be too expensive - after all, it's not much more than a hash-table lookup (depending on the implementation details)
* more memory needed: scales, may be restricted. additionally, i don't even expect memory cost to be very high. a cached path of, say, n steps may cost n+c bytes, where c is constant (store directions, not coordinates). n can even easily RLE-compressed.
so, assuming paths to use an average of ~200 bytes, you could store over 5k paths per megabyte (yes, that's mega, not giga).
now, the questions are:
a) are the costs of cache management and lookup smaller than those of an a*-recalculation?
b) does this increase performance of ... YAY LUNCH BREAK WOOOOOOO!
b) does this performance increase justify the added complexity?
c) does it work on systems with less memory?
update: back from lunch
Post by: Sunken on October 21, 2009, 06:37:02 am
Sure, but "frequented locations" will be whole stockpiles, farm plots and other multi-tile regions, not single tiles. The restriction of one log or stone per stockpile tile, for example, ensures that you will never get a single cache hit when pathing to get logs or stones from a stockpile.
i assume that most movements are not between quasi-random destinations (e.g. a hunter wandering around until he finds game) but few highly frequented locations.
I still conclude that abstract locations will be absolutely crucial for path caching to have any utility. But feel free to prove me wrong!
Post by: Shades on October 21, 2009, 06:40:58 am
I still conclude that abstract locations will be absolutely crucial for path caching to have any utility. But feel free to prove me wrong!
The only case I can think of is stuff that is moved from workshop to workshop without going to a stockpile in between. I agree with your generate point though, but personally I don't think generating the abstract locations will be hard so I'm not overly worried about it.
Post by: sir monko on October 21, 2009, 07:39:13 am
i understood "abstract locations" more in the sense of the often heard "automatically divide your fortress into meaningful sections and magically add waypoints where it makes sense", an idea i'm not all too fond of. i always thought performant management of sections in an ever-changing map is a lot more complicated than most people realize. maybe i'm wrong.
actually, in the old caching-thread i had the idea of not doing path-selection from (x1, y1, z1) -> (x2, y2, z2), but (x1±1, y1±1, z1) -> (x2±1, y2±1, z2), because moving to the adjacent field would be trivial in almost all situations. 3x3 would roughly covers a workshop and already be a huge improvement.
eager to test that :)
Post by: zwei on October 21, 2009, 08:09:57 am
3x3 would roughly covers a workshop and already be a huge improvement.
eager to test that :)
Actually, it would be useless, because workshops only ever use central tile: that is where all the workers and haulers path to, eight other tiles are there to just make it bigger and pretty and you only have to path to them should something fairly unusual happen - i.e. some items happen to get there.
Post by: sir monko on October 21, 2009, 08:24:47 am
Actually, it would be useless, because workshops only ever use central tile: that is where all the workers and haulers path to
great, that strengthens my point of path caching being useful :)
still, 3x3 greatly reduces the numbers of paths needed for piles and farms, though it introduces the a problem, namely that adjacent tiles are not necessary coverable by one step (e.g. if the slope's too steep).
i agree: grouping adjacent pile and farm tiles would be a huge win. someone should implement it :)
Post by: Kardos on October 21, 2009, 08:40:04 am
If the game knows where the creature and the desired location are before hand, using a higher level caching system might be feasable until the creature is within, lets say, 10 tiles of the destination, upon which the creature begins moving on a 1x1 system.Actually, it would be useless, because workshops only ever use central tile: that is where all the workers and haulers path to
great, that strengthens my point of path caching being useful :)
still, 3x3 greatly reduces the numbers of paths needed for piles and farms, though it introduces the a problem, namely that adjacent tiles are not necessary coverable by one step (e.g. if the slope's too steep).
i agree: grouping adjacent pile and farm tiles would be a huge win. someone should implement it :)
During the initial pathfinding calculation, the destination could do a floodfill up to 10 (or however many tiles the gamemaker desires) tiles away from iteself, and then that barrier is held in memory for acess until the creature finishes pathing.
Post by: Starver on October 21, 2009, 08:58:27 am
How much space should a cache use up? Is it cell-by-cell movement, or abstraction across unarguable open areas? What search routine might be employed to hunt down what cached paths are affected by changes in the linkages at one particular point in the map? (i.e. find paths that are now useless, so junk that path and immediately recalculate the routing for any agents that are affected... Although the converse insofar as some way that an route that opens up can be recognised as useful to an agent currently going around the previous and longer route would be an almost miraculous addition, without complete repathing every time something changes)
Path caches with dynamic awareness could of course feature multiple branchings in their definitions for whether certain features are opened, and a flag to alert an agent on the path ("if (not yet past <feature>) && (past <lastbranchpoint>) then retrace to <branchpoint> and try <otherbranch>", with various alternate handlers for generating brand-new paths or re-using/reversing various parts and swapping in other alternatives (with or without testing for newly opened 'alternate alternatives').
And how about optimising the cached paths (by memory, if not by overhead calculations) by comparing all newly generated paths with pre-existing ones and checking for shared elements, and as soon as shared partial paths can be identified, and stored as 'sub-solution' paths that can thus potentially shorten future path-searches drastically, or at least identify that the shortest-paths across one particular possible tree of travel are longer than those along the other (or can be considered only solutions for when the bridge is raised/door locked/invading army imposing their presence on the alternate route).
Some of my initial thoughts this morning (I'm not awake yet[1]) were that something LZW-like would be simple to implement, 'streaming' each newly established route through to generate a look-up table and then later routes discovering shared lookups not by going "we already know this much, lets see how much more we know", at the encoding end as we stream the points through (after all, there will be no repetition of points within a single route, unless there's a 'mesh' of possible routes in a dynamics-aware multi-route version that isn't internally optimised), but more about taking into account each past route and identifying shared routing. A particular strength in tunnels (although I think it'd be a mistake to make this an end-game target).
(The main strength of LZW, that the encoder does not need to pass the encoding table to the decoder but that each can dynamically generate it upon sending/receiving the streaming data, is also a moot point given that we aren't wanting to reduce 'transmission overhead'. So we would probably fall back upon standard 'pre-processed' compression techniques, as inspiration, that searches for commonalities throughout the raw information that can be considered frequent or extrapolatable and building our 'sub-solution' library from that. Mixing this in with the actually routefinding. While this, in itself, is not a part of the routefinding algorithm, a routefinder that has knowledge of common paths can avoid some degree of repetitious calculation for novel whole-routes with close matches to a previously understood one, e.g. workshop-to-stockpile carrying, for each sockpile tile and a possibly a grouping of similar workshops nearby, or just for progressive retrieval of stones from a mined-out area. But of course done so as to avoid the local minima situation.)
[1] Now it's past dinnertime, but I still don't think my mental reserves are up to strength, still.
justify all that effort for this engine. (It also couldn't handle reverse-matches.)
Post by: Slogo on October 21, 2009, 09:54:12 am
It will increase the # of hits as even if the dwarf is targeting an uncommon tile they can still use cached routes to move between the regions.
It will help prevent dwarves from moving to an obstacle then backtracking. Since the moving entity does not store the complete route, only the relevant information about what region they're in and what region they're targeting they can get the specific path once they enter a region. That way if a region's is updated and the path changes they'll know immediately on entering the specific region.
It will make keeping cached paths up to date a lot easier. It's trivial to know what regions are affected by a change in a tile (just check what regions the tile is in) so it also lets us update far fewer cached paths than checking each one to see if it is affected by the change.
It will allow us to store less. Rather than storing 10 similar paths we can store the region to region paths once.
Post by: sir monko on October 21, 2009, 10:08:51 am
http://www.bay12games.com/forum/index.php?topic=24790.0
~~~
@starver:
imo, the most important thing is to KISS (at least in the beginning).
* i wouldn't make it update-aware (too expensive). paths get invalidated when dwarves that follow it run into a (permanent) problem. i bet that works well enough; also, it's a bit like how it works now. also, cheap.
* paths with unstable tiles (e.g. doors or drawbridges) could be marked as such and evaluated when needed. i think even that is overkill.
* cell by cell. if you store directions and RLE-compress them, it has roughly the same effect as open area abstraction (though it makes a lot of other optimizations impossible).
* cache invalidation: kill the cached path when it's used > X times or it's age > Y (so new shortcuts aren't ignored)
* shared partial paths: MADNESS! my brain starts to smell like burnt bacon if i try to think about the consequences. what else? subpath reference counting!? pathcache-virtual machine with trace-JITting? ;) maybe lars bak and peter norvig play dwarf fortress. hey, if one of you reads this, could you please implement that part?
@starver & @slogo:
yes. yes! yes. of course, increasing the complexity of the cache will almost certainly make it more efficient. but it will also prevent the cache from being ever implemented.
moreover, i'm pretty sure the cache would improve things a lot, even if no subpathtracingjitmumboregionpartinioning was applied.
group piles together, group fields together, and the dumbest cache would pay for itself may times, while being easy to implement (and thus, less susceptible to bugs).
while complexity may make DF interesting in the first place, you certainly want it in its codebase.
Post by: Shades on October 21, 2009, 10:16:35 am
* i wouldn't make it update-aware (too expensive). paths get invalidated when dwarves that follow it run into a (permanent) problem. i bet that works well enough; also, it's a bit like how it works now. also, cheap.
Fairly sure it, if not being update-aware, does recalculate paths before it runs into a problem. If you, for example, lock a door as critters move towards it they switch to a new route very quickly.
Post by: Sunken on October 21, 2009, 10:25:01 am
Post by: shadow_slicer on October 21, 2009, 10:33:42 am
I like the idea of just using a hierarchical A* algorithm. Basically we would do what someone (can't find the post :-[) suggested and use a flooding algorithm to define "rooms" by limiting them based on the number of neighbors/size of the room (to block off choke points). This will correctly group most rooms (whether convex or not) and will group the exterior as one giant room.
The "rooms" will keep a list of their border tiles and which other rooms are connected to those tiles. Additionally they will keep a cache (most recently used) of the cost to travel from one border tile to another border tile. These costs will be calculated using A* within the room.
In order for a dwarf to find a route on the map, the pathfinder will use A* over a modified map consisting of the tiles in the room the dwarf is in, the tiles in the room the dwarf's destination is in, and all the other rooms on the map. Travel between rooms can use the cached values, so it will be much faster than simple A* over the entire map.
Additionally any changes will only affect the room they are in (and possibly an adjacent room). If the modifications are trivial, the room's cache can simply be cleared. For more complicated changes the room can be either be split or merged and the neighboring rooms notified. And that's all that would need to be done!
If we want to support pets, we will need to modify things slightly, since certain doors may be impassible. We could make it so that doors/hatches will automatically end a room, and then just keep a flag on that boundary marking it pet impassible. Then when we perform pathing we can ignore those adjacencies for pets.
This in itself will solve the vast majority of the pathing problems currently plaguing DF. Unfortunately, this will not solve the multi-tile, swimming, flying, digging monster issues we have discussed so far, but we can deal with that in the future. The best way I can see to solve that issue though, is to build a separate pathing map for them (or just use non-hierarchical A* over all of the tiles).
Post by: Silverionmox on October 21, 2009, 12:36:55 pm
That code can be reused for things like ducklings behind their mother, a chain gang, caravans, military formations etc. The distances that need to be pathed are very short and shouldn't pose a significant burden, while still allowing the complete flexibility to move out of the way, should there be a problem with that route.
Cells probably can have multiple connections (eg. two rooms with a wall with two doors between them, or a row of pillars). How is the choice between them made if they are at equal distance? And will the other one ever be used? If something (eg. a catsplosion) needs to be evaded in one cell, will that effect the choice of connection?
Post by: shadow_slicer on October 21, 2009, 12:56:32 pm
This could be made random if in the implementation of A*, potential next-nodes are added to the priority queue such that their relative position among other nodes with the same score is random. (i.e. Find position of queue of first node with that score, and last node with that score and choose a random number between 0 and 1 + [the difference between the two positions], insert the potential next-node at position [position of first node]+[random number]).
This could be made to select for larger hallways if a secondary score consisting of the average width or minimum width of the path was used as a tie-breaker. Also the presence/absence of cats/goblins/miasma could also be used similarly. Or an heuristic taking into account all of the above could be used (anything you can describe mathematically is possible).
Post by: Fieari on October 21, 2009, 01:47:53 pm
Edit: Wait, no... I can compile the test program Boost's tutorials provide, and that finds the proper includes just fine. Weird. And yet when trying to compile this program, I get:
Code: [Select]
9 C:\programming\pathing\graph.h boost/iterator/iterator_facade.hpp: No such file or directory.Not to mention a bunch of other errors that may be related. Now, I'm not using any of the .hg stuff... is there something I'm missing there?
Post by: Slogo on October 21, 2009, 01:53:04 pm
Pets just need to path towards their owner. There should be a parameter how close they need to be (i.e. within how many squares). A pet following the dwarf should only get through the door if the dwarf wants it to stay close, and that means stay directly adjacent.
That code can be reused for things like ducklings behind their mother, a chain gang, caravans, military formations etc. The distances that need to be pathed are very short and shouldn't pose a significant burden, while still allowing the complete flexibility to move out of the way, should there be a problem with that route.
That's starting to get beyond the scope of pathfinding though. There are some crowd and leader/follower based pathfinding optimizations which would be worth looking into but for the most part it's the goal selection that's going to make pets behave the way they do. I think some help with goal finding work may be within what we're doing I don't think we really need to modify how pets pick which tile they're going to head towards, only how they get there.
Post by: Combatjuan on October 21, 2009, 02:05:43 pm
#include <limit>
#include <algorithm>
and comment out one of the printfs on account of 64bitness.
This was on Ubuntu 9.04, x64. I lack mercurial foo and made no attempt to update them.
Post by: Silverionmox on October 21, 2009, 03:09:42 pm
Yes, rooms should have multiple connections. The choice between them would be made with A*. If they are along paths with the same distance to the destination, then the choice of which door is implementation dependent.
This could be made random if in the implementation of A*, potential next-nodes are added to the priority queue such that their relative position among other nodes with the same score is random. (i.e. Find position of queue of first node with that score, and last node with that score and choose a random number between 0 and 1 + [the difference between the two positions], insert the potential next-node at position [position of first node]+[random number]).
This could be made to select for larger hallways if a secondary score consisting of the average width or minimum width of the path was used as a tie-breaker. Also the presence/absence of cats/goblins/miasma could also be used similarly. Or an heuristic taking into account all of the above could be used (anything you can describe mathematically is possible).
(http://imgur.com/rgnon.png)
Assuming this situation with two cells, picking a random entrance: the yellow route is the normal one, the red route is the actual one that is taken when the water and the miasma are avoided. That leads to zigzagging if avoidance is only considered within cells. The choice of connection should be re-evaluated if taking another path is necessary, but only after half of the distance of the new part has been traversed.
Post by: shadow_slicer on October 21, 2009, 03:35:25 pm
There are several ways to deal with the problem you are bringing up.
1) In my previous post I mentioned that if the shortest path was absolutely necessary we could add some heuristic tie-breaking rules to prefer certain paths over others. If one of the rules was to choose the path farthest from miasma/water, then it would avoid the problem depicted in the picture.
2) If the shortest path is not necessary, we could simply raise the path cost for traveling over water and miasma tiles. Then hierarchical A* would find the best path with no modifications.
I don't really see the need to re-evaluate the path after a portion of the path has been traversed. If you really need to for some reason, then the hierarchical A* should be fast enough. You could use a distance-vector based method instead. A distance-vector pathing algorithm would have the rooms record their distance to all other rooms, and a dwarf would path towards the room closest to the destination. This would be sub-optimal, but allows for incremental pathing (though it would actually be more prone the the problems you describe!)
Post by: Impaler[WrG] on October 21, 2009, 04:07:02 pm
Invalidation of cached paths on map changes seems to me to be very tricky, while its possible to know that a path has been blocked by a map change theirs no way to know if the opening of a shortcut has made the path undesirable. As was pointed out already any cache should be set up to occasionally do a confirmationary search after a hit, if the search finds a different path then the one that's in the cache it updates it. This should allow shortcuts and blockages to be picked up rapidly.
Post by: Slogo on October 21, 2009, 04:30:33 pm
As a second note if the red route is more efficient than a smoother route going through the top part of the divider it should still probably be used, I don't think it would make more sense for the dwarf to go all the way around.
Disregarding that I'd handle the situation in a few ways but I need a little more time to write them out properly. Suffice to say that I don't think it'll be a major hurdle to using a hierarchy and I'm sure there's already some pre-existing algorithms to handle it.
Post by: numerobis on October 21, 2009, 10:43:10 pm
What version boost? iterator_facade has been around a while, but some versions of boost out there are *ancient*.Code: [Select]9 C:\programming\pathing\graph.h boost/iterator/iterator_facade.hpp: No such file or directory.
#include <limit>Which file needs those includes? And which printf? All the printfs are printing either unsigned with a %u format, or uint64_t with a %llu format, so those should be clean -- they compile 32 or 64 bit on my machine, but I recognize these things differ across platforms.
#include <algorithm>
and comment out one of the printfs on account of 64bitness.
Post by: Nexii Malthus on October 22, 2009, 01:14:55 am
(http://imgur.com/rgnon.png)I'd go for the artificial stupidity that the red path brings.
I think that attempting to implement behavioural quirks in pathfinding is a good thought but might be far beyond technical limits. [Regarding avoiding miasma specifically]
Post by: Sunken on October 22, 2009, 02:26:08 am
Post by: Impaler[WrG] on October 22, 2009, 02:44:43 am
Post by: Fieari on October 22, 2009, 04:03:49 am
Now that I've removed the makefile, I'm down to just one error.
Code: [Select]
C:\programming\pathing\main.cpp In function `point randomPassablePoint(const grid*)':
16 C:\programming\pathing\main.cpp `random' undeclared (first use this function) If you change random() to rand(), it compiles runs and works. Now just to see how it works and tinker away on my day off...
Post by: Slogo on October 22, 2009, 09:10:44 am
Traffic costs dose make me think how were going to combine that with movement types, a flying creature should have the same cost at all time ware as a walking creature would be slowed down by mud and other unfavorable conditions. Each movement type might require its own cost which may present a significant memory overhead.If you sum up the movement cost of a tile on demand you could also add some sort of ignore flag to just skip these types of factors.
Post by: numerobis on October 22, 2009, 11:33:23 am
If you change random() to rand(), it compiles runs and works. Now just to see how it works and tinker away on my day off...Ugh. I haven't suffered from the lack of random() in about 10 years. In real code we'd probably end up rolling our own PRNG, but the real lesson is that cross-platform stuff like this bites. Maybe I'll set up automake this weekend, which should help a bit.
My weekend goal: add in a graph sparsifier along the lines of what I mentioned, and hierarchical A* based on that (we can do different sparsifiers later, like the one in the ualberta paper). And make things more cross-platform-safe.
Post by: Fieari on October 22, 2009, 01:51:03 pm
Been looking up some options, and I think the Mersenne twister is too slow for our needs. Perhaps the MWC method would be better?
Anyway, another thing I've never used but would probably be helpful would be a CVS repository. Shall we set something up at sourceforge?
Post by: numerobis on October 22, 2009, 07:10:29 pm
Post by: numerobis on October 22, 2009, 07:33:44 pm
Oh, hey, look at this, public-domain library, fast, state of the art. Search done: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/
Post by: Fieari on October 22, 2009, 08:45:27 pm
That is, if a square were WALK and MIASMA, and your movement types were WALK only, you'd be able to move through the miasma fine because it allows walkers, and the fact that you don't have miasma movement is ignored.
For testing purposes, I also only put into the map reader the ability to combine the first 4 types so I wouldn't have to muck about with binary files, keeping the maps human readable.
The only annoying part is I'm having to change just about every function to pass the integer possessing the movement types allowed by the actor. Tracking down every spot that this is needed is a bit annoying, but it's preferable by far to using a global, and this is a fundamental enough priority of the project that it probably does need to be pervasive.
Post by: Fieari on October 22, 2009, 09:53:39 pm
Post by: Nexii Malthus on October 26, 2009, 12:29:47 am
The wow power leveling usual point where break evenNNNNNNNNNNOOOOOOOOOOOOOOOOOOOOOOoooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
happens is around 4,000 copies. If you're printing wow gold
4,000 copies of a given item, aion power leveling it's
almost always better aion gold to go to an offset press for
the job, though cheap wow power leveling this will vary
considerably buy wow power leveling by region and services
offered.When looking at the cost effectiveness aion kinah
of colour laser printing,
Post by: Sunken on October 26, 2009, 03:57:35 am
...or, wait, what?
Post by: eerr on October 26, 2009, 09:55:14 am
not a problem with one path, but rather with many simultaneous paths that must all recalculate when they intercede.
Post by: Nadaka on October 26, 2009, 10:37:47 am
This is dead, also, the problems with pathfinding are rather... derivative.probably right. The best way to optimize path finding is caching or parallelization.
not a problem with one path, but rather with many simultaneous paths that must all recalculate when they intercede.
Caching could improve performance for most systems, except those with limited memory.
Parallelization could improve performance for most systems, except those with a single core.
Post by: Sunken on October 26, 2009, 10:43:47 am
Post by: Starver on October 26, 2009, 11:26:55 am
16cable: 4.75->10.75 6 2.7kg
This is dead, also, the problems with pathfinding are rather... derivative.I find the announcement that this is a dead project a little premature for my taste. The fact that different people (and even the same people, at different times, puts hand up... "Guilty!") have been discussing different aspects from those that other people consider core has made it a bit less concentrated than it perhaps could have been, but I, for one, have learnt that some of the terms I 'made up' for my own thought experiments were surprisingly similar to some that actually dedicated problem solvers had layed down in full-on published papers. Which is gratifying to know.
not a problem with one path, but rather with many simultaneous paths that must all recalculate when they intercede.
While I haven't yet got involved in getting down and dirty with actual coding, I still have Plans (TM) to either get the provided platform set up at home or at least make test implementations of my own.
(Ok, so I've been busy over the last few weekends, and the few hours I had to spare this last one that I could have done something practical with I devoted to playing a bit of Oolite. No, I've not not had much practical DF time, recently, just some theory.)
I'd boil all the discussed issues down to:
- Algorithm (A* or derivative or something novel), with or without 'multi-terrain' ability[1]
- Dynamic environment's effects on routes[2], and a little advanced speculation about alternate movement possibilities.
- Zoning/segmentation methods to streamline the routefinding methods (and the given efficiencies/inefficiences involved).
- Caching methodology (especially regarding potential sharing between at least transiently compatible movement-classes).
- An environment/engine for testing and comparing (which I really don't know enough about to comment)
- Eventually (long-term) the possibility of integrating it into a DF build. If not the DF build.
[Post-edit rewording @Sunken: I split the caching and clustering (a.k.a. zoning/segmentation, at least the way I understand what you mean) apart from each other, above, but of course there's a link between all the above and the rest, and that one's a particular strong one. If you don't take the approach of resolving to abstract nodes as an act of separation between 'physical' and logical so-called-reality. Hmm, that's not explained well. YKWIM, I hope.)
[1] Personally, I'd say popular movement classes use their own specific mono-/multi-terrain capabilities to create parts of the cached routing systems for their own class, leaving opportunities for sharing. Any rare or minority movement class gets asked to work their own route out on-the-fly, but could still gain advantage through suitable opening/closing opportunities message-passing to the agent's routine. Or at least potential route-closing changes.
[2] For now, I'm leaving out the concept of routing based upon an individual agent's "knowledge". That would probably mean a completely different approach.
Post by: CobaltKobold on October 26, 2009, 10:25:03 pm
(c/c++ argument) ASM! ::)
Then the path finding could just treat non-passable as inf cost if that critter cannot pass that way. Not perfect, but fairly simple.returns bad routes if no route exists-the current problem people ha'e with "restricted" not really stopping dwarfs if it's the only way.
TASK: FETCH IRON BOOT
LOCATION: MAGMA PIPE
Urist: Hey, I found the best route at infinite cost! Let's go!
Urist has died in the heat x180
Your fortress has crumbled to its end.
Not only are rectangles the most common thing people build in DFBAHAHAHAHAHAHHAAAAA...er, sorry.
Obviously, this makes for trouble if the traffic zones are changed a lot or are badly shaped (dots spread out in the middle of rooms and so on).I'm not the only one who Restricts or Lows saplings when wood is a problem, or Highs valuable sand tiles.
But what about larger agents like wagons, two close trees can form an impenetrable barrier. Their would need to be a TCZ for each agent size which will probably be prohibitive.this is one of the main reasons Toady is off multi-tile critters atm, I believe. (Balance issues, nailing down tile size are others). Wagons are currently in because you need, basically, to check the Depot Access twice per wagon merchantgroup- plus when you hit shift-D, ne'er actively...well, I think. Prolly wrong- I hear dead pathfinding kills(scuttles) a wagon.
You can also, for example, precalculate map for next season because you know that tree saplings will become obstacles.uh, trampling.
Alternatively you could assume everything above a 'top level' zone is a trivial connection to critters passing through these can move freely, which although means a special case is probably a cleaner solution overall.That's a possibility, although it's not trivial to carry out in practice.
[/quote]There is a huge problem with this, namely, underworld layers. Certainly treating fliers as special case with 'Outside Light' as a connected zone (not necessarily trivially, terrain ridges an be problematic) should be worth something.
On to constructiveness.
==Path Changers==
The Good
Building completions, all sorts.
Buildings destroyed.
Mining, most sorts
Door locks
Channeling
The bad/moderate
Lever-action stuff (take note: bridges can cut off paths when down, too- for fliers, or by ramps)
Caveins- precipitated by a Good, can be quite a lot of changes.
NEW: Exploration
The ugly
Water flowing. (swimmers, nonswimmers. non-waterbreathing swimmers.)
Magma flowing. (magmaswimmers, the rest): Flows are also a problem, possibly in part because they affect pathing of dwarfs..tracking them's gonna be a bitch.
Water freezing.
Obsidian forming.
creatures moving (so you can avoid running into them an dhaving to go low/high) - this one is, I hear, a nasty FPS hit if you don't ha'e sufficient traffic space, as it causes them to (seek other path) and FAIL, which is slow.
it seems relatively trivial to maintain parallel route-maps for such as aerial, aquatic, magma-surviving,Except these features are nonexclusive! You can perfectly well have flying amphibious fireimmunes. (We don't, yet, I think). This would be able to use a path that, in theory, existed in none of the individual maps. So, maintaining each would be 2^flags. your 5x becomes 32x, which, considering the problem looked at...is nasty. And I count 8+ (okay, really 7 and a tristate, for 384x)
[snip]
The above possibilities means multiplication 5-fold of the map (or, rather, 5 maps if better optimisation of each map can be made),
Also, by using a full byte, you add support for conditional blockage at a future date(8 bits per tile).Um, we're already past 8 options
Terrain Move Abilities (starver got some, but not all (http://www.bay12games.com/forum/index.php?topic=43265.msg816787#msg816787), Fieari picked up on building destroyer)
surface swim
underswim
fireimmune/magmaswim (does this ha'e an underswim variation, or just one? Might want three variants for Future.)
air-breathe
fly
open-doors(ugly. Can slip by when something else does)
lockpicking
BUILDINGDESTROYER:1
BUILDINGDESTROYER:2
"outdoors"-the order
Buroughs
FUTURE/less definitely useful
Trapavoid?
jump(N)
climb
fall-acceptable
glide
avoid-plants(Keep off the live shrubs/saplings?)- could be more a weighting factor, but I can see it causing walls.
digger
LIGHT (if faultily stored atm)- think vampires
Knowledge (faction X)
miasma
steam
mist
magmamist
dust
Nasty Pathing Operations
Flows?/Pressure?
Building (potentially)- find closest up to 100 of eachstone/wood/metal/blocks! Remember distances!
Post by: Shades on October 27, 2009, 03:56:07 am
Then the path finding could just treat non-passable as inf cost if that critter cannot pass that way. Not perfect, but fairly simple.returns bad routes if no route exists-the current problem people ha'e with "restricted" not really stopping dwarfs if it's the only way.
I said non-passable, not restricted. The point was to avoid having lots of maps by limiting zones to connection types. A critter can only pass through a whole zone or they can't at all. Some zones might be one square doing this (a single door for example). I obviously wasn't clear enough with how I explained it, never been good at that. :)
Post by: CobaltKobold on October 27, 2009, 11:46:43 am
Like the steel boot in the magma pipe.
Post by: Fieari on October 27, 2009, 01:40:24 pm
Terrain Move Abilities (starver got some, but not all (http://www.bay12games.com/forum/index.php?topic=43265.msg816787#msg816787)Not all of these should be movement types. At least not "inclusive" movement types, where a creature can move through the tile if he has -any- of the movement options available. But if we must add the complexity of exclusionary movement types, where the creature MUST have a movement type to move through it, we can cut down on even more.
For instance, fireimmune/magmaswim/swim. These are three good inclusive tags, but with exclusive mapping in addition to inclusive, you could drop it to just swim and fireimmune, with magmaswim being "any with swim, but exclusive to fireimmune". You could also add in airbreath to the exclusive mix, removing the need for seperate under and surphace swim tags.
Also... what would be the point of mist as a movement type? Or magma mist given that you already proposed fireimmune? Why would someone avoid dust, unless you mean cave in dust, and that kind of dust isn't the sort that you path around because it's so momentary.
I still do dispute the need for a seperate BUILDINGDESTROYER:1 or 2. I would really prefer just weighting the cost of going through different types of constructions and materials. Moving it from a "is it possible" question to a "is it really worth the effort" question. If you're the sort of creature that will bust through stuff to get places, the only question is how difficult busting through that stuff is.
...
But all this is assuming we use exclusionary movement types. I'm not entirely convinced that's really a good idea though, simply because it would add a lot more divisions to the connectivity map part of the calculation, but it does offer benefits, it is true. Doors would be easier to add to the pathfinding, for instance. But the question is-- would it add to CPU load for pathfinding? The algorithms I have in my head suggest that yes, it would, which is why in my implementation above I use inclusive movement types only. But if anyone has an idea on how to add exclusionary movement types without increasing CPU load, I'm all ears.
Of course, the best part of exclusionary movement would be that we could add SIZE:xx movement types, for the pathfinding benefit of multitile creatures... I dunno. The pros and cons need to be weighed.
Post by: numerobis on October 27, 2009, 04:05:45 pm
One method, if we believe that most paths only need to use a small number of the terrain or creature types, is to treat the hierarchies as caches: so the hierarchies for kittens and for dwarves would be loaded up pretty much always, but that for dragons pretty much never, and that for goblins only when there are a lot of goblins on the map. If we do it by terrain type, we'd choose to maintain hierarchies over terrain types that get used a lot (floors, retricted zones) and not store hierarchies for the rarely-used ones (magma, open space).
Then pathing is: if there's a hierarchy for the current creature, use it. If not, just do A* like in olden times. I'm not 100% clear on the multiple terrain type version of this game, but the HA* paper was playing exactly that.
Post by: Idles on October 27, 2009, 04:13:57 pm
Terrain Move Abilities (starver got some, but not all (http://www.bay12games.com/forum/index.php?topic=43265.msg816787#msg816787)Not all of these should be movement types.
...
But if anyone has an idea on how to add exclusionary movement types without increasing CPU load, I'm all ears.
Read the hierarchical annotated A* paper linked multiple times throughout the thread. It allows for multiple movement types and varying creature sizes. If we were to start off using that as a basis for this pathfinding project, we would want to keep the list of "movement ability" categories as small as possible in order to avoid using a lot of memory for the hierarchical pathing graphs. In that system, your abstract "high level" pathing graph takes up an amount of memory proportional to the number of movement categories you want to have. This "high level" abstract graph has far fewer nodes and edges than the low-level tile-based graph--thus, pathfinding in the "high level" graph saves you CPU cycles because the search space is far smaller.
Thus, to fit current DF movement types into categories, you'd have:
walker ("floor", "ramp", "stairs", "unlocked door" tiles with no water or magma)
flyer (walker tiles + "open space" tiles -- this maps to giant eagles)
swimmer (any tile with water level > 4 -- this maps to carp)
amphibious ( walker + swimmer -- this maps to snakemen/lungfish)
dwarf( walker + tiles with water level <= 4 + "pet-impassible doors")
magmaphibian(walker tiles (+ any magma) + "open space" tiles with magma > 4 -- this is imps, etc.)
destroyer(walker + "locked doors" + "buildings" -- this maps to trolls, dragons, etc.)
digger( walker + "wall" tiles)
Right there you've basically got 8 movement types, which fits in a byte. A creature has just a single movement type, which in turn describes all the possible tile states that the creature can move through.
I can't think of anything that doesn't fit into these categories, besides I think spirits of fire, which if I remember correctly can fly as well as swim through water and magma.
Basically, keep these movement types in mind while you read the Hierarchical Annotated A* paper and you'll have a good starting point for this pathfinder.
Post by: Fieari on October 27, 2009, 04:31:42 pm
I read the powerpoint version of the HAA* paper above. I didn't see anything about multitile creatures though...
Post by: Shades on October 27, 2009, 05:16:35 pm
Thus, to fit current DF movement types into categories, you'd have:
walker ("floor", "ramp", "stairs", "unlocked door" tiles with no water or magma)
flyer (walker tiles + "open space" tiles -- this maps to giant eagles)
swimmer (any tile with water level > 4 -- this maps to carp)
amphibious ( walker + swimmer -- this maps to snakemen/lungfish)
dwarf( walker + tiles with water level <= 4 + "pet-impassible doors")
magmaphibian(walker tiles (+ any magma) + "open space" tiles with magma > 4 -- this is imps, etc.)
destroyer(walker + "locked doors" + "buildings" -- this maps to trolls, dragons, etc.)
digger( walker + "wall" tiles)
This list is not 8, you even list how some of the flags combine to make others (amphibious would just have both walker and swimmer set for example) seems excessive to record them as well. Also I wouldn't worry so much about how many we have as long as a sensible structure is used to store the information.
I'd use:
walker (as above but should water < 4 be included here too?)
flyer (open space)
swimmer
magmaswimmer
pet passable / door opener
destroyer ("buildings" what is the difference between destroyer 1 and 2)
Not sure if we need digger? Adding a new movement type is cheap so we can when we need. Note that my swimmer groups include climbing out of water/magma if walker is enabled so adjacent zones would only need to mark which they are.
As we only have one critter larger than a tile I'd suggest just doing the current pathfinding for it as needed, if we come up with a clever system great but otherwise just ignore it for now.
Post by: Idles on October 27, 2009, 06:31:02 pm
This list is not 8, you even list how some of the flags combine to make others (amphibious would just have both walker and swimmer set for example) seems excessive to record them as well.
....
As we only have one critter larger than a tile I'd suggest just doing the current pathfinding for it as needed, if we come up with a clever system great but otherwise just ignore it for now.
About your first point: if you read the HAA* paper, the important thing to take away from it is that each creature is able to move over only a part of the map. A "movement type", as I defined it, is a set of rules from which you can figure out whether or not a creature can move through a tile (given that you have all the information that DF currently stores about tiles). By using these "movement types", you construct the abstract "high level" path graph, where the annotations of edges include information defining which one (and only one) of the "movement types" is valid for that edge. That information could take the form of a byte which stores flags (1 flag for each of the 8 movement types I defined).
For the low-level tile-map (which already exists inside DF), you obviously wouldn't store that flag annotation byte, since you would be duplicating information already present in the tile map. Instead, you'd just rely on the information that DF already provides, and have a function that takes a tile and spits out a byte (or word or whatever) with the flags for which "movement types" are valid for that tile.
About your second point: The HAA* paper already has a clearly defined method for handling multiple tile creatures. Perhaps you should read it? That said, we could always special-case them, by just letting them use regular old A* without any enhancements. However, since the point of this project is to improve the current pathfinder, we might as well use the HAA* method for multiple-tile creatures.
About dragons, and other creatures that don't clearly fall into a single "movement type": we can just let them use regular A*, because having to store an abstract path graph for each possible permutation of tiles that a creature can move through would quickly eat up all of our memory. My list of movement types was an attempt to shoehorn all the common creatures into as few categories as possible.
Here's a link to the HAA* website with accompanying paper that I'm referring to: here. (http://harablog.wordpress.com/2009/02/05/hierarchical-clearance-based-pathfinding/)
Post by: CobaltKobold on October 27, 2009, 08:29:54 pm
I still do dispute the need for a seperate BUILDINGDESTROYER:1 or 2. I would really prefer just weighting the cost of going through different types of constructions and materials. Moving it from a "is it possible" question to a "is it really worth the effort" question. If you're the sort of creature that will bust through stuff to get places, the only question is how difficult busting through that stuff is.I'd agree if it were actually attacking and stuff, but DESTROYER1 can only destroy doors and a few other similarly flimsy things, while 2 can get through any nonconstruction.
Why do people suggest weighting paths that cannot be traversed?
Post by: Idles on October 27, 2009, 08:50:00 pm
Why do people suggest weighting paths that cannot be traversed?
Probably lack of understanding about how path finding algorithms work.
Post by: Shades on October 28, 2009, 04:22:46 am
Here's a link to the HAA* website with accompanying paper that I'm referring to: here. (http://harablog.wordpress.com/2009/02/05/hierarchical-clearance-based-pathfinding/)
The last HHA* (well technically HPA*) differed from that in that routes across zones where stored by type and not grouped so a critter would pick whichever route worked for it, combining them would be faster but at the cost of memory. Probably a cheap enough trade off mind you.
I still think the zoned A* will probably be more efficient though (although again at the cost of memory as you don't have regular cluster sizes)
Edit: of course trying different methods is the point of this :)
Post by: qwertyuiopas on October 28, 2009, 07:18:03 am
Post by: Sunken on October 28, 2009, 07:29:55 am
Post by: Shades on October 28, 2009, 07:42:50 am
If the system only supports weighting, make the last bit set if it CAN be pathed at all. No silly mucking around in infinites, especially infinite improbability(Though with that, dwarves wouldn't have problems...)
In most pathing systems I've used/written hitting an infinite weighted value for a link for a given movement type meant it couldn't be taken and so wasn't added to the list of solutions. Don't worry overly about terminology I think, basically blocked is blocked however the under laying algorithm handles it.
8 movement types will never cut it, even if we don't try to solve problems that are not currently solved (and we want to, don't we?). Soldiers, pets, invaders and ordinary dwarfs are all different walker subtypes, and we'll want more if certain doors are going to be accessible only to certain people, for example. It's a challenge we have to deal with in any solution we come up with.
I don't think the number is relevant as long as we know what they are at some point. However it depends what you meant by movement types, the HAA* suggested wanted combinations of movement styles which means we'd need more types and more memory used but saves us calculations in real time and so increases performance. Other methods might only need a flag of what movement styles are possible in which case I have a hard time thinking of even eight.
Post by: shadow_slicer on October 28, 2009, 11:40:29 am
For example, the client would enable a movement class (so we would create/modify our hierachy to support it) by calling some function:
int movementclass = PathLibrary::CreateMovementClass( pathcostfuncptr)
where pathcostfuncptr is a function pointer pointing to a function of the form:
int DwarfPathCost(point p1, point p2),
with point p1 and p2 being adjacent points (i.e. p2 is one of the 27 points surrounding and including p1). This DwarfPathCost will return MAX_INT if it is not passible, otherwise the path cost. Note that for the case of p1=p2, this determines whether or not a movementclass is allowed to be in this square, so it should be either 0 or MAX_INT.
Then the client could get routes using PathLibrary::GetPath(p1, p2, movementclass) to use the hierarchical model, or PathLibrary::GetPath(p1, p2, pathcostfuncptr) to use plain A*.
Post by: Shades on October 28, 2009, 11:43:34 am
Might be able to do the HAA* clusters like that though.
Post by: Sunken on October 28, 2009, 01:55:50 pm
It's a real problem but we can perhaps tackle it by problem relaxation and heuristics - like maintaining only some connectivity maps and supplementing them on demand, somehow. Or merging them on demand.
Post by: CobaltKobold on October 28, 2009, 04:20:30 pm
From what I read around here, the current solution is to just have one real connectivity map (for walking)- and those with odd abilities path to the nearest affected passable- to destroy or pass through it- and then path onward after they do so. Fliers and swimmers seem to be a bit wonky, not sure how they're handled.
I can think of easy ways to out-engineer this solution- making a support with a brick wall on it for buildingdestroyers, making a locked doorway to magma for lockpickers-so it's clearly granting suboptimal results.
So, what ways of MOSTLY ignoring these edges will still produce good (This may be a bad idea.)
Post by: Footkerchief on October 28, 2009, 06:23:22 pm
From what I read around here, the current solution is to just have one real connectivity map (for walking)- and those with odd abilities path to the nearest affected passable- to destroy or pass through it- and then path onward after they do so. Fliers and swimmers seem to be a bit wonky, not sure how they're handled.
It just runs A* between the goal and destination. Ramps, stairs, water and flight are just ways to move from tile to tile. It won't try an A* path if the connected components don't match up, which is where some problems arise (as with flight-modded dwarves in fort mode and so on). Regular flying creatures use something more like short-range flood fills to combat this, if I remember, but they can't use arbitrary A* if they don't have a component check or else the processor will die on pathing failures.
I think swimmers do the same thing, since they only aggro for nearby creatures.
Post by: qwertyuiopas on October 28, 2009, 07:59:22 pm
Why bother with movement types at all? Why can't we just use a callback function to determine the cost/if pathable of some opaque movement class? There's really no reason to store all the path costs (27 of them per each tile!) since they're easy to calculate. This way we don't worry about bitmasks and composite movement classes or things like that, we only ask the client "what is the cost from here to here".
For example, the client would enable a movement class (so we would create/modify our hierachy to support it) by calling some function:
int movementclass = PathLibrary::CreateMovementClass( pathcostfuncptr)
where pathcostfuncptr is a function pointer pointing to a function of the form:
int DwarfPathCost(point p1, point p2),
with point p1 and p2 being adjacent points (i.e. p2 is one of the 27 points surrounding and including p1). This DwarfPathCost will return MAX_INT if it is not passible, otherwise the path cost. Note that for the case of p1=p2, this determines whether or not a movementclass is allowed to be in this square, so it should be either 0 or MAX_INT.
Then the client could get routes using PathLibrary::GetPath(p1, p2, movementclass) to use the hierarchical model, or PathLibrary::GetPath(p1, p2, pathcostfuncptr) to use plain A*.
That is written as if this was going to be an internal library, I think, but most of the other discussion assumes an external library(DLL, I think?)
That, or I have no clue how inter-program function calls work in C++.
Post by: shadow_slicer on October 28, 2009, 10:36:04 pm
I'm not saying we're calculating everything on demand. What I mean is that the client program defines a movement class with a function instead of a capabilities bit-vector. Everything else is pretty much the same. This means that when the client creates a movement class, we make the connectivity map for this class (or adapt our connectivity map to support this class). The client will still need to notify us of tile changes so we can update our connectivity map. The primary advantage is not having to store as much per-tile information (which would be duplicated from the client).
As Shades mentioned, the biggest drawback of this method is that it makes things more complicated for algorithms that combine movement types. Though, I think with as few movement types as we are dealing with, it might not make sense to focus on optimizing that.
@qwertyuiopas:
What I described should work as an external library. I've worked with libraries that do the same thing (standard C library, C++ STL, Boost, glib, gtk), so I don't expect that to be a problem.
Post by: Shades on October 29, 2009, 03:44:39 am
I'm not saying we're calculating everything on demand. What I mean is that the client program defines a movement class with a function instead of a capabilities bit-vector.
Can you expand upon that? The programic side of passing such things is simple but for actual path-finding I don't see how that works if your not calculating on demand, if you can expand upon that so I can understand what you mean it would be most helpful :)
Post by: Sunken on October 29, 2009, 09:34:44 am
If this is what he means I sort of agree, but also point out that connectivity is not local in nature and it might be hard to maintain efficiency if that, too, is made modular.
Post by: Shades on October 29, 2009, 10:02:59 am
Post by: shadow_slicer on October 29, 2009, 10:38:42 am
@Sunken: Yes, you seem to understand my idea. I'm not sure what you mean though about "connectivity is not local". I'm only proposing to use the callback to obtain information on local connectivity. Obviously the library will need to maintain some state for connectivity between zones or whatever, but this is going to be built on the local connectivity information regardless of how this local information is obtained.
Post by: Shades on October 29, 2009, 10:51:20 am
@Shades: It was passed in the GetPath function only to get uncached paths. The cached paths use a movementclass which was set up using PathLibrary::CreateMovementClass. CreateMovementClass will set everything up.
So assuming you needed a local cache of routes of any potential request coming later how do you interrogate the movement class to see what is possible. Or would it return some array of all possible distance calculations between whichever squares the local zone needs to check?
Post by: numerobis on October 29, 2009, 11:50:31 am
Why bother with movement types at all? Why can't we just use a callback function to determine the cost/if pathable of some opaque movement class?Different movement types imply different hierarchies for running the pathfinding over. Until we know how to compute the cost, we can't build anything and we're stuck with the basic A*.
Post by: shadow_slicer on October 29, 2009, 03:10:38 pm
If the library needs to generate a cache of any potential path requests that may come later, then it would use some pathing algorithm (could be HAA*, A*, HHA*, etc) and the cost data from the callback to generate the paths.
If the client needs to generate a cache of potential paths, then it would call getPath for each of the paths it needs.
@numeroids: What I meant by not having movement types was not having a bitvector that determines whether a tile is passable or not by creatures with certain movement capabilities as was previously discussed. Instead, both the movement type and creature capability are wrapped into the result of a single function call, which also returns the cost. Since we know the cost, we can build whatever zones or hierarchies we want. I think that this is better than the alternative of storing the capabilities and costs for moving any of the roughly 27 possible directions from each of the tiles on the map, we only need to store a function pointer and simply ask the client every time we need it (since presumably the client can calculate it quickly).
Post by: numerobis on October 29, 2009, 04:39:14 pm
Post by: shadow_slicer on October 29, 2009, 07:23:07 pm
I actually do think we should explicitly represent all 26 of the edges between grid squares. If you don't then it becomes more difficult to figure out which directions are allowed. Suppose you have a path under a cliff. Both the ground under the cliff and the top of the cliff are passable for walkers, but walkers shouldn't be allowed to "teleport" up to the top of the cliff. You could try restricting this movement to special cases, but that simply makes it more difficult to model jumping off ledges, climbing ramps, and numerous other movements. Jumping off ledges is a particularly strong example: you need to allow the path to go diagonally down, but going diagonally down probably needs to have a different cost and require different capabilities than going to the next tile over horizontally. You can't add this information to the tile at the bottom because it only should effect paths that jump down from the ledge. You could handle it as a special case, but that seems messy. In summary, if you do not explicitly represent all the edges, it is extremely difficult to apply different costs and requirements to different directions, and simply putting the cost and requirements inside the tile is not flexible enough to handle many cases that we need to support without ugly hacks.
As you said though, in the abstract graph movement types would need to be represented on the edges unless we have separate abstract graphs for each movement type.
Post by: Mike Mayday on October 30, 2009, 06:09:11 am
Post by: Mel_Vixen on October 31, 2009, 04:52:49 am
[edit]By the way i think a platform independend implementation would be needed.
The "world-tiles" should be modify-able in a way that new "flags" can be added flexible. The flags would be handled like in a sql DB with "type" and "set" attributes plus the id of the tile as primary key. This could be very handy for new travel methods and new obstacle variants.
[/edit]
Quote
Not only are rectangles the most common thing people build in DF their also efficient at covering the unmodified terrain as well and are much simpler to set up and can aid in doing multi-tile creatures as we can be assured that a creature thats longest dimention is shorter then the rects shortest can move through the rectangle.
Rectangles also would support a simple idea i had a bit Better. A little discussion about shading gave me it. Idealy The shortest connection between two (not moving) points is a straight line. Since we know the accesnodes between two mapchunks/Leaves we could use Bresenhams algorithm to calculate The shortest way in a leave betwee two nodes. If a soft obstacle, like a dorf is on the path we can use at this point a short A* to path around the obstacle to continue the precalculated path or we just dodge (1 step forward in a certain angle) and use a bresenham again.
3D paths ("cubes" instead rectangles) like for swimmers/flyers can also make great us of that because the bresenham would return a (in
Quote
qoute by DeathofRats
That's part of the problem, and why I'd say it's important for whatever algorithm is chosen to have two characteristics: that segmentation works locally (i.e., I just mined a tile, I don't have to recalculate all the pathing for the whole damn world), and that different areas can be re-zoned (divided or merged). If those two criteria are met, it shouldn't be a problem to deal with terrain modifications.
In my opinion we should implement apart from heuristics multiple takes on (in mapchunk) pathing because depending on the pathing task and outer influences (like the sidelenghts/radius of a mapchunk) a different pathing variant could be more efficient then the usual one. For example bresenham might be enought to traverse 3 mapchunks and in the 4rd a A* might be needed.
Quote
AI settlements will need to be generated with their own set of room definitions, traffic designations, etc.. That will break save compatibility again, so it's not trivial - but something like that has to happen if the townsfolk ever are not to act like they all suffer from terrible amnesia, don't really know what they're doing there, and just amble around wondering whether they are just a randomly generated npc in a big simulation.
Actually the save-comp doesnt need to be destroyed. IIrc for villages the "room-devs" already exist and the "paths" could be saved in a own file which actually would also benefit the ADV mode because then the paths for the Ai-Agents "ofscreen" can be still calculated without loading the entire junk of data like objects (not needed items, animals, etc.).
Since the rooms already exists the routes could be caculated on the first load of an old save-file.
Post by: shadow_slicer on October 31, 2009, 09:49:34 am
Quote
quote by Heph:
Rectangles also would support a simple idea i had a bit Better. A little discussion about shading gave me it. Idealy The shortest connection between two (not moving) points is a straight line. Since we know the accesnodes between two mapchunks/Leaves we could use Bresenhams algorithm to calculate The shortest way in a leave betwee two nodes. If a soft obstacle, like a dorf is on the path we can use at this point a short A* to path around the obstacle to continue the precalculated path or we just dodge (1 step forward in a certain angle) and use a bresenham again.
Actually I think A* with a good heuristic should be equivalent (in both result and computational cost) to Bresenhams algorithm for the case where that algorithm is applicable. As long as the heuristic is such that a straight-line is always seen as better than the curve, A* will not explore any of the "off line" paths. Since diagonal travel is allowed, a good "remaining cost" heuristic is "max(dx,dy,dz)". Additionally, if you use a straight-line technique until you hit an obstacle, there is no way to ensure optimality of the resulting path. If you look into the literature on "Greedy Geographic Routing", which is a somewhat similar idea, you'll find the worst-case (and in result the average-case) performance is very poor. Also, I'm not sure there's a good way to change back from A* after getting around the obstacle. There are two problems: 1) how will you know when you've gotten around an obstacle 2) which candidate next step from A* should you start with? Really just using A* is significantly simpler, and shouldn't be that much more costly.
As for using bezier curves or other algorithms to represent behavior, I don't think the pathing library should be responsible for handling AI.
Quote
quote by Heph:
In my opinion we should implement apart from heuristics multiple takes on (in mapchunk) pathing because depending on the pathing task and outer influences (like the sidelenghts/radius of a mapchunk) a different pathing variant could be more efficient then the usual one. For example bresenham might be enought to traverse 3 mapchunks and in the 4rd a A* might be needed.
While this is definitely possible to use different pathing algorithms in different zones, but I'm still not convinced we need something other than A*.
Post by: Sunken on October 31, 2009, 10:35:44 am
The Bresenham algorithm also wouldn't work with TCZs, as a side note ; )
Post by: qwertyuiopas on October 31, 2009, 08:01:25 pm
The cliff case is meaningless: You could have a tile group for the upper path, and one for the lower. Where the edges of the groups cross, you could describe the movement costs between them. The advantage is that you don't have to store the data of every identical sky tile in an empty 10x10x10 cube, only where it connects to other groupings. When breaking up the grouping, you simply transfer the edge data, either to the tile itself(unlikely) or to the new edge.
Also, a suggestion: maybe put a slight extra weight against direction changes? Not tile-based, but a flier going up over a mountain wouldn't follow the ground exactly... (Of course, such behaviour might develop naturally...)
Post by: CobaltKobold on October 31, 2009, 10:54:22 pm
I don't think the pathing library should be responsible for handling AI.At some le'el, it becomes the AI. "Do I take the path nearer the goblin archers or the one past the spiked pits?"
Post by: Urist McDepravity on November 01, 2009, 05:16:15 am
"Do I take the path nearer the goblin archers or the one past the spiked pits?"Thats a good point, actually. Although it should not be handled by pathing lib, but rather it should provide ALL possible paths, one for each way of traversing the graph.
Post by: Shades on November 01, 2009, 05:27:13 am
"Do I take the path nearer the goblin archers or the one past the spiked pits?"Thats a good point, actually. Although it should not be handled by pathing lib, but rather it should provide ALL possible paths, one for each way of traversing the graph.
No it should return the single shortest* path based on whatever criteria as passed to it. If this includes a don't go within X of goblins that is fine.
*probably shortest or close enough to make no difference.
Post by: Silverionmox on November 01, 2009, 05:34:26 am
Earlier all-edge comment:Arguably, the flier will automatically fly at least one level above ground as soon as rock, trees etc. are proper obstacles. Also, air currents will make some directions easier than others so they might be inclined to soar up along with warm air and then glide down towards their destination and descend. Personal preferences also could play a role. Prey birds would habitually stay sufficiently far above ground as a matter of habit, to avoid predators.
The cliff case is meaningless: You could have a tile group for the upper path, and one for the lower. Where the edges of the groups cross, you could describe the movement costs between them. The advantage is that you don't have to store the data of every identical sky tile in an empty 10x10x10 cube, only where it connects to other groupings. When breaking up the grouping, you simply transfer the edge data, either to the tile itself(unlikely) or to the new edge.
Also, a suggestion: maybe put a slight extra weight against direction changes? Not tile-based, but a flier going up over a mountain wouldn't follow the ground exactly... (Of course, such behaviour might develop naturally...)
How well does A* handle different costs depending on (corr.) direction? This will be useful for streams, air currents, differentiating uphill/downhill, etc.
Post by: Mel_Vixen on November 01, 2009, 05:48:37 am
Quote
The Bresenham algorithm is fundamentally aesthetic in nature and doesn't help you find the shortest way between points in any meaningful way. Sure, it might look better if creatures moved in what looks like straight lines towards their goal in open areas, but it has no effect on the cost of the path, and it is more expensive in implementation than just "if x < targetx && y < targety then up-right" (since Bresenham would require storing extra values and performing computations slightly more complex than the above, each step).
I know that it was created for shading but actually it can help you to find the shortest path between two points since in nature the shortest path between 2 nonmoving points is a straight line. Our points would be the access and exit points of a TCZ - given that said TCZ is rectangular.
You might now answer that In tile based games the a straight line isnt the shortest path,thanks to the "New york" Problem, but toady iirc circumvents the "New york" Problem by having diagonally movement being less expensiv then pure vertical and horizontal movement.
Secondly the the dwarfes iirc already remember their paths so they dont have to calculate them in every turn/frame. The Bresenham can return the tiles of the path while your way ignores that posibility.
And now comparing your method with my method.
your variant: (a made it a bit more pseucode like)
Code: [Select]
x, xtarget, y ytarget // get the coords of dorf and dorfs target
xst, yst // x and y steps
if x = targetx
then
xst = x;
else
if x < targetx
then
xst = x+1;
else
xst = x-1;
if y = targety
then
yst = y;
else
if y < targety
then
yst = y+1;
else
yst = y-1;
store (xst, yst)
the problem with your code is that it works fine in a "New york" world but its rather hard to incooporate a non "New york" world where a straight line is the better movement. It works also nice for moving targets actually.
In the worst case you compare 4 times 2 variables and you have 2 additions per iteration.
my variant:
Code: [Select]
get xa, ya // get start coordinate
get xe, ye // get end coordinate
d = 2×Δy – Δx // Δy = ye - ya ; Δx = xe - ye
ΔO = 2×Δy
ΔNO = 2×(Δy − Δx)
y = ya
store tile (xa, ya)
for every x of xa+1 till xe
if d ≤ 0
then
d = d + ΔO
else
d = d + ΔNO
y = y + 1
store tile(x, y)
The worst case here are 10 calculations for initialisation and , per iteration, 2 times comparing variables and 2 calculations. The bresenham would also work fine with DFs non "New york" world. The contra is that it doesnt work for moving targets.
if we say that calculations and comparings are each 1 clockcicle your code is better on the first 4 steps the dwarf does while in the 5th they reach an equilibrium and in the 6th the abused Bresenham gets better.
But actually i am not very far in this stuff yet since i do normaly other stuff.
Post by: Sunken on November 01, 2009, 06:22:41 am
Edit: Scratch that, there's a sqrt(2) factor in there
Edit 2: Heh, scratch that too, I misread your claim.
Still, Bresenham will require the storage of the d parameter, and the final path will not actually cost any less than the silly-move one; it will just distribute the diagonal movements differently. Plus, it disallows TCZs (as presented by me, as opposed to rectangles or other strictly convex regions) and, as you say, isn't optimal with moving targets.
Post by: Mel_Vixen on November 01, 2009, 06:49:52 am
Code: [Select]
x, xtarget, y ytarget // get the coords of dorf and dorfs target
xst, yst // x and y steps
As long x!=xtarget && y!=ytarget
do
if x = targetx
then
xst = x;
else
if x < targetx
then
xst = x+1;
else
xst = x-1;
if y = targety
then
yst = y;
else
if y < targety
then
yst = y+1;
else
yst = y-1;
store (xst, yst)
Which would haven even more (2) compartions.
But as i said i am not that good at this stuff. normaly i do Assembler where i can count my clockcicles.
Post by: Sunken on November 01, 2009, 06:53:47 am
Still, Bresenham will require the storage of the d parameter, and the final path will not actually cost any less than the silly-move one; it will just distribute the diagonal movements differently. Plus, it disallows TCZs (as presented by me, as opposed to rectangles or other strictly convex regions) and, as you say, isn't optimal with moving targets.
Post by: Sunken on November 01, 2009, 06:55:34 am
Post by: Mel_Vixen on November 01, 2009, 07:07:26 am
Post by: Sunken on November 01, 2009, 07:21:02 am
The problem is the search. What is needed is a reduction in branching factor and search tree depth (and a good heuristic). And it's my belief that TCZs will have a better impact on these key factors than rectangles will.
But it's true that in principle it would be possible to mix zones of different types. It would require more programming work, and a disciplined, modular structure, but it's not infeasible.
Post by: Mel_Vixen on November 01, 2009, 07:28:27 am
Right now toady uses a algo that supports 200 on 2GHZ. (he uses a normal A* without heuristics etc. iirc) if we find ways to make the pathfinding 10% faster we can support with 2GHZ 220 dwarfs. The tradeofs just need to be acceptable as in that they schouldnt be to memory expensiv and schould give atleast semiperfect paths.
Heuristics, trees and branches are one point in the equeation. but what and how things happen inside zones, tcz, and what not is i think important too.
Post by: shadow_slicer on November 01, 2009, 03:04:14 pm
In analyzing the algorithm you are starting from the basic assumption that you want to go from point A to point B in some convex shape which contains no obstacles. You don't describe how the points A and B get determined (which is important), how the convex shape is defined and fits together with the rest of the map, or how it is known that the shape contains no obstacles. Additionally, you are assuming that the only shortest path is a "straight" line. In fact there are a large number of similar paths which are not straight lines (which consist of some combination of diagonal and non-diagonal movements) which all share the same cost.
As Sunken so aptly put it:
Planning movement within a rectangle is not a problem, no matter which algorithm you select. Nor is it within TCZs, using silly-move. The computational costs for these are all absolutely negligible...
The problem is the search. What is needed is a reduction in branching factor and search tree depth (and a good heuristic).
The thing is the that good zone design and branching is probably going to provide 95% of the improvement we'll see. Whatever happens in zones (if they are suitably designed), is not going to help even 1% (probably less than 0.1%). Additionally, as Sunken has mentioned, silly-move is much simpler than Bresenham's algorithm (and has fewer problems in zones that are not obstacle-free).
As far as the zones and branching goes, however, I'm not so much of a fan of the TCZ's that Sunken is proposing. I don't see the advantages of silly-move over A* for the scenarios silly-move is applicable for outweighing the flexibility of A*. Additionally I think we can do better if we allow for zones that do not satisfy TCZ (such as a windy passage that doubles back), but which are trivially pathable with A*. I also think that zones should be 3D since this allows for similar windy passages in the z-direction (and the sky could be one gigantic zone).
Post by: Richards on November 01, 2009, 04:30:43 pm
Post by: CobaltKobold on November 01, 2009, 04:34:02 pm
Has anyone corresponded with Tarn about supporting a team to create an official path finder utility for DF?I'd wait until something actually appeared.
Post by: zchris13 on November 01, 2009, 08:59:00 pm
Post by: Sunken on November 02, 2009, 04:04:43 am
As far as the zones and branching goes, however, I'm not so much of a fan of the TCZ's that Sunken is proposing. I don't see the advantages of silly-move over A* for the scenarios silly-move is applicable for outweighing the flexibility of A*. Additionally I think we can do better if we allow for zones that do not satisfy TCZ (such as a windy passage that doubles back), but which are trivially pathable with A*. I also think that zones should be 3D since this allows for similar windy passages in the z-direction (and the sky could be one gigantic zone).
There's a trade-off: The more complex the intra-zone navigation is allowed to be, the larger the zones can be -
straight-line -> convex zones
silly-move -> TCZs
A* -> more convoluted zones (entire maps, in fact)
We simply have to experiment to see which trade-off is most beneficial.
I'll say this, though: There has been no automatic cut-off criterion proposed for A* zones. As I just said, they can be as big as you want - but unlike the other two, the intra-zone pathing becomes more and more expensive for A* as the zones grow. Silly-move and straight-line are constant-time no matter how big the zone.
So, saying that we should use A* inside zones just begs the question - what zones to use? I thought up TCZs because they seemed a reasonable compromise.
Post by: Draco18s on November 02, 2009, 06:57:29 am
Post by: Fieari on November 02, 2009, 07:25:32 am
Post by: Sunken on November 02, 2009, 07:27:53 am
I'm just popping in now to point out that in a tile based game, convex zones are rectangles. Any other shape ceases to be convex. Diagonals in a square area are an odd case, as the square the line passes through may or may not be traversable (eg. diagonal movement around a corner). Currently such diagonals are allowed at normal cost (1.4), but this may change.Here's a convex zone to disprove your claim:
Code: [Select]
##
####
####Edit: I suppose you mean that a tiled world superimposed on R² only allows (axis-aligned) rectangles. But that's a useless definition of convexity in a tiled world. I, and I think most others, have been assuming convex to mean "the shortest path from two tiles in the set lies in the set", where "path" is something a dwarf might follow.
Post by: Starver on November 02, 2009, 09:22:38 am
As for using bezier curves or other algorithms[...]
[Actually, the response of mine has been retro-ninjaed later, especially regarding the Bresenham function. Which I admit I never knew the name of.]
Post by: Sunken on November 02, 2009, 09:36:19 am
Now, if costs were changed to make the staggered diagonal actually cheaper than the alternatives, then that would be a different story.
As for curves - a curve is never the shortest way (in Euclidean space) and so again I think aesthetics must give way to efficiency.
Post by: Starver on November 02, 2009, 09:40:07 am
As for curves - a curve is never the shortest way (in Euclidean space) and so again I think aesthetics must give way to efficiency.
Although because of the non-Euclidean nature, a 'nice' curve can be no less efficient than a drastic non-curve. (As can be a total zig-zag, of the kind we might actually use in Adventure mode while trying to sweep more of the field...)
Post by: Starver on November 02, 2009, 09:47:29 am
Is there? Oooh, I assumed it was equal in all eight basic directions.1: You're working from an incorrect assumption. Diagonal movement and orthogonal movement cost the same in DF.
Edit: Scratch that, there's a sqrt(2) factor in there
Might put the scuppers on any assumptions I've made, if this is true. Of course, if anyone was going to make this matter, Toady would.
(Not that it would matter if a route was 10E,5NE or 5NE,10E or 5x(2E,NE), but it would mean that 20E would be quicker than 10x(NE,SE) or 5x(2NE,2SE) or other versions. Now, how does one test. Actually, never mind, for now I'm not too bothered. Just speculating.)
Post by: Starver on November 02, 2009, 10:07:03 am
I'm just popping in now to point out that in a tile based game, convex zones are rectangles. Any other shape ceases to be convex. Diagonals in a square area are an odd case, as the square the line passes through may or may not be traversable (eg. diagonal movement around a corner). Currently such diagonals are allowed at normal cost (1.4), but this may change.If you enforce wossisnames[1] algorithm, then a diagonal in a zone might preclude a 'convexity', as the 'ideal stagger' hits the wall, but otherwise shouldn't matter.
For example:
Code: [Select]
########
######b a#
########c #
#d #
# #
a->d 17W,2S which would be a sum of 15W,2SW, or "ideally" 5W,SW,5W,SW,5W, which would hit the wall next to 'c'. (This was quickly rendered in ASCII, I may not have made the wall stagger B-algorithmically correct! It might more correctly miss the wall.) But it could be made to sidestep SW earlier and delay one of the W-steps. To equal weight (with or without the root-2 cost l;evied against a diagonal).
b->d and c->d would be required to start with the SW movement (and the route from b may need to premeturely enact a second SW, compared to the B-algorithm's idealisation), but to the same equality. That shape would still be topologically convex in all respects. Treat the 'inconvenient' corners on the diagonal wall much the same as inconvenient single-square 'step-around' blocking elements in the middle a zone.
Ironically, if the 1.4 weighting exists, then a straight orthagonal path that has to make a diagonal-and-back 'jink' around an obstruction on the path will cause more problems to (strict) convexity than otherwise. Although minimal in the grand scheme of things.
[1] Begins with a 'B', forgotten it already. Sorry.
Post by: Shades on November 02, 2009, 10:21:19 am
As there is only 8 squares in the plane of movement the combinations for order of testing are few and can probably be handled cheaply with a lookup against the links (rather than processing links in order).
I have no idea if it would get the effect you desired, or how much extra processing it would really be but it would be a very minor change to any of the A* implementations (or any graph based search) and so easy to test.
Post by: Starver on November 02, 2009, 10:47:49 am
For example:Code: [Select]########
######b a#
########c #
#d #
# #
a->d 17W,2S which would be a sum of 15W,2SW, or "ideally" 5W,SW,5W,SW,5W, which would hit the wall next to 'c'. (This was quickly rendered in ASCII, I may not have made the wall stagger B-algorithmically correct! It might more correctly miss the wall.)[...]
Actually, re-reading my own item, the whole point is that corner-to-corner of the region should have been a 'regular' stagger, the wall that bounds it is beyond the limit. However consider the zone bounded by...
Code: [Select]
###
#####################a#
#b c #
# #
# d #
#######################...that would be as much convex as the rectangle minus the alcove, given that no shorter route exists between all points within than the shortest route that must avoid the boundary. The route a->b would have to be SW, then a number of Ws, but would not be longer than the stagger that travelled 'through' the wall then emerged at or around 'c'.Of course, if 'c' was an obstacle, then the 1.4 weighting would cause minor problems (without, one might even stretch the definition to still be convex), and ditto if there was a further alcove in the wall at 'c'. But (at least in the latter case) it would be a trivial matter of making the 'a' or 'c'-adjacent alcove a separate 1x1 zone attached to the main zone. For a "'c' is an obstacle block", then if your algorithm bothered about the weighting disadvantage you might be forced to split the zone. Either straight down (shortest distance) or across to hypothetical obstacle 'd' should that also spring into existence.
But that's all... well, a different solution/direction-of-solution to what is being worked on with the A* algorithm stuff, as far as I can tell, so this is just waffle on my part.
Post by: Sunken on November 02, 2009, 11:03:17 am
Post by: Starver on November 02, 2009, 11:40:30 am
Sounds like you're on your way to inventing the TCZ ;-)Well, to be honest I had invented something prior to hearing of the TCZ that was very TCZ-ish, but then came in after the weekend/night-off to see almost all of my considerations[1] mentioned. (Straight off, I must have skimmed over what TCZ was, and thought the "C" was for "Convex", but rewound after a while when I decided I really needed to know the title.)
But even back then I had assumed it was to be a Bresenham line on all sides (albeit that there were several possible B-lines, or rather that various possible B-line segments could be adopted as the zone border, where it wasn't a fully constrained border, just a path between corners as defined by other constraints. And I'd also never considered the possibility of single blocking cells in the centre not (necessarily) preventing a zone's trivial connections. To my eye, it would have forced the bissection of a potential zone.
[1] Tell you what, here's something of what I'd composed. This was not a finalised version (appears to have a few BBCode-inaccuracies, being composed off-line), but contained the basis of the "from the ground up" conglomeration of tiles into zones, which would have been maintained but in turn grouped as sub-zones, and included the stipulation that zone borders overlap (by one tile) with relevant adjacent zone borders, and that zone-group edge-zones overlap with all relevent adjacent zone-groups' edge-zones (i.e. copying) which I felt was an inefficiency of storage that could be made up for by the possibility of giving better routings between zones/meta-zones/meta-meta-zones/etc.:
Spoiler (click to show/hide)
Post by: shadow_slicer on November 02, 2009, 12:00:05 pm
There's a trade-off: The more complex the intra-zone navigation is allowed to be, the larger the zones can be -
straight-line -> convex zones
silly-move -> TCZs
A* -> more convoluted zones (entire maps, in fact)
We simply have to experiment to see which trade-off is most beneficial.
I'll say this, though: There has been no automatic cut-off criterion proposed for A* zones. As I just said, they can be as big as you want - but unlike the other two, the intra-zone pathing becomes more and more expensive for A* as the zones grow. Silly-move and straight-line are constant-time no matter how big the zone.
So, saying that we should use A* inside zones just begs the question - what zones to use? I thought up TCZs because they seemed a reasonable compromise.
I'm not sure that intra-zone pathing is difficult for A* than it is for silly-move for the cases where the two are applicable. If your silly-move algorithm is simply "move to the tile that is closest to the destination," then A* will have similar complexity since the only tiles A* would consider are the same ones the silly-move algorithm would (So both are algorithmic on the order of the zone size). A* has some additional overhead since it handles non-uniform costs, but this does not affect the algorithmic complexity.
Having smaller zones can actually decrease performance as well. As zone size decreases, the ratio "surface area" (number of external links) to the enclosed volume increases linearly (or is it even superlinearly?). In order to perform A* routing over the set of zones, the source link must know the cost to each of the other links in the zone. This means that the required calculation and storage grows as O(n^2) in terms of the number of links, resulting in the total algorithmic complexity increasing as O(n^3) in terms of the "smallness" of the zone. Note that this will be true regardless of what method of pathing is chosen inside the zones.
I'm not saying that the zones should be too large. A large zone with lots of chokepoints or doubling back or something like that would perform horribly, regardless of the pathing algorithm chosen. There definitely needs to be a tradeoff between zone size and complexity. I still haven't found a good algorithm to define the zones. All I really have so far is that the edges should follow chokepoints as much as possible to reduce external links. And I would probably use rectangular prisms as zones since this makes it easier to determine which zone a tile belongs to.
Edit: (instead of posting again)
Also, A* can be modified to produce the diagonal pattern that several posters have suggested seems more natural. If A* was adjusted to order (ascending) candidate nodes with equal heuristic cost by "sqrt(dx^2+dy^2+dz^2)", where dx, dy, and dz are the differences between that node's x,y, z and z positions to that of the destination, the paths found by A* would look like those "straight" lines.
Post by: Sunken on November 02, 2009, 02:05:50 pm
I'm not sure that intra-zone pathing is difficult for A* than it is for silly-move for the cases where the two are applicable. If your silly-move algorithm is simply "move to the tile that is closest to the destination," then A* will have similar complexity since the only tiles A* would consider are the same ones the silly-move algorithm would (So both are algorithmic on the order of the zone size). A* has some additional overhead since it handles non-uniform costs, but this does not affect the algorithmic complexity.OK, amortized over the entire path you are correct, since silly-move has to either check the next step at each step, or precompute the entire path at cost O(N). A* has to do the latter.
A* will incur more overhead as you say.
The main point is not that silly-move is cheaper, but that it is part of the definition of a TCZ, which in turn guarantees that both movement strategies will find the path. Larger zones - you'll have to find another proof of linear complexity. And you have to come up with a way of creating those larger zones.
Quote
Having smaller zones can actually decrease performance as well. As zone size decreases, the ratio "surface area" (number of external links) to the enclosed volume increases linearly (or is it even superlinearly?). In order to perform A* routing over the set of zones, the source link must know the cost to each of the other links in the zone. This means that the required calculation and storage grows as O(n^2) in terms of the number of links, resulting in the total algorithmic complexity increasing as O(n^3) in terms of the "smallness" of the zone. Note that this will be true regardless of what method of pathing is chosen inside the zones.Yes, that was the whole point of TCZs. They are strictly larger than convex zones or rectangles, while guaranteeing a constant-per-step complexity intra-zone. Again, even larger zones would be better, but not if that means losing intra-zone complexity guarantees.
Quote
I'm not saying that the zones should be too large. A large zone with lots of chokepoints or doubling back or something like that would perform horribly, regardless of the pathing algorithm chosen. There definitely needs to be a tradeoff between zone size and complexity. I still haven't found a good algorithm to define the zones. All I really have so far is that the edges should follow chokepoints as much as possible to reduce external links. And I would probably use rectangular prisms as zones since this makes it easier to determine which zone a tile belongs to.Rectangles will always be smaller than TCZs, that's what I've been trying to tell you. I'd still use rectangular bounding boxes to speed up TCZ lookup though.
And to my mind TCZs will be better at isolating chokepoints than rectangles. The biggest problem with TCZs is their tendency to overlap each other.
Quote
Also, A* can be modified to produce the diagonal pattern that several posters have suggested seems more natural. If A* was adjusted to order (ascending) candidate nodes with equal heuristic cost by "sqrt(dx^2+dy^2+dz^2)", where dx, dy, and dz are the differences between that node's x,y, z and z positions to that of the destination, the paths found by A* would look like those "straight" lines.Even assuming that computing that polynomial (leaving out the sqrt) is not too expensive, I think that would tend to make the algorithm try suboptimal paths first? Because that formula is not, in fact, the straight-line distance to the goal as the dwarf would walk in an open world. I'm not sure about that.
Anyway, it means adding computational complexity for an essentially cosmetic effect. It would work, though, and could be tried easily enough on top of any other approach AFAICS.
Post by: Sunken on November 02, 2009, 02:21:10 pm
Path-planning-wise this will have all the attributes of a convex subdivision, which has been part of the discussion for a bit now: trivial to path across, larger and thus potentially more efficient than rectangles; smaller and (or so I maintain) potentially less efficient than TCZs.
And you're suggesting a structure to create and maintain that convex subdivision, correct?
I'm doubtful as to the tree approach, though. Won't it be inefficient wherever the edges of the convex zone runs in the diagonal direction? This zone
Code: [Select]
#####
#### #
### ##
## ###
# ####is convex enough but would not benefit from the hierarchy you propose, I think? Correct me if I'm wrong.
Post by: shadow_slicer on November 02, 2009, 03:34:10 pm
...
Yes, that was the whole point of TCZs. They are strictly larger than convex zones or rectangles, while guaranteeing a constant-per-step complexity intra-zone. Again, even larger zones would be better, but not if that means losing intra-zone complexity guarantees.
*Edit: (forgot to comment here)
I'm not sure that I agree with you about the necessity of guaranteeing the intra-zone complexity be low. With hierarchical and cached pathing, the intra-zone stuff can be complex. It might be useful to constrain the zone such that the path from any tile in the zone to each of the zone edges have some maximum complexity* (or even just the nearest zone edge), since we're going to be pathing to and from edges a lot, but otherwise the intra-zone complexity doesn't seem such a big deal.
*I would define complexity of pathing as the ratio of number of tiles searched to path length.
....
Rectangles will always be smaller than TCZs, that's what I've been trying to tell you. I'd still use rectangular bounding boxes to speed up TCZ lookup though.
And to my mind TCZs will be better at isolating chokepoints than rectangles. The biggest problem with TCZs is their tendency to overlap each other.
I actually think that A* rectangles will be larger than TCZs. TCZs need a region of a) uniform cost and b) trivial complexity. How many TCZ regions (and purely A* regions) will be necessary to span the following design:
Code: [Select]
###########
a # # #
# # # # # #
# # # # # #
# # # b
###########
Spoiler (click to show/hide)
I am thinking though that pure rectangles may not be best. I was thinking that some sort of subtractive geometry might be efficient. We could define zones as "all of this rectangle except for those in the following
Even assuming that computing that polynomial (leaving out the sqrt) is not too expensive, I think that would tend to make the algorithm try suboptimal paths first? Because that formula is not, in fact, the straight-line distance to the goal as the dwarf would walk in an open world. I'm not sure about that.The way I mean it, it shouldn't cause suboptimal paths to be tried first. I realize this formula isn't the distance as the dwarves see it (for them its actually max(dx,dy,dz)). It's actually more of a tie-breaking rule. We already have the cost (which in A* is equal to cost so far plus estimate of cost to go) for each candidate node. All I'm suggesting is if there is more than one node with the same cost, they can be sorted based on Euclidean distance (purely for aesthetic reasons).
Anyway, it means adding computational complexity for an essentially cosmetic effect. It would work, though, and could be tried easily enough on top of any other approach AFAICS.
Post by: Sunken on November 02, 2009, 05:08:53 pm
I actually think that A* rectangles will be larger than TCZs. TCZs need a region of a) uniform cost and b) trivial complexity. How many TCZ regions (and purely A* regions) will be necessary to span the following design:You obviously don't mean the same by rectangles as I do. I mean unoccupied rectangles - unoccupied being the prerequisite for search-less pathing such as the Bresenham algorithm. You allow obstacles in your rectangles. It would take 11 unoccupied rectangles.Code: [Select]###########
a # # #
# # # # # #
# # # # # #
# # # b
###########Spoiler (click to show/hide)
To be clear:
axis-aligned obstacle-free rectangles
subset of
convex obstacle-free zones
subset of
TCZs
and
convex obstacle-free zones
subset of
arbitrary axis-aligned rectangles
subset of
arbitrary connected zones
The first three types of zone have a constant-per-tile complexity guarantee through their definition. The last two don't (I guess it's order-of-total-tiles, worst case), unless you add some further restriction (which is certainly possible).
Quote
I am thinking though that pure rectangles may not be best. I was thinking that some sort of subtractive geometry might be efficient. We could define zones as "all of this rectangle except for those in the followingCan't say if this would work or not. It seems like it might be sensitive to changes in the map, the way quadtrees et al. are, and the diagonal-tunnel example I put forth a couple of posts ago applies here too. But these are just hunches on my part. If you think it will work then you should try it out.rectangleszones". This could be represented as a "tree" (actually more of an acyclic graph -- since I would allow nodes to have multiple parents) where zones further down the "tree" have higher priority over the ones higher up. To determine what zone a node is, the algorithm goes down the path of the "tree" with zones containing the node until there are no zones further down containing the node. Pathing proceeds normally just using zones. This hierarchy is only used for determining a node's zone (though if we require zones to be completely contained in their parent zone: 1) we would have a multi-level hierarchy we could use to optimize pathing 2) the "tree" is a tree).
Quote
Yes, as a tie-breaking rule it would do the trick, though I'm not sure how you'd make sure the modification doesn't go beyond pure tie-breaking. But it's probably not a hard problem to solve.Even assuming that computing that polynomial (leaving out the sqrt) is not too expensive, I think that would tend to make the algorithm try suboptimal paths first? Because that formula is not, in fact, the straight-line distance to the goal as the dwarf would walk in an open world. I'm not sure about that.The way I mean it, it shouldn't cause suboptimal paths to be tried first. I realize this formula isn't the distance as the dwarves see it (for them its actually max(dx,dy,dz)). It's actually more of a tie-breaking rule. We already have the cost (which in A* is equal to cost so far plus estimate of cost to go) for each candidate node. All I'm suggesting is if there is more than one node with the same cost, they can be sorted based on Euclidean distance (purely for aesthetic reasons).
Anyway, it means adding computational complexity for an essentially cosmetic effect. It would work, though, and could be tried easily enough on top of any other approach AFAICS.
And the cost isn't max(dx,dy,dz) (or max(abs(dx), abs(dy), abs(dz)), if one is to nitpick), but actually:
max(abs(dx), abs(dy))-min(abs(dx),abs(dy)) + sqrt(2)*min(abs(dx), abs(dy)) + abs(dz)
"straight bit plus diagonal bit plus vertical". Or at least I think so - I think movements in the "vertical diagonal" aren't possible.
Post by: Draco18s on November 02, 2009, 05:37:53 pm
You obviously don't mean the same by rectangles as I do. I mean unoccupied rectangles - unoccupied being the prerequisite for search-less pathing such as the Bresenham algorithm. You allow obstacles in your rectangles. It would take 11 unoccupied rectangles.
Which solution has more overhead in terms of memory cost:
11 rectangles
5 regions
?
Post by: numerobis on November 02, 2009, 08:02:48 pm
I'm just popping in now to point out that in a tile based game, convex zones are rectangles. Any other shape ceases to be convex. Diagonals in a square area are an odd case, as the square the line passes through may or may not be traversable (eg. diagonal movement around a corner). Currently such diagonals are allowed at normal cost (1.4), but this may change.It's not hard to define a discrete version of convexity that would obviate your worry (straight-line obstacle-free path from the center of each tile to the center of all other tiles, for example).
But all we seem to be using of convexity is that there is a closed-form solution for the cost from one tile in the region to another, looking just at their coordinates -- this is a larger class than what you'd normally call convex. For example, TCZs have this property when diagonals are the same cost as the other directions (which is not the case here, but it might still form a good approximation).
Post by: numerobis on November 02, 2009, 08:07:41 pm
It's not clear to me how to improve on that, but maybe someone has an idea.
Post by: Cloaked on November 02, 2009, 09:52:18 pm
Both vastly and rapidly changing terrain, hundreds of agents constantly demanding paths, multiple movement types and nearly completely user controlled/designed terrain. You can't assume anything! Terrifying.
The stuff laid out in the thread so far makes sense to me though, especially how it makes sense to divide the world into zones to simplify the graphs involved. TCZ's sort of remind me of navigation meshes, only squashed into this 2D grid-world. The basic idea is the same, they define interconnected trivially walkable areas with local agent steering filling in to walk around unexpected obstacles. (Here's a easy to digest article about em: http://www.ai-blog.net/archives/000152.html Obviously, nothing from that is really directly applicable, but having more buzzwords to google is always good.)
It seems like no matter how you define your zones though, there are going to be situations that will be less than optimal. It'd be nice if there was a way to intelligently recognize those cases and reduce them appropriately. (That winding passageway posted previously, for example, has only one possible path through it, so it'd be nice if it was able to merge those into a neat single black box path from a to b instead of a long chain of TCZs.)
On the logistics side, though, are you guys going to go full-steam open source project on this thing? It'd be nice if people outside of a handpicked team could download and fiddle with the source, with any changes still going through an approval process controlled by the team. It might be a little early for that, though, if you're still designing a framework. But I know that's what I'd be doing right now (fiddling with stuff, either framework-wise or algorithm-wise) if I could.
Post by: Sunken on November 03, 2009, 03:16:05 am
But all we seem to be using of convexity is that there is a closed-form solution for the cost from one tile in the region to another, looking just at their coordinates -- this is a larger class than what you'd normally call convex. For example, TCZs have this property when diagonals are the same cost as the other directions (which is not the case here, but it might still form a good approximation).Exactly. "Convex" doesn't have an unambiguous definition on discrete sets, AFAICS, but if one can define a closed-form next-step algorithm (math people would probably use "ordering" here somehow), then a convex region is one where that algorithm takes you on a path that never leaves the region to your goal.
TCZs are just a special case, more general than more natural algorithms that just step in a straight line for the goal, doing no obstacle avoidance at all (such as Bresenham).
Indeed, the silly-move strategy is not guaranteed to produce the shortest path within a TCZ with obstacles - it can be a factor sqrt(2) inefficient in really contrived cases.
But note - even if we form our zones as TCZs, we don't have to use silly-move to actually path-plan within them. We can use A*, and we'll know the search will not run into the usual traps with doubling-back. That will give us an optimal path with the benefits of a more or less intelligent zone decomposition.
Post by: Sunken on November 03, 2009, 03:17:59 am
5 TCZs have more overhead than 11 rectangles, obviously. But I will stick my neck out and claim most fortresses don't look much like the example. Only testing in the wild will tell which is the more practical.You obviously don't mean the same by rectangles as I do. I mean unoccupied rectangles - unoccupied being the prerequisite for search-less pathing such as the Bresenham algorithm. You allow obstacles in your rectangles. It would take 11 unoccupied rectangles.
Which solution has more overhead in terms of memory cost:
11 rectangles
5 regions
?
Post by: Sunken on November 03, 2009, 03:22:45 am
A question not yet raised in this thread: what heuristic to use. One issue with traffic designations is that if you're actually using them heavily, the standard heuristic (euclidean distance) can make you sad: it assumes it costs 1 to cross each unit distance, when in fact it probably costs 2 or maybe 25.It's a tough one. I think it will be extremely hard to come up with a heuristic that is more efficient than the obvious one - not Euclidean, but
It's not clear to me how to improve on that, but maybe someone has an idea.
max(abs(dx), abs(dy))-min(abs(dx),abs(dy)) + sqrt(2)*min(abs(dx), abs(dy)) + abs(dz)
...and still remain both admissible and monotonic.
I think we're probably stuck with this heuristic. We'll have to concentrate on making the tree itself more efficient, through using good segmentation of the map. And maybe using caching, though I'd go for that only as a secondary.
Post by: Sunken on November 03, 2009, 03:48:15 am
Both vastly and rapidly changing terrain, hundreds of agents constantly demanding paths, multiple movement types and nearly completely user controlled/designed terrain. You can't assume anything! Terrifying.At least our world will always be a grid of discrete squares! That's a great strength compared to pathfinding in, say, reality. :)
Quote
The stuff laid out in the thread so far makes sense to me though, especially how it makes sense to divide the world into zones to simplify the graphs involved. TCZ's sort of remind me of navigation meshes, only squashed into this 2D grid-world. The basic idea is the same, they define interconnected trivially walkable areas with local agent steering filling in to walk around unexpected obstacles. (Here's a easy to digest article about em: http://www.ai-blog.net/archives/000152.html Obviously, nothing from that is really directly applicable, but having more buzzwords to google is always good.)Interesting, even though it's mostly a comparison of two systems none of which is on a discrete grid. Doesn't mean one cannot take inspiration from it, though. Convex polygons do have their interpretation in discrete space, as we've found.
Quote
It seems like no matter how you define your zones though, there are going to be situations that will be less than optimal. It'd be nice if there was a way to intelligently recognize those cases and reduce them appropriately. (That winding passageway posted previously, for example, has only one possible path through it, so it'd be nice if it was able to merge those into a neat single black box path from a to b instead of a long chain of TCZs.)Indeed! I have no idea how to adjudicate that automatically, nor whether the overhead would mean it wasn't worth it, but a system that could pick its zone types optimally would be golden.
To encourage attemts at this, as well as to make it easier to compare different types of zones and their performance, any framework should make "zone" a generic module that could be instantiated as a TCZ, rectangle, convex zone or A* block, transparently.
Post by: Starver on November 03, 2009, 06:25:22 am
Starver, I just skimmed over your description;[...]Just bear in mind that was a v0.1 (Alpha) thing that I wrote before I integrated and evolved what everyone else was talking about, if you're talking about the Spoilered bit. (I just thought "I spent at least 3/4 hour writing it, way back when, maybe I should give it a belated airing... :))
Quote
but essentially, you're proposing a tree-like decomposition into convex zones, right? A non-overlapping, mutually exclusive decomposition?Essentially, although crucially I was going on the premise of the boundaries actually overlapping by exactly one unit (at whatever level of [meta-^n]zoning we're at).
Thus
Code: [Select]
<--Zone 1-->
......01234567890......
<--Zone 2-->...in a 1D example. In 2D that boundary would be a B-line (consistently defined for both zones) which would allow a calculation of where the segment "[point-in-Zone1] to [point-in-Zone2]" intersected the "boundary between Zone1 and Zone2" segment (or, indeed, if it did not intersect, and thus had to be doglegged around the end of the "bbZ1&Z2" segment closest to the off-segment intersection point. The [piZ1] and [piZ2] coordinates possibly being the entry point to the Z1 or Z2 from/to Z0 or Z3, as guided by the meta-zone path-finding. However, I later saw problems with that.Quote
Path-planning-wise this will have all the attributes of a convex subdivision, which has been part of the discussion for a bit now: trivial to path across, larger and thus potentially more efficient than rectangles; smaller and (or so I maintain) potentially less efficient than TCZs.Again, that's historic. But you've got the essential flavour of what was proposed, with a couple of minor differences that don't bear going into.
And you're suggesting a structure to create and maintain that convex subdivision, correct?
Quote
I'm doubtful as to the tree approach, though. Won't it be inefficient wherever the edges of the convex zone runs in the diagonal direction? This zoneWith travel between the (inserted) "a" to "b" locations may (with the 1.4 multiple penalty for diagonality) be less quick to travel between than the (equivalent in crow-flies) "c" to "d" route. My original concept would have split the constrained diagonal containing a and c into one convex area and the large one leading to b and d into another, with a shared boundary somewhere a couple of cells to the right of "c" (and however far off-diagram you needed to go). Which would have meant a path dog-legging around the top of that "/|" shape from either a or c, and then straight to d or stagger to b. An a->b journey would actually be longer than a vector-based approach, of course, due to the disproportionate acumulation of 4x(root(2)-1) additions to the pure horizontal distance compared with the proper trigonometric adjustment for the angling across.Code: [Select]#####c dis convex enough but would not benefit from the hierarchy you propose, I think? Correct me if I'm wrong.
#### #
### ##
## ###
#a #### b
(If I understand your query (and diagram) correctly, of course.)
Post by: Sunken on November 03, 2009, 06:35:44 am
Code: [Select]
###############
########### #
########## ##
######### ###
######## ####
####### #####
###### ######
##### #######
#### ########
### #########
## ##########
# ###########
###############I was afraid the tree structure wouldn't be able to efficiently represent that as one unit. A quadtree wouldn't, but if you can then that's cool.
Post by: Starver on November 03, 2009, 07:27:04 am
(Here's a easy to digest article about em: http://www.ai-blog.net/archives/000152.html Obviously, nothing from that is really directly applicable, but having more buzzwords to google is always good.)Interesting. That's very close to the approach I never-quite-implemented back when I was so impressed by Doom, but thought it needed a proper third dimension (and the rest of the funny tweaks, like rotational symmetries of 1/2)... Looks like another thing I could have gotten rich from but didn't. :)
Quote
(That winding passageway posted previously, for example, has only one possible path through it[...]Yes/No (*delete as inapplicable). There's only one shortest path, but there are others. c.f:
Code: [Select]
##### #####
#+++# ##+##
#+#+# and #+#+#
#+#+# #+#+#
++#++ +###+
##### #####
7.2 to 11 only 7.2
steps steps
(based on 1.4 for diagonal)
And if you're splitting those into rectangles (minimum of 5, in either case), how do you ensure the former only takes the 7.2 step route?
(But, drawing in my prior mentioned 1-cell overlap system, it'd be 2 convex shapes, which would optimise the travel into the overlapped cell. However, there are other problems with that.)
Post by: Starver on November 03, 2009, 07:44:17 am
I'm afraid you misunderstood my diagram... I meant an indefinitely long diagonal corridor surrounded by rock. I'll make it clearer:It would from the B-line "vectorised" format. The tree structure was just the start but after merging based upon adjacent tree-ends that are mergable (in a manner I could spend ages explaining, but won't[1]) that would end up with something not so much a classic quad-tree.
<snip>
I was afraid the tree structure wouldn't be able to efficiently represent that as one unit. A quadtree wouldn't, but if you can then that's cool.
It would actually end up saying something like: "[1,1]->[4,1]:southwall; [4,1]->[14,11]:southeastwall; [14,11]->[11,11]:northwall; [11,11]->[1,1]:northeastwall", in whatever shorthand data format is required, and the ":southwall" being shorthand for what might (in reality) be something like ":{walkable,fliable}tozone<otheropenzoneID>" (when not enclosed in that direction) and ":southeast" maybe ":{diggable}tozone<undergroundzoneID>" (when that still is). Noting that this concept of mine has subtly changed since, anyway.
edit-added-footnote: [1] Not because you wouldn't understand, but because I know I'd spend ages on it, when you can probably work it out... sorry, just re-read what I said, and was afraid it looked a bit arrogant.
Post by: shadow_slicer on November 03, 2009, 07:57:38 am
And the cost isn't max(dx,dy,dz) (or max(abs(dx), abs(dy), abs(dz)), if one is to nitpick), but actually:
max(abs(dx), abs(dy))-min(abs(dx),abs(dy)) + sqrt(2)*min(abs(dx), abs(dy)) + abs(dz)
"straight bit plus diagonal bit plus vertical". Or at least I think so - I think movements in the "vertical diagonal" aren't possible.
I'm still not sure I understand why the metric needs to be like that. I thought the cost for diagonal movement was the same as cardinal movement (and it seems the dwarves move just as quickly in either cardinal or ordinal). Also, dwarves can't move diagonally upward? I can understand the case for staircases, but for ramps and flying/swimming creatures it seems really silly.
Post by: Sunken on November 03, 2009, 08:34:36 am
You may be right about vertically diagonal movement. I'm not sure, but I have a hunch the vertical movement actually doesn't add to the cost at all.
Post by: Starver on November 03, 2009, 09:14:02 am
Also, dwarves can't move diagonally upward? I can understand the case for staircases, but for ramps and flying/swimming creatures it seems really silly.I understood that as like up the following diagonal-on-the-horizontal-plane as well as ramp-traversal:
Code: [Select]
Z0 Z+1
### ##+
#^# #v#
+## ###
And I wasn't sure enough about it to say, but I thought it was possible. I'm sure my Adventurer can do it, wandering diagonallly across landscapes) but haven't got the game in front of me to test, right now.. (Plus Alt-Move climb at that angle, where otherwise permssable.
Post by: Sunken on November 03, 2009, 09:56:37 am
Post by: Xgamer4 on November 03, 2009, 03:14:57 pm
To the best of my knowledge, diagonal movements in 2D cost sqrt(2) times as much as orthogonal ones. At least during search.
While I have absolutely no proof, as Toady is a mathematician it really wouldn't surprise me if he obeyed the Pythagorean Theorem.
Post by: CobaltKobold on November 03, 2009, 06:57:43 pm
And yeah, no clue about the ramp travel cost-define, though it'd clearly be root2 for straight and root3 for diagonal normally...but that doesn't really model the cost right for a creature in a gravity well. (Likewise < > as 1 cost)
Post by: Shades on November 04, 2009, 03:48:13 am
It's funny, because a fair number of FPSs...do not cap your speed properly, and thus it is faster to proceed at an angle to your view (gi'ing you that sqrt(2) factor boost, if strafing is as fast as walking)
And yeah, no clue about the ramp travel cost-define, though it'd clearly be root2 for straight and root3 for diagonal normally...but that doesn't really model the cost right for a creature in a gravity well. (Likewise < > as 1 cost)
Actually most FPS's handle it correctly. A number of engines (quake and doom based ones for example) purposefully allow this though as there was a bug back in Quake 1, that when fixed was complained about so they ended up putting in code to replicate the effect under the corrected physics engine. I think in the halflife (counterstrike) engine it is also intentional but I don't know so much about that one.
Post by: Starver on November 04, 2009, 08:24:23 am
The length of your stride is a nominal amount, and your legs also have a lateral movement that can be added to that (pythagorously) to increase the stride length.
Yes, I'm treating the ball-and-socket joint at the hip as a two-way axial joint, and discounting the entire twisting of the pelvic bone that's probably what happens. And if you avoid all that, how you do it (and the presumed maximum speed you're attempting) without tripping over ones own feet, is another matter, as well as the fact that there's an energy cost and soft upper limit to moving the mass of your legs along the longer diagonal (which is not 45°, so won't give you strict root(2) 45° movement for the cost of unit forward motion) and the fact that the lateral muscles probably aren't as strong and various other issues...
But apart from that, one could argue that there's a speed-boost. :)
Post by: CobaltKobold on November 04, 2009, 05:30:34 pm
Post by: zchris13 on November 04, 2009, 06:00:23 pm
Post by: shadow_slicer on November 04, 2009, 08:40:15 pm
Besides, it's not like development occurs on the forums. (After all Toady seems to do quite a bit of development, but is on the forum less than most...).
As for me, I've played around a little bit with the code numerobis posted earlier (http://www.bay12games.com/forum/index.php?topic=43265.msg824354#msg824354). I made the following changes:
- changed to use SConstruct to build (since I couldn't use the build script provided)
- made some modifications so it would compile
- changed the grid to support multiple tile types (other than just passable): now we have to have stairs and ramps to go up and down, instead of flying everywhere...
- changed the A* algorithm to match the one on wikipedia and adapted it to use a heap and hash set for fringe and visited respectively
- changed the A* algorithm to return both the path cost and the path
- some other bug fixes and changes that I'm probably forgetting to mention
- Generated a few more test cases
For the curious, my code is available on my personal webspace (http://www.prism.gatech.edu/~bhamilton3/pathing.tar.gz), though it's still in-progress. Next task will probably be the hierarchical stuff, which I'm still planning out.
Post by: zchris13 on November 04, 2009, 09:22:01 pm
Post by: Starver on November 05, 2009, 05:41:56 am
Doesn't make your leg any longer ::)Ok, I wandered offtopic a bit, but I should point out that leg length isn't an issue. People with shorter legs don't hover, after all. Larger subtended angles just means a lower centre of body mass during the stride. (And awkwardness of stride, at extreme examples, just to auto-pedant myself.)
Ontopic: I've fallen foul of my usual development hell, in that I've been messing around with my own solution (no, not the "trinary tree" bit, but a bit of hand-coded TCZ-like bit 'after' that) but got stuck on minutiae and got diverted on the thought of "I know, I'll covert one of my active forts into a testable map". I'm glad to see that practical efforts have progressed despite me.
Post by: qwertyuiopas on November 05, 2009, 08:20:22 am
Post by: Starver on November 05, 2009, 11:27:37 am
Why not a SPZ(Single Path Zone) to complement the TCZ. It would cover any area with exactly two exits, and cache the fastest path between them. Pathing within it would use A*.Two exits as in two exit-tiles, or two 'walls of exit'?
Sorry, probably a stupid question, but I've been looking at a problem related to travel over several (ostensibly linearly-connected) zones where the width and placement of the apertures reduces the triviality. (Not merely the Z-bending situation.)
Although as long tunnels/causeways (and long straight tunnels being a nicer case, but not exclusively so) are one of the common features of a DF setup (alongside open countryside and rectangular rooms) and these have a single-tile aparture (and you can possibly also add long tunnel systems that are the sole entry/exit for rooms along their length), maybe it's not an issue.
Post by: qwertyuiopas on November 05, 2009, 05:52:06 pm
Code: [Select]
############################
# ######################
# ##### ###########
# ##########
####### #########
####### #########
####### #
####### # #############
######## # # ###########
######### ############
############ ###############
############ ###############
Could be seen as a single SPZ containing many lesser zones, but as one(or more) node up the heiarchy(I think), it could reduce massive quantities of pathing, as long as neither endpoint is inside of it.As the lowest level, it would take the place of multiple lesser zones in an arbitrarily shaped single tile corridor, or as in the example, a parent node to many other types to represent a more complex room.
It could be firther expanded to a Limited Exit Zone, that has N exits, and stores a path between each, but only exists if all exits are single tile.
Post by: Starver on November 06, 2009, 10:52:20 am
Single-tile.Now, I think there's merit in something there. (If not for the system being worked on, another one.) Assuming your Zoning system likes differently 'headered' zones, they can be packed and optimised better, and (if push comers to shove) travel between points within such a convoluted zone can be left to ad-hoc calculation (assuming that there's not such a regular occurance that it deserves to be put upon and stay in any cache) as the load per-calculation would be trivial.
<snip>
On a tree-of-meta^N-nodes model, no differentiation would be necessary, as it could be a single 'tile' in the first meta-organisation, and storing the popular entrance->exit route(s, i.e. bi-directional) probably as the main contents of any internal cache (followed by workshop or lever from/to each entry/exit, where applicable).
(Not that I think the meta-node would be exactly that area, but I know your diagram is an interesting example partially designed for expediency of drawing. At a quick count, although I am not an algorithm and so might not be reading it the same, I can see three "TCZ-like" zones, not counting entry and exit corridors, which might be give themselves up to being part of a meta-zone roughly covering the area you give. Unless "diggable routing" concerns arise, when at six 'underground traversal' ones (not counting the NW corner and a small number of tiles that might be technically a zone, or at least an overlap of zone) you may have a different number of non-underground ones to give edge-to-edge or one-tile-border connectivity, according to how you can arrange your B-Line joins.)
Post by: Sunken on November 06, 2009, 03:48:59 pm
Post by: shadow_slicer on November 10, 2009, 10:14:09 am
For the curious, my code is available on my personal webspace (http://www.prism.gatech.edu/~bhamilton3/pathing.tar.gz), though it's still in-progress. There's some sort of bug in the hierarchy code that makes it return a suboptimal path (depending on grid size).
Spoiler (click to show/hide)
Post by: zchris13 on November 10, 2009, 04:54:12 pm
Post by: SmileyMan on November 10, 2009, 07:39:05 pm
Also, when it comes to dividing zones, why not leverage user brainpower and divide zones based on door placement? Any one door has zone a on one side and zone b on the other (zone a and b may of course be the same). A bit of smart map searching could look for known door-like construction patterns as well.
Post by: zchris13 on November 10, 2009, 07:54:43 pm
Post by: Silverionmox on November 11, 2009, 10:25:30 am
Haven't had the time to read the whole thread, but has anyone mentioned "pheromone" style caching for the pathfinding? Could be useful for a colony-style game like DF.In eternal voting as: "Automatically adjust pathfinding for traffic (http://www.bay12games.com/forum/index.php?topic=29716.0)".
Also, when it comes to dividing zones, why not leverage user brainpower and divide zones based on door placement? Any one door has zone a on one side and zone b on the other (zone a and b may of course be the same). A bit of smart map searching could look for known door-like construction patterns as well.I agree that we should use the player's brain cycles whenever we can, especially since doors, zones etc. are likely concentrated in areas where the most dwarves and hence the most pathfinding requests are.
Post by: Shades on November 11, 2009, 10:41:34 am
Haven't had the time to read the whole thread, but has anyone mentioned "pheromone" style caching for the pathfinding? Could be useful for a colony-style game like DF.
The only way this really works is a combination of allowing full pathfinding some of the time (even as low as 1 in every hundred times) to force capture of new pathways and to store a pheromone map rated for each game task (or at least each related set of tasks).
No reason it can't be tested the same way as other methods though :)
Post by: SmileyMan on November 11, 2009, 12:34:30 pm
But, if we re-imagine the map as a series of subdivisions, then we could make the pheromone data universally applicable, based on subdivision heirarchy.
A Small 2D example
First, a standard Cartesian coordinate map
Code: [Select]
11 12 13 14 15 16 17 18
21 22 23 24 25 26 27 28
31 32 33 34 35 36 37 38
41 42 43 44 45 46 47 48
51 52 53 54 55 56 57 58
61 62 63 64 65 66 67 68
71 72 73 74 75 76 77 78
81 82 83 84 85 86 87 88
And here it is, as a binary subdivision map, divided xyxyxy
Code: [Select]
000000 000010 001000 001010 100000 100010 101000 101010
000001 000011 001001 001011 100001 100011 101001 101011
000100 000110 001100 001110 100100 100110 101100 101110
000101 000111 001101 001111 100101 100111 101101 101111
010000 010010 011000 011010 110000 110010 111000 111010
010001 010011 011001 011011 110001 110011 111001 111011
010100 010110 011100 011110 110100 110110 111100 111110
010101 010111 011101 011111 110101 110111 111101 111111
Critter 1, starting at (7,7) wants to move to (2,2). There are no pheromones
stored at (7,7), so it gets a path from the standard pathfinder, which will
trivially be 77->66->55->44->33->22
At each of those tiles, it places a marker, containing the final destination,
and the direction of travel. The destination is stored as the most significant
different bit pair from the current coordinate:
Code: [Select]
(7,7) -> 111100
(2,2) -> 000011
Stored: 110000 = (-1,-1)
(6,6) -> 110011
Stored: 110000 = (-1,-1)
(5,5) -> 110000
Stored: 110000 = (-1,-1)
(4,4) -> 001111
Stored: 001100 = (-1,-1)
(3,3) -> 001100
Stored: 001100 = (-1,-1)
(2,2) = Destination
Critter 2, starting at (6,6) wants to move to (3,1). The subdivision coordinate
for (3,1) is 001000, and therefore the most significant bit pair from (6,6) is
110000. There is already a marker on (6,6) telling us how to move to 110000, so
we don't need to go to our expensive pathfinder. The movement goes as follows:
Code: [Select]
(6,6) -> 110011
(3,1) -> 001000
Search: 110000
Found: 110000 = (-1,-1)
(5,5) -> 110000
(3,1) -> 001000
Search: 110000
Found: 110000 = (-1,-1)
(4,4) -> 001111
(3,1) -> 001000
Search: 000100
Not found!
So we go to the pathfinder for a path from (4,4) to (3,1), which will be something
like twice as fast as finding a full path from (6,6) to (3,1). The path returned will be
44->33->32->31, so we place markers as before:
Code: [Select]
(4,4) -> 001111
(3,1) -> 001000
Stored: 000100 = (-1,-1)
(3,3) -> 001100
Stored: 000100 = (0,-1)
(3,2) -> 001001
Stored: 000010 = (0,-1)
(3,1) = Destination.
Critter 3, wanting to travel from (7,7) to (3,1) will now follow markers all the way, and will make no pathfinder calls at all.
For each tile, it makes sense to store a limited number of markers in a cache,
depending on memory usage.
The nice thing about this is that while there is a chance of a sub-optimal path,
it will create quite naturalistic behaviour in that dwarves follow the paths other
dwarves make.
I hope the above makes sense. Given that it's obviously optimal in square areas
of power-of-two edge length, it might be best as a sub-routine for final delivery
within the (existing?) 64x64 regions, and then a macro path-finder can find routes
for the inter-region transport.
I'm sorry I don't have the time to stick some code behind this. :(
PS - I never did formal Computer Science, but if any compscis out there can tell me if this has been done before, that would be nice! If not, I henceforth name it the SmileyMan Algorithm!!!!!! :P
Post by: Sunken on November 11, 2009, 04:10:19 pm
That is all.
Post by: Silverionmox on November 12, 2009, 08:32:42 am
The Department for Unwanted Enlightenment would like to inform the readers of this thread about the word Stigmergy (http://en.wikipedia.org/wiki/Stigmergy).That just closed my browser.
That is all.
Post by: Starver on November 12, 2009, 10:52:06 am
The Department for Unwanted Enlightenment would like to inform the readers of this thread about the word Stigmergy (http://en.wikipedia.org/wiki/Stigmergy).
That is all.
Of course, we're dealing with globally-omniscient agents. Given that we have these[1] and not limited-knowledge agents with[2] or without[3] communicative learning, Stigmergy isn't relevent to the current situation, though I know it would be an interesting thing to put into the world simulation. With associated change of game balance.
[1] "Oh, I feel sad. I need to speak to the Mayor. He's deep in the caverns and I will now calculate how to get to him, and recalculate arbitrarily to his new position he moves around, despite being way beyond my vision or hearing."
[2] "Hoi! Freind McMiner, I have been tasked to cut some fine amethyst, and I understand that you can tell me down which long tunnel they may be found... Pray tell me, good sir, which way hither?"
[3] "Where is that log? Where is that log? I've been asked to get a log, and I don't know where it is. It wasn't to the east of this wall last time I looked, but is it there now.... No, first I'll check the northern forests. It may be there..."
[edit@Silverionmox: Didn't close mine, F.Y.I. Try again, reboot machine, check browser... Probably not Wiki's fault, unless it's a Compatability Mode issue.]
Post by: shadow_slicer on November 19, 2009, 09:48:05 pm
I am running three test cases and measuring the time it takes to calculate 100,000 random paths with different hierarchies. A plot of the initial results is shown in the spoiler below.
Spoiler (click to show/hide)
Post by: Impaler[WrG] on November 19, 2009, 10:26:03 pm
Post by: shadow_slicer on November 20, 2009, 12:12:28 am
Post by: eerr on November 20, 2009, 05:49:45 pm
Toady has the code set to recalculate pathfinding for every creature that paths through a tile.
can you find a more efficient method?
Post by: PermanentInk on November 21, 2009, 03:56:46 pm
Of course, we're dealing with globally-omniscient agents. Given that we have these[1] and not limited-knowledge agents with[2] or without[3] communicative learning, Stigmergy isn't relevent to the current situation, though I know it would be an interesting thing to put into the world simulation. With associated change of game balance.
Just because we're not constrained to a limited-knowledge approach doesn't mean we can't use one. Consulting a global knowledge bank costs cycles, more so the larger it is. An effective limited-knowledge technique, of which stigmergy is a good and intriguing example, can be more efficient by structurally ignoring parts of the knowledge bank not likely to be relevant to an agent's decision.
Anyway, it's not so much a question of "what we have," if you mean the current pathfinder, because the point is to rip it out and replace it with something better. There's nothing inherently globally omniscient about DF critters; it's just a question of them being programmed that way or not.
Quote
[1] "Oh, I feel sad. I need to speak to the Mayor. He's deep in the caverns and I will now calculate how to get to him, and recalculate arbitrarily to his new position he moves around, despite being way beyond my vision or hearing."
[2] "Hoi! Freind McMiner, I have been tasked to cut some fine amethyst, and I understand that you can tell me down which long tunnel they may be found... Pray tell me, good sir, which way hither?"
[3] "Where is that log? Where is that log? I've been asked to get a log, and I don't know where it is. It wasn't to the east of this wall last time I looked, but is it there now.... No, first I'll check the northern forests. It may be there..."
I like your examples, though. They illustrate an idea that keeps nagging at me the more of this thread I read, which is that it's not necessarily appropriate for dwarves to be omniscient. Aren't there all kinds of interesting story possibilities that point toward pathing based on individual entity knowledge?
- You awaken bloodied and sore amid a corpse-strewn battlefield. Rubbing a massive lump on the back of your head, you wonder where the hell you are and how to get home.
- The road is blocked ahead by a rockslide, but the caravan doesn't know that. The bandits, however, do; they close in from behind for an ambush at the dead end.
- Only the ancient vizier knows of the secret passage behind the throne room. That is, until one day a cleaning dwarf trips on the lever that opens the door, and his curiosity gets the better of him...
Obviously, these are pretty rich examples of such a system compared to what the game's capable of today, but the point is that any pathfinder that's going to be viable long-term should be consistent with the long-term direction of the game.
btw, I believe you mean "thither," not "hither." "Hither and thither" is like "here and there," and I believe the usage parallels this in general. TMYK
Post by: shadow_slicer on November 22, 2009, 12:07:20 am
Just because we're not constrained to a limited-knowledge approach doesn't mean we can't use one. Consulting a global knowledge bank costs cycles, more so the larger it is. An effective limited-knowledge technique, of which stigmergy is a good and intriguing example, can be more efficient by structurally ignoring parts of the knowledge bank not likely to be relevant to an agent's decision.That's not really true. It is significantly simpler (and faster) to solve problems using global knowledge. Any problem that can be solved with limited information can be solved at least as fast (and in the vast majority of cases faster and better) using global information. This is one of the central ideas of information theory. Techniques such as stigmergy are useful when the problem MUST be solved in a distributed manner. They are slow to converge and require significantly more overhead.
Quote
Anyway, it's not so much a question of "what we have," if you mean the current pathfinder, because the point is to rip it out and replace it with something better. There's nothing inherently globally omniscient about DF critters; it's just a question of them being programmed that way or not.But there is a significant difference between the kind of AI you're suggesting and pathfinding. Pathfinding really doesn't care where the knowledge comes from or how it is used. What you're talking about is AI.
Post by: Nexii Malthus on November 22, 2009, 02:54:37 am
The problem is that pathfinding pretty much is one of the core heart pieces of artificial intelligence, it is about finding a preferred path, where to go, where to move, how to get there, and why that path.
I do feel that Stigmergy is incredibly Awesome, and will be perfect for the dwarf swarm intelligence.
But I feel that the global information pathfinding should still be -the- underlying foundation for now, I just have a bad instinctive feeling about going fully with stigmergy. Stigmergy is a big, big job and inevitable goal. But the omniscient system will have to do for now, after all, what we are seeking is pure, liquid, clean performance. Once we have that, we can build on it.
Post by: zchris13 on November 22, 2009, 12:33:06 pm
Post by: shadow_slicer on November 22, 2009, 03:12:56 pm
The problem is that pathfinding pretty much is one of the core heart pieces of artificial intelligence, it is about finding a preferred path, where to go, where to move, how to get there, and why that path.This is probably where our difference of opinion comes from. I see pathfinding as finding the lowest cost path on an arbitrary graph. This is a mathematical property of the graph and has nothing really to do with AI. The AI determines where to go, and by defining the costs for going different routes, defines the graph. The costs themselves define "why" that route. If a dwarf is operating under limited information, some tiles could be "unexplored". These would have some default cost (a relatively high "exploration cost"). In essence all aspects of AI you want to include are simply found through defining an appropriate cost function.
Quote
I do feel that Stigmergy is incredibly Awesome, and will be perfect for the dwarf swarm intelligence.I actually disagree about this. I think there are significantly easier ways to achieve the effects you want to achieve. Stigmergy is complicated and poorly understood. It makes sense if you want to build cheap, simple hardware to perform complex distributed tasks, but not when you're using a single powerful processor with global information on a centralized task.
Post by: PencilinHand on November 22, 2009, 04:07:23 pm
--------
If I have read and understood this thread right, the code that is being thrown around right now just arbitrarily divides a map along groupings of Cartesian coordinates forming a hierarchical system. Then what would be most useful(for a pathfinding dwarf) would be a system of methods for more objective/less arbitrary grouping within the hierarchy.
Using the example given on the previous page, if "|" and "_" were borders for different zones at a mid level in a hierarchy of zones then something like the following might be more effective than arbitrary division.
Code: [Select]
############################
#c ######################
# ##### | ###########
# | | ##########
####### | #########
#######______|_____#########
####### #| | b
####### # |_#############
######## # # ###########
######### _ ############
############ ###############
############a###############Assuming my understanding is correct then what is really needed are efficient algorithms for forming and maintaining these zones intelligently so as to minimize path calculations from a to both b and c.
Am I way off base or is that basically the crux of the problem?
Post by: Impaler[WrG] on November 22, 2009, 06:43:23 pm
Post by: SmileyMan on November 23, 2009, 07:52:38 am
Especially if it's one of the header-only files, I'd do that, especially as a few of them are slated to end up in C++0x anyway.
Personally, I couldn't program without the boost smart pointers anymore.
Post by: Starver on November 23, 2009, 10:51:19 am
Well, "which way thither" would be a valid way of "which way to get there", but I'm sure I was writing something more like "which way from here". So "...which way, hither?"Quote[2] "Hoi! Freind McMiner, I have been tasked to cut some fine amethyst, and I understand that you can tell me down which long tunnel they may be found... Pray tell me, good sir, which way hither?"
btw, I believe you mean "thither," not "hither." "Hither and thither" is like "here and there," and I believe the usage parallels this in general. TMYK
But that's not a pathfinding point. (Well, not algorithmically, only contextually. :)) And, besides, I misspelt/misgrammared various other parts of that post ("Freind"? Sheesh!)
Post by: SmileyMan on November 23, 2009, 12:38:20 pm
Archaic English, but possibly useful as OO method names!
Post by: Impaler[WrG] on November 23, 2009, 01:12:03 pm
I'm going to try to modify slicer's code to eliminate the boost dependencies, possibly implementing some simplified versions of the boost classes that were used.
Post by: shadow_slicer on November 23, 2009, 01:50:22 pm
As for the iterator in graph.h, I think it's not used in STL containers, so it might not be as difficult as I initially feared. You can simply remove the boost stuff completely and add operator++(), and operator*(), which call the existing functions. (I've done it and posted it here (http://www.prism.gatech.edu/~bhamilton3/graph.h)).
I'm currently still looking over my code. There are a couple more optimizations to the caching I could do: 1) when a path is cached, the reverse path is also cached, 2) when a path cannot be found, all nodes in the region that haven't been visited yet also are disconnected (so we can cache them as well), 3) when a path cannot be found, A* has calculated (or at least done 99% of the work necessary to calculate) the minimum path to all nodes in the region searched, so we should cache this information. I think (1) may be working in my newest version, but (2) and (3) are doing really weird things. I'm going to keep working at it. The public interfaces should stay the same, so any changes I make shouldn't make a large impact.
Edit (instead of double replying):
@PencilinHand: Yes, that is now where we stand. We were talking earlier about TCZs and other zone types. Once Impaler[WrG] has combined the code with Khazad we will have both some pathfinding code to work with and a set of realistic scenarios. Then we can all implement our zone ideas and see how they actually perform, instead of just arguing back and forth.
Post by: Impaler[WrG] on November 23, 2009, 03:18:35 pm
Post by: SmileyMan on November 24, 2009, 03:46:54 am
I got halfway through drawing out a plan for a hybrid pheromone/zone map system that would be quite lightweight. No time for it today, but maybe finish off the paper code later in the week.
A few quick questions: How are we expecting calls from the program to the pathfinder to work? Given that some critters may be moving towards a moving target, I'm assuming that it's a series of "I am at location X and I need to get to location Y, what is the next step I should take?" and the return value will be a direction in some format (sub-question: is there a standard format for these vectors?). The primary goal is then to return from that call as quickly as possible, without returning an impossible move and without causing "waggle dance" or other atrtfacts.
Post by: Impaler[WrG] on November 25, 2009, 04:48:52 am
http://theory.stanford.edu/~amitp/GameProgramming/index.html
Post by: Starver on November 25, 2009, 10:48:20 am
A few quick questions: How are we expecting calls from the program to the pathfinder to work? Given that some critters may be moving towards a moving target, I'm assuming that it's a series of "I am at location X and I need to get to location Y, what is the next step I should take?" and the return value will be a direction in some format (sub-question: is there a standard format for these vectors?). The primary goal is then to return from that call as quickly as possible, without returning an impossible move and without causing "waggle dance" or other atrtfacts.
Personally, I think the return call should be the entire path from current location to intended destination, and possibly some more jiggery pokery (as noted belote). On an trip that takes 100 steps, a mythical beyond-theoretically-possible pathfinding alorithm that takes N cycles to work out the optimally short 'N'-step journey that must be taken, and tells the agent the next step, would need 100*101/2 cycles to get them to the destination. And, in reality, we know that it's probably somewhere between N Log N and something truly exponentional, for a single N-step, which then gets added. So, best to calculate a single path, once, and follow it. At least until something tells you that a recalculation is needed.
Obviously, recalculation at each step of the way handles dynamic environments exceedingly well. And also simplifies things in an 'intuitive' system where only the next step (or two) is queried, which would involved back-reference to rule out "been there, and it's blocked off" possibilities). To that end, the additional 'jiggery pokery' could involve waypoints where conditions should be queried, or even (at a memory-mapping level) implementing memory pointers that mean that when obstructions are created/removed they can create an 'event'/interupt to be picked up by all relevent pathing agents, to force them[1] or encourage them[2] to re-evaluate their path.
This latter would reduce re-pathing needs, length of any particular repathing need[3] and act almost as if the omniscient repathing were being reassessed after every single tick. Not quite, because unforseen new paths from multi-tile landscape destruction are among the trickiest things to consider (and probably not worth the trouble), although an event-initiated link to currently routing agents might also be considered when a new agent plans its own path and discovers not only the new and better routing possibility, but that it shares significant waypoints with those remaining in the currently travelling agents queue of intentions. The triggered events can then prompt the more senior router to reassess their position, with most of the required calculation already done.
Of course, this is the omniscient solution. Some people, I know, would rather it be reduced to discovery/trial-and-error. And I certainly think that could be factored in, but not in a pheremone-style way. An ant's nest has a whole mass of active agents, running their own instantaneous partial-path routing, not externally controlled by a solid purpose but creating an apparent purpose from the combined 'waveforms' of purposelessness within the hive mind and hive body. I don't see enough of those aspects being sufficiently relevent to dwarfkind.
[1] Where a blocking affects the currently intended path, but allow them to remember this for point [2]...
[2] When a path segment opens that was previously blocked, or (if you want to get complicated) by some means remember "could have, if only" shorter viable paths prevented only by a temporary style of blockage. By adding a little extra effort/information storage, it might even streamline travel through zones such as repeater-led cyclic barriers across a path. And if you allow natives (and possibly, by experience, long-term transient entities like a multi-season beseiging force) this skill, but leave opportunistic thieves/fauna/etc to continue their current method of pathing, it could mean that the creation of a 'dynamic labyrinth' could be a viable means of 'defence by confusion'.
[3] Imagine wandering half way round the mountain to find that the area previously impassible from a straight-through route and needing the detour was now only passable via the straight-through route. Something you might easily get if setting up a satelite 'watchtower' or outlying habitation/cafeteria specifically for certain resource-handlers. The overhead of recalculation (and 'dwarf energy' in this (extreme) example could have been avoided if, shortly after they set off on their way round, there had been a 'psychic summons' bringing them back again to the now-opened direct route (before or after the original indirect route was shut off[4]).
[4] Technically, a shut-off before the official short-cut actually opening should trigger relevent "cannot reach, abandoning job" events. But it's up to the player to work out how they want to synchronise the activities of blocking masons/channellers/etc with permitting miners/bridgers/etc.
Post by: SmileyMan on November 25, 2009, 11:40:30 am
Each critter has to make a call that says "What is my next move" - whether it makes that to its own stored copy of the path or to a pathfinding engine is largely irrelevant. I'm not suggesting that the engine recalculate every path for every call, although we can program a reference engine that does that if necessary. But the call to the engine should be (pseudo-typed):
Code: [Select]
DirectionVector GetNextStep(const CoOrdinate& CurrentLocation, const CoOrdinate& Destination);and everything inside there should be a black box to the client, returning DirectionVector in as few cycles as possible (averaged over many calls).
BTW - would you ppl like to work on this in Google Wave? I've got a bunch of invites, so PM me your email address if you want one!
Post by: Starver on November 25, 2009, 12:22:22 pm
Returning the entire path works very poorly for a moving target (e.g. a meeting), and also means that if several dwarves are following a common path (e.g. trade-depot -> stockpile) then you have multiple memory copies of the same path.Apologies, I meant to add that to the "advantages of a single move" side. With the caveat that if it's converging paths (at least temporarily) the two pathers could be given event-driven instructions to pop the successively nearer destination locations of of each other, and if one is chasing the other considerations might be given to adding the 'done' movements of the latter to the former's path. Worst case scenario is when the disinterested 'target' agent is moving in a topologically 'perpendicular' manner and the changes to the seeker's path are not simple splice-on/off functions.
Quote
Each critter has to make a call that says "What is my next move" - whether it makes that to its own stored copy of the path or to a pathfinding engine is largely irrelevant. I'm not suggesting that the engine recalculate every path for every call, although we can program a reference engine that does that if necessary. But the call to the engine should be (pseudo-typed):I see, so pathing is initiated by the GetNextStep function if there's no acknowledged (and irrefutable) path already in existence, but the next step merely shifted from the required path store (not sure what syntax your pseudo-code would use, but some sort of hybrid of a "state %path; $path{$agentID} = GetNewPath($src,$dst) unless (defined $path{$agentID} && !PathChangeNeeded{$agentID});" command that I might use in my Perl testbed's version of the above[1]) whenever possible. With the PathChangeNeeded() function covering a variety of relevent sins...Code: [Select]DirectionVector GetNextStep(const CoOrdinate& CurrentLocation, const CoOrdinate& Destination);
and everything inside there should be a black box to the client, returning DirectionVector in as few cycles as possible (averaged over many calls).
[1] I've forgotten how I'd do this in the mainstream C-family of languages, to my shame. Might also be "state", but it's so long since I've employed this particular trick of the trade. Apologies if Perl syntax is similarly strange to you.
Post by: shadow_slicer on November 25, 2009, 12:38:54 pm
As for the interface, I think there would be more state required than simply the current location and destination.
Perhaps instead of returning the next step in the path, it should return a "path iterator" which it can use to find the next step.
If a method such as A* is used, the path can be completely calculated, and an iterator to that path can be returned. If a more dynamic or adaptive method is used, the iterator could actually calculate the next step. Further the iterator could implement Starver's jiggery pokery without the client needing to have any knowledge of the actual jiggery pokery involved.
If the destination changes, we could call something like "changeDestination(iterator, newdestination)" and get a new iterator for the new path. This would allow dynamic or adaptive algorithms to be used efficiently.
Post by: SmileyMan on November 25, 2009, 01:54:04 pm
The iterator idea gives us the advantage that it gives a hint to the engine as to which paths are currently in use, so shouldn't be dropped from the cache. The iterator should be valid as long as the destination didn't change.
That doesn't help us for moving targets, and if we just used the single function, it would involve the constant creation and destruction of iterators, which might be expensive.
However, we could have different functions so that we can optimise the calling code depending on whether we are aiming for a moving or a static target:
Code: [Select]
PathIterator GetFirstStep(const CoOrdinate& CurrentLocation, const CoOrdinate& Destination); // for paths to static destinations
DirectionVector GetNextStep(const CoOrdinate& CurrentLocation, const CoOrdinate& Destination); // for paths to mobile destinations
where PathIterator has an "increment" operator, and is convertible to a DirectionVector.If we code that properly, the path finding engine can keep a register of the current PathIterators, and make sure that the nothing in the path cache will be dropped if it is on a current iterated path.
That might be quite an overhead to maintain though - you could achieve the same effect just by having the GetNextStep function and a sufficiently large path cache. The tradeoff is memory optimisation versus performance optimisation, always a fun debate :P
One other bonus of just having the GetNextStep function is that you can be vague at the beginning of a long path - how you implement this in code is up to you, but for instance if you are outside in the far south and you know you need to get to the far north, then you can tell the critter to start moving north, and get more specific the closer you get. See my heirarchical coordinate system for details of why this might be useful, also moving targets again.
Back to the interface, I think the engine needs a "MapDataChange" function, which (ideally) the game could call when map data changes (would obviously require Toady's input), but in the meantime, critters could call when the direction specified in the return from GetNextStep was impassable. You could use this as a very inefficient (in the beginning) way of building up the map data without having to read memory at all, or alternatively have a sub process which scans the map at intervals and checks for changes:
Code: [Select]
void MapDataChange(const CoOrdinate& Location, const bool IsPassable, const Cost& = 0);or similar.
By just having two functions in the interface, the barrier to entry is very low (a raw A* uncached engine would be pretty easy to write as a "worst case performance" reference model), and a test program could be run up quickly; it also makes it easier for Toady to integrate into his code later on.
For something I've got no time to spend on, I seem to be spending quite a lot of time on this! :) More fun than work though......
Post by: shadow_slicer on November 25, 2009, 02:25:12 pm
That doesn't help us for moving targets, and if we just used the single function, it would involve the constant creation and destruction of iterators, which might be expensive.I was thinking something more like:
However, we could have different functions so that we can optimise the calling code depending on whether we are aiming for a moving or a static target:Code: [Select]PathIterator GetFirstStep(const CoOrdinate& CurrentLocation, const CoOrdinate& Destination); // for paths to static destinationswhere PathIterator has an "increment" operator, and is convertible to a DirectionVector.
DirectionVector GetNextStep(const CoOrdinate& CurrentLocation, const CoOrdinate& Destination); // for paths to mobile destinations
Code: [Select]
PathIterator path(const point& CurrentLocation, const point& Destination); //for all pathfinding
PathIterator changeDestination(Pathiterator currentIterator, const point& newDestination); //only used when destination changes
I think we will definitely need state for each active path. Otherwise it becomes impossible to use A* or other methods that need to calculate the entire path in one step.Quote
If we code that properly, the path finding engine can keep a register of the current PathIterators, and make sure that the nothing in the path cache will be dropped if it is on a current iterated path.
It appears you are thinking of having a path cache that stores the entire path for all active paths. I don't think that storing entire paths would be very useful. While dwarves tend to go to and from nearby places, they almost never go to/from the exact same tile. The odds are the path would be broken by a topology change before it could ever be reused. And while we could use these full paths for path splicing, this becomes very complicated very quickly.
Quote
One other bonus of just having the GetNextStep function is that you can be vague at the beginning of a long path - how you implement this in code is up to you, but for instance if you are outside in the far south and you know you need to get to the far north, then you can tell the critter to start moving north, and get more specific the closer you get. See my heirarchical coordinate system for details of why this might be useful, also moving targets again.
This can also be done quite easily with the iterator.
Quote
Back to the interface, I think the engine needs a "MapDataChange" function, which (ideally) the game could call when map data changes (would obviously require Toady's input), but in the meantime, critters could call when the direction specified in the return from GetNextStep was impassable. You could use this as a very inefficient (in the beginning) way of building up the map data without having to read memory at all, or alternatively have a sub process which scans the map at intervals and checks for changes:In the current code base the client creates a "grid" class which implements two functions:Code: [Select]void MapDataChange(const CoOrdinate& Location, const bool IsPassable, const Cost& = 0);
or similar.
Code: [Select]
cost_t edgeCost(const point& a, const point&b);
unsigned max(unsigned dim);
where max returns the size of the grid along dimension dim and edgeCost returns the cost to travel from point a to an adjacent point b. A cost of -1 is used to represent disconnection. You can use edgeCost(a,a)!= -1 to determine if point a is passable.Then the client passes this grid to the library, which does stuff based on that.
For the cached pathing methods there is another function "registerChanged(const point &p)" which needs to be called for every tile that has changed.
Post by: Impaler[WrG] on November 25, 2009, 02:46:51 pm
Post by: Nadaka on November 25, 2009, 03:49:36 pm
This will:
A: eliminate the cost of finding a valid path to optimize.
B: seed h(estimated distance function) with an accurate approximation of the remaining distance to reduce the amount of branching required to optimize the already known path.
You could even use the same method when recalculating paths to static goals after sanity checking to ensure the previous path is not blocked in order to find new paths that are shorter if desirable.
Post by: dyze on November 25, 2009, 03:57:48 pm
are you guys working on actual df code, or are you just theorizing as of now?
Post by: SmileyMan on November 25, 2009, 05:36:10 pm
I was thinking something more like:OK, the PathIterator with the ChangeDestination method is good, that's a clean single interface.Code: [Select]PathIterator path(const point& CurrentLocation, const point& Destination); //for all pathfindingI think we will definitely need state for each active path. Otherwise it becomes impossible to use A* or other methods that need to calculate the entire path in one step.
PathIterator changeDestination(Pathiterator currentIterator, const point& newDestination); //only used when destination changes
In the current code base the client creates a "grid" class which implements two functions:Code: [Select]cost_t edgeCost(const point& a, const point&b);where max returns the size of the grid along dimension dim and edgeCost returns the cost to travel from point a to an adjacent point b. A cost of -1 is used to represent disconnection. You can use edgeCost(a,a)!= -1 to determine if point a is passable.
unsigned max(unsigned dim);
Then the client passes this grid to the library, which does stuff based on that.
For the cached pathing methods there is another function "registerChanged(const point &p)" which needs to be called for every tile that has changed.
Should ChangeDestination be a method of the iterator class or should it be a method of the engine? If it's an engine method, should it take and return a reference, allowing the engine to re-use the iterator (or create a new one), for maximum flexibility?
Post by: Impaler[WrG] on November 25, 2009, 06:02:22 pm
As for the system it will use a central PathManager which will store the Map graph/s. It will act as a factory for MovementControllers and keep a link with all of them. The PathManager will receive map-change updates and can then signal certain Controllers to change paths if desired. It will also perform caching of paths. The Controllers will request new paths from the Manager when their in need, though it will also be possible to request paths directly from the Manager.
Post by: peterix on November 25, 2009, 06:30:18 pm
1. the ant approach.
2. network routing protocols (real-world pathfinding)
3. space subdivision
So, you have agents (ants, packets... blah) running around, doing space subdivision as they go. One sub-part is equivalent to a router in a network. The network is a graph of nodes (routers) and connecting paths.
The obvious + here is that you don't have to recalculate the whole thing when a tile changes. It will get recalculated when the agents discover the changes, and the changes will be only local at first, propagating through the network as they become known.
Would make an interesting experiment I think :)
Post by: qwertyuiopas on November 25, 2009, 07:29:43 pm
-GetShortestPathLength (for stockpile distances and stuff)
-GetFullPath (should be rarely used, but still exist)
-GetStep (As mentioned before)
-GetNSteps (Allows program to cache a few steps at a time)
-GetLOSSteps (Gets the steps between the unit and where the path leaves LOS. This would be closest to real life, since I doubt you think "Okay, next I must go that way for a step. Okay, next I must go that way for a step. Okay, next I must go that way for a step. Okay, next I must go that way for a step. " When all 4 are in the sime direction)
Post by: Starver on November 26, 2009, 09:09:48 am
It's C++, with CoOrdinate and DirectionVector being as-yet-unidentified types.What I meant was "not sure how I'd do this Perlish statement in C-Code". See "Persistent Private Variables" in the appropriate PerlSub perldoc (http://perldoc.perl.org/perlsub.html#Persistent-Private-Variables) for the mechanism I was employing. In leiu of making it a global (i.e. ($main::<foo>) implementation or needlessly package/object-wide. Sorry, minor distraction, that.
Quote
One other bonus of just having the GetNextStep function is that you can be vague at the beginning of a long path - how you implement this in code is up to you, but for instance if you are outside in the far south and you know you need to get to the far north, then you can tell the critter to start moving north, and get more specific the closer you get. See my heirarchical coordinate system for details of why this might be useful, also moving targets again.
Actually, I'd take some of that idea and think about the possibility of globally using "Just In Time" techniques. For the origin in the far south, the mere hint of the destination in the far north should prompt the topmost layer of routing hierarchy to spew "move through zone immediately to north" as the lowest-definition necessity of routing. If using a hierarchical zone (and meta^N-zone?) structure to the algorithm, one could easily buffer the detail of the journey until (at the latest) entry to the zone[1]. In a true 'real time' game, consideration would need to be given to using 'slack ticks' to get the backlog of future necessary workings precalculated so as not to pile up JIT requisitions, but minor lag-hiccups could be absorbed by DF, I'm sure, should things get complicated. (Though "every tick, do a given proportion of queued pathfinding details" could be implemented without excessive management overheads, and could clear the still-to-do queues long before the JIT philosophy rquires it.)
But. Whatever the system, it should be able to check its assumptions and not say "wander north, don't worry about the detail for now" when you get something like the typical 'megaprojecty' conditions where the map is cleaved by impassible terrain (Dwarfmade, natural or from player-led/hostile-presenced travel restrictions/reprioritisations) and a straight-mined tunnel has been helpfully comissioned by the player (and completed by the workforce) especially for this eventuality. A meta-hierarchical system should handle that with grace, of course.
[1] Or, where zone transitions are N tiles wide, N tiles prior to the zone transition, in order to allow the foresight to to handle optimal zone transitioning. This may be hwolly contained within the predecessor area, or leach into ones further back.
Post by: Starver on November 26, 2009, 09:23:07 am
half of the stuff in this thread goes way over my head. the other half goes way way over.Personally, I'm working on a testbed system of my own, having not yet grappled the published testbed system (also standalone, merely simulating[1] the general conditions of DF, not tapping into its code. But this is to address one particular design feature of a pathing algorithm.
are you guys working on actual df code, or are you just theorizing as of now?
For the rest, I am theorising (some might say "pontificating") about various pathing issues being discussed. Some of which apply to 'my' personal project, and some don't even apply to the collaborative work at hand.
(The trouble is, my brain works far faster than my hands can type code/explanations. And, if I say so myself, I can type code and explanations very quickly. If occasionally inaccurately. :))
[1] Simulating a simulation! Ha! I suddenly find my approach even more amusing...
Post by: SmileyMan on November 26, 2009, 09:29:46 am
How aboutMost of those things are just moving the GetNextStep call to various points in the algorithm by making assumptions about algorithm behaviour that the implementor may not wish to conform to.
-GetShortestPathLength (for stockpile distances and stuff)
-GetFullPath (should be rarely used, but still exist)
-GetStep (As mentioned before)
-GetNSteps (Allows program to cache a few steps at a time)
-GetLOSSteps (Gets the steps between the unit and where the path leaves LOS. This would be closest to real life, since I doubt you think "Okay, next I must go that way for a step. Okay, next I must go that way for a step. Okay, next I must go that way for a step. Okay, next I must go that way for a step. " When all 4 are in the sime direction)
However, the PathLength you mention is an important function for the engine. It could be a method of the PathIterator concept.
Post by: tomato on November 27, 2009, 08:12:01 am
How aboutMost of those things are just moving the GetNextStep call to various points in the algorithm by making assumptions about algorithm behaviour that the implementor may not wish to conform to.
-GetShortestPathLength (for stockpile distances and stuff)
-GetFullPath (should be rarely used, but still exist)
-GetStep (As mentioned before)
-GetNSteps (Allows program to cache a few steps at a time)
-GetLOSSteps (Gets the steps between the unit and where the path leaves LOS. This would be closest to real life, since I doubt you think "Okay, next I must go that way for a step. Okay, next I must go that way for a step. Okay, next I must go that way for a step. Okay, next I must go that way for a step. " When all 4 are in the sime direction)
However, the PathLength you mention is an important function for the engine. It could be a method of the PathIterator concept.
We're talking about interface and DF needs if someone wants to implement an algorithm for pathfinding they have to implement such and such functions, wants and whims have nothing to do with it.
Now, I've been thinking about the interface a bit lately, taking into consideration the way that DF stores its map internally and what it needs from the PF.
Movement maps
First thing, I don't think that the callback mechanism for detecting passability for tiles is a good idea - remember, the very paths we look for are directly influenced by it and to calculate it we would have to run it thousands of times. As such IMO the single tile in the map data that the library gets should be a bit field with each bit setting passability for certain type of transport, for example:
Code: [Select]
pool:
00000100
least significant bit - dwarfs
2nd - flyer
3rd - swimmer - water
4th - swimmer - salt water (do fresh water fish die in salt water?)
5th - building destroyer
6th - digger
etc.
up to 32 or 64 bits with dedicated bits for traders and siegers, pets, etc.
workshop: (hexadecimal)
30 30 30
33 33 33
33 33 30
the 2nd row and first 2 bottom fields are passable for dwarfs, flyers, building destroyers and diggers,
the top row and bottom right corner is passable only for building destroyers
we can use 64bit ints for 64 different kinds of movement. Now, if we want to get a path, we set a single bit in the bit mask and tell what kind of transport the creature wants and from where to where it wants to go.To optimise, we can additionaly set cost of movement for creature type, in the above example, we could get a additional cost maps looking like this:
Code: [Select]
dwarf:
2 2 2 2 2
2 0 0 0 2
2 5 5 5 2
2 5 5 0 2
2 2 2 2 2
flyer:
2 2 2 2 2
2 0 0 0 2
2 2 2 2 2
2 2 2 0 2
2 2 2 2 2
building destroyer:
2 2 2 2 2
2 10 10 10 2
2 2 2 2 2
2 2 2 10 2
2 2 2 2 2
the upside of this approach is that the algorithm doesn't know that 0x01 is dwarf and 0x02 are flyers, it's completely arbitral and thus portable. By using bitfields instead of ints we can get unlimited number of movement types.
miasma and shallow water may be represented by higher path cost.
Resource maps
The second thing is finding nearest x-thing, to do this, we need to know which square contains which resources.
To do this, we can get ID-s (32bit ints should be enough) and squares they are on.
For updates simple add() and substract() should suffice:
Code: [Select]
typedef coord_t int32_t;
add(coord_t[3] square, int ID, int count);
substract(coord_t[3] square, int ID, int count);
optionally functions for mass adding and removingagain, we not need to know that ID=10 are obsidian rocks, ID=14 are plump helmets and ID=20 are goblin axemen.
so, to find closest stuff the interface would look something like this:
Code: [Select]
coord_t[3] findID(coord_t[3] starting_point, movement_t bit_mask, int ID);
Interface
I think, that the functions qwertyuiopas mentioned:
Code: [Select]
movementVectors* GetFullPath(movement_type, beginningxyz, endxyz);
movementVector GetStep(movement_type, beginningxyz, endxyz);
movementVectors* GetNSteps(movement_type, beginningxyz, endxyz);
have to be included.GetLOSSteps() is optional, GetShortestPathLength() is findID()
Beside them, we need:
Code: [Select]
setDefaultMovementPenalty(movement_cost[32]) - sets default movement cost for different movement types
updateMapData(cubestartxzy, movement_mask[][][]) - takes cube of space, it's starting field and puts it to the movement graph with fields being movement masks
updateMapMovementPenalty(cubestartxyz, cost_type, movement_costs[][][]) - takes movement type and cube of space with movement cost as filelds in the cube
all coordinates should be signed to enable update only of parts of world and non uniform world.
Post by: bartavelle on November 27, 2009, 08:38:09 am
* this seem like a great idea
* it would be cool to have the first post link to something where the current work is. It looks that there is already stuff done (i just read the last 2 pages), but i'm not willing to parse 20 pages just to find it :/
* did anyone think about penalties for path going through creatures ? this is really important in DF
And for a not so great contribution : for the people who wonder how to cut the map in zones that would reduce the quantity of nodes to take into account, there are already others who solved this.
http://en.wikipedia.org/wiki/Binary_space_partitioning
Post by: Starver on November 27, 2009, 09:00:40 am
* did anyone think about penalties for path going through creatures ? this is really important in DF
Yesno* (*delete as inapplicable). When planning an arbitrarily long route, any creature you might want to penalise moving through (over/under/charge through with battleaxe whirring like a rotary mower blade) is almost certainly not going to be there when you get there.
It may be in the same length of tunnel, of course, and I suppose it could be factored in as a 'smeared-out' movement penalty if chained down in a tunnel (1/3rd penalty over each of three adjacent tiles, or 3x1/9th penalty if a 3-wide tunnel with it in the centre, etc) but that's a more complex thing to deal with than potential presence/absence of bridge/barrier by the time your agent finds its way to a transient pinch-point/dynamic landscape feature of some other kind.
OTOH, factoring in the level of traffic a corridor currently has (not tying down to any particular creature, but using the current creature occupation/transit details through a zone to discourage use of that zone) might result in intelligent (and 'SatNav with realtime updates'-like) traffic re-routing options that reduces such congestion.
And (of course) hostiles, the presence of least-favourite-vermin and miasmic outbreaks could modify the decision tree while searching for a path, to weight against certain pathing areas. That may already be a considered point. (Certainly a lot of the external links have this mentioned, so I suppose those working on this more actively/collaboratively have it in mind, even if it's missing from my own testbed, which currently is 'agent and landscape only'.)
Post by: shadow_slicer on November 27, 2009, 11:18:52 am
First thing, I don't think that the callback mechanism for detecting passability for tiles is a good idea - remember, the very paths we look for are directly influenced by it and to calculate it we would have to run it thousands of times. As such IMO the single tile in the map data that the library gets should be a bit field with each bit setting passability for certain type of transport
The problem is that you don't have to have a single bitvector for each tile, you have to have a bitvector for all 27 possible directions from the tile! This is necessary to implement stairways, adjacent floors, ramps, etc. without nasty special cases. Then we also need separate costs for each movement type, creature size and everything else (possibly for each direction!).
I don't see a way of storing all that information without using gobs of memory. But if we simply use a call-back function, the client can calculate that information procedurally. This function doesn't have to be slow. Odds are if Toady implemented one for DF it would probably only consist of some modulo arithmetic, bit tests, and table lookups -- all of which is pretty fast.
Quote
Resource mapsI don't think something like this belongs in a general purpose pathing library. I think it should be enough if we have something like
Code: [Select]
cost_t estimatePathCost(Point source, Point destination)
which finds a reasonable estimate of the cost of traveling between two points without actually calculating the path. This could be implemented by finding the path cost between the zones containing the source and destination (which is cached).Then the client can loop over all the items they want to consider and choose the shortest themselves.
Quote
InterfaceI don't think that these are really necessary at all. They can all be implemented by successive calls to GetStep or whatever. In fact if we implement GetStep as an STL-compatible iterator instead of as a function, we can use STL methods to implement each of them in a single function call. This means that they don't really add anything useful and complicate the interface. Remember KISS (http://en.wikipedia.org/wiki/KISS_principle)!
I think, that the functions qwertyuiopas mentioned:Code: [Select]movementVectors* GetFullPath(movement_type, beginningxyz, endxyz);have to be included.
movementVector GetStep(movement_type, beginningxyz, endxyz);
movementVectors* GetNSteps(movement_type, beginningxyz, endxyz);
GetLOSSteps() is optional, GetShortestPathLength() is findID()
Post by: numerobis on November 27, 2009, 02:31:05 pm
- iterate over the edges out of a vertex
- cost of an edge
- admissable heuristic for the distance between two vertices
At the moment, it doesn't seem like we need anything more for any of the extensions we've come up with.
It shouldn't be our job to determine how the graph is stored internally; anything that supports the interface should be acceptable.
What a game needs of our pathfinder is probably just:
- compute a path from s to t, returning a forward iterator over edges
Post by: tomato on November 27, 2009, 04:04:19 pm
First thing, I don't think that the callback mechanism for detecting passability for tiles is a good idea - remember, the very paths we look for are directly influenced by it and to calculate it we would have to run it thousands of times. As such IMO the single tile in the map data that the library gets should be a bit field with each bit setting passability for certain type of transport
The problem is that you don't have to have a single bitvector for each tile, you have to have a bitvector for all 27 possible directions from the tile! This is necessary to implement stairways, adjacent floors, ramps, etc. without nasty special cases. Then we also need separate costs for each movement type, creature size and everything else (possibly for each direction!).
Every direction, for every movement type!? What for!? If two adjectant tiles are passable for creature of type 0x01, then it can move there. Add ceiling as an additional level (with whatever cost for movement - or 0 for DF) and we have all directions covered. It would be nice to have a path finder this complicated, but for a game like DF, where maps have thousands of tiles already (and we will get additional z-levels in couple of weeks/months making it tens/houdreds of thousands) with houndreds of pathfinding entities and we simply can't allow ourselfs the privilege of such compexity.
Spoiler (click to show/hide)
As for movement/cost - in 90% of cases the penalty will be exacly the same for each and every tile, movement cost different than default will be the exception, I see no problem with that (you just keep second 3d table with pointers for data about non default values - pointer=NULL - defaults).
Quote
I don't see a way of storing all that information without using gobs of memory. But if we simply use a call-back function, the client can calculate that information procedurally. This function doesn't have to be slow. Odds are if Toady implemented one for DF it would probably only consist of some modulo arithmetic, bit tests, and table lookups -- all of which is pretty fast.still, pushing a 3D table through a single function call will be much faster, and this doesn't inform the pathfinder when the passability is invalidated. So we would have to run it over each and every field when map change occures. So if we have to get this data one way or another, why not force the programmer to give it to us in the first place?
Quote
In what game that needs pathfinding you don't need to search for nearest enemy or resource?QuoteResource mapsI don't think something like this belongs in a general purpose pathing library.
Quote
I think it should be enough if we have something likeAnd that's the problem: the application has to loop, if it has no idea which, for example - a stone, is closest, it may run our function thousand times and it still won't find really the nearest one (or one even close to). There are many situations when stones that are near in cartesian space may well be most remote ones. (even the ones that are in a 20x20x20 cube) and for a 50x50x50 cube you get 125000 calls. "oops!" (simple situation - each and every level mined, mason on a level with a stair case in far corner, with stones near it, you get 50x50x49 stones to check and you still don't find the closest one - the one 26 fields south and you get exacly the same situation as now, only a bit later with much bigger computational cost)Code: [Select]cost_t estimatePathCost(Point source, Point destination)which finds a reasonable estimate of the cost of traveling between two points without actually calculating the path. This could be implemented by finding the path cost between the zones containing the source and destination (which is cached).
Then the client can loop over all the items they want to consider and choose the shortest themselves.
add to this the ability for player to select which exact materials the goods have to be made from and you get situation where a stupid legendary mason is a cause of milion of calls a second to our function
Quote
But this doesn't allow for the programmer to not make the creatures onmniscient - creature gets a path (it is saved in the creature), it tries to follow it, if it can't return to it in, let's say 4 steps (to allow passing other creatures in corridors) using simple A*(because a door it just got to is closed or cavern collapsed), it only then asks for another one.QuoteInterfaceI don't think that these are really necessary at all. They can all be implemented by successive calls to GetStep or whatever. In fact if we implement GetStep as an STL-compatible iterator instead of as a function, we can use STL methods to implement each of them in a single function call. This means that they don't really add anything useful and complicate the interface. Remember KISS (http://en.wikipedia.org/wiki/KISS_principle)!
I think, that the functions qwertyuiopas mentioned:Code: [Select]movementVectors* GetFullPath(movement_type, beginningxyz, endxyz);have to be included.
movementVector GetStep(movement_type, beginningxyz, endxyz);
movementVectors* GetNSteps(movement_type, beginningxyz, endxyz);
GetLOSSteps() is optional, GetShortestPathLength() is findID()
But this is not as important as the way the map data is imported to the library and later updated
Post by: Impaler[WrG] on November 27, 2009, 04:14:05 pm
Quote
The problem is that you don't have to have a single bitvector for each tile, you have to have a bitvector for all 27 possible directions from the tile! This is necessary to implement stairways, adjacent floors, ramps, etc. without nasty special cases. Then we also need separate costs for each movement type, creature size and everything else (possibly for each direction!).
Yes we can have a terrain type bitvector for each tile AND a separate bitvector for connectivity to adjacent tiles. To establish that their is an edge between two tiles first check the terrains are both compatible with the movement type which should be a simple &. Then determine the direction type for one tile relative to the other, north for example and & that with the connectivity, then do the same for the other tile.
This will work for anything except movement that can pass though walls and floors, people have long theorized such a digger type ability. This would probably be best handled with a different system.
As for edge costs, because were assuming a grid system the costs are going to be attached to the tiles and the edge cost is either the value from the tile being moved into or the sum of half the in and out tiles. It should be a simple switch to change the behavior as well.
Post by: tomato on November 27, 2009, 04:29:08 pm
This will work for anything except movement that can pass though walls and floors, people have long theorized such a digger type ability. This would probably be best handled with a different system.
My system solves it - a tile can be passable for a digger and be impassable for every other creature type (still using the bits I showed earlier) a tile with passability mask of 0x20
Post by: Footkerchief on November 27, 2009, 04:49:21 pm
QuoteIn what game that needs pathfinding you don't need to search for nearest enemy or resource?QuoteResource mapsI don't think something like this belongs in a general purpose pathing library.
1. Determining a destination is a separate task from finding a path, and should be treated as such.
2. A general purpose pathfinding library can't determine the destination on its own, because it's not necessarily just "nearest." It can involve lots of game logic, creature preferences, weird item requirements for jobs, anything.
Post by: Impaler[WrG] on November 27, 2009, 04:50:54 pm
Quote
Add ceiling as an additional level (with whatever cost for movement - or 0 for DF) and we have all directions covered.
I wouldn't want to treat the z axis differently but applying the same logic to each axis it would simply things a good deal, and I already have a very similar system in Khazad, but I'm not sure if it would handle diagonal movement well.
Post by: tomato on November 27, 2009, 05:19:40 pm
Then what would be your solution for a lone stonecrafter on a level with stones that are as far as a ladder to higher and lower levels that are full of stone?1. Determining a destination is a separate task from finding a path, and should be treated as such.QuoteIn what game that needs pathfinding you don't need to search for nearest enemy or resource?QuoteResource mapsI don't think something like this belongs in a general purpose pathing library.
Quote
2. A general purpose pathfinding library can't determine the destination on its own, because it's not necessarily just "nearest." It can involve lots of game logic, creature preferences, weird item requirements for jobs, anything.if you have only few items that are very specific then you can easly iterate over them, ask for path length to each and every one of them and this way find the nearest one.
Problem starts when you have hundreds or thousands of items that are exacly the same - then the engine can't easly decide which is nearest (as with our lone stonecrafter).
Post by: shadow_slicer on November 27, 2009, 06:05:19 pm
Every direction, for every movement type!? What for!? If two adjectant tiles are passable for creature of type 0x01, then it can move there. Add ceiling as an additional level (with whatever cost for movement - or 0 for DF) and we have all directions covered.If you add an extra level to represent the ceiling, then you can't support vertical diagonal movement, which is already used for things such as ramps (or later jumping off of cliffs).
Quote
As for movement/cost - in 90% of cases the penalty will be exacly the same for each and every tile, movement cost different than default will be the exception, I see no problem with that (you just keep second 3d table with pointers for data about non default values - pointer=NULL - defaults).While it's true that the penalty is likely to be the same for each tile, keeping a second table doesn't save anything. A 32 bit pointer is just as big as a 32 bit float.
Quote
still, pushing a 3D table through a single function call will be much faster, and this doesn't inform the pathfinder when the passability is invalidated. So we would have to run it over each and every field when map change occures. So if we have to get this data one way or another, why not force the programmer to give it to us in the first place?Pushing the table through would not be faster. Unconditional function calls are dirt cheap. Modern CPU's can execute them without even a single cycle of extra delay. Copying and maintaining a working set that does not even fit in L3 cache is expensive. With the function call, we have less memory overhead so we displace much less of the client's working set out of cache.
And that's still not mentioning the size of the table! If you decide to devote 8 bits for each tile (not for each edge, each tile!) just to determine is this tile passable by movement type x, a 256x256x16 map would require 1MB of memory. If we add ceilings, it goes to 2MB. Note that this is a really small map. A 6x6 I am currently using is 288x288x32, which is 2.5 or 5 MB. Note that this does not include any cost information, and only supports 8 movement types. If you add terrain types or other features then this only gets worse. And, as you said the number of tiles on the map is only going to increase. If you get up to hundreds of z-levels with larger embark sizes, things get nasty very quickly.
Quote
In what game that needs pathfinding you don't need to search for nearest enemy or resource?Honestly, I can think of quite a few. But not to dismiss your point, I understand what you're saying, and I admit that the problems seem quite coupled from a certain point of view. We definitely need to take into consideration the needs of users, but I feel that if we choose a simpler interface and let the client glue together the pieces he needs, the library will be much more flexible, robust and faster.
Quote
And that's the problem: the application has to loop, if it has no idea which, for example - a stone, is closest, it may run our function thousand times and it still won't find really the nearest one (or one even close to). There are many situations when stones that are near in cartesian space may well be most remote ones. (even the ones that are in a 20x20x20 cube) and for a 50x50x50 cube you get 125000 calls. "oops!" (simple situation - each and every level mined, mason on a level with a stair case in far corner, with stones near it, you get 50x50x49 stones to check and you still don't find the closest one - the one 26 fields south and you get exacly the same situation as now, only a bit later with much bigger computational cost)First of all, loops are not slow --especially small repeated loops such as iterating over a list -- even if they have unconditional function calls. Everything is already in L1 and branch prediction will be near perfect, so this should be rather fast.
Second, once you have your interface, how would your method solve it? Would you do a breadth first search of the entire graph until at least one of the items is found? We could just as easily do this without storing the locations of items. Just provide a method:
Code: [Select]
Path EstimateCostToNearest(Point Start, Point[] Targets)
Quote
That's also not true. However much information you want the creature to have should be reflected in the map you provide to the pathfinder. If you don't provide omniscient information, then the creatures won't be omniscient. It's up to the client to decide what they want.QuoteI don't think that these are really necessary at all. They can all be implemented by successive calls to GetStep or whatever. In fact if we implement GetStep as an STL-compatible iterator instead of as a function, we can use STL methods to implement each of them in a single function call. This means that they don't really add anything useful and complicate the interface. Remember KISS (http://en.wikipedia.org/wiki/KISS_principle)!But this doesn't allow for the programmer to not make the creatures onmniscient - creature gets a path (it is saved in the creature), it tries to follow it, if it can't return to it in, let's say 4 steps (to allow passing other creatures in corridors) using simple A*(because a door it just got to is closed or cavern collapsed), it only then asks for another one.
Edit: fix quote tags
Post by: Footkerchief on November 27, 2009, 06:35:01 pm
Then what would be your solution for a lone stonecrafter on a level with stones that are as far as a ladder to higher and lower levels that are full of stone?
if you have only few items that are very specific then you can easly iterate over them, ask for path length to each and every one of them and this way find the nearest one.
Problem starts when you have hundreds or thousands of items that are exacly the same - then the engine can't easly decide which is nearest (as with our lone stonecrafter).
I see what you're getting at, but the scheme you described:
Code: [Select]
coord_t[3] findID(coord_t[3] starting_point, movement_t bit_mask, int ID);is not going to work, if ID simply refers to "object type." Maintaining the client-side index would be extremely messy (does a given number refer to enemy goblins, or enemy goblin archers, or all goblin archers? do left-handed redheads get their own ID? what about thrones decorated with platinum but not with steel?) and it would be awkward for the client to request the nearest of an inhomogeneous set of objects.
There's no reason for the pathfinding library to care about object IDs at all. It's much cleaner for the client to simply pass in a set of candidate destination coordinates and for the library to find the nearest of those coordinates and pathfind to it. Obviously the client should first do some trivial pruning of the coordinate set.
Post by: CobaltKobold on November 27, 2009, 06:51:56 pm
And remember the dwarven pathfinding truism: sock.value>>elephant.cost
The problem is that you don't have to have a single bitvector for each tile, you have to have a bitvector for all 27 possible directions from the tile! This is necessary to implement stairways, adjacent floors, ramps, etc. without nasty special cases. Then we also need separate costs for each movement type, creature size and everything else (possibly for each direction!).You could....take the wordsright out of my mouth.
I don't see a way of storing all that information without using gobs of memory. But if we simply use a call-back function, the client can calculate that information procedurally. This function doesn't have to be slow. Odds are if Toady implemented one for DF it would probably only consist of some modulo arithmetic, bit tests, and table lookups -- all of which is pretty fast.
26 directions though. When is it ever impermissible for a creature to move to where it already is? Also...I am fairly sure that flying or swimming are required to make a move to up-and-direction (without ramp, natch)
I'd suggest some manner of preclusions,
I see why DF is loath to path fliers/swimmers right, now...for walkers, it's dead easy to see where you can go- up if on(up stair || up/down stair) && up(down stair || up/down stair) up+direction if on(ramp) && direction(wall), direction if directionIsFloored, and the downs are inverse of the ups (...actually, it won't check for the wall, as you're standing on it, leading to one-way ramps.)
Post by: Footkerchief on November 27, 2009, 07:04:01 pm
Footkerchief: So long as it's a 'pure' pf thing, yeah. But, eventually, pf = ai.
Even if that does happen, it makes sense just from a modularity standpoint to separate the concerns of generating a set of candidate objects and finding the nearest of them. It won't ever be practical to rely on an "object type ID" as a substitute for an arbitrary set of objects.
Post by: tomato on November 27, 2009, 07:06:08 pm
Ramps don't use vertical diagonal movement, you can't have such situation:Every direction, for every movement type!? What for!? If two adjectant tiles are passable for creature of type 0x01, then it can move there. Add ceiling as an additional level (with whatever cost for movement - or 0 for DF) and we have all directions covered.If you add an extra level to represent the ceiling, then you can't support vertical diagonal movement, which is already used for things such as ramps (or later jumping off of cliffs).
Code: [Select]
(cross section)
__#
##↑
ramps work just like stairsQuote
IMO you're a bit over estimating the capability of CPUs, but that's not the point.QuoteAs for movement/cost - in 90% of cases the penalty will be exacly the same for each and every tile, movement cost different than default will be the exception, I see no problem with that (you just keep second 3d table with pointers for data about non default values - pointer=NULL - defaults).While it's true that the penalty is likely to be the same for each tile, keeping a second table doesn't save anything. A 32 bit pointer is just as big as a 32 bit float.Quotestill, pushing a 3D table through a single function call will be much faster, and this doesn't inform the pathfinder when the passability is invalidated. So we would have to run it over each and every field when map change occures. So if we have to get this data one way or another, why not force the programmer to give it to us in the first place?Pushing the table through would not be faster. Unconditional function calls are dirt cheap. Modern CPU's can execute them without even a single cycle of extra delay. Copying and maintaining a working set that does not even fit in L3 cache is expensive. With the function call, we have less memory overhead so we displace much less of the client's working set out of cache.
Most of the map is static, you displace client cache once, when loading the data, not every time it's called. And the data has to be stored somewhere - if we want the library to be opaque it needs to duplicate the data
Quote
And that's still not mentioning the size of the table! If you decide to devote 8 bits for each tile (not for each edge, each tile!) just to determine is this tile passable by movement type x, a 256x256x16 map would require 1MB of memory. If we add ceilings, it goes to 2MB. Note that this is a really small map. A 6x6 I am currently using is 288x288x32, which is 2.5 or 5 MB. Note that this does not include any cost information, and only supports 8 movement types. If you add terrain types or other features then this only gets worse. And, as you said the number of tiles on the map is only going to increase. If you get up to hundreds of z-levels with larger embark sizes, things get nasty very quickly.since when 5MiB of memory for such huge tool is much?! We're trying to trade in memory for cycles, 200MiB would be nothing!
and a 200x400x200 map with 32 movement types and 2 non standard penalty values per tile on average would take only about 244MiB, and first: that's a huge map, second: the amount of other game info that's needed will still dwarf it.
Remember, I told that the pathfinding world doesn't need to be uniform, the engine has to push to the lib only the fragments that have known resources or pathfinding creatures
Quote
i said nothing of such sort, i know that loop is faster than unrolled loop on current CPUs, that was not the pointQuoteIn what game that needs pathfinding you don't need to search for nearest enemy or resource?Honestly, I can think of quite a few. But not to dismiss your point, I understand what you're saying, and I admit that the problems seem quite coupled from a certain point of view. We definitely need to take into consideration the needs of users, but I feel that if we choose a simpler interface and let the client glue together the pieces he needs, the library will be much more flexible, robust and faster.QuoteAnd that's the problem: the application has to loop, if it has no idea which, for example - a stone, is closest, it may run our function thousand times and it still won't find really the nearest one (or one even close to). There are many situations when stones that are near in cartesian space may well be most remote ones. (even the ones that are in a 20x20x20 cube) and for a 50x50x50 cube you get 125000 calls. "oops!" (simple situation - each and every level mined, mason on a level with a stair case in far corner, with stones near it, you get 50x50x49 stones to check and you still don't find the closest one - the one 26 fields south and you get exacly the same situation as now, only a bit later with much bigger computational cost)First of all, loops are not slow --especially small repeated loops such as iterating over a list -- even if they have unconditional function calls. Everything is already in L1 and branch prediction will be near perfect, so this should be rather fast.
the point is repetitive calling of non tryvial function. As I demonstrated, for a 50 tile cube a legendary mason will cause 1 million calls per second to the function, if the function returns in 100 cycles (that's fast) we get a 10% CPU usage on a 2GHz CPU. And that's a single mason, for a non-optimal algorithm
Quote
Second, once you have your interface, how would your method solve it? Would you do a breadth first search of the entire graph until at least one of the items is found? We could just as easily do this without storing the locations of items. Just provide a method:For starters it would search the item on accessible areas, simple 3D floodfill would be better than the current DF implementation.Code: [Select]Path EstimateCostToNearest(Point Start, Point[] Targets)
Then we could optimise for stuff that's in abundance - by storing additional info whatever an item of such and such ID is in reduced movement graph of the map.
Besides I'm suggesting an interface, not a solution. When the library has full picture of the situation, we can optimise the whole situation, not just find local minima.
Quote
it was just a simple use case of the API, not the only way to use it.QuoteThat's also not true. However much information you want the creature to have should be reflected in the map you provide to the pathfinder. If you don't provide omniscient information, then the creatures won't be omniscient. It's up to the client to decide what they want.QuoteI don't think that these are really necessary at all. They can all be implemented by successive calls to GetStep or whatever. In fact if we implement GetStep as an STL-compatible iterator instead of as a function, we can use STL methods to implement each of them in a single function call. This means that they don't really add anything useful and complicate the interface. Remember KISS (http://en.wikipedia.org/wiki/KISS_principle)!But this doesn't allow for the programmer to not make the creatures onmniscient - creature gets a path (it is saved in the creature), it tries to follow it, if it can't return to it in, let's say 4 steps (to allow passing other creatures in corridors) using simple A*(because a door it just got to is closed or cavern collapsed), it only then asks for another one.
Edit: fix quote tags
As I suggested - that's the interface for stuff that's in the hundreds or thousands, not single items.Then what would be your solution for a lone stonecrafter on a level with stones that are as far as a ladder to higher and lower levels that are full of stone?
if you have only few items that are very specific then you can easly iterate over them, ask for path length to each and every one of them and this way find the nearest one.
Problem starts when you have hundreds or thousands of items that are exacly the same - then the engine can't easly decide which is nearest (as with our lone stonecrafter).
I see what you're getting at, but the scheme you described:Code: [Select]coord_t[3] findID(coord_t[3] starting_point, movement_t bit_mask, int ID);
is not going to work, if ID simply refers to "object type." Maintaining the client-side index would be extremely messy (does a given number refer to enemy goblins, or enemy goblin archers, or all goblin archers? do left-handed redheads get their own ID? what about thrones decorated with platinum but not with steel?) and it would be awkward for the client to request the nearest of an inhomogeneous set of objects.
There's no reason for the pathfinding library to care about object IDs at all. It's much cleaner for the client to simply pass in a set of candidate destination coordinates and for the library to find the nearest of those coordinates and pathfind to it. Obviously the client should first do some trivial pruning of the coordinate set.
For items that are uncommon, a function that takes all coordinates of them and returns the closest one would be better.
Post by: CobaltKobold on November 27, 2009, 07:17:10 pm
Post by: Footkerchief on November 27, 2009, 07:20:47 pm
As I suggested - that's the interface for stuff that's in the hundreds or thousands, not single items.
For items that are uncommon, a function that takes all coordinates of them and returns the closest one would be better.
Ah, okay. That makes more sense.
Post by: numerobis on November 27, 2009, 08:11:12 pm
What data would need to be replicated?
Post by: shadow_slicer on November 27, 2009, 09:27:54 pm
Ramps don't use vertical diagonal movement, you can't have such situation:Are you sure about that? Because I've tested this on 40d15. No creature is ever on the top floor of a ramp. They all travel horizontally as well as vertically. Even wagons from trade caravans skip the top floor. If you put a dog on a chain next to a ramp, he can be on the top floor diagonal from the ramp, but never on the top floor of a ramp. If this is an item of dispute, I can take a screen cap.Code: [Select](cross section)ramps work just like stairs
__#
##↑
Quote
IMO you're a bit over estimating the capability of CPUs, but that's not the point.I think you're overestimating the memory/cache system of modern CPUs. If something is not in the cache, you're talking about a stall of hundreds of clock cycles. If you read any recent works on optimizing program execution speed, it's all about reducing the size of the working set and increasing locality.
Most of the map is static, you displace client cache once, when loading the data, not every time it's called. And the data has to be stored somewhere - if we want the library to be opaque it needs to duplicate the data.
Have you considered what sort of interface Toady currently uses for pathfinding? That's an important consideration if we ever hope he will ever consider adopting our library. I guarantee you he calculates the connectivity and path costs on demand. Separating that out to a function call from our library is one just additional instruction. Duplicating the data is tens to hundreds of megabytes.
Further, duplicating the data is entirely unnecessary, even for an opaque library. There are lots of opaque libraries that allow you to register callback functions. And that is all the getEdgeCost function is. Further this design adds significant flexibility: All we need is the costs from tile A to tile B. The client already stores this data and can calculate it easily. Why not let them decide whether they want to calculate it each time or store it. If you want, you can even write a wrapper around this API that stores its own copy of that data (and we could even include it in the library).
Quote
since when 5MiB of memory for such huge tool is much?! We're trying to trade in memory for cycles, 200MiB would be nothing!5MiB is a huge amount when it's sacrificed for no perceivable gain. I wouldn't mind using 200MiB of path caches if it was able to significantly speed things up, but that's a ridiculous amount just to obtain basic functionality. If you read the literature and look at other pathfinding implementations, none of them duplicate the connectivity map. Some (modern ones) even talk about pushing the path-finding footprint below 500kiB! On that 200x400x200 map you mentioned, the other method would only require 32 bytes(!) to achieve the same thing. Your design just doesn't scale.
and a 200x400x200 map with 32 movement types and 2 non standard penalty values per tile on average would take only about 244MiB, and first: that's a huge map, second: the amount of other game info that's needed will still dwarf it.
The point is exactly that there is such a large amount of game information already, that we can't afford to add more. Each byte of memory in use by the pathing library is one that can't be used to improve the game.
Quote
Remember, I told that the pathfinding world doesn't need to be uniform, the engine has to push to the lib only the fragments that have known resources or pathfinding creaturesAnd how will the engine know which portions of the map are relevant to pathfinding? The library is supposed to be opaque.
Quote
A 3D flood-fill, also known as a breadth first search. You could also use a multi-destination version of A* (though you'd need to be careful to keep the fringe from getting too large). You could even use the cache to estimate the distance and only path the shortest ones. The point is that, for the vast majority of potential optimizations, *what* the item is, is irrelevant. I agree with you (and a basic tenant of Information Theory) that the more information you have, the better and faster your estimates are. The problem is that speed is not the only goal, and if you're going to spend your memory on caches to help speed up pathing, you're better off storing partial paths or connections between zones than you are storing the location of all blue socks on the map.QuoteSecond, once you have your interface, how would your method solve it? Would you do a breadth first search of the entire graph until at least one of the items is found? We could just as easily do this without storing the locations of items. Just provide a method:For starters it would search the item on accessible areas, simple 3D floodfill would be better than the current DF implementation.Code: [Select]Path EstimateCostToNearest(Point Start, Point[] Targets)
Then we could optimise for stuff that's in abundance - by storing additional info whatever an item of such and such ID is in reduced movement graph of the map.
Besides I'm suggesting an interface, not a solution. When the library has full picture of the situation, we can optimise the whole situation, not just find local minima.
Post by: qwertyuiopas on November 27, 2009, 10:20:52 pm
That WOULD be a meaningful trade between memory and CPU usage.
Post by: numerobis on November 28, 2009, 12:30:27 pm
What A* needs of the graph is:
- iterate over the edges out of a vertex
- cost of an edge
- admissable heuristic for the distance between two vertices
At the moment, it doesn't seem like we need anything more for any of the extensions we've come up with.
It shouldn't be our job to determine how the graph is stored internally; anything that supports the interface should be acceptable.
What a game needs of our pathfinder is probably just:
- compute a path from s to t, returning a forward iterator over edges
I guess people are also thinking we want:
- compute a path from s to the nearest node in T, where T is a small set
- compute a path from s to the nearest node that matches the predicate p
The last one doesn't support A*, but we can still do Dijkstra's algorithm (or BFS, but I'm not convinced that's really any better).
There's gnashing of teeth about function call overhead. If it's a template library that we're building, the use of a function object is usually inlined, so it's not a problem.
Post by: PencilinHand on November 28, 2009, 01:36:52 pm
Footkerchief: So long as it's a 'pure' pf thing, yeah. But, eventually, pf = ai.
And remember the dwarven pathfinding truism: sock.value>>elephant.costThe problem is that you don't have to have a single bitvector for each tile, you have to have a bitvector for all 27 possible directions from the tile! This is necessary to implement stairways, adjacent floors, ramps, etc. without nasty special cases. Then we also need separate costs for each movement type, creature size and everything else (possibly for each direction!).You could....take the wordsright out of my mouth.
I don't see a way of storing all that information without using gobs of memory. But if we simply use a call-back function, the client can calculate that information procedurally. This function doesn't have to be slow. Odds are if Toady implemented one for DF it would probably only consist of some modulo arithmetic, bit tests, and table lookups -- all of which is pretty fast.
26 directions though. When is it ever impermissible for a creature to move to where it already is? Also...I am fairly sure that flying or swimming are required to make a move to up-and-direction (without ramp, natch)
I'd suggest some manner of preclusions,but I already found a corner case I'd like preserving: swimming up waterfalls.solvable if swimming is permitted to go up.
I see why DF is loath to path fliers/swimmers right, now...for walkers, it's dead easy to see where you can go- up if on(up stair || up/down stair) && up(down stair || up/down stair) up+direction if on(ramp) && direction(wall), direction if directionIsFloored, and the downs are inverse of the ups (...actually, it won't check for the wall, as you're standing on it, leading to one-way ramps.)
I could think of one or two but they are fringe cases that are closer to a game mechanic than pathfinding.
EDIT: Here is a hint (http://www.bay12games.com/forum/index.php?topic=45390.0).
Post by: Impaler[WrG] on November 28, 2009, 03:15:37 pm
If we wanted to get more sophisticated I'd go with a zone tagging scheme, the client (main game) can tag zones as having some abstract quality and the server will do a hierarchical search for the closest such zone. If the placement of things like items sets a value in the zone then you can get a rough idea of ware the desired items are, even if the categories are broad and further checks need to be made it still rules out the vast majority of the map.
The idea of using a global list for rare items is also an excellent idea, a heuristic can be run on each one and the best one selected. Its not as efficient when a great number of items are present in which case the breath-first search would be faster. Doing the right kind of search for each request will be key.
Post by: numerobis on November 28, 2009, 03:41:09 pm
The idea of using a global list for rare items is also an excellent idea, a heuristic can be run on each one and the best one selected. Its not as efficient when a great number of items are present in which case the breath-first search would be faster. Doing the right kind of search for each request will be key.Actually, I don't see why the pathfinder should be in the business of keeping that list. I suggest two possibilities:
1. let the caller designate a range (begin and end iterators). In each step, when computing the heuristic, A* will take the min over all destinations.
2. Or let the caller designate a heuristic function and a success function. A* will call the heuristic, not knowing the destination; the game will take min over the destinations if it has a set of destinations, or just compute the single heuristic for one destination, or return 0 if it has no destination in mind.
Formulation 2 means we can implement a single A* routine and have it handle the case of not knowing where we actually want to go (in other words, we'd be running Dijkstra's algorithm).
BFS is a bit faster than Dijkstra's, but I feel that Dijkstra's is more the answer that people would expect. Sure, we can provide BFS as another routine.
Post by: Impaler[WrG] on November 28, 2009, 05:49:24 pm
As for ware a global item list would be stored, a client side store would probably be better and is indeed what I had in mind. The client loops over it and calls to the the server to get the distance.
Post by: numerobis on November 28, 2009, 06:04:18 pm
It is a waste to do:
Code: [Select]
for each destination, A*(src, dest)Rather than:
Code: [Select]
A* (src, heuristic, success) where heuristic and success loop over the destinations.The former will spend lots of time looking for the best path to the very far away destination, whereas the latter will prune the search and stop as soon as it's found the closest one.
Post by: CobaltKobold on November 28, 2009, 06:54:32 pm
I could think of one or two but they are fringe cases that are closer to a game mechanic than pathfinding.And yet...when it's doing that, it won't be looking for paths, so I think it can safely be ignored for groundlings?
EDIT: Here is a hint (http://www.bay12games.com/forum/index.php?topic=45390.0).
I do wonder if there is any way for a [FLIER] to recover from projectile status more than others. but tht's off-topic.
Post by: PencilinHand on November 29, 2009, 01:22:31 am
I could think of one or two but they are fringe cases that are closer to a game mechanic than pathfinding.And yet...when it's doing that, it won't be looking for paths, so I think it can safely be ignored for groundlings?
EDIT: Here is a hint (http://www.bay12games.com/forum/index.php?topic=45390.0).
I do wonder if there is any way for a [FLIER] to recover from projectile status more than others. but tht's off-topic.
In response to your question, it is likely that Toady would handle all potential situations such as that(or something like a bird losing its wings while flying and fighting) with the physics engine. That is why I said it was closer to a game mechanic than relevant to pathfinding but could give cause for a pathable subject to attempt to path from a position which it should not be able occupy(due to a bug, basically).
---------
My mind keep wandering back to how pathfinding through a hierarchy of zones would work and the most economical and useful methods for forming the hierarchy. Toward that end, I do have a question for shadow_slicer.
1. Is it a necessity or a convenience there for the divisions in the example some pages back to be base 2 or a divisible of 2?
2. How are zones ensured to be useful without potential misdirection?
Spoiler (click to show/hide)
I have tried to follow the whole thread but I think some of it went over my head due to jargon and I haven't looked into the specific code. So, I am wondering how the more inconvenient cases are handled.
Post by: shadow_slicer on November 29, 2009, 01:29:46 pm
1. Is it a necessity or a convenience there for the divisions in the example some pages back to be base 2 or a divisible of 2?Convenience. With everything evenly divisible things just work out nicely in my testing scripts. I could have used 3 instead of 2 just as easily.
Quote
2. How are zones ensured to be useful without potential misdirection?Zones, (at least in the current gridzone) just contain edge nodes (nodes on the edge of the zone that can lead to other zones). These edge nodes cache paths to all the edge nodes in their zone. The hierarchical A*, basically is just A* over these edge nodes. Of course we also need to add edge nodes for the source and destination temporarily. Since all the edge nodes are used and the path costs come directly from the original graph, this preserves optimality.Spoiler (click to show/hide)
Post by: Starver on November 30, 2009, 08:20:09 am
26 directions though. When is it ever impermissible for a creature to move to where it already is?My internal objection to the original figure was that "staying where one is" is never a viable movement option.
Although, I then wondered, could dynamic landscapes ever develop to the point where shifting tiles (elevators (especially paternosters), 'tramway tiles' and man(/dwarf)-rated conveyors, Thunderbird 1 entry styled bridges, even Hogwarts moving bridge/stairways) could reasonably be expected to introduce the "wait until safe to board/alight" movement.
Of course, then pathing would be far more complex. We are already faced with the near impossibility of navigating "magma-curtain" features (how would the pathing feature know that travelling onto a dwarf-activated pressure pad mean that a viable path will open after within X cycles and close again within Y cycles[1], but otherwise be closed), but then that's a whole different future problem. :)
[1] Never mind complex interactions of other dwarfs/invaders/pets/cat-powered-RNG-pressure-plates upon any system.
Post by: Sunken on November 30, 2009, 09:51:49 am
2. How are zones ensured to be useful without potential misdirection?Zones, (at least in the current gridzone) just contain edge nodes (nodes on the edge of the zone that can lead to other zones). These edge nodes cache paths to all the edge nodes in their zone. The hierarchical A*, basically is just A* over these edge nodes. Of course we also need to add edge nodes for the source and destination temporarily. Since all the edge nodes are used and the path costs come directly from the original graph, this preserves optimality.Spoiler (click to show/hide)
Optimality is not preserved if there's only one node per edge, mind; indeed, hierarchical A* is not optimal.
Post by: shadow_slicer on November 30, 2009, 10:04:56 am
Optimality is not preserved if there's only one node per edge, mind; indeed, hierarchical A* is not optimal.In the current zoneManager implementation there are as many nodes per zone edge as there are tiles on the edge (eg. 1 node for each tile with at least one edge leaving the zone), so it is still optimal.
Post by: PencilinHand on November 30, 2009, 01:45:06 pm
Optimality is not preserved if there's only one node per edge, mind; indeed, hierarchical A* is not optimal.In the current zoneManager implementation there are as many nodes per zone edge as there are tiles on the edge (eg. 1 node for each tile with at least one edge leaving the zone), so it is still optimal.
So then, for example, in a simple open(no obstructions, no z axis considerations) 4x4 there are 12 nodes/tiles around the edge and each node caches a path to every other node in the same zone.
11+10+9+8+7+6+5+4+3+2+1=66 cached paths with an average length of ~3 for a simple open 4x4 zone. Is this right?
Post by: shadow_slicer on November 30, 2009, 02:07:36 pm
Optimality is not preserved if there's only one node per edge, mind; indeed, hierarchical A* is not optimal.In the current zoneManager implementation there are as many nodes per zone edge as there are tiles on the edge (eg. 1 node for each tile with at least one edge leaving the zone), so it is still optimal.
So then, for example, in an open(no obstructions, no z axis considerations) 4x4 there are 12 nodes/tiles around the edge and each node caches a path to every other node in the same zone.
11+10+9+8+7+6+5+4+3+2+1=66 cached paths with an average length of ~3 for an open 4x4 zone. Is this right?
The current version is both worse and better than that would imply. It's worse than that because the path cache is stored directionally (though only calculated once for each direction), so all nodes have their own copy. Additionally nodes have paths to their adjacent neighbors in other zones, so there would be a total of 168 cached paths, of average length ~2.14.
It's actually better than it seems because 1) we don't usually deal with zones smaller than 8x8, and 2) the cache is actually calculated on demand, so paths that aren't ever considered won't be calculated or stored in cache. Since the cache is filled on demand, a zone in a completely open area will probably only have to calculate a single path (if not yet cached) each call. Further this path would simply be a portion of the path non-hierarchical A* would have needed to calculate anyway.
Anyway, I'm thinking in the long run, we'll probably have to sacrifice optimality. The current implementation uses too much memory and is only twice as fast as A* for a typical load. We really need to use less memory and calculate paths faster, which is not really possible if we want to maintain optimality.
Post by: PencilinHand on November 30, 2009, 02:58:58 pm
The current version is both worse and better than that would imply. It's worse than that because the path cache is stored directionally (though only calculated once for each direction), so all nodes have their own copy. Additionally nodes have paths to their adjacent neighbors in other zones, so there would be a total of 168 cached paths, of average length ~2.14.
It's actually better than it seems because 1) we don't usually deal with zones smaller than 8x8, and 2) the cache is actually calculated on demand, so paths that aren't ever considered won't be calculated or stored in cache. Since the cache is filled on demand, a zone in a completely open area will probably only have to calculate a single path (if not yet cached) each call. Further this path would simply be a portion of the path non-hierarchical A* would have needed to calculate anyway.
Anyway, I'm thinking in the long run, we'll probably have to sacrifice optimality. The current implementation uses too much memory and is only twice as fast as A* for a typical load. We really need to use less memory and calculate paths faster, which is not really possible if we want to maintain optimality.
I had a feeling it was going to be something like that.
I do have an idea that might keep perfect to near perfect optimality by carefully managing and taking advantage of zone definitions. Assuming the idea can work at all and isn't completely baseless with regards to CPU and memory considerations. ???
Let us set that aside for the moment, though.
------
Aside from the inter/intra zone node caching, how are the paths cached? What I mean is, are zone meta-paths cached or are the semi-explicit tile paths cached?
-------
I am sorry for these odd questions. I have coded in C++, Java, and different flavors of Basic before but don't have the time(read as the willpower to learn more about C++) to really dig into and wrap my head around code someone else wrote.
Post by: shadow_slicer on November 30, 2009, 03:16:44 pm
Aside from the inter/intra zone node caching, how are the paths cached? What I mean is, are zone meta-paths cached or are the semi-explicit tile paths cached?The only path caching currently implemented is between the border nodes on the edge.
Quote
I am sorry for these odd questions. I have coded in C++, Java, and different flavors of Basic before but don't have the time(read as the willpower to learn more about C++) to really dig into and wrap my head around code someone else wrote.It's fine, really. This is a really hard problem, and I had to think carefully to come up with even the imperfect prototype we have now. The more we talk about it, and the more we think about it, and the closer we come to a good solution.
Post by: atomfullerene on November 30, 2009, 07:46:33 pm
When is it ever impermissible for a creature to move to where it already is?I could think of one or two but they are fringe cases that are closer to a game mechanic than pathfinding.
Here's a good one: If you use a hatch to drop a dwarf on top of a tree, he'll just stand there 1 z-level above it, not climbing down or anything. But if you build a floor next to him on the same z-level, he'll walk right off onto it. But he won't ever walk onto a tree.
Post by: PencilinHand on November 30, 2009, 09:16:15 pm
Seriously, this is a bit dense to explain(there may be one or more MAJOR logical errors I am not seeing as well) so I am warning you to get comfortable before you start reading.
Suppose we have a 3x3 grid of tiles.
Spoiler (click to show/hide)
Continuity is guaranteed(that is that all pathable tiles are connected) within this grid if and only if the center tile is not blocked. This should be easy to check for.
Spoiler (click to show/hide)
If the center tile is blocked, and if any two of the four cardinal points are blocked then there may be a pathable tile that is not connected within the grid. This too should not be difficult to check for.
Spoiler (click to show/hide)
If this 3x3 grid were then split along the division to form an L and a 2x2 cube then they would be pathable within themselves. If the geometry ever changes(suppose the center becomes unblocked) then the whole grid could be checked and merged back together. A check for this should be relatively simple(knowing that 3x3 should start at xyz coordinates).
Of note: At most, a 3x3 grid could be divided into 4 2x2 cube(a sub-zone) with shared walls. So sub-zones would either need to share unpathable tiles or only include pathable ones.
Spoiler (click to show/hide)
As for connected zones and subzones(preferentially). It only be necessary to note if a zone or sub-zone is connected to its neighboring zones without having to calculate every possible connection. All that is important is that it is or isn't connected. So in the following example, 2 is only connected to 1 and 3 but the number of shared tiles doesn't matter.
Spoiler (click to show/hide)
All of this allows for the following assumption. Any given path through one zone or sub-zone to a neighboring zone or sub-zone on the same z-level will cost approximately the same**. So, each zone and subzone can be treated as a if it were an extra large tile itself. With this A* should be able to be run at any level of the hierarchy without resolving down to the explicit per-tile path. Resolve the path to the nearest desired zone size(suppose a 3x3 resolution) then path within each known zone that must be visit and reevaluate the overall meta-path periodically.
I haven't included stairs and ramps in this example specifically as it would confuse the attempted explanation, but they would be treated simply as more neighboring zones though the cost of moving through different z-levels would be scaled.
Under ideal circumstances, this should be able to resolve to perfect optimality. Under less ideal circumstances I think(I haven't proven this to myself) the Delta from perfect optimality should be bound and not deviate frequently nor significantly.
A further advantage that this offers is you can then cache whatever resolution meta-path is desired. Be it on the 3x3 or the 81x81 level.
This is, of course, assuming that the method described in this post is not only possible to implement but more CPU and memory efficient than current methods. Admittedly, I haven't fully thought out all possibilities and shortfalls so this there is probably a major hole in this proposal.
**This isn't necessarily true. The exception(weighing a single tile sub-zone as equal to a 3x3 zone), by my reckoning, should be relatively rare and the difference should be relatively minor on the macro scale.
------------
I hope somebody smarter than me actually reads all of this and gets the idea I am trying to communicate.
Here is some thoughts on zones and subzones.
Spoiler (click to show/hide)
Post by: shadow_slicer on December 01, 2009, 02:45:14 pm
I do have a few criticisms of your method:
Any given path through one zone or sub-zone to a neighboring zone or sub-zone on the same z-level will cost approximately the same **.This isn't really true. The path length through one of your 3x3 zones could be as little as 1, or as much as 5. This means that the path found can be more than twice the optimal path length. Additionally, I don't think your method supports non-uniform path costs.
Also, you're only reducing the graph by a factor of 3. The overhead from keeping track of the new abstract graph (and its subgraphs) is likely to be larger than any gains you have from abstraction. I don't see a way to easily increase the amount of abstraction, so I'm not sure we can solve this without significant changes.
Post by: Impaler[WrG] on December 01, 2009, 03:08:26 pm
Post by: PencilinHand on December 01, 2009, 04:08:36 pm
That's certainly an interesting idea. The subzone portion sound similar to the "Memory Efficient Pathfinding" (https://www.aaai.org/Papers/AIIDE/2007/AIIDE07-006.pdf) paper Impaler linked to earlier.
I do have a few criticisms of your method:Any given path through one zone or sub-zone to a neighboring zone or sub-zone on the same z-level will cost approximately the same **.This isn't really true. The path length through one of your 3x3 zones could be as little as 1, or as much as 5. This means that the path found can be more than twice the optimal path length. Additionally, I don't think your method supports non-uniform path costs.
Also, you're only reducing the graph by a factor of 3. The overhead from keeping track of the new abstract graph (and its subgraphs) is likely to be larger than any gains you have from abstraction. I don't see a way to easily increase the amount of abstraction, so I'm not sure we can solve this without significant changes.
I was hoping for criticism.
It is an idea, not sure about interesting but it is an idea. I haven't read that paper, yet, I will try to in the near future. I may have to just break down and write the code to test it so I can get it out of my head. At the least it will allow me to do something somewhat more productive compared to my more recent endeavors. Besides, it should be good for a few laughs if I ever finish it and make it public, so that is a bonus.
I did note that the assumption isn't always true(see the "**" note). My argument for the assumption is that exceptions would generally be rare and relatively minor. Generally not more than a difference of a few tiles in the whole path when the exception happens. It is a potentially unruly assumption, off hand I can think of one case especially. However, I think it is too important for the abstraction to work to do without it.
I think non-uniform pathcosts could be supported but would have to be generalized(so it wouldn't be exact to the tile) in a similar fashion to the zone/tile connection abstraction between hierarchy levels. In fact, it could be used to allow a user to optimize the most glaring exceptions.
I was thinking that layering the abstracting by 3x3 for each level(suppose 3 or 4 levels?) of a hierarchy. The memory overhead would be higher but it might significantly reduce the CPU and memory cost during the actual pathing. If you are feeling that it would merely balloon memory cost without noticeably reducing execution cost then it is likely to be a dead end.
You didn't mention the lack of consideration for novel modes of travel, like digging/destroying obstructions, swimming, flying etc. Or was that what you meant by the non-uniform cost comment?
For novel modes of travel, I would think that there would need to be a different method all together.
I think my posts are excessively wordy without improving clarification or communicating significantly(much like the idea, HA!).
Post by: zchris13 on December 01, 2009, 08:24:19 pm
Post by: Impaler[WrG] on December 02, 2009, 01:01:03 am
Post by: shadow_slicer on December 02, 2009, 02:40:55 am
Spoiler: Screenshot (click to show/hide)
Post by: Nexii Malthus on December 02, 2009, 06:22:40 am
Post by: PencilinHand on December 02, 2009, 12:23:55 pm
Trees still aren't fixed, but here's a screenshot of a path below ground.Spoiler: Screenshot (click to show/hide)
Simply awesome!
--------
EDIT:
I have thought more about what I was proposing and have done a little research on it. What if, instead of assuming exactly identical cost for all zones at the current hierarchy, what if A* were run one hierarchy level below what was being pathed over to generate a better approximate cost for a specific zone?
Post by: CobaltKobold on December 03, 2009, 11:27:22 am
Post by: Footkerchief on December 03, 2009, 11:35:01 am
Post by: CobaltKobold on December 03, 2009, 11:46:17 am
Post by: Draco18s on December 03, 2009, 12:40:56 pm
This isn't really true. The path length through one of your 3x3 zones could be as little as 1, or as much as 5.
6, actually:
Regions:
a b
c d
MetaPath: a -> b -> c
region layout:
###|...
###|...
###|.##
-------
..#|...
..#|##.
...|...
The path through region D costs 6.
65.
##3.5
12.
Post by: Nadaka on December 03, 2009, 12:53:07 pm
This isn't really true. The path length through one of your 3x3 zones could be as little as 1, or as much as 5.
6, actually:
Regions:
a b
c d
MetaPath: a -> b -> c
region layout:
###|...
###|...
###|.##
-------
..#|...
..#|##.
...|...
The path through region D costs 6.
65.
##3.5
12.
it actually doesn't. You can pass from position 5 to the next tile diagonally. You also count movement into and out of the area as movement through it, meaning the initial and final area have no movement cost. You should only count one of those two values for each area (it does not matter wich one, as long as it is consistant). The following diagram demonstrates the actual longest path through d.
##.|#..
##.|#..
##.|###
-------
###|...
..#|##.
...|...
edit:hehe, oops.
Post by: Footkerchief on December 03, 2009, 12:57:54 pm
##.|#..
##.|#..
##.|###
-------
###|...
..#|##.
...|...
Post by: PencilinHand on December 03, 2009, 02:12:35 pm
Footkerchief: So long as it's a 'pure' pf thing, yeah. But, eventually, pf = ai.
And remember the dwarven pathfinding truism: sock.value>>elephant.costThe problem is that you don't have to have a single bitvector for each tile, you have to have a bitvector for all 27 possible directions from the tile! This is necessary to implement stairways, adjacent floors, ramps, etc. without nasty special cases. Then we also need separate costs for each movement type, creature size and everything else (possibly for each direction!).You could....take the wordsright out of my mouth.
I don't see a way of storing all that information without using gobs of memory. But if we simply use a call-back function, the client can calculate that information procedurally. This function doesn't have to be slow. Odds are if Toady implemented one for DF it would probably only consist of some modulo arithmetic, bit tests, and table lookups -- all of which is pretty fast.
26 directions though. When is it ever impermissible for a creature to move to where it already is? Also...I am fairly sure that flying or swimming are required to make a move to up-and-direction (without ramp, natch)
I'd suggest some manner of preclusions,but I already found a corner case I'd like preserving: swimming up waterfalls.solvable if swimming is permitted to go up.
I see why DF is loath to path fliers/swimmers right, now...for walkers, it's dead easy to see where you can go- up if on(up stair || up/down stair) && up(down stair || up/down stair) up+direction if on(ramp) && direction(wall), direction if directionIsFloored, and the downs are inverse of the ups (...actually, it won't check for the wall, as you're standing on it, leading to one-way ramps.)
I almost hate to bring this up but the thought occurred to me.
It is actually 10 directions, for a dwarf in the current implementation at least and up to 18 for a flying creature. The reason it is 10 for a dwarf is when they are moving between z-levels at an angle(can only happen at ramps) they CAN NOT move in the direction of the ramp and stay on the same z-level. Also, they CAN ONLY chose to go either up or down a given ramp from one direction not both. This is because of the way that ramps are defined, they MUST have empty space on the z-level above them if they are going down. They MUST have a blocked tile on the same z-level if they are going up.
In the case of flying creatures, they can choose to either walk down the ramp or fly in the empty space on the z-level immediately above it.
To test this, construct a small wall(a 3x2 will work) and build a ramp in the middle of one of the long sides. Then deconstruct the single wall tile that the ramp is attached to. Then try to station someone on the wall. The test for down ramp is equally simple, just build a wall on the z-level above the ramp and watch the guy get stuck.
For a flying creature, it could be argued that there is no need for the extra eight slanting down paths. All pathfinding should be accomplishable for every possible creature with only 10 direction movement anyway because of the way ramps are defined in game.
Depending on the way Toady has defined the construction options and if he adds any, will depend on if more than 10 directions would be needed in the future.
Post by: Impaler[WrG] on December 03, 2009, 04:20:01 pm
Post by: qwertyuiopas on December 03, 2009, 06:52:23 pm
HOWEVER, 5 + same-level = unchanged, making the true number 26, or solving the "stay in place" question.
Post by: Draco18s on December 03, 2009, 08:37:11 pm
Quote
##.|#..
##.|#..
##.|###
-------
###|...
..#|##.
...|...
Ah, thanks for the fix resulting in what I'd computed. I'd originally had a distance of 7, that that too ignored diagonals.
Post by: qwertyuiopas on December 03, 2009, 09:45:16 pm
27 is numpad's 9 times 3(up, down, and same-z)
HOWEVER, 5 + same-level = unchanged, making the true number 26, or solving the "stay in place" question.
Clarification:
By
"solving the "stay in place" "
I meant that it would make not moving a "valid" direction, so such concerns wouldn't be a problem.
On the other hand, it might be a good idea that, if all 27 directions are included, you ensure it can't "explore" what happens if a creature doesn't move, how it affect's it's path. Some very possible infinite loops there, unless not moving is automatically discarded by the pathing, and that makes the pathing check for moving to the source tile from it utterly worthless...
Post by: alway on December 03, 2009, 10:48:21 pm
Start with your basic searching or whatever you decide is the fastest method. Then, record where dwarves move and create a seperate list for this. The first list will have all the world areas, and the second will have areas which dwarves have recently walked on. The second list will have some sort of counters assigned to the areas which will count down, and when the number reaches zero, the area will be removed from the list. This will make the list into a sort of short term memory for where dwarves have walked recently. This assumes your dwarves will move in relatively the same areas and pathways a majority of the time, which unless they are fleeing from goblins is a correct assumption. For example, most forts have a main hallway through which dwarves will walk. Using this method, the second list will very quickly acquire paths from common tasks, such as food stockpile to dining hall or storing mugs in their stockpile. This second list will be checked first for a path from point a to point b, and since it will be much smaller than the main world list it should take quite a bit less time to calculate a heuristic for. Another benefit is if you have dwarves running through a massive stockpile room. Take for example one of my forts which has a huge 1 z level room carved out which is probably about 100 by 100 in size and filled with stockpiles. Dwarves running from room entrance A to room entrance B will almost always take the same path or only 1 or 2 squares off the same path. So rather than generating a heuristic for the whole room, the path will be charted once, then followed by all further dwarves to come down that particular path with a much smaller list of areas over which the heuristic must be done. There are some problems with the method, such as the paths being potentially inefficient and doubling back as well as the extra processing done to make and maintain the 2nd list, and they may or may not cripple this technique.
Knowing C++ decently enough to try out this approach, I think I may start trying to implement such a search tommorow...
Post by: Impaler[WrG] on December 04, 2009, 03:50:54 am
Quote
This assumes your dwarves will move in relatively the same areas and pathways a majority of the time, which unless they are fleeing from goblins is a correct assumption.
While it is generally true that new paths will be in the same areas as past paths this is to use when finding a path. We can't exclude low-traffic area from any search nor can we use it as a valid heuristic.
As for pathing across that large cavern this is what a hierarchical approach is designed to achieve. Pathing is more efficient over the fewer number of nodes and its possible to store paths inside the zone and them link them up. This is all covered in the HAA* paper at the start of the thread (I should copy it too the first post to make it more obvious.)
Now that the basic pathfinder is up and running I'll be working on the logging, profiling and creating a more full test suite.
Post by: Starver on December 04, 2009, 06:51:42 am
On the other hand, it might be a good idea that, if all 27 directions are included, you ensure it can't "explore" what happens if a creature doesn't move, how it affect's it's path. Some very possible infinite loops there, unless not moving is automatically discarded by the pathing, and that makes the pathing check for moving to the source tile from it utterly worthless...I would have thought that any state-sensitive (all of them? by one means or another?) pathing algorithm would discount a movement of "no movement" on the same basis as moving back to a previously visited position is discounted.
i.e. on arrival at a tile (or simulated arrival, during a path-finding), the tile is marked as "visited", and while it might not yet have the "if you were to go here, it would be X paces to your destination" details (though it would have "Y paces from your origin" ones) attempts to revisit next turn or by moving onwards and back, or onwards, sideways and diagonally back, or any other combination would know it was an invalid new position.
Speaking generically for all algorithms, as there'd be different ways to implement this, e.g. pulling from a stack of unvisited locations all possible adjacent locations, or flagging the array of all locations, and those methods not strictly breadth-first (by cumulative weighted distance, if applicable) have the opportunity to interogate and re-allocate to themselves previously visited tiles with new, more optimal routing (and then have to worry about updating the visiting points following onwards from the original, longer, earlier discouvered route). But the latter could never apply to 'same tile, no movement' moves, without some form of 'negative weighting' (which, transiently, could be used for certain very special circumstances, but is outside the scope of the DF pathing concerns, at least right now).
Of course, the caveat is the aforementioned (predictably) dynamic environment, where waiting for a door to open allows a quicker transit than keeping on moving around the longer route. If standing still were still disallowed, then hopping between two or more nearby cells could be 'allowed' as a movement. Either way, one has to assume that a state-sensitive system would essentially deal with this as "entire world, with 'gate' opening in N moves" state on one turn and "entire world, with 'gate' opening in N-1 moves" as the next (and differing) state, thus treating either standing still or movement as also moving to T+1 in the time dimension, regardless of overall effect. And have to have some other rather more prosaic method of not emulating an XYZ loop/useless return even while progressing into infinite T, all of which would obviously mean resource hogging keeping track of an XYZT visited array compared to a simple XYZ array, roughly proportional to the total T allowed before a 'time out' limit is applied, or other solution found.
IYSWIM. :)
Post by: shadow_slicer on December 04, 2009, 10:34:56 am
I would have thought that any state-sensitive (all of them? by one means or another?) pathing algorithm would discount a movement of "no movement" on the same basis as moving back to a previously visited position is discounted.As far as I know, this is correct.
Quote
Of course, the caveat is the aforementioned (predictably) dynamic environment, where waiting for a door to open allows a quicker transit than keeping on moving around the longer route. If standing still were still disallowed, then hopping between two or more nearby cells could be 'allowed' as a movement.Actually, I would probably not implement waiting in the pathfinder. I would adjust the cost function so that the cost of an edge was related to the traveling time. This means that traveling through a door that is about to open includes the time cost of waiting for it to open. Basically, if the cost of every tile is 1, and there's a door that will open in T steps from now, 5 tiles away from here, the cost of going through that door should be (T-5). This will add significant complexity to the cost function and will require doors and other predictable close-able things to be within their own zones (and carried all the way up the hierarchy). Anyway, we probably won't support anything like that for a while (if ever).
Also, Impaler has gotten the connectivity rendering working on the current Khazad svn head.
Spoiler: Screenshot (click to show/hide)
Post by: Nadaka on December 04, 2009, 10:49:29 am
27 is numpad's 9 times 3(up, down, and same-z)
HOWEVER, 5 + same-level = unchanged, making the true number 26, or solving the "stay in place" question.
Clarification:
By
"solving the "stay in place" "
I meant that it would make not moving a "valid" direction, so such concerns wouldn't be a problem.
On the other hand, it might be a good idea that, if all 27 directions are included, you ensure it can't "explore" what happens if a creature doesn't move, how it affect's it's path. Some very possible infinite loops there, unless not moving is automatically discarded by the pathing, and that makes the pathing check for moving to the source tile from it utterly worthless...
no movement is not a valid concept in A* unless your algorithm is modified to predict the location of moving obstructions. However, dwarf fortress does not really have these kinds of moving obstructions.
Post by: Starver on December 04, 2009, 11:32:54 am
Actually, I would probably not implement waiting in the pathfinder. I would adjust the cost function so that the cost of an edge was related to the traveling time. This means that traveling through a door that is about to open includes the time cost of waiting for it to open. Basically, if the cost of every tile is 1, and there's a door that will open in T steps from now, 5 tiles away from here, the cost of going through that door should be (T-5).You'd need to modify the algorithm following the calculated path to avoid a juggernaut-like rush up to the not-yet-open pathway only for it to trigger on the current impassibility.
That is, if it doesn't use a locally suboptimal, but not actually looping, time-wasting path in order to not arrive there in time, but in my mind it's like arriving at a T-junction of 1-wide tiles, where travel to the left (+wait) would get through the door, but travel to the right is a (currently) guaranteed-but-lengthier route. If it's a two-wide corridor, then you might be able to generate a saw-tooth traversal along it to 'spend the time', if you could get past the non-optimality of the journey up until the arrival at the transient obstruction.
(Of course, the model could be similar to that of the clustering of tame animals against pet-impassable doors. Although that's a different "vague pathing" method, and I assume related to invader pathing in both "attack" and "flee" mode insofar as heading for "somewhere inside" and "virtually any map edge" respectively for said invaders.)
Quote
[...] Anyway, we probably won't support anything like that for a while (if ever).Yes, I'm being a bit speculative in the above. (A bit? Typical English understatement, some might contend. :))
Quote
Also, Impaler has gotten the connectivity rendering working on the current Khazad svn head.Because I haven't commented (I think... If I have, I've slept since): Nice.
Oh, and @Nadaka: Some might say "Yes we have, bridges and floodgates and things", but I know you mean (as I've previously qualified) predictable bounding/unbounding features.
Back to the predictable location of moving obstructions, I'm also minded of the trap-laden routes of the kind most memorably parodied (to me, at this moment in time) in the film Galaxy Quest. Also the Five Doctors-style "walk only on certain tiles" thingy, among various other versions of that trick. Actually, I suppose the Indianna Jones And The Last Crusade examples are better, if you could imagine unifying the timing (Penitant Man) and positioning (In The Word Of God) problems. But I waffle. Again/More.
Post by: Impaler[WrG] on December 04, 2009, 01:58:34 pm
Quote
This will add significant complexity to the cost function and will require doors and other predictable close-able things to be within their own zones (and carried all the way up the hierarchy). Anyway, we probably won't support anything like that for a while (if ever).
Actually that's exactly what I was thinking of doing, The client would have to report these locations, then these would indeed be kept out of the zones. But I don't think they can be carried all the way up, I'd merge normal nodes until a normal node has only a single link to a door node then allow merger. This makes it really simple to later sever the connection and allows the high level graph to compress down as efficiently as possible.
Post by: Footkerchief on December 09, 2009, 10:36:50 pm
Post by: Impaler[WrG] on December 10, 2009, 12:04:01 am
Post by: CobaltKobold on December 10, 2009, 12:11:05 am
Post by: Blacken on December 10, 2009, 12:28:02 am
Synchronous and single threaded currently, when the client sends a map change or a path request execution passes to the server (the pathfinder code) and has to return before the client can continue. While multi-threading is interesting I'm not up too speed on it and I think a lot of performance can be squeezed out by optimizing the actual path finding code, perhaps far more then just multi-threading the old code.Optimizing your proposed pathing doesn't mean it must remain single-threaded, though. I haven't looked at any of your code, just skimmed the thread, but it seems to me that the closer to shared-nothing you can get (minimizing any critical execution segments) in a single-threaded iteration that you can get, the easier you can take advantage of a multithreaded option later.
Talking about big-Oh when multithreaded stuff is involved gets weird, but in a perfectly shared-nothing system you can approach (never reach, of course, but approach) a 1/n multiplier to your entire execution time. That strikes me as being something that might be very, very useful down the line.
I'm not an algorithms guy, but it seems intuitively that you'll fairly quickly reach a point of diminishing returns in terms of the actual algorithm, whereas spreading it across multiple cores can reduce it further. It's not an either/or proposition.
(There's also graph knitting and all sorts of other stuff that can create a performance/accuracy tradeoff, but I'm sure that's already come up. :) )
Post by: Nadaka on December 10, 2009, 01:24:42 am
Synchronous and single threaded currently, when the client sends a map change or a path request execution passes to the server (the pathfinder code) and has to return before the client can continue. While multi-threading is interesting I'm not up too speed on it and I think a lot of performance can be squeezed out by optimizing the actual path finding code, perhaps far more then just multi-threading the old code.Optimizing your proposed pathing doesn't mean it must remain single-threaded, though. I haven't looked at any of your code, just skimmed the thread, but it seems to me that the closer to shared-nothing you can get (minimizing any critical execution segments) in a single-threaded iteration that you can get, the easier you can take advantage of a multithreaded option later.
Talking about big-Oh when multithreaded stuff is involved gets weird, but in a perfectly shared-nothing system you can approach (never reach, of course, but approach) a 1/n multiplier to your entire execution time. That strikes me as being something that might be very, very useful down the line.
I'm not an algorithms guy, but it seems intuitively that you'll fairly quickly reach a point of diminishing returns in terms of the actual algorithm, whereas spreading it across multiple cores can reduce it further. It's not an either/or proposition.
(There's also graph knitting and all sorts of other stuff that can create a performance/accuracy tradeoff, but I'm sure that's already come up. :) )
I am an algorithm and optimization guy.
For A* each path needs to read edges from the map and store them in a queue, and that will be distinct for each path. The only potential conflict might be in the estimation algorithm and that will vary from implementation, but probably wont have any conflict anyway.
The overhead of multi-threading in this case will be minimal in terms of performance, resulting in a nearly 2 to 4 fold performance improvement on modern cpu's and negligible or avoidable performance cost on single core machines.
The overhead of multi-threading in terms of developer cost will also be minimal as almost no modification to the algorithm is required.
However, since the goal is to and optimize and measure the time to calculate each path, multi-threading in this way will be counterproductive to those stated goals.
A* can also use parallelism for a single path as the evaluation of each choice in its edge queue is independent. Each thread will be adding and removing from the queue, so that queue will need to be synchronized. Each task is relatively trivial, and so spawning one thread per task will not be efficient. Instead you would have a number of worker threads processing edges off the queue while your main thread waits for the workers to finish.
This will be less efficient that per path parallelization, but should give you somewhat less than linear speedup per core.
Post by: numerobis on December 10, 2009, 03:34:10 am
Assuming you don't do that, then independent path queries are indeed perfectly independent -- just read-after-read dependences that don't count for much. The parallel depth is the length of the longest path query, and everything is work-efficient. The problem is figuring out how to write the code that calls the pathing code so that it knows to do lots of path queries at once.
Within an invocation of A*, I don't see how you're going to get any parallelism at all though. You're going through a priority queue one element at a time; you can't get much more sequential than that. With a hierarchy I suppose you could do something, but it wouldn't be entirely straightforward.
Post by: Shades on December 10, 2009, 03:48:59 am
Within an invocation of A*, I don't see how you're going to get any parallelism at all though. You're going through a priority queue one element at a time; you can't get much more sequential than that. With a hierarchy I suppose you could do something, but it wouldn't be entirely straightforward.
There isn't much point doing it within the A* invocation if you ask me. Better to leave is as just able to calculate multiple routes at a time.
The problem is figuring out how to write the code that calls the pathing code so that it knows to do lots of path queries at once.
You don't actually have to worry about this either, in fact it's more likely to be included by Toady if he doesn't have to refactor a lot of his code to do it. The trick here is to take advantage of the fact a single creature will request it's path multiple times before it gets to it's target.
On the first request you return a quick, possibly wrong, where to go next style path and spawn off your A*. Later requests the A* will have hopefully finished and you can pass the remaining route (and spawn off the recalculation for next time). It's not perfect as technically the dwarf will follow a blocked route for a short time, we return the last route not a current one, but it's probably good enough. If not then validating a route is very quick to do.
Post by: Footkerchief on December 10, 2009, 03:53:24 am
Within an invocation of A*, I don't see how you're going to get any parallelism at all though. You're going through a priority queue one element at a time; you can't get much more sequential than that. With a hierarchy I suppose you could do something, but it wouldn't be entirely straightforward.
Well, you can go bidirectional at least, right?
edit: here we go. (http://stackoverflow.com/questions/1640633/multithreaded-a-search-in-java-or-lisp-or-c)
Here's a PDF of the second paper mentioned, (https://bioinformatics.cs.vt.edu/~guos/parallel/00205103.pdf) couldn't find a free copy of the first.
Post by: Starver on December 10, 2009, 06:21:51 am
could gain lots by pathing in background of queued but not yet chosen jobs...the relevant portion anyway (from material to workshop, say, and leaving the dwarf-to-material until a dwarf takes on the job)Possibly, but unless there's a minimal number/closely packed set of raw material items, what's the probability that the precalculated material-to-workshop path is relevent to the best accessible raw material for whatever worker picks up that task?
Yes, if there's just one worker, currently in the workshop, then its likely that the closest raw item to the workshop is the closest to the worker when they pick it up, but you have to consider drink/food/socialising/complaining breaks, other workers taking over production just after wandering out into the wilderness for a game of hide-and-seek, etc. And we know the little buggers tend to mess us around enough with that sort of thing already. :)
Post by: Rose on December 10, 2009, 06:25:44 am
I'm just posting here so I get updates.
Post by: qwertyuiopas on December 10, 2009, 08:15:33 am
Post by: Shades on December 10, 2009, 09:11:10 am
Perhaps have a single low-priority thread running searches for pathing optimizations by seeking out areas where an abstract zone may be placed, like a TCZ, SPZ, or anything else that anyone theorizes may help?
To be honest the really simple change of allowing pathing in the 'background' to the main DF thread would improve performance significantly for the majority of players without being a major amount of work. I'd suggest do that first then improve later. (following the "there is always time to improve later" mantra)
Post by: Starver on December 10, 2009, 10:32:21 am
To be honest the really simple change of allowing pathing in the 'background' to the main DF thread would improve performance significantly for the majority of players without being a major amount of work. I'd suggest do that first then improve later. (following the "there is always time to improve later" mantra)
My doubts about this are related to the fact that when I run DF, it's machine speed rather than framerate capping that limits the timely progression. Maybe if the routing could be optimised to the extent that it doesn't freeze the world when immigrants or invaders are about to enter the world (for example... I've always assumed that the mass path-calculation is what's caused the freeze, just before the relevent notification comes up) then maybe I would have free cycles to spare to do some speculative "just in case" path searching as opposed to the "just in time" that takes exactly as long as it needs to, anyway, because the world can't help but wait[1].
But unless you have got that luxury, adding the overhead of a management process for multithreading[2] and adding in the pursuit of speculative not-necessarily-usable calculations (rather than the not-eventually-usable calculations already inherant in nature of path searching, and which caching/etc should help to avoid or keep to avoid later unnecessary redoing) seems to me to be a less than optimal approach, overall.
But that's an impression, only. I'm probably misrepresenting the process.
[1] The exception to this general rule is the occasional notification of things that should only be happening while the world is running (e.g. strange moods and births) that interupt the world-frozen resource management phases. Of course, these are rare enough events that it's hard to replicate and properly analyse. I've always assumed that there's already a multithreading queue for some things, that initiated an enquiry just before changing to the management mode and continue for a second or two in the background to give the new baby its character, etc, and then announcing the arrival. But then I'm not too familiar with Toady's code.
[2] Noting my comment in the above footnote that there may already be one in place... Could that be used for minimal extra overheading?
Post by: Thief^ on December 10, 2009, 11:00:50 am
this is all fascinating, but I have nothing to contribute.You can just press the "notify" button instead of "reply" and you get updates without having to have a post in the thread. Try that in future.
I'm just posting here so I get updates.
Post by: Shades on December 10, 2009, 11:04:39 am
stuff :)I think you misunderstood my post, I wasn't talking about doing any extra/speculative pathfinding, only to 'background' the current pathfinding so it doesn't tie up main game thread. Like you say when a number of new dwarfs enter the map there is no reason for the game to freeze.
You can just press the "notify" button instead of "reply" and you get updates without having to have a post in the thread. Try that in future.
I have to say I don't find the notify display of threads anywhere near as nice as the one you get for threads you have replied to.
Post by: Nadaka on December 10, 2009, 11:35:25 am
There are a couple important things you can do to optimize performance.
1: work less.
2: work smart.
3: do more work at a time.
From what I have seen, you guys are trying to work smart, improve on A*. Unfortunately there are only 3 ways to do this from an algorithmic perspective. 1: produce an estimation heuristic closer to the shortest path using domain specific knowledge. 2: reduce your search space by combining nodes. 3: accept paths that are less than ideal, typically done by having an estimation heuristic higher than the shortest path.
Parallelization is doing more work at a time, its maximum performance multiplier is a little less than your number of cpu cores. However, it is low hanging fruit, its minimum performance multiplier is also a little less than your number of cpu cores. Parallelization can provide a benifit more than your number of cores with special planning. For instance, you don't need to wait for all paths to be calculated to move on to the next step, you can keep calculating new paths while you are consuming the first paths generated.
Working less usually produces the greatest returns. Here is an example, in a 2x2 single level map, hypothesize a path 40 tiles long ( a little less than half the maps width) for each step taken the path is recalculated. For each step taken one tile somewhere on the map becomes blocked (that is a fairly high rate of construction imo). Each tile has a 1 / (9216 - step) chance of becoming blocked. The chance that that tile is in the remaining path is (40 - step) / (9216 - step). There is only an 8.9% chance of the path becoming blocked while the path is being run. Caching that path will save you 39 path calculations 91.1% of the time, It will also save you 38 path calculations more than 99% of the time.
For paths with an average length as short as 5 or 6 tiles, caching should produce at least equal gains compared to improved algorithm or multi-threading, and if the average length is longer, it will become progressively more effective.
Post by: numerobis on December 10, 2009, 01:54:50 pm
Your idea for a massive speedup is how A* is pretty much always implemented in a "real-time" game, and even in a number of turn-based games.
Post by: numerobis on December 10, 2009, 02:09:33 pm
Bidirectional, true. There's a common issue that your two paths may not usefully link up (e.g. the shortest path through a rectangle -- do you first go diagonally, then up, or first up, then diagonally? Making sure both paths meet somewhere is not necessarily easy, but maybe there's tricks.)Within an invocation of A*, I don't see how you're going to get any parallelism at all though. You're going through a priority queue one element at a time; you can't get much more sequential than that. With a hierarchy I suppose you could do something, but it wouldn't be entirely straightforward.
Well, you can go bidirectional at least, right?
Here's a many-processor A* paper much more recent than that vintage one you found, with a bunch of references: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.50.6224&rep=rep1&type=pdf
I have to note wryly that the authors have searched far and wide, and discovered that they have written half of the important papers on the topic.
Post by: Nadaka on December 10, 2009, 02:14:06 pm
I also make stuff fast for a living, and I have to say, parallelizing the solution to one problem is rarely low-hanging fruit. If you can rewrite things to that you need to solve the same problem repeatedly, and you can spawn off a bunch of threads to do it, then it can be pretty easy -- assuming that "rewrite things" is itself easy.
Your idea for a massive speedup is how A* is pretty much always implemented in a "real-time" game, and even in a number of turn-based games.
Generally I agree with you about parallelization rarely being low hanging fruit if you don't start developing a solution with it in mind. However, for the problem "produce a lot of paths quickly, using a standard implementation of A*" it is pretty low fruit.
Post by: Footkerchief on December 10, 2009, 02:54:20 pm
Bidirectional, true. There's a common issue that your two paths may not usefully link up (e.g. the shortest path through a rectangle -- do you first go diagonally, then up, or first up, then diagonally? Making sure both paths meet somewhere is not necessarily easy, but maybe there's tricks.)
It appears someone found a way of handling it. (http://www.springerlink.com/content/w68p612q2m155413/) Can't find a free version of the article as usual.
Post by: Impaler[WrG] on December 10, 2009, 03:01:04 pm
Better algorithms utterly blows that out of the water, improvement can be factors of 10 for an optimized hierarchical pathfinder vs an optimized non-hierarchical approach, all the papers I've read confirm this. Add a cache on top of that and we can probably get another factor of 10 improvement.
I'm going to focus on algorithms for now but will try to keep each path search isolated so the potential exists for parallelization in the future. One thing in particular that were already doing is storing the parent node data produced in each search outside of the grid. Most pathfinders just write this to the grid their searching on in a 'throw away' layer that overwritten with each search, clearly a solution which will fail if multiple searches are occurring over the same space. Theirs also the issue of modifying the grid, as the current implementation will never be modifying the grid and searching at the same time theirs no issue. But if multi-threaded theirs would need to be a signaling system to keep searches and grid modifications from colliding, grid modifications would probably need to be handled singularly as well.
Post by: numerobis on December 10, 2009, 03:06:28 pm
Post by: qwertyuiopas on December 10, 2009, 05:26:09 pm
...
There are a couple important things you can do to optimize performance.
1: work less.
2: work smart.
3: do more work at a time.
...
There is a fourth option in situations such as this:
4: Use ALL of your time.
For example, when the game is paused, or most of the dwarves are asleep, or any other low-pathing-required situation, that is when you should have it seek for slight optimizations that take time for minimal gain, since you can afford to spend a whole second of game-paused time to optimize the placement of a zone, since the game is paused, thus you have free processing power, not being used anywhere.
With "new ore" announcements, that is common.
If you make the slow-optimization-search a pausable background thread, then it can be paused during any pathing and resumed when there is processing break. That way you don't waste idle time, even if the gains are minimal.
Post by: CobaltKobold on December 10, 2009, 05:42:26 pm
With "new ore" announcements, that is common.Mind that that's like to go away next version.
Post by: Shades on December 11, 2009, 03:44:49 am
There is a fourth option in situations such as this:
4: Use ALL of your time.
This is 'free' if we have multiple pathing threads, although only for paths already requested.
Most home multi-core systems are around 2 or 4 cores and though a jump to 8 cores is inevitable it's not going to happen for a while. This means your maximum performance gain is around a factor of 4, realistically a factor of 3.
Better algorithms utterly blows that out of the water, improvement can be factors of 10 for an optimized hierarchical pathfinder vs an optimized non-hierarchical approach, all the papers I've read confirm this. Add a cache on top of that and we can probably get another factor of 10 improvement.
Better algorithms are always good, but in this particular case multiple threads should be minimal effort as it's an isolated situation (calling pathing finding for a single path) that will trigger it. So although I agree that a better gain is possible and would encourage you to work on a better algorithm we might as well grab low hanging fruit where we can.
Also it won't be wasted work because even if you find an incredible algorithm it can still take advantage of the multiple threads.
Post by: Starver on December 11, 2009, 11:04:19 am
Sorry, there'd been talk about such stuff[1], and I misread your post as a continuation of same. :)stuff :)I think you misunderstood my post, I wasn't talking about doing any extra/speculative pathfinding, only to 'background' the current pathfinding so it doesn't tie up main game thread. Like you say when a number of new dwarfs enter the map there is no reason for the game to freeze.
[1] Assuming I didn't misread that wrongly!
Post by: Impaler[WrG] on December 12, 2009, 03:12:10 am
I'm going to work next on some connectivity management tracking so these situations are handled gracefully, this should bump up performance considerably, I'll also explore bidirectional searches and other optimizations that can be done to plain non-hierarchical A*.
Post by: piupau on December 12, 2009, 06:41:58 am
Ok got an automated test suite running, it tests combinations of randomly selected passable points. A bunch of relavent info is written to file (I'll put it on the screen too soon).
This is a very interesting project. I just read through the thread, and suggest that you'd put that test suite up for grabs in the first post -- It would give a much more 'upbeat' feeling to reading through all the posts knowing that there was something released in the end. :)
Post by: Impaler[WrG] on December 12, 2009, 01:51:27 pm
Post by: Impaler[WrG] on December 18, 2009, 12:02:26 am
(http://img163.imageshack.us/img163/1341/screenshot2oy.png)
Here is a screen shot of the new output info, this is a test of 100 random point pair path attempts. As you can see 3 paths are rejected for non connected regions while the rest succeed. Theirs a great deal of information on nodes and graph reads here but the Time efficiency is the simplest thing to understand, its milliseconds per tile in the path. This can be made a lot better, when doing a relativly easy path on open ground I can get about 100th the time per step, this is because of pathing in internal areas ware dead ends mean that simple heuristics breakdown, a hierarchical solution should reduce this.
Post by: sir monko on December 18, 2009, 05:10:00 pm
if someone's interested in what a* looks like in action, i put a demo together (and maybe caused a firefox 3.6 delay, due to the code revealing a performance bug):
here it is:
http://jsastar.tapirpirates.net/dfdemo/
this is just something i hacked together in my free time in the last 2 days, so it's definitely not optimized or anything. it works for drawing wiggling lines though.
i recommend google chrome for best performance. i tested it in recent versions of FF > 3.5, chrome, safari and opera. ie does - of course - not work (it needs the canvas element, and i shudder if i think about ie performance).
disclaimer: i'm not responsible if it locks up your browser!
Post by: Impaler[WrG] on December 19, 2009, 07:55:24 pm
UPDATE: 0.23 ;D
Post by: zchris13 on December 19, 2009, 08:52:42 pm
UPDATE: 0.23 ;DWhat does this mean?
Post by: Draco18s on December 19, 2009, 09:39:40 pm
UPDATE: 0.23 ;DWhat does this mean?
I think that his algorithm is so efficient that it takes an average 0.23 milliseconds per tile in a path, down from 0.33 (down from 1.16).
Post by: zchris13 on December 19, 2009, 09:43:56 pm
Post by: Fieari on December 19, 2009, 09:46:45 pm
Now we just need to add in connectivity maps for more movement types...
Post by: Draco18s on December 19, 2009, 10:02:46 pm
Can I assume that that is blazing "set the atmosphere on fire" fast?
Not quite. We'd have to get down to roughly 0.01 milliseconds per tile first.
Post by: CobaltKobold on December 19, 2009, 10:06:51 pm
Post by: Impaler[WrG] on December 19, 2009, 10:25:16 pm
The problem is it's unknown what the equivalent speed in DF would be, I suspect were still a good deal slower because theirs are a few layers of indirection necessary to retrieve the Connections and Edge Costs from the Grid. This has to be their because were going to use a Client/Server structure ware as DF probably integrates all this into the map structure and thus has less indirection. So even fully optimized I don't think the simple AStar will be equal to DF, but the Hierarchical solution will be and the Cache on top of that will be another gain so by the end we could be looking at an order of magnitude better performance.
Quote
.01 ms...10000 cycles per tile?
Remember this is per tile in the FINAL PATH, as you can see on the output were expanding ~18 nodes per path tile and examining the graph ~70 times, this is actually rather good considering what something like Dijikstra's algorithm would do. I'm testing on a map with a relatively large and complex fort which is generating plenty of dead ends and maze like structures for the Pathfinder to get lost in.
UPDATE:
Got the performance down to 0.19 by using an Object Pool, I think is as good as it will get without using real profiling tools which I'm not experienced with. I'm going to move towards the Hierarchical approaches and to create a path Corridor through which to run a slightly modified AStar through, this should reduce the nodes being expanded to make things faster.
Post by: Impaler[WrG] on December 21, 2009, 03:19:39 am
Today I reworked the AStar implementation into one which is interpretable, basically the main loop is function call with a second layer determining the number of times to loop the inner portion. I can call for a specific number of Nodes to be expanded or for it to continue until the path is found or found to be impossible. And because everything is saved I can at any time request a 'best path'. The plan is to have moving agents spread their path finding calculations over time which will help prevent those 'lag spikes' when every dwarf rushes out of the fort to pickup a sock.
Next I'm going to work on the MovementController class and see if I can get a whole slew of constantly moving agents running around and following paths.
Post by: qwertyuiopas on December 21, 2009, 04:41:19 am
When it runs out of pending paths to calculate?
Post by: cooz on December 21, 2009, 04:47:44 am
What will it do when DF is paused?
Could it keep caching some random point-to-point paths so they can be used when you unpause game? On the other hand it might not be so great idea in DF universe where everything's constantly changing...
Post by: Foa on December 21, 2009, 05:28:42 am
It'd probably finish all pathfinding in that particular frame, unless you unpause before that, but a path is only defined for one instance, AND I DO MEAN EXACTLY ONE!What will it do when DF is paused?
Could it keep caching some random point-to-point paths so they can be used when you unpause game? On the other hand it might not be so great idea in DF universe where everything's constantly changing...
Post by: Impaler[WrG] on December 21, 2009, 03:41:14 pm
Though were allowing the Client to request and receive a full path the preferred interaction is going to be through the movement controller which would be called every time an agent needs to know what direction to move in. The Client would request a Controller for each agent which would then keep the pointer to its own private Controller, it of course has to input current location and goal but after that the Controller is just a black box from the Clients perspective.
Post by: PermanentInk on December 23, 2009, 08:31:57 pm
Post by: Impaler[WrG] on December 25, 2009, 01:09:24 am
Post by: Pie on December 27, 2009, 06:56:34 pm
To those brave souls who dare to accept this challenge, I salute you (especially Impaler, who seems to have got down and made progress, which is always difficult with a project of this magnitude).
Post by: Vigilant on December 27, 2009, 11:55:12 pm
Btw, this offer isn't usually the kind people like, but I'm kinda a rogue programmer that drifts in and out. I can't really commit to a project but if you guys run into problems I can often come up with something useful ;)
Post by: Footkerchief on December 28, 2009, 01:45:52 am
I don't know if anyone's tackled this but one of the other major slowdowns on DF is when no path exists it keeps trying to find one anyways. What needs to happen is potentially request future requests for that path for some time before allowing searches again.
The game does prevent repeated requests in most cases -- the ones where it doesn't are bona fide bugs, like these:
# 000184 □ [dwarf mode][creatures] animals locked in a room lag the game heavily from path-finding, additional lag can be caused by assigning them to be butchered
# 000631 □ [dwarf mode][idle behavior] (Report) forbidding pet passage through the door of an enclosure creates lag
Post by: Nadaka on December 28, 2009, 03:31:14 am
Post by: Footkerchief on December 28, 2009, 03:41:34 am
Post by: CobaltKobold on December 28, 2009, 10:01:34 am
Post by: Vigilant on December 28, 2009, 03:40:22 pm
I don't know if anyone's tackled this but one of the other major slowdowns on DF is when no path exists it keeps trying to find one anyways. What needs to happen is potentially request future requests for that path for some time before allowing searches again.
The game does prevent repeated requests in most cases -- the ones where it doesn't are bona fide bugs, like these:
# 000184 □ [dwarf mode][creatures] animals locked in a room lag the game heavily from path-finding, additional lag can be caused by assigning them to be butchered
# 000631 □ [dwarf mode][idle behavior] (Report) forbidding pet passage through the door of an enclosure creates lag
It also doesn't do it sometimes for skulkers and possibly siegers? Sometimes the entrance to my fortress temporarily gets impassable due to flooding, and if there's like 3 thieves, the FPS tanks -_-
Post by: Footkerchief on December 28, 2009, 03:44:17 pm
It also doesn't do it sometimes for skulkers and possibly siegers? Sometimes the entrance to my fortress temporarily gets impassable due to flooding, and if there's like 3 thieves, the FPS tanks -_-
Couldn't find anything in the bug list, but yeah, I think I remember reports of this happening with siegers and HFS.
Post by: qwertyuiopas on December 28, 2009, 04:04:20 pm
Post by: Impaler[WrG] on December 29, 2009, 05:36:34 am
In other news, I've implemented simple Pawns in Khazad that I can spawn on the map by pressing 'a'. Each one gets a MovementController object which is created by the Central PathManager, each Controller acts as a personalized access point for Pathing from the perspective of the Pawn which can set a Destination and then repeatedly request its next movement direction without ever being aware of the Path. I've also begun separating the Controller itself from the path by implementing a PathWalker which is a tiny class that just iterates the Path and abstracts away the internal storage details of the path, most importantly because its read only with respect to the path their will be no conflict if multiple Walkers reference the same Path which could will allow units to share a Path or for an effective PathCache to work without collision risks.
Post by: Xgamer4 on January 10, 2010, 01:33:50 am
Post by: Impaler[WrG] on January 10, 2010, 03:47:17 am
Post by: cooz on January 10, 2010, 09:41:17 am
http://mm.soldat.pl/development-log/937
Post by: PencilinHand on January 10, 2010, 01:26:06 pm
That is something I have seen consistently. Most pathfinding I have seen considers something like a single level 512x512 node space(~262k nodes) to be the upper practical limit for use(often with only cardinal movement allowed). Most don't go that high in practice and limit their pathfinding to, at most, the length of one side of the map.
The smallest DF map possbible is almost 100x100x20(rough numbers for a simple 2x2, about 200k nodes) with a maximum search length 3 or 4 times the length of a side. I mean, look at dwarf heaven, a 6x6 with like 50 z levels, that is almost 4.5 million possible dwarf passable nodes. And that is before mega projects get involved, then the sky is quite literally the limit. DF is working on a scale all its own in most ways, pathfinding included as far as I have seen.
I find I am sometimes bothered by the comparisons drawn between other implementations and the discussions here.
Post by: CobaltKobold on January 10, 2010, 10:28:42 pm
Post by: zchris13 on January 10, 2010, 10:29:36 pm
EDIT: Possible nodes. My bad.
Post by: CobaltKobold on January 10, 2010, 10:35:49 pm
But that's tiles, not necessarily nodes. There are smarter ways. (Also, average map would be 6x6 or so, which would be 288x288x~60 z on your average mountainside?)
Post by: PencilinHand on January 11, 2010, 06:53:05 pm
Which will be needed when flying/(under)swimming pathing goes in.
But that's tiles, not necessarily nodes. There are smarter ways. (Also, average map would be 6x6 or so, which would be 288x288x~60 z on your average mountainside?)
Every tile is possibly a dwarf passable node if given enough time. Yes, there are other smarter ways of designating nodes than on a one to one with the number of tiles. In my opinion, the most useful discussion in this thread has to do with ways of abstracting or limiting the number of searchable nodes.
However, I worry that efforts to reduce the number of nodes become increasingly less useful the further development goes toward things like invasion tunneling and multi-tile creatures. Granted, the vast majority of pathfinding is done by dwarfs or dwarf-like creatures now and anything we can do to improve dwarf-like pathfinding will be a great boon to the project(assuming any of this actually gets adopted). However, I know if we can completely get around the need to maintain a one tile per node map which includes potentially critical information like blocked/unblocked, water, magma, and foor IFF the goal is to provide for ALL pathfinding needs. Which it appears it is.
Create a flexible BSD licensed C++ path-finding library specialized for grid based systems like that of Dwarf Fortress and able to efficiently handle rapidly changing map geometry and other challenges such as but not limited too multiple travel domains, multi-tile creatures, variable movement cost terrain and path caching.
shadow_slicer gave a very good summery of the problem back at the end of November
If you decide to devote 8 bits for each tile (not for each edge, each tile!) just to determine is this tile passable by movement type x, a 256x256x16 map would require 1MB of memory. If we add ceilings, it goes to 2MB. Note that this is a really small map. A 6x6 I am currently using is 288x288x32, which is 2.5 or 5 MB. Note that this does not include any cost information, and only supports 8 movement types. If you add terrain types or other features then this only gets worse. And, as you said the number of tiles on the map is only going to increase. If you get up to hundreds of z-levels with larger embark sizes, things get nasty very quickly.
Post by: Xgamer4 on January 12, 2010, 12:20:06 am
shadow_slicer gave a very good summery of the problem back at the end of NovemberIf you decide to devote 8 bits for each tile (not for each edge, each tile!) just to determine is this tile passable by movement type x, a 256x256x16 map would require 1MB of memory. If we add ceilings, it goes to 2MB. Note that this is a really small map. A 6x6 I am currently using is 288x288x32, which is 2.5 or 5 MB. Note that this does not include any cost information, and only supports 8 movement types. If you add terrain types or other features then this only gets worse. And, as you said the number of tiles on the map is only going to increase. If you get up to hundreds of z-levels with larger embark sizes, things get nasty very quickly.
This was talked about earlier, but I don't remember the verdict. Is this actually a problem, though? Dwarf Heaven, mentioned on the last page, was apparently 6x6 gridsize, with tile size around 288x288x50, going off the same numbers for 6x6 that shadow_slicer mentioned. That's 4 - 8MB of RAM. That's a pittance. Most computers in use have 1GB of RAM min, and most current computers come with 3-4GB. 8MB of RAM won't be missed, especially if it helps speed up pathing.
Which I think was the main concern. That it wouldn't actually have any improvement. Is that right?
As a sidenote, there was a link, to github I think, with current development. Does anyone have it, and is it actually up-to-date? I'd like to look at the code and pretend I know what I'm doing before the feeling of being in way over my head sinks in. <_<
Post by: shadow_slicer on January 12, 2010, 01:35:21 am
Yes, my point was that this (1) was a lower bound/ conservative estimate for memory usage under that architecture and (2) would not significantly increase pathfinding speed.shadow_slicer gave a very good summery of the problem back at the end of NovemberIf you decide to devote 8 bits for each tile (not for each edge, each tile!) just to determine is this tile passable by movement type x, a 256x256x16 map would require 1MB of memory. If we add ceilings, it goes to 2MB. Note that this is a really small map. A 6x6 I am currently using is 288x288x32, which is 2.5 or 5 MB. Note that this does not include any cost information, and only supports 8 movement types. If you add terrain types or other features then this only gets worse. And, as you said the number of tiles on the map is only going to increase. If you get up to hundreds of z-levels with larger embark sizes, things get nasty very quickly.
This was talked about earlier, but I don't remember the verdict. Is this actually a problem, though? Dwarf Heaven, mentioned on the last page, was apparently 6x6 gridsize, with tile size around 288x288x50, going off the same numbers for 6x6 that shadow_slicer mentioned. That's 4 - 8MB of RAM. That's a pittance. Most computers in use have 1GB of RAM min, and most current computers come with 3-4GB. 8MB of RAM won't be missed, especially if it helps speed up pathing.
Which I think was the main concern. That it wouldn't actually have any improvement. Is that right?
I later demonstrated that a single bit/two bits per tile was too low to implement the current behavior exhibited by DF. To implement the current behavior you need at least 13 bits/tile (or even 26bits if bidirectionality is not assumed). This pushes the total to 52-208MB, just for the connectivity map. If instead of simple connectivity you want to include movement costs (let's just use 3 bits so we have 8 possible costs), we have 156MB-624MB of memory. And this is simply necessary to achieve the same behavior DF currently exhibits (while supporting 8 movement types).
Further this method is likely to be slow. That 624MB of data is competing with DF data and code for space in the L1 and L2 cache of the processor. It is likely that whenever pathfinding functions are called, the necessary data will not be in the on-CPU caches so we will have to wait at least 100 clock cycles for each read to non-cached data. Then, when pathfinding has finished (or reached a point where it decides to return), DF is likely to be completely purged from the cache so we have another ~100cycle/read delay until the cache fills again.
So yes, representing the connectivity/costs in an explicit table has serious scalability problems. If, however, the pathing library instead had DF calculate the connectivity/costs on demand, this would access the same data that DF normally uses (which is stored much more compactly than is possible in a general-purpose library), keeping it in the cache and avoids both the additional delay and the extra memory consumption. This leaves us with a whole lot of memory left over to do useful things like path caches, connected zones, and multiple hierarchy levels, etc. which will have a positive impact on pathfinding speed.
As a sidenote, there was a link, to github I think, with current development. Does anyone have it, and is it actually up-to-date? I'd like to look at the code and pretend I know what I'm doing before the feeling of being in way over my head sinks in. <_<The current code base is part of Khazad, so you can find it on Khazad's sourceforge page (http://sourceforge.net/projects/khazad/).
Post by: Nadaka on January 12, 2010, 08:09:43 am
Post by: Starver on January 12, 2010, 10:07:27 am
actually, there is a barrier for data structures where increasing their size will reduce performance. Once a data structure can no longer fit in available cache space, it takes hundreds of cpu cycles to retrieve it from main memory. Its called cache thrashing, and its probably the biggest reason why AMD chips are less powerful than Intel, they just have smaller cache.a) A number of AMDs have less cache than a number of Intels, more like. Unless Intel's chucked all the budget/low-cache systems recently and AMD hasn't bothered with top-end variants, but I haven't cared to check the current situation in detail recently, so I could be wrong and my real comment is:
b) The time taken to transfer data (raw data[1] blocks, not data structures) is limited by bandwidth to the processor, which is narrower than the cache may be small on all systems I've worked with where there is a cache that's anything more than just a bus-width register.
But I am a bit out-of-date with modern architectures and methodologies, so I suppose someone could have developed an efficient sub-caching system on a system with an ultra-wide and/or ultra-fast pipeline into it, or even manage to do a L2/L1 staging system for data (and sub-set of data) of differing priorities. And of course taking into account multi-core requirements where applicable.
Personally, based upon older phenomena such as "Pagefile Thrashing" and similar (except, I suppose, in reverse), I would have used the term "Cache Thrashing" to indicate that, having loaded one or more complete sets of data into the cache the computations then indicate that another bit of data is required, displacing data that (unbeknownst to the cachig algorithm) is subsequentally to be called back upon by the new data, which displaces another cache item but (once back) facilitates computations that ask for that old item, etc. Thus shoving whole pages (or cache-level equivalent) of data in and out of the quick-but-small storage area (cache for memory, memory for pagefile) taking more time to access than it might have been if it was just a matter of reading the large-but-slow memory in-situ in the first place.
Well, those are my thoughts, for what it's worth. Probably worth nowt. :)
[1] As the processor won't have any idea it what groupings it should cache, unless you've programmed at Assembler level and deliberately worked within MMX-style arrayed data references (or whatever they use these days), or have a very good compiler that can manage to optimise that for you despite your being unaware of what's required. If you have a structure containing a rapidly changing/accessed flag/connectivity byte[2] next to a fairly static array[3] only occasionally queried, then does the cache (and data pipeline bringing it to the chip for caching) shove a block area of memory across or pick and choose bits and pieces (e.g. just the flag bytes) from a number of different cachable areas? Depending on the microcode, assembler and whatnot, it could go either way, but my hunch is that a blocks would be cached just for the sake of their flag-bytes, in this kind of circumstance. Less extreme variants possible, but with the same idea.
[2] Maybe saying "If trying to walk through here, you can go from $adjacent_area1 to $adjacent_area2 in $steps_for_1to2 cycles" in a DF context, and used to determine if it's worth pathing through that area.
[3] The internal routing details of a DF 'zone' which, due to it being not the most promising route and rejected as a possible path, isn't actually explicitly queried to find how to pass from Adjacent Area 1 to Adjacent Area 2 through this.
Post by: qwertyuiopas on January 12, 2010, 08:39:28 pm
A better method probably exists, but being able to ignore unpathable and empty tile blocks would save on memory and provide a useful abstraction layer for other improvements.
Post by: peterix on January 12, 2010, 09:11:58 pm
Post by: Impaler[WrG] on January 12, 2010, 11:34:06 pm
Post by: Djohaal on January 14, 2010, 01:05:48 pm
First thing that comes to my mind is that in large, uniform open floors (like tower cap farms and such), a quadtree subdivision could speed up the pathfinding a lot. Perhaps it could lead the dorfs to the nearest point on the quad subdivision, then from there on the "real" pathfinding system would start.
Now for the fortress itself, an interesting idea would be for the software to analyze and interpret the pathing in archtectural terms in order to optimize it.
Most fortress designs have corridors, rooms and such. If an analyzer was to interpret and abstract them into simplified elements (say, a "corridor" with 20 doors, instead of a 60-tile long node cloud with branching units that would be the rooms) it could speed up the pathfinding. For instance, if Urist McLost was to go to his bedroom, the pathfinder would trace the path across a pre-analyzed and already laid out corridor system to his bedroom's door. Once he made it there, the "brute force" system would path him to his bed, closet, throne, etc.
Doors and DF room definitions could help the analyzer to lay out its corridor/room tree. A significant "pathing mutation" would have to be implemented though, to avoid the corridor system squeezing all your dwarves in a single lane on that 5-wide hall you did though. Another issue I can foresee is dealing with rooms in corridors. (I for instance tend to make lateral expansions to my corridors to make my dining halls) and doors in the corridors themselves.
The main issue is that constantly analyzing the fortress's layout to draw and redraw the corridor tree would require so much processor power. Perhaps there could be settings on it so it'd auto-update the corridor tree after a certain number of new tiles were dug out. A secondary system that looked up significant changes in areas that are "already mapped" on the corridor tree could also avoid unecessary processing.
Last but not least though, this analyzer/simplificator system could be coded on such a contained way that it could be threaded independently of the main pathfinder. This would greatly enhance performance if we focus on paralelism.
Post by: PencilinHand on January 14, 2010, 04:26:24 pm
stuff
I understand 30+ pages is a long read so I will simply tell you that all of this has already been discussed, most of it at quite some length. Virtually all of it has been discarded as it would involve so much pre-processing as to be a fools errand and beyond the scope and of intend for the project. The only one that I feel has any potential usefulness in the near future is the implementation of parallelism on a per path level(should help minimize the headaches). However, parallelism has been "set aside" as overly ambitious for a first go, which this is.
Part of the point of this project is to be able to present a superior, "simple" to integrate pathfinder, for Toady to adopt if he so chooses. Offering a pathfinder that requires extensive rewriting every time Toady wants to change something is NOT an option he will take.
Post by: numerobis on January 14, 2010, 10:43:02 pm
Parallelism has indeed been discarded twenty times over, because it's unlikely to be helpful compared to just working hard on the sequential code.
Post by: PencilinHand on January 15, 2010, 01:03:01 am
I'm not so sure the concept of having an overlay has been discarded. That's kind of the entire point here. It's not clear that quadtrees are the way to go for the hierarchical structure: a twisty windy path with only one entrance and exit would be a lot of nodes in a quadtree, but might conceivably be contracted to a single node using some other partitioning of space.There have been many discussions about different partitioning systems but the biggest problem is the lack of a convenient way for those interested to implement and compare them. Last I head this was being worked on.
Parallelism has indeed been discarded twenty times over, because it's unlikely to be helpful compared to just working hard on the sequential code.Yes and no. Good sequential code will always be useful. Parallelism is also useful to a point. Somethings would work well parallel and some wouldn't. Map preprocessing could probably be made parallel, and reprocessing after a map change may also be doable. Each path if individually treated as a thread might be a workable means of implementing a parallel method with minimal collusion and overhead. However, I am not doing the coding so I can not be sure of the details. Besides those who are doing the coding have said they aren't going to do anything parallel, at least for now.
Post by: ggeezz on February 10, 2010, 02:37:55 pm
Just wondering. I found it rather fascinating, if a bit over my head.
Post by: eternal on February 19, 2010, 05:38:11 pm
Is this still being worked on?
Just wondering. I found it rather fascinating, if a bit over my head.
I'm also interested and bumping this. Going to read through the whole thread, but I'm wondering what the status of the project is. Real life definitely takes a toll on these things :-[
Post by: Sagabal on February 28, 2010, 11:36:32 am
I'm also interested and bumping this. Going to read through the whole thread, but I'm wondering what the status of the project is. Real life definitely takes a toll on these things :-[
Bumping again. Has any progress been made? I'd like to see this happen.
Post by: Impaler[WrG] on February 28, 2010, 02:38:52 pm
As of right now Pathfinding is on hold as I turned back to the Graphics engine in Khazad which I've decided to re-build in Ogre. The work done so far has been good but I've failed to attract a dedicated developer who would continue building it in my absence. Splitting the code into it's own sourceforge project was predicated on having that dedicated developer though as it is right now the code very ready to cleave off as I've followed the kind of encapsulation that allowed the same to be done with DFHack. If anyone is interested in working on picking up the project just PM me and I can get you access the SVN repository.
Post by: deek0146 on March 01, 2010, 04:22:26 am
Post by: ArPharazon on March 05, 2010, 01:56:01 pm
Post by: Draco18s on March 05, 2010, 02:24:58 pm
I apologize for not having read the thread, but has anyone tried something like ant trails, at least for civilian dwarfs?
Was discussed, yes.
Post by: Pie on March 05, 2010, 07:58:48 pm
Now I'm sure I'm missing something obvious due to my total lack of competence in this area, but I just thought that it might be worth thinking about (if it hasn't already been thought of).
Post by: Nadaka on March 05, 2010, 10:08:13 pm
I realise that this is not within the scope of this project (as it involves feature implementation by Toady), but in terms of Dwarf Fortress Pathfinding in general, if there was some option for the user to paint a rough route from point A to point B, would this make Pathfinding faster? The "brush" would be several tiles large, to give the system some flexibility. The basic idea would be that there would be far fewer tiles to search through, and even some of the most sub-optimal routes would be acceptable for most players (since they painted them). If the route was sliced in half (say by some deadly magma), it would simply revert to the standard Pathfinding method. Another advantage of this method is that you don't have to calculate total distances with objects many z-levels above. The main disadvantage is of course the lack of automation to this and the fact that you'd still need a much better Pathfinder for situations not involving stockpiles and workshops (or other specific start/end points).
Now I'm sure I'm missing something obvious due to my total lack of competence in this area, but I just thought that it might be worth thinking about (if it hasn't already been thought of).
You can already somewhat do this with traffic zoning.
Post by: PencilinHand on March 06, 2010, 12:54:14 am
I realise that this is not within the scope of this project (as it involves feature implementation by Toady), but in terms of Dwarf Fortress Pathfinding in general, if there was some option for the user to paint a rough route from point A to point B, would this make Pathfinding faster? The "brush" would be several tiles large, to give the system some flexibility. The basic idea would be that there would be far fewer tiles to search through, and even some of the most sub-optimal routes would be acceptable for most players (since they painted them). If the route was sliced in half (say by some deadly magma), it would simply revert to the standard Pathfinding method. Another advantage of this method is that you don't have to calculate total distances with objects many z-levels above. The main disadvantage is of course the lack of automation to this and the fact that you'd still need a much better Pathfinder for situations not involving stockpiles and workshops (or other specific start/end points).
Now I'm sure I'm missing something obvious due to my total lack of competence in this area, but I just thought that it might be worth thinking about (if it hasn't already been thought of).
You can already somewhat do this with traffic zoning.
The idea will be expanded upon somewhat by the burrows concept. Which will prevent dwarfs from taking jobs etc. outside their assigned burrows thus limiting the destination possibilities. However, pathfinding will not take burrow boundary's into account.
Post by: tigrex on March 06, 2010, 01:44:07 am
Post by: PencilinHand on March 06, 2010, 08:09:44 am
That might lead to some exasperating situations, such as a dwarf entering a burrow with the aim to go through it and into the burrow next door and then to its destination, only to find that the second burrow is forbidden to it. It reverses, and then sees burrow 1 again, turns around and walks through it, leading to what people will soon call burrow dancing.
Dwarfs ignore burrows when pathing.
Post by: jaked122 on March 06, 2010, 03:41:00 pm
or you *could* have embedded pathing nodes that players can place in important areas that would link to nearby reasources(much like the unreal engine does) and the doors could easily function as conditional nodes (if connected to a lever)
Post by: Mel_Vixen on March 06, 2010, 03:55:31 pm
edit:
Anyway btt: The first post says:
Quote
Create a flexible BSD licensed C++ path-finding library specialized for grid based systems like that of Dwarf Fortress and able to efficiently handle rapidly changing map geometry and other challenges such as but not limited too multiple travel domains, multi-tile creatures, variable movement cost terrain and path caching.
Algorithms for multitile creatures could be extend to "Swarms" (Boids etc.). The only difference is that a swarm is flexible in form. I dont know how hard it would be (its just a fix idea) but i think much of the code for multi-tile creatures could work for swarms too. Thus you could safe much time by calculating a path for one swarm then a path for every single member.
The "Swarm" concept could apply to things like Herds of animals, squads, armys, working-groups, etc.
Anyway that is only suggestion based on a educated guess.
Post by: ggeezz on March 08, 2010, 11:16:43 am
I realise that this is not within the scope of this project (as it involves feature implementation by Toady), but in terms of Dwarf Fortress Pathfinding in general, if there was some option for the user to paint a rough route from point A to point B, would this make Pathfinding faster? The "brush" would be several tiles large, to give the system some flexibility. The basic idea would be that there would be far fewer tiles to search through, and even some of the most sub-optimal routes would be acceptable for most players (since they painted them). If the route was sliced in half (say by some deadly magma), it would simply revert to the standard Pathfinding method. Another advantage of this method is that you don't have to calculate total distances with objects many z-levels above. The main disadvantage is of course the lack of automation to this and the fact that you'd still need a much better Pathfinder for situations not involving stockpiles and workshops (or other specific start/end points).
Now I'm sure I'm missing something obvious due to my total lack of competence in this area, but I just thought that it might be worth thinking about (if it hasn't already been thought of).
You can already somewhat do this with traffic zoning.
The idea will be expanded upon somewhat by the burrows concept. Which will prevent dwarfs from taking jobs etc. outside their assigned burrows thus limiting the destination possibilities. However, pathfinding will not take burrow boundary's into account.
There's a difference in intent here of using less CPU cycles vs. telling dwarves where to walk.
You can already tell your dwarves where to walk, somewhat, but you can't already save CPU cycles by doing manual pathfinding.
Post by: Shades on March 08, 2010, 11:25:54 am
There's a difference in intent here of using less CPU cycles vs. telling dwarves where to walk.
You can already tell your dwarves where to walk, somewhat, but you can't already save CPU cycles by doing manual pathfinding.
Actually that is what your doing when you 'tell your dwarves where to walk', the high traffic zones are a lower weighting to normal traffic and so will be searched in preference to other areas. Assuming your traffic zones connect that start and end of the path then the solution is found with fewer steps and so saves CPU cycles.
You could enhance this affect by increasing the weight of 'normal' in the ini file however this will slightly increase the search space in other situations so probably isn't worth it.
Post by: Draco18s on March 08, 2010, 11:44:40 am
::)
Post by: ggeezz on March 08, 2010, 03:31:42 pm
There's a difference in intent here of using less CPU cycles vs. telling dwarves where to walk.
You can already tell your dwarves where to walk, somewhat, but you can't already save CPU cycles by doing manual pathfinding.
Actually that is what your doing when you 'tell your dwarves where to walk', the high traffic zones are a lower weighting to normal traffic and so will be searched in preference to other areas. Assuming your traffic zones connect that start and end of the path then the solution is found with fewer steps and so saves CPU cycles.
You could enhance this affect by increasing the weight of 'normal' in the ini file however this will slightly increase the search space in other situations so probably isn't worth it.
Ahh, I hadn't thought about that.
Still though, there's a rather large difference between making the pathfinding somewhat faster in some situations vs. eliminating it altogether.
But I guess saying "you can already do this somewhat" is still technically accurate and moreso than I realized.
Post by: CobaltKobold on March 08, 2010, 05:57:29 pm
*Game cancels run (now paused): Urist McMason needs path to masonry shopSounds like a game genre!
::)
Post by: qwertyuiopas on March 08, 2010, 09:09:48 pm
By putting restricted zoning all over the external areas that dwarves shouldn't access anyway, the pathing would be less likely to spill out into the external areas and flood-fill the map during an attempt to navigate a massive maze.
Of course, if it already does minor pathing on the macro-grid level, such a waste of CPU might be discounted anyway...
Post by: KillHour on March 09, 2010, 01:27:47 am
Well, if I understand the pathing system correctly, it tries all least-cost unexplored tiles first(If that tile has, assuming the most optimal path starting at that tile to the destination, the least cost of all other tiles that the pather hasn't explored yet), that using restricted pathing zones where you don't want dwarves going often means that the path must be much longer before it will check that route for a connection, so if it would have taken another route entirely anyway, it will ignore that section for longer, and save cycles.
By putting restricted zoning all over the external areas that dwarves shouldn't access anyway, the pathing would be less likely to spill out into the external areas and flood-fill the map during an attempt to navigate a massive maze.
Of course, if it already does minor pathing on the macro-grid level, such a waste of CPU might be discounted anyway...
Basically, it works like this:
Say we have a fort:
Code: [Select]
██████████████████████
█++++++█θ+█θ+█θ+█++++█
█+y++++█++█++█++█+x++█
█++++++█+██+██+██++++█
█++++++++++++++++++++█
█++++++██████████++++█
██████████████████████
You want to get from X to Y.
Each square in the fort is labeled with high traffic (1)
Code: [Select]
██████████████████████
█111111█11█11█11█1111█
█111111█11█11█11█1111█
█111111█1██1██1██1111█
█11111111111111111111█
█111111██████████1111█
██████████████████████
Each 'tick', the path is flooded 1 square outwards.
The numbers and letters represent the number of ticks the pathfinder takes to get to each specific square. (after the 10 digits, I switch to letters, so a=10, b=11, etc.)
It stops when it reaches its destination.
*'s are squares the computer doesn't have to calculate.
Code: [Select]
██████████████████████
█*gfeee█cc█99█66█1112█
█*gfedd█bb█88█55█1012█
█*gfedc█a██7██4██1112█
█*gfedcba987654322222█
█*gfedc██████████3333█
██████████████████████
Lets say the bedrooms are forbidden (10)
Each forbidden square waits 10 ticks before being checked.
Code: [Select]
██████████████████████
█*gfeee█**█**█**█1112█
█*gfedd█**█**█**█1012█
█*gfedc█*██g██d██1112█
█*gfedcba987654322222█
█*gfedc██████████3333█
██████████████████████
Because of this, the bedrooms don't need to be checked, saving cycles. However, actually trying to go to a bedroom takes more cycles to calculate a path.
Post by: qwertyuiopas on March 09, 2010, 08:08:46 am
So if it got to a T junction, the path slightly towards the goal would get a bias, unless some other weighting factor made that tile cost slightly more.
So similar to that diagram except natually following fewer paths, I think.
Post by: tylor on March 09, 2010, 08:23:16 am
Post by: PencilinHand on March 09, 2010, 11:44:36 pm
A* adds heuristical minimum possible path cost from added tile to goal, which is not necessarily straight-line cost. More effective A*-based algorithms mostly works with other heuristics.
True, but they typically have a smaller node space(512^2 = 262k nodes is big in pathfinding, DF can be 50*288^2 = ~4.1 mill nodes) and limited connectivity(mostly just the 4 cardinal directions, sometimes 6 x-y-z axis directions, sometimes 8 compass directions, DF has 10 + special cases such as doors, ramps, flying, and swimming).
This has all been discussed back in the thread. The main conclusion of this thread was that the best course was to try and limit the ground pathable space by transforming it into a smaller and quicker to path node space(+95% of pathing is over ground so it makes sense to focus there). Several white papers with different methods were put forward, and some other novel ideas were suggested towards this end. A bit of code was written in the vein of one of the white papers and it was partly integrated with Khazad(there were pictures of paths posted), a few suggestions were put forward for how to make a benchmark for different methods but nothing was really put together.
Really, read the thread, or one of the dozen of other threads. Many things have already been discussed and it is just not an easy problem to solve. Significant concessions and game shaping choices may need to be made to really improve the pathfinding.
Knowing one way or the other would require Toady who has indicated he isn't really interested in this, at least currently. This is due to the necessity of waiting on/being pressured by other programmers and schedules which Toady has said repeatedly would steal the joy out of it for him. It is something of a miracle that the graphics code is being revamped like it is. Besides, Toady already feels like he may be looking into pathing related issues(siege improvements, multi-tile creatures, digging enemies, workshop and item hauling) in the "near future."
Post by: qwertyuiopas on March 10, 2010, 07:57:46 am
The resulting improvement would be minimal, but it is a tactic that can be used *now* without modifications to the game itself, and if you already have 300 dwarves and assorted pets, a 0.5% gain per path may add up to a noticable speed gain.
The best solution, however, would require Toady to edit the game itself, and that must wait at least until the end of the current release, hopefully soon.
Post by: ggeezz on March 10, 2010, 10:39:57 am
Post by: tylor on March 11, 2010, 02:24:08 am
Post by: Shades on March 11, 2010, 04:28:18 am
Post by: The Bismuth on March 11, 2010, 08:52:49 am
I was thinking of Fuzzy Voroni Diagrams.
PencilinHand has been advocating a hierarchical structure. This makes a lot of sense to me given the size of the problem. I was thinking that you could do this by focusing on landmarks. A subnode would be in the vicinity of the landmark which is easiest to reach. It may also be fuzzily in the vicinity of another landmark if that were nearly as easy to reach. This dual status would help determine the connectedness and distance between two landmarks.
By definition these regions zones would be connected internally, though probably not convex if they form a warren of tunnels. In that case though, the point where paths to the local landmarks diverged should give you a good hint that it was time to find a more specific route.
The blocking and opening of paths would alter which neighbourhood a sub-node was in, but this change would be localised. Similarly the landmark nodes would gain or lose neighbours, but the numbers of connections made and broken would be small.
(So Now I say it) I am not a computer scientist, more like a mathematician. I am more used to dividing problems up so they will be easy for people to think about. I don't know enough about the computer algorithms you are using to know if there is any efficiency gain from this.
Post by: Shades on March 11, 2010, 09:47:17 am
However to be honest we don't have a very big search space so it is unlikely methods other than A* will be more optimal. Not that I think that should stop us looking of course.
Post by: The Bismuth on March 11, 2010, 10:41:12 am
Post by: praguepride on March 11, 2010, 04:17:03 pm
So instead of trying to create this super-advanced efficient algorithm, you instead create specialized algorithms for specific problem points. Issues like:
Shutting down constant pathfinding when a unit gets trapped.
Creating seperate algorithms for flying/swimming/walking units that can bypass doors/walking units that are "dumb"
Quickly recalculating paths when disconnects happen.
I'm not an algorithm guy so I have no real clues to add, but it sounds like there are too many issues to deal with as a comprehensive system. So instead, try creating smaller systems that can join together to "fake" a comprehensive system...
For example (and again, forgive my ignorance). Certain actions could trigger different path-finding algorithms.
Take this scenario: The user switches a draw brdige that restructures the world. An algorithm that is very good at quick mapping new paths could be loaded to do a quick pass, then a different one that is better at continous path optimization could be restored to refine and control the more "normal" routes.
Perhapse the re-map could be triggered to certain major events: (seasonal changes for example) or could be triggered if X number of rejected paths crop up. So if a single door is locked and only one dwarf is effected by it, the whole map doesn't need to be redrawn just because one or two dwarves can't enter that room.
However, when the user floods the world and suddenly nobody can path, the huge number of "rejected" paths would trigger a remap. Perhaps a break should be put in so if flooding is progressing it doesn't continually redraw.
On that note, realism could cover some flaws. A dwarf trying to go to a draw bridge that's no longer passable might do so because he doesn't realize the bridge is closed. It's exusable for him to not have an instant knowledge of the entire map at all times.
During a massive event like a flood, perhaps pathfinding is heavily diminished to represent people in shock and awe at the coming tide of water/magma/goblins. Something could be put into place to recognize that a huge map-altering event is ongoing and that the map doesn't need to be re-calculated every second as it's just going to change again.
Perhaps activate a "safety" pathfinding that steers clear of flooding water/magma. So instead of constantly skirting close to it, the pathfinding just blocks off that whole section of map as inaccessible and encourages dwarves to go "the long route"
Post by: PencilinHand on March 12, 2010, 07:13:43 pm
Gosh, this is a long thread. I've skimmed it and reckon I understand where you are at well enough not to be completely irrelevant.
I was thinking of Fuzzy Voroni Diagrams.
PencilinHand has been advocating a hierarchical structure. This makes a lot of sense to me given the size of the problem. I was thinking that you could do this by focusing on landmarks. A subnode would be in the vicinity of the landmark which is easiest to reach. It may also be fuzzily in the vicinity of another landmark if that were nearly as easy to reach. This dual status would help determine the connectedness and distance between two landmarks.
By definition these regions zones would be connected internally, though probably not convex if they form a warren of tunnels. In that case though, the point where paths to the local landmarks diverged should give you a good hint that it was time to find a more specific route.
The blocking and opening of paths would alter which neighbourhood a sub-node was in, but this change would be localised. Similarly the landmark nodes would gain or lose neighbours, but the numbers of connections made and broken would be small.
(So Now I say it) I am not a computer scientist, more like a mathematician. I am more used to dividing problems up so they will be easy for people to think about. I don't know enough about the computer algorithms you are using to know if there is any efficiency gain from this.
What I was advocating and what was actually used by the code that was written are different. Read the white papers. Your idea about Voronoi diagram is interesting but it seems likely to be no less arbitrary and more likely to be less convenient than most of the other method of ambiguation listed in this thread. My idea was partly inspired by a method for approximating the length of a non-trival curve. The simplest visual approximation is to use this java program (http://www.me.vt.edu/chaos/vonkoch.html) to approximate the dimension of a Von Koch curve by iterating a series of boxes of decreasing size. I was originally proposing a little more than just that but if you are really interested in my poor explanation of it you can go back and read it without much difficulty.
I haven't checked through the 35-page thread completely yet, but has anyone thought about just "cheating" it?Almost all of your suggestions would require a level of interactivity with the "non-pathfinding" code that only Toady would be able to do any of it if he ever spun off the pathfinding code from DF. A notable exception, however, is treating every movement model(swimming, flying, lava-passable, invader digging) as a special case, which is inevitable. However, it would almost certainly be an overloaded function or pre-programed "dumb" AI behavior.Spoiler (click to show/hide)
The "realism covering flaws" bit is already in place to an extent. A dwarf will only repath once it runs into a block in its path. The idea of a "safety" pathfinding would have very limited practical usage and may be more likely to cause frustration on the user's end than any kind of real speed improvement.
Post by: qwertyuiopas on March 12, 2010, 11:44:34 pm
First, it can be *paused* at any time without losing the current state/location in the function.
Second, that pause/unpause command could be handled directly by DF itself, so that, for example, if DF hits both the FPS cap and the grapthical FPS cap consistantly, it could activate the pathing optimizer during the delay. Furthermore, DF could let it run when the game is paused.
Also, it's existance could easily be controlled with an external setting.
(Recently finished a 7DRL where the AI was on a second thread from the interface, so you could quit even when it was not your turn, and even if the AI froze.)
Post by: PencilinHand on March 13, 2010, 02:01:08 am
I *still* retain my opinion that a background optimization thread might be a good thing.
First, it can be *paused* at any time without losing the current state/location in the function.
Second, that pause/unpause command could be handled directly by DF itself, so that, for example, if DF hits both the FPS cap and the grapthical FPS cap consistantly, it could activate the pathing optimizer during the delay. Furthermore, DF could let it run when the game is paused.
Also, it's existance could easily be controlled with an external setting.
(Recently finished a 7DRL where the AI was on a second thread from the interface, so you could quit even when it was not your turn, and even if the AI froze.)
Except that there is nothing to optimize. The project as it was proposed would update its nodes and connectivity map when it is told there is a change and only develop a path when one was called for. There are limitless problems with trying to guess useful information before it is needed. Organizing any kind of cache of recent paths is more likely to be done on an as needed basis anyway. What could be optimized, or what work could be done that isn't already certain to be done as needed?
If what you are really pushing for is the library to function as an independent thread then that seems possible but likely that the benefits would be negligible most of the time. The processes would run regardless of if DF was paused, locked up, or otherwise limited but there would be some performance overhead associated with it plus some other likely complications.
If the paths that were being generated required some sort of iterative post-processing smoothing to approach optimality then I might see a use for something like that. A few of the white papers behaved something like that but nothing like that had been adopted.
Post by: immibis on March 13, 2010, 02:08:37 am
If what you are really pushing for is the library to function as an independent thread then that seems possible but likely that the benefits would be negligible most of the time. The processes would run regardless of if DF was paused, locked up, or otherwise limited but there would be some performance overhead associated with it plus some other likely complications.Consider:
1. The rest of DF is single threaded (except PRINT_MODE:2DASYNC, I think, in 40d#)
2. A lot of people have multiple cores now.
3. Pathfinding is one of the slowest parts of DF.
If pathfinding ran in another thread it could use another core, greatly speeding up DF.
Post by: tylor on March 13, 2010, 03:22:23 am
I think same aaproach could work for DF too...
Post by: immibis on March 13, 2010, 03:35:25 am
Post by: Impaler[WrG] on March 13, 2010, 03:55:41 am
tylor: I already have something like that I call it 'interrupt-able' A*. Each search and all of its nodes remain in memory until some kind of completion is reached (a full path or failure) only then are resources freed. The actual node expansion loop is controllable, I can tell it to run for some number of expansions or until completion. At any time I can query the search to get back the best partial-path found so far.
But I still need work out having an agent that started moving on that partial path smoothly transition to the final one if it turns out that the initial path was flawed and the agent was sent in the wrong direction, most likely I'd just request another path from scratch.
For those of you not in the know, the overall effect is simply to 'smooth' that 'hang' that occurs when a larger number of agents all start pathing simultaneously. Though their are some potential side benifits, like terminating searches when they grow past a certain length but that get a bit more complex.
Post by: tylor on March 13, 2010, 05:38:34 am
Btw, where can I check current state and source of Pathfinder?
Post by: Impaler[WrG] on March 13, 2010, 06:59:03 am
Post by: Draco18s on March 13, 2010, 10:18:45 pm
I remember doing quasi-threaded pathfinding in Flash. Game is called Garden Raider, and mobs there are moving using A*. If some mob wants a path, it addstask in queue, and wait. Then each frame some pathes are calculated for some limited time. That way sometimes mobs where standing there doing nothing, but that's better than lags, I think.
Sort of reminds me of AI War: Fleet Command. The more units there were, the less often each one performed a targeting update. The game is in fact so slick it can run 2,000 unit on 2,000 unit combats with no slowdown over a networked game (was just doing that an hour ago, actually, though my computer did chug when there were a huge number of ships traveling through wormholes, likely due to the number of ships and the transparency computation).
Post by: tylor on March 14, 2010, 12:13:33 am
Hierarchical pathfinding algorithms boas up to 10x performance optimization, so it seems possible...
Post by: Teiwaz on March 14, 2010, 01:02:03 am
I think generating a complete path to your destination is a waste. You're too likely to have the path invalidated (by map changes, something getting in the way, Dwarf deciding he's thirsty and dropping his current task, etc.)
Generate zones (areas of limited size which are contiguous for the most limited movement type possible in them - i.e. a normal floor area would be part of a zone where the entire area is contiguous for a creature which can walk but not open doors, an area of water for one which can swim but not open doors, a tile with a door on it for creatures which can walk and open doors, etc) with connectivity links (not specific paths, just info that they zones are connected) to adjacent zones. Generate an initial path on this (comparatively very simple) zone network, and then just generate a path to reach the next adjacent zone as you need it.
You'll end up generating a lot more paths, but the cost will be diffused over time, (this doesn't seem so important in DF as in games that demand a smooth framerate - but caravans and sieges arriving can still cause huge, nasty hitches from the cost of the initial long pathfind for a bunch of new creatures entering the map all at once) and with relatively small zones the individual paths should be fairly trivial to solve. Additionally, as you know you are only pathing as far as the next adjacent zone, you can limit pathfinding to within your current zone - so in the worst case, of a failed pathfind, you'll only floodfill your current zone. (Though this should never happen if your connectivity info is good.) Shorter pathfinds also reduce the number of paths which are invalidated and have to be regenerated when the map changes (a map change would only affect creatures currently pathfinding in the same zone, and would only affect creatures pathfinding through the zone but not currently in it if the connectivity changes).
Also, I'd put aside multi-tile creatures for now. They're an edge-case, and even when they're there, there will generally be very few of them. It's not worth optimizing for this at the cost of a better algorithm for single-tile creatures. Not to mention the current lack of multi-tile creatures...
Post by: immibis on March 14, 2010, 01:16:50 am
Not to mention the current lack of multi-tile creatures...Aren't wagons considered creatures? (except the one you embark with)
Post by: bombcar on March 14, 2010, 01:30:44 am
Not to mention the current lack of multi-tile creatures...Aren't wagons considered creatures? (except the one you embark with)
Yes, traders' wagons are 3x3 creatures, but they really travel around the "heart" of the wagon; you can see this in "D" mode. The starting wagon is treated as a creature in some ways; you can get error messages on some embarks that make this obvious.
Post by: praguepride on March 14, 2010, 08:27:15 pm
Post by: CobaltKobold on March 15, 2010, 01:46:29 am
This is why the fire trick works, indeed. It just does the DePOT-ACCESSIBLE thing for them to determine pathable space, I'm imagining.Not to mention the current lack of multi-tile creatures...Aren't wagons considered creatures? (except the one you embark with)
Yes, traders' wagons are 3x3 creatures, but they really travel around the "heart" of the wagon; you can see this in "D" mode. The starting wagon is treated as a creature in some ways; you can get error messages on some embarks that make this obvious.
Post by: immibis on March 15, 2010, 01:55:22 am
What's the fire trick?Yes, traders' wagons are 3x3 creatures, but they really travel around the "heart" of the wagon; you can see this in "D" mode. The starting wagon is treated as a creature in some ways; you can get error messages on some embarks that make this obvious.This is why the fire trick works, indeed. It just does the DePOT-ACCESSIBLE thing for them to determine pathable space, I'm imagining.
Post by: Shades on March 15, 2010, 03:21:39 am
3. Pathfinding is one of the slowest parts of DF.
Has anyone actually tested this? Because the pathfinding for a hundred or so critters on a map as small as we have should not be such a bottleneck, if it is a problem then simply improving the implementation of the A* used might be all that is needed.
Post by: praguepride on March 15, 2010, 09:14:05 am
But I guess that's another arguement for another thread...
Post by: random51 on March 15, 2010, 09:37:33 am
Post by: Shades on March 15, 2010, 09:59:04 am
Might have been mentioned before, but in terms of pathfinding performance in DF, pathfinding for invisible creatures applies a HUGE performance hit.
Do we know that? I've not seen anything to show that. Why would a few extra units cause so much more pathfinding because they are invisible than a siege causes?
I agree the game slows down as there are more units (but it does with more rocks too).
Also I'll agree breaking into the fun stuff causes it to slow down as well.
But before we actually try and build all kinds of fun, paralleled processed, highly optimised and specialised super pathfinding system shouldn't we actually run some sort of test?
Post by: random51 on March 15, 2010, 10:57:16 am
Might have been mentioned before, but in terms of pathfinding performance in DF, pathfinding for invisible creatures applies a HUGE performance hit.
Do we know that? I've not seen anything to show that. Why would a few extra units cause so much more pathfinding because they are invisible than a siege causes?
I agree the game slows down as there are more units (but it does with more rocks too).
Also I'll agree breaking into the fun stuff causes it to slow down as well.
But before we actually try and build all kinds of fun, paralleled processed, highly optimised and specialised super pathfinding system shouldn't we actually run some sort of test?
I did test. All it takes is a door. Close the door, my system runs fine, open the door and it drops down to almost nothing.
Did the same test with a siege which has even more units and the door test produces far less of a change in FPS.
Of course I'm assuming it is because they are invisible. It could just be their red noses and big feet.
Post by: Shades on March 15, 2010, 11:10:41 am
I did test. All it takes is a door. Close the door, my system runs fine, open the door and it drops down to almost nothing.
Okay so there should have been little difference other than scope of search space between those two states.
Did the same test with a siege which has even more units and the door test produces far less of a change in FPS.
Of course I'm assuming it is because they are invisible. It could just be their red noses and big feet.
However this makes no sense at all :) if it was purely pathfinding a siege should be as bad. Unless the invisible demons where fliers? but even then they should probably be able to path to a dwarf before needing to fly much.
Post by: random51 on March 15, 2010, 11:22:19 am
Post by: PencilinHand on March 15, 2010, 02:17:38 pm
I did test. All it takes is a door. Close the door, my system runs fine, open the door and it drops down to almost nothing.
Okay so there should have been little difference other than scope of search space between those two states.Did the same test with a siege which has even more units and the door test produces far less of a change in FPS.
Of course I'm assuming it is because they are invisible. It could just be their red noses and big feet.
However this makes no sense at all :) if it was purely pathfinding a siege should be as bad. Unless the invisible demons where fliers? but even then they should probably be able to path to a dwarf before needing to fly much.
The answer is special cases. WE don't know and can't test the actual differences between "invisible" creatures and visible creatures. Only Toady knows. There is probably something in the code that a profiler(which Toady recently started using) would show is causing a performance bottleneck. The detect test could be the source of the slowdown for all we know. Kind of like how the way a forbidden door doesn't stop pets from trying to path through a door but does stop the pet from going through it with the end result being a pet that is constantly trying to repath through the door. That is a special case that we can understand because it is visible and obvious if you are paying attention. It is thought that the "invisible creature" slowdown cause might be similar to the pet/forbidden door problem.
That pathfinding is a major source of slowdowns is not an assumption. To test this, make a simple fort and turn weather, economy, temperature, and invaders off, grow the fort to 200 dwarfs then remove all jobs so they are all idle and set a 200+ item stockpile to be moved from one place to another. You will observe a significant difference in the FPS between 200 idle dwarfs and 200 dwarfs hauling things even a short distance, and people have recorded this. I personally did a similar test with assigning 80 dwarfs to stone detailing and ordered a large area smoothed and watched as my FPS dropped from the low 40's to below 20.
Additionally, there is a general observation that the more pathable creatures there are on a map the slow the game runs, this is one of the reasons why caging animals improves game performance. Historically, pathfinding is a non-trivial problem to code and, our own observations, along with Toady's admissions, indicate that there are problems related to the pathfinding code. Like how flying units won't fly to something unless there is a valid ground path to their destination, which is a problem related to the implementation of the connectivity map. Other behavior also causes slowdowns, like how off-duty dwarves check un-owned chests and cabinets for jewelry and trinkets, which can cause dwarfs to spam for paths as every idle dwarf checks and rechecks every empty and unclaimed chest repeatedly.
Sieges, specifically, are another special case, it is our understanding that the target of a siege is the dwarf closest to the siege when it spawns. But for invisible units, the target may be an item, the closest child, or a random dwarf, which might be 15-z levels down and on the other side of the map AND may need to be repathed EVERY SINGLE frame because it has moved.
All of that said, there are other major causes of slowdowns. The graphics code has historically been one, but it is being addressed. Another know source of slowdowns is the number of items/objects on a map. The assumption there is that the dynamic tables being used to keep track of each item and all it's details is big enough that accessing them invariably leads to multiple cache misses and multiple calls to main memory. A little math, some knowledge of computer architecture, and a few assumptions about how item information is stored supports this assumption.
-----
I couldn't find the current algorithm (is there one? Conversation seems to go in circles a lot) but anyway, an observation:Toady has confirmed that the algorithm used in DF is a slight modification of A*. If you are asking about the implementation that Impaler[WrG] and shadow_slicer put together, read this paper (http://webdocs.cs.ualberta.ca/~nathanst/papers/mmabstraction.pdf), this power point (http://webdocs.cs.ualberta.ca/~nathanst/talks/mmabstraction_talk.pdf), this paper (http://webdocs.cs.ualberta.ca/~mmueller/ps/hpastar.pdf), and pages 23 through 26 of this thread.Spoiler (click to show/hide)
Post by: Shades on March 15, 2010, 02:54:25 pm
That pathfinding is a major source of slowdowns is not an assumption. To test this, make a simple fort and turn weather, economy, temperature, and invaders off, grow the fort to 200 dwarfs then remove all jobs so they are all idle and set a 200+ item stockpile to be moved from one place to another. You will observe a significant difference in the FPS between 200 idle dwarfs and 200 dwarfs hauling things even a short distance, and people have recorded this. I personally did a similar test with assigning 80 dwarfs to stone detailing and ordered a large area smoothed and watched as my FPS dropped from the low 40's to below 20.
Likewise take a new map, one dwarf and mine out a few thousand squares of rock, also watch the framerate drop. People have recorded this too.
To be honest my gut feeling is also that there is an issue with pathfinding (hence my posts towards the start of this thread), however I realised I couldn't prove it to myself so posed the question so see if anyone could. So far I'm not convinced we can show it is.
The main reason I want to prove that it is pathfinding is that I work with A* on a game with node spaces orders of magnitude larger than in DF and we don't have anything like this slowdown, we don't do quite so many calls but the time per call shows we could easily with a hundred or so units 50 times a second.
This leads me to wonder that assuming it is the pathfinding then maybe there is something bigger than mearly the algorithm used, such as pathfinding allocating node paths on the heap rather than the stack (which I know was an issue for our first implementation).
Post by: Teiwaz on March 15, 2010, 07:45:59 pm
The main reason I want to prove that it is pathfinding is that I work with A* on a game with node spaces orders of magnitude larger than in DF and we don't have anything like this slowdown, we don't do quite so many calls but the time per call shows we could easily with a hundred or so units 50 times a second.
I worked on a game which uses a variant of A*, with substantially fewer units (< 10) pathing over a much smaller network (< 2K nodes) than DF, and found pathfinding to be a huge expense when dealing with situations that basic A* doesn't handle very well. (Multi-story buildings, u-bends, etc. Anything which causes the heuristic to break down by forcing you to move substantially way from your objective to get there causes havoc with A*, as it quickly ends up flood-filling the network. DF is rife with these situations.)
Post by: qwertyuiopas on March 15, 2010, 08:42:45 pm
And if such recalculations turn out to take a lot of processing, then having a background thread do such processing and have the pathing in those areas temporarily revert to plain A* would suddenly seem more sensible.
Additionally, flagging each area with a general volatility rating, and only reclaculating for areas that generally don't change would keep recalculation to a minimum.
Most importantly, if optimization zone recalculation were a distinct thread, then pathing wouldn't totally halt during a long rezoning, it would merely be slower.
Now, if there were a very time-consuming way to further triple the speed of pathing in nonvolatile areas, wouldn't it make sense to calculate it when nothing else is happening?
And at the very least, specifying a framework where DF could signal the pather when to slow down or pause and when to go ahead with such background optimization would leave future decisions for the system more open.
Post by: immibis on March 15, 2010, 10:33:09 pm
Consider that every tile of a liquid under pressure needs to pathfind as well.3. Pathfinding is one of the slowest parts of DF.
Has anyone actually tested this? Because the pathfinding for a hundred or so critters on a map as small as we have should not be such a bottleneck, if it is a problem then simply improving the implementation of the A* used might be all that is needed.
Post by: Shades on March 16, 2010, 03:26:36 am
Anything which causes the heuristic to break down by forcing you to move substantially way from your objective to get there causes havoc with A*, as it quickly ends up flood-filling the network. DF is rife with these situations.
No it doesn't, no it's not. If you where getting these problems then either your implementation was broken or the heuristic you picked was flawed. A* is designed to solve exactly this problem, it's what it is good at, it's the one reason you use it.
Consider that every tile of a liquid under pressure needs to pathfind as well.
Actually only the top level (and pumps), and then it only needs to pathfind through a liquid, your node space is minimal in this case.
Post by: Impaler[WrG] on March 16, 2010, 06:32:09 am
Quote
Generate zones (areas of limited size which are contiguous for the most limited movement type possible in them - i.e. a normal floor area would be part of a zone where the entire area is contiguous for a creature which can walk but not open doors, an area of water for one which can swim but not open doors, a tile with a door on it for creatures which can walk and open doors, etc) with connectivity links (not specific paths, just info that they zones are connected) to adjacent zones. Generate an initial path on this (comparatively very simple) zone network, and then just generate a path to reach the next adjacent zone as you need it.
The flaw with pathing only from zone to zone is that some arbitrary goal needs to be put in each zone which results in a "choppy" path in which the agent unnecessarily moves to that goal point. My plan is to use a zone tree to both confirm connectivity and to progressively whittle-down a to a series of connected zones which will form a 'corridor' through witch an A* search will be confined. That prevents most wasted node expansions but still gives the optimum path.
Quote
If you are asking about the implementation that Impaler[WrG] and shadow_slicer put together, read this paper, this power point, this paper, and pages 23 through 26 of this thread.
Unfortunately we never progressed to the point of zone implementation, currently were at a decent tile-level A* implementation, a test-suite that runs large batches of randomly generated searches and reports time efficiency and a path-following class that serves as the interface between the Path engine and the agent and will navigate for an agent.
Post by: shadow_slicer on March 16, 2010, 11:51:09 am
Unfortunately we never progressed to the point of zone implementation, currently were at a decent tile-level A* implementation, a test-suite that runs large batches of randomly generated searches and reports time efficiency and a path-following class that serves as the interface between the Path engine and the agent and will navigate for an agent.
There was also a hierarchical A* implementation, but zones were statically determined square regions of size N by N. This method had some performance improvement over the tile-level A* for most maps, but wasn't significantly better. The framework should be general enough to use TCZs or other zones, but that wasn't implemented. It should even support multilevel hierarchies as well. It doesn't yet implement a breadth-first search or multi-item search or connectivity checking, but those could probably be added.
I had hoped to implement shortest path zones (zones that completely contain the shortest path between each of the nodes in the zone), but I haven't had so much time to work on it this year because of school. This is particularly difficult because the zones need to change appropriately in response to map changes. I haven't found a way to adjust the zones in such a way that I can guarantee they would achieve the same structure as if they were simply generated from the new map instead of simply being modified from the old one. This is not theoretically necessary, but it is sufficient for algorithm correctness and stability (preventing fragmentation) and would make it a whole lot easier to debug zone placement. It turns out this is actually a really difficult problem :-\.
tylor: I already have something like that I call it 'interrupt-able' A*. Each search and all of its nodes remain in memory until some kind of completion is reached (a full path or failure) only then are resources freed. The actual node expansion loop is controllable, I can tell it to run for some number of expansions or until completion. At any time I can query the search to get back the best partial-path found so far.Why don't you instead use 'interrupt-able' A* to path from the destination to the current source position (or even the expected source position after N moves)? That way you don't have to request another path if the node went in the wrong direction -- they would just back track a bit. Since the dwarves already appear to do this anyway because of the delay assigning jobs, it probably wouldn't be that noticeable. You could even terminate early if the current source position becomes one of the explored nodes in the shortest path tree that you keep for A*.
But I still need work out having an agent that started moving on that partial path smoothly transition to the final one if it turns out that the initial path was flawed and the agent was sent in the wrong direction, most likely I'd just request another path from scratch.
For those of you not in the know, the overall effect is simply to 'smooth' that 'hang' that occurs when a larger number of agents all start pathing simultaneously. Though their are some potential side benifits, like terminating searches when they grow past a certain length but that get a bit more complex.
Post by: Teiwaz on March 16, 2010, 12:30:12 pm
The flaw with pathing only from zone to zone is that some arbitrary goal needs to be put in each zone which results in a "choppy" path in which the agent unnecessarily moves to that goal point. My plan is to use a zone tree to both confirm connectivity and to progressively whittle-down a to a series of connected zones which will form a 'corridor' through witch an A* search will be confined. That prevents most wasted node expansions but still gives the optimum path.
Pathing to the arbitrary zone reference point is unnecessary. All you have to do is path into the zone. Depending on performance, you either path to the reference point, but stop as soon as you enter the zone and generate a path to the next one or you generate a path to a "best guess" point which is nearest to you inside the target zone (and then generate a new path as soon as you enter the zone for whatever reason, in case your guess was wrong.) Obviously, which approach is more efficient depends on zone size.
You may not get a perfectly optimal path in all conditions (I expect most of the problems would exist outside) but in the normal environment of Dwarf Fortress - a series of rooms and corridors usually connected by only 1 or 2 tiles, which would generally be the boundary of a zone as they often have doors - there's not very much variation between a perfectly optimal path and one that works. (In fact, I'd bet there wouldn't be a difference at all in most cases.) In any case, we're dealing with a game with performance issues. Generating a path quickly and efficiently is *far* more important than generating the perfectly optimal path that saves a dwarf a tile or two of travel time.
While restricting the network over which you need to path will help, generating a complete path still means you're generating a LONG path, and also one that is likely to be discarded (the case of targeting something that is itself moving, like siegers going after a dwarf and having to regenerate their path every frame, for instance), and is therefore wasteful. Pathing between zones means that you only have to re-generate the path if the moving target is in the same or an adjacent zone as you, which means it's a short path by definition and therefore cheap. (If they're in a different zone, your path doesn't change. Even if the target moves to a new zone, you only have to regenerate the "high level" zone path, and even then, you just need to generate it from the second-last zone in your path, with a maximum path length of 2. Cheap!)
Post by: Teiwaz on March 16, 2010, 12:52:09 pm
No it doesn't, no it's not. If you where getting these problems then either your implementation was broken or the heuristic you picked was flawed. A* is designed to solve exactly this problem, it's what it is good at, it's the one reason you use it.
Yes, the heuristic was flawed for that particular path network, but it worked fine for 95% of the levels in the game. (It was the default implementation used by Unreal 3, I believe. I suspect the heuristic was based on straight-line distance between the node and the destination, just like 90% of A* implementations out there.) That's the problem with game environments - they aren't very predictable. And once you let players start screwing with the environment, (DF lets you do this a little :P) things get worse.
Imagine the case of a dwarf standing on the floor immediately below the barrel of tasty Dwarven Wine he wants to get to. Somewhere on the level is a staircase leading up to the next floor. How do you avoid flood-filling the entire floor the dwarf is on until you find the staircase without doing some prior-to-runtime analysis of the map?
Post by: Martin on March 16, 2010, 01:45:58 pm
The approach I'd initially take is the hierarchical one - using certain elements to serve as reference points and build out a cache only using these points. The reference points would be:
door
stair
bridge
hatch
floodgate with mechanism
trade depot
etc.
I'd concentrate on the elements that the player can control (let the players interested in running massive fortresses design them with the pathfinder in mind), which routinely serve as chokepoints and are variable in state. This might also include workshops and stockpiles. Toady could even add the ability for the player to designate a reference point using the zone tool, or some such.
I overall agree with Teiwaz here. I wouldn't get obsessed with keeping this cache perfectly calculated for one main reason - we don't do that in real life, and because we know that actual travel costs in DF rarely match theoretical costs once the per-tile cost of traffic is included. IRL, we know we want to get to some place over there. Let's say we decide to ride our bike to the store. We know where we are going and we predetermine the reference points we're going to move through - street intersections and so on, but we don't have a priori knowledge of the state of these reference points. We know they presumably exist and we might have some probabilistic knowledge of their reliability, but that's about it. When we embark on our trip, we then discover the state of each of these reference points and recalculate accordingly. Construction may have a street blocked and we then recalculate from that observation to the next reference point on our journey. A gate might be locked and we recalculate again. Each time, we store that information so we don't repeat the mistake the next time. By and large, we use something that resembles a hierarchical A* system, but with different reference points, weightings and so on for each person.
So it's okay to have imperfect pathing, so long as it's imperfect in reasonable ways. Further, it's okay to embark with an incomplete path, so long as you have reasonable reference points - and that's where I'd put all the focus. There are some gameplay benefits to doing it this way, in fact. Right now, a dwarf can set a path across the map, only to find 99% of the way there that something has changed and they need to recalculate the path - perhaps having to retrace a huge distance. You can interrupt them by drafting them, etc. but otherwise they'll happily wander into a siege that every other dwarf is rapidly fleeing from. If there were a series of reference points the dwarf were navigating among, you'd only calculate a full path from the current position to the next reference point (much less A* work) and upon reaching that reference point, you'd check the updated state of the reference points in case something changed (such as someone pulled the 'stay inside' alarm), find the new optimal path from the cached reference distances (very fast), and then take the cached path. This way, the dwarf would adjust their path periodically based on new information arriving in the cache. By assuming that dwarves communicate such things freely, the game gets more realistic. Normally in a hierarchical A*, you don't have much discovery of new information in mid-path but with 200 dwarves running around, it's not unreasonable to have that occur, so let's call it a feature.
Each time a dwarf encounters a path which is no longer valid (a wall was put in, for example) the cache is updated for the distance between those two reference points (and possibly for all paths that share one of those two references, since there may be more), and dwarves will update accordingly.
One further benefit to this is that, memory permitting, the game can maintain two (or more) different sets of caches - one for dwarves and one for enemies. Dwarves will know all the shortcuts, but enemies should not, so their cache should be incomplete. Early sieges and squads may bumble about the place, but each encounter will fill in the cache a little more and cause them to act more efficiently. Depending on how memory efficient it is (it can even be saved to disk since it's only periodically invoked) different caches can be in place for gobbos, orcs, etc. so each one can have their own knowledge of your site and provide the player with differing behaviors, all for a relatively low cost. Further, species with different abilities would have caches better suited to their abilities.
Memory permitting, one other bit of data can be accumulated to influence pathing. In addition to the theoretical travel cost, the real cost can be stored for each reference node pair. The theoretical cost to go from A to B might be 100, but if it's a heavily travelled route, the last real cost might be 1000 as dwarves constantly climb over each other. That cost could be factored into the algorithm and also discounted over time so that as time passes, the stored real cost naturally declines, encouraging dwarves to try the route again. Again, this is more similar to how people behave.
Personally, I'd encourage people to embrace a slightly sloppy and fast solution over an ideal one.
Post by: Sizik on March 16, 2010, 01:46:24 pm
Post by: Draco18s on March 16, 2010, 01:49:02 pm
It just occurred to me, does Kobold Quest use pathfinding in any way similar to DF?
I think so, yes. But its not dynamic.
Post by: bombcar on March 16, 2010, 03:41:49 pm
If I have 20 dwarves sitting in the meeting area, and the designate 20 stone to be dumped, each dwarf paths to the stone individually.
But if the stones are near each other, in theory the "group" could path to the room, and then split up.
Post by: Martin on March 16, 2010, 05:50:13 pm
It should be possible to do "group" pathfinding.
If I have 20 dwarves sitting in the meeting area, and the designate 20 stone to be dumped, each dwarf paths to the stone individually.
But if the stones are near each other, in theory the "group" could path to the room, and then split up.
That's harder than it seems as the game doesn't realize that it's given 20 jobs at a similar location to 20 different dwarves starting at a similar location. If a hierarchical pathfinder can be worked out, that'd take care of much of the problem. In what I described, you'd make the meeting area a reference point, so half of the pathing solution would now be done. The other end could be dicier, but then the idea of the interruptible a* could help. If it turns out that paths tend to clump in that way, hold onto the last path nodeset (or the last few) and see if the destination lands in that nodeset. If so, just use that information recently calculated. If those search nodesets can be kept small due to a good hierarchical approach, then it's easier to hang onto them for a bit. Not sure how much Toady can trade memory consumption for CPU, but it seems to me he can afford a fair bit given the current state of things. A lot of these things are easily parallelizable as well.
Toady really needs libdispatch to be widely deployed. That'd be perfect for him to knock these things out. Geeks can read about it here (http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/12) (Just the 2 pages on Grand Central Dispatch). Each job assignment would spin off into its own thread, match it to the dwarf, do the pathing, and off we go.
Post by: Teiwaz on March 16, 2010, 06:01:22 pm
Post by: Martin on March 16, 2010, 06:09:23 pm
Post by: Andir on March 16, 2010, 07:16:02 pm
The game already provides a means to set up pathfinding routes. I'm not suggesting that players would be responsible for doing it, just that if an experienced player wanted to drop a reference point out where they do a lot of tree cutting or something, having the capacity to do so wouldn't be a bad thing.It's not unlike dropping a flag, beacon or lighthouse in real life. In fact, I think it would add a little bit of cool realism to have some sort of beacon system but I understand the above point. Some players wouldn't.
Post by: Martin on March 16, 2010, 08:32:43 pm
Some players wouldn't.
Oh, most players wouldn't. But then most players don't use the traffic tools now, either, but I'm glad that they're available for those of us who do.
The reason I say to give the user the option to do is is that players will work out the system and find a way to put it there anyway. If bridges turned out to be automatic reference points (and they might now as part of the connectivity map) then the more advanced users will just plop a bridge out in the middle of nowhere simply to serve the pathfinding function. We do all manner of nutty things now to label levers, force dwarves or enemies down particular paths, get water or magma to do something we want it to do, and so on. I'm just suggesting that the game might as well formalize the interaction with the pathfinding system rather than leave players to come up with some goofy scheme.
Post by: Starver on March 17, 2010, 04:56:11 am
One further benefit to this is that, memory permitting, the game can maintain two (or more) different sets of caches - one for dwarves and one for enemies. Dwarves will know all the shortcuts, but enemies should not, so their cache should be incomplete. Early sieges and squads may bumble about the place, but each encounter will fill in the cache a little more and cause them to act more efficiently. [...]different caches can be in place for gobbos, orcs, etc. so each one can have their own knowledge of your site[...] [saved to disk, etc]
While this isn't currently how knowledge is handled (in fact, it isn't) should it be so, a note I would add to this is that should the attack or siege result in every last man (/orc/goblin/elf/swarm of flies/whatever) dying, then knowledge gained during that attack is invalid (except maybe if there's something like a [psychic_link] tag on the creature raws?) and fortress access information reverts to none/the information known and 'loaded' up for the race just prior to their most recent arrival.
A good defence against invaders with this sort of limit (even if some live to head back on home) could be an ever-changing entryway. Although you'd really have to apply it to caravans, as well. In the absence of the extra-map dynamics promised in future versions anyway this could mean that caravans position their appearance on (accessible) parts of the edge closer and closer to the Depot location (as, of course, would invading forces better position themselves). At least while it (and the route to it) was static. Messing with the route causes delays for good and bad visitors alike.
But, of course, this leads to the whole idea of knowledge-based linkage asking whether dwarfs have their particular brand of omniscience at all, within their own fort. Something that I think is being shied away from, given how unbalancing that would be to the player (assumed to be inspirational guidance, however bad they are playing or deliberately destructive are their guiding instructions to the current noble-of-disfavour) without making it a completely different aspect to the game (e.g. Adventure-mode fortress building).
(Also, you could argue about when information known by resident dwarfs also applies to immigrants. Maybe the Hamlet/Etc Liaison's knowledge gets taken back after their visit, or a current native is tasked to wander over to within shouting distance to at least get them coming in the right direction... unless other changes to the game mean that immigrants can arrive after the total death of a fortress, when they have to start from whatever 'scratch' the game provides for your original settlers. However, as I said, that's a different game.)
Sorry, it's been a while since I offered my opinion, and I can't really compete with those who have been able to actively participate with the coding tests (my own lie covered in virtual dust, and there's no point digging them out now, given their isolation of development w.r.t. the main PFP occuring here).
Post by: CobaltKobold on March 17, 2010, 10:19:25 am
That, because of the way wagons are done, going up a one-thick ramp/wall/magma will put the donkeys through the magma, setting them on fire. But only going up.What's the fire trick?Yes, traders' wagons are 3x3 creatures, but they really travel around the "heart" of the wagon; you can see this in "D" mode. The starting wagon is treated as a creature in some ways; you can get error messages on some embarks that make this obvious.This is why the fire trick works, indeed. It just does the DePOT-ACCESSIBLE thing for them to determine pathable space, I'm imagining.
Post by: PencilinHand on March 17, 2010, 12:21:05 pm
... Many things have already been discussed and it is just not an easy problem to solve. Significant concessions and game shaping choices may need to be made to really improve the pathfinding.
Knowing one way or the other would require Toady who has indicated he isn't really interested in this, at least currently. ....
If you guys want to speculate that is fine with me, I would even be happy to see some code get produced from it. However, anything needing the level of interactivity with the game that is being proposed would require Toady to implement it and maintain it. We should not forget that Toady has said he would like to properly implement climbing and jumping, along with many other pathing algorithm breaking things.
Again, speculation is fine, but keep in mind that we are speculating over minutiae of an incomplete project.
Post by: ggeezz on March 17, 2010, 01:25:31 pm
With all of the recent talk I feel it is necessary to quote myself:... Many things have already been discussed and it is just not an easy problem to solve. Significant concessions and game shaping choices may need to be made to really improve the pathfinding.
Knowing one way or the other would require Toady who has indicated he isn't really interested in this, at least currently. ....
If you guys want to speculate that is fine with me, I would even be happy to see some code get produced from it. However, anything needing the level of interactivity with the game that is being proposed would require Toady to implement it and maintain it. We should not forget that Toady has said he would like to properly implement climbing and jumping, along with many other pathing algorithm breaking things.
Again, speculation is fine, but keep in mind that we are speculating over minutiae of an incomplete project.
The reason I bumped the thread quite a while back is that I thought Toady would be addressing the issue in the "not too far off" future. I would imagine this speculation would be to his benefit when that time comes. Not that he doesn't know what he's doing, but having had some people thoroughly hash out the topic is likely to be of benefit.
Post by: Shades on March 18, 2010, 03:10:45 am
... Many Significant concessions and game shaping choices may need to be made to really improve the pathfinding.
I don't see why, I didn't the first time you mentioned it either.
We should not forget that Toady has said he would like to properly implement climbing and jumping, along with many other pathing algorithm breaking things.
And why would these break a decent algorithm, it's just more nodes in the node map with different costs and/or movement flags.
Post by: Draco18s on March 18, 2010, 08:26:18 am
movement flags.
Storing that many movement flags per tile would be a huge amount of data, relatively.
Post by: shadow_slicer on March 18, 2010, 09:56:17 am
movement flags.
Storing that many movement flags per tile would be a huge amount of data, relatively.
If you think about it the amount of memory needed to store an entire embark square is already huge. Toady uses some optimizations such as not storing empty space or areas that haven't been modified since they were generated.
In any case, there's no real reason we need to store the movement flags at all. They can likely be calculated on the fly based on the data Toady already needs to store about the terrain.
Post by: Shades on March 18, 2010, 10:25:03 am
movement flags.Storing that many movement flags per tile would be a huge amount of data, relatively.
Movement flags could easily be a bitset, if the unit pathing has the appropriate flags it can use that link if not it can't. Ideally we can link directly to the map anyway. You have to specify some form of system to allow pathing under different conditions. Either that or have a whole node map for each set of movement types, which I imagine would be more space.
Post by: Teiwaz on March 18, 2010, 11:07:08 am
Movement flags could easily be a bitset, if the unit pathing has the appropriate flags it can use that link if not it can't. Ideally we can link directly to the map anyway. You have to specify some form of system to allow pathing under different conditions. Either that or have a whole node map for each set of movement types, which I imagine would be more space.
Why would you bother? Just get the information from the adjacent tile. You only need connectivity data for a simplified node network (pathing zones, TCZs, reference points) and there would be several orders of magnitude fewer nodes on the path network than there are tiles in the game, so memory wouldn't be an issue.
Post by: Shades on March 18, 2010, 11:09:56 am
Why would you bother? Just get the information from the adjacent tile. You only need connectivity data for a simplified node network (pathing zones, TCZs, reference points) and there would be several orders of magnitude fewer nodes on the path network than there are tiles in the game, so memory wouldn't be an issue.
You need to know if your allowed to path to the adjacent tile, as mentioned linking directly to the map is the best option but we do need some way to tell.
Post by: tylor on March 18, 2010, 11:29:49 pm
Non-military dwarves should also rely on their designated leader if possible for tasks like mining and hauling. It will encourage to stick miners and haulers in groups (with military screen) even if player is not planning to fight with them.
2. Pathfinding should not affect FPS. Everyone has processor allowance for pathfinding. Followers can transfer their allowance to leader. If something can't find it's path soon enough, let it wait.
Post by: smokingwreckage on March 18, 2010, 11:39:04 pm
Post by: Shades on March 19, 2010, 03:57:22 am
2. Pathfinding should not affect FPS. Everyone has processor allowance for pathfinding. Followers can transfer their allowance to leader. If something can't find it's path soon enough, let it wait.
Of course it will effect FPS, it uses cycles and you only have so many. Any taken up with pathfinding can not be used for other things. DF is CPU heavy so it's not like we are waiting on the graphics card (unlike most modern games)
Would it be feasible to "stack" units, in the spirit of tile-based strategy games? So, a group would all jump in the same tile as their leader or other assigned pathfinder, path once and all move simultaneously to the destination, then de-stack on arrival?
Feasible, and relatively easy, however I'm not sure the amount of saving you'd get would be worth the special casing. The majority of pathfinding is fairly independent. Certainly worth trying though.
Post by: Andir on March 19, 2010, 06:23:26 am
It doesn't have to though... it's much more realistic to have a dwarf pause for a while than it is to abort fluid simulations. (I'm referring to the frame calculation time.)2. Pathfinding should not affect FPS. Everyone has processor allowance for pathfinding. Followers can transfer their allowance to leader. If something can't find it's path soon enough, let it wait.
Of course it will effect FPS, it uses cycles and you only have so many. Any taken up with pathfinding can not be used for other things. DF is CPU heavy so it's not like we are waiting on the graphics card (unlike most modern games)
I tend to agree with tylor on this. Pathfinding can be a secondary event in a closed loop game engine. IMHO, pathfinding should be given a finite time to complete and if it's not complete in that time, further processing should be deferred until the next round. All that looks like to the player is that their dwarf is planning their trip. Maybe they "pull out a map and draw a route." Not many people care if that takes more than one frame to complete. Of course, minimizing this is important, but I feel that making the game wait while one dwarf decides what direction to take is a little asinine.
Personally, I'd love to have the actor mechanic shoved off to another thread that is independent of the world processing. The main thread has enough tasks dealing with the proper placement of fluid, calculating cave ins, lever actions, and other minutia that forking AI into one or more threads would make the game feel more visceral even if the dwarf did take a second longer to calculate a path.
Post by: Shades on March 19, 2010, 07:02:55 am
It doesn't have to though... it's much more realistic to have a dwarf pause for a while than it is to abort fluid simulations. (I'm referring to the frame calculation time.)
It still has to be processed, it will effect FPS, it uses cycles and you only have so many.
You can limit it only so many calculations an update if you like and that will cause the dwarf to pause briefly but don't think that means it's not effecting FPS. The best you can do is avoid the processing spikes (where it drops a lot for a couple of frames) so it runs at a more constant, but lower overall, FPS.
Post by: Draco18s on March 19, 2010, 08:29:35 am
The best you can do is avoid the processing spikes (where it drops a lot for a couple of frames) so it runs at a more constant, but lower overall, FPS.
Which is preferable in 99% of cases.
Post by: Shades on March 19, 2010, 08:45:00 am
The best you can do is avoid the processing spikes (where it drops a lot for a couple of frames) so it runs at a more constant, but lower overall, FPS.Which is preferable in 99% of cases.
I agree completely, although with DF I don't get that many spikes myself so I'm not sure how bad this problem is. Spreading the calculations across multiple turns only works if you get spikes of requests. Either way it still effects the frame rates.
Post by: Exponent on March 19, 2010, 10:19:34 am
There are ways to keep determinism while pathing in the background, but the efficiency improvements would probably be limited, and would come almost entirely from the presence of multiple cores. And the code would probably become significantly more complicated in the process.
Post by: Andir on March 19, 2010, 10:29:38 am
Have you ever used the "All dwarfs except military are forbidden outside" at all? When you trigger that, lock a major door in your fort or several other things that cause a mass recalculation there's a spike in processing (or in the case of DF, a severe drop in FPS.) If not that, try assigning all dwarfs to military duty once. They will path somewhere and stop. Your FPS will skyrocket. Heck, even in the current build all you have to do is designate an area to be ramped out and you can watch the pathfinding throttle your fortress or forbid/allow collecting stone outside.The best you can do is avoid the processing spikes (where it drops a lot for a couple of frames) so it runs at a more constant, but lower overall, FPS.Which is preferable in 99% of cases.
I agree completely, although with DF I don't get that many spikes myself so I'm not sure how bad this problem is. Spreading the calculations across multiple turns only works if you get spikes of requests. Either way it still effects the frame rates.
I'd much rather have a consistent game/simulation FPS than have pathfinding impact general performance.
Post by: qwertyuiopas on March 21, 2010, 12:53:21 am
One relatively simple solution would be to only calculate some number of paths per frame, based on laziness of the individuals and length of the paths, so that a few will be delayed a frame rather than drop the framerate.
Also, when finding jobs like mining, maybe a short-range flood fill might be useful? As mining often runs in veins, checking the nearest 5 tiles for further jobs and skipping pathing entirely would be a great improvement, both in expected behaviour and overall speed. Some similarily clustered jobs could also benefit from such a system.
Post by: Richards on March 21, 2010, 03:53:20 pm
Post by: zwei on March 21, 2010, 03:54:36 pm
Active player pauses game quite often and it is ideal time to run more expensive pathing optimizations or to run pather preventively (example for miner was given. It is not so far fetched for pather to precalculate his path for dozens on mined squares.)
Also, lag spikes usually happen right after player unpauses after giving commands that result in huge amount of pathing (i.e. moving goods to trade depot, mass dumping something). For example as player marks goods to be transfered, pather can begin to calculate path for hauler to item and further patch with item to trade depot. When player leaves dialog and unapauses, precalculated paths are committed.
Especially useful for wagons as game pauses when caravan arrives and wagon pathing is expensive.
Post by: NoahTheDuke on March 21, 2010, 05:41:20 pm
What about making use of player-pauses?
That's been discussed and rejected at least five times so far.
Noah
Post by: Akhier the Dragon hearted on March 22, 2010, 12:41:49 am
Spoiler (click to show/hide)
Post by: Andir on March 22, 2010, 06:53:07 am
In a lot of the post I see people talking about breaking areas down into small subsections and what not. if from there you made just ........ tip of the tongue and it just wont come off here have an example of what I am trying to say, sorry if this is what someone has already suggested and I just did not understand it because of it going over my head or me just missing it becuase I just read this whole thread in a couple hours straight.I have also thought about this, but apply your same logic to a flat open plane. Where you enter each block would determine the shortest path.this is the best I can explain what I am thinking. Most of my programming experience is with vb.net and c# so all the talk about A* is over my head by a bit. Also as a question I am currently learning programming in collage and in my free time, what language should I learn if I want to make games more specifically roguelike games?Spoiler (click to show/hide)
Post by: shadow_slicer on March 22, 2010, 08:38:06 am
In a lot of the post I see people talking about breaking areas down into small subsections and what not. if from there you made just ........ tip of the tongue and it just wont come off here have an example of what I am trying to say, sorry if this is what someone has already suggested and I just did not understand it because of it going over my head or me just missing it becuase I just read this whole thread in a couple hours straight.The pathfinding code we're currently playing around with already implements such caching. In some cases it does significantly increase pathing speed. The bigger benefit from hierarchical pathing will be in the way it limits the size of the search space (i.e. if you know that B and C are not connected but D is connected to B and C, then you won't try to find a path to c from within B. Instead you should path towards D first). I don't think that part is fully implemented :-\.this is the best I can explain what I am thinking. Most of my programming experience is with vb.net and c# so all the talk about A* is over my head by a bit. Also as a question I am currently learning programming in collage and in my free time, what language should I learn if I want to make games more specifically roguelike games?Spoiler (click to show/hide)
Post by: Draco18s on March 22, 2010, 10:42:29 am
Spoiler (click to show/hide)
Of course, the paths you're suggesting be cached there are extremely trivial to recalculate (probably more so than to look them up). They're strait-line in a 1 tile wide corridor.
Post by: SkyRender on March 22, 2010, 10:43:17 am
Basically, it works something like this: for east, north, west, south, up, and down, perform an expanding cone of searches out 1 tile "forward", "up-forward", "down-forward", above, and below (for cardinal directions; up/down it'd get a lot messier since you'd have to do a genuine 9-tile radial search). If an acceptable item is found (or the trailer in that direction is impassible), its location and distance are noted (if there was an item; otherwise, it returns null) and that trailer of the search is cut off. Repeat until all trailers either locate something or null out, then do a comparison to find the genuinely closest object. This would lead to a LOT of trailers, and some noticeable search overlap, but it would also have very thorough coverage. Its main issue (besides the massive number of small checks) would be very odd layouts with lots of corners.
Post by: Exponent on March 22, 2010, 11:19:29 am
...Sounds like a straight-forward breadth-first search, which unfortunately tends to be very slow and consumes a lot more memory.
The theme for this thread has largely been about how to improve pathfinding performance, with pathfinding quality being secondary, so improving the nearest item search has taken a bit of a back seat. And rightly so, in my opinion. It's a tough nut to crack, but might be easier if general pathfinding is significantly improved. Especially if a lot of useful data gets cached in such a way that it can be used to also make some quick guesses about which items are truly closest. The hierarchical stuff could help with that, for example, by providing very quick upper/lower bounds or estimates of costs from one area (dwarf) to another (potential item to grab). That way, multiple potential items could be scanned quickly, and the ones that obviously involve a long path (despite being physically close) can be skipped over without a large hit to performance.
Post by: Shades on March 22, 2010, 11:29:14 am
Sounds like a straight-forward breadth-first search, which unfortunately tends to be very slow and consumes a lot more memory.
Iterative depth first is probably a better option to solve this as although it takes longer the memory overhead is minimal.
Post by: Akhier the Dragon hearted on March 22, 2010, 01:03:14 pm
Of course, the paths you're suggesting be cached there are extremely trivial to recalculate (probably more so than to look them up). They're strait-line in a 1 tile wide corridor.If games really did always do straight line paths then okay, I just used them to make it easier on me, for instance I could have made the blocks be 100x100 and filled with the most odd and winding corridors you ever did see. also I look back on my idea and have realized that it should have been check that those paths where clear first because if it was not then you would not have already pathed anywhere you did not need to like in my example where you would have already pathed from point a to side of block B and so on when you might end up not even going into block B at all depending.
I figured that any of the precalc could be done during load up like everything else and that the goal was to make the pathfinding faster so I would be fine add a little extra time at start up for faster speed while playing....Sounds like a straight-forward breadth-first search, which unfortunately tends to be very slow and consumes a lot more memory.
The theme for this thread has largely been about how to improve pathfinding performance, with pathfinding quality being secondary, so improving the nearest item search has taken a bit of a back seat. And rightly so, in my opinion. It's a tough nut to crack, but might be easier if general pathfinding is significantly improved. Especially if a lot of useful data gets cached in such a way that it can be used to also make some quick guesses about which items are truly closest. The hierarchical stuff could help with that, for example, by providing very quick upper/lower bounds or estimates of costs from one area (dwarf) to another (potential item to grab). That way, multiple potential items could be scanned quickly, and the ones that obviously involve a long path (despite being physically close) can be skipped over without a large hit to performance.
Post by: peterix on March 22, 2010, 01:14:40 pm
I'm not sure how practical it is (or if anything like it has been suggested), but it seems to me that the main issue here is the search algorithm for "nearest item" more than the actual pathfinding to the object. It would make sense to improve that first, instead of using the existing "find nearest in terms of sheer physical distance" algorithm. What I came up with for that is a sort of radial trailer pathfinding algorithm. It's directly fractal at the moment and could use some refinement as a result, but I think it's on the right track.This assumes that looking up what item is at which coordinates is a free operation. It's a very expensive one. O(n), where n is the length of the item vector. It could make sense if it was O(log n) or O(1). Not like this.
Basically, it works something like this: for east, north, west, south, up, and down, perform an expanding cone of searches out 1 tile "forward", "up-forward", "down-forward", above, and below (for cardinal directions; up/down it'd get a lot messier since you'd have to do a genuine 9-tile radial search). If an acceptable item is found (or the trailer in that direction is impassible), its location and distance are noted (if there was an item; otherwise, it returns null) and that trailer of the search is cut off. Repeat until all trailers either locate something or null out, then do a comparison to find the genuinely closest object. This would lead to a LOT of trailers, and some noticeable search overlap, but it would also have very thorough coverage. Its main issue (besides the massive number of small checks) would be very odd layouts with lots of corners.
I believe (based on my knowledge of the reverse-engineered internals), that locating the item is done by scanning the vector once and using some distance equation like this one: http://en.wikipedia.org/wiki/Taxicab_geometry
So, searching for the item works like this:
Code: [Select]
for each item
test if the item is reachable
if(reachable)
if(distance to item < distance to final item)
final item = itemI also believe that the 'reachable' test can be done 'for free', because DF stores information about the connected areas as part of its map (if item and dwarf are in disconnected areas, item is not reachable). This explains the lag from manipulating doors, floodgates and bridges using levers - the connected areas are recalculated.
Post by: Draco18s on March 22, 2010, 01:38:28 pm
Post by: Teiwaz on March 22, 2010, 03:05:31 pm
Most of my programming experience is with vb.net and c# so all the talk about A* is over my head by a bit. Also as a question I am currently learning programming in collage and in my free time, what language should I learn if I want to make games more specifically roguelike games?
Language doesn't really matter all that much - A* is just an algorithm, and can be done in any language. (It, and variants, tend to be the most commonly used pathfinding algorithms as it's pretty much the best thing we have). I'd look into XNA (www.xna.com) - it's what "Xbox Live Arcade" games are built with, and it uses C#, so your experience with that language isn't wasted. (It can also be used to make windows games, windows phone games, etc.)
It's an excellent platform to develop on as a hobbyist, as it does a lot of the painful, low-level stuff for you so you can get on to the more interesting parts of the game. (Things like input, and it has an excellent content pipeline to get sprites, or even 3d models into the game easily, etc.)
Post by: Akhier the Dragon hearted on March 22, 2010, 05:49:37 pm
next question should I learn some C and C++? it seems that many of the examples and stuff that I can find for roguelikes is in them and as I have learned from this thread DF is in C++.
On topic now I think that the movement forms could be done in a different way from what everyone seems to want with a different thing for walking flying and all that. you could build more of a tree with different forms of walking because flying is just walking with more area and walking is just where pets can walk plus doors that are tightly shut and so on. the only stand alone form of moving is swimming and even that ties in with amphibious creatures so you would not really have to build a giant list of where each movement form can go because that would have to much duplication of data and be inefficient.
also amphibious and magma swimmers are basically the same thing except for where they swim so the movement is the same, just in different liquids.
Post by: Teiwaz on March 22, 2010, 06:47:27 pm
thanks Teiwaz!
next question should I learn some C and C++? it seems that many of the examples and stuff that I can find for roguelikes is in them and as I have learned from this thread DF is in C++.
Don't bother with C, C++ will teach you everything C can do, and more. C can be a good stepping stone for C++, but C# will do just as well for that and will get you into an object-oriented way of thinking.
C++ is really, really powerful, but can be simply brutal. Absolutely learn it if your plans are to go into the industry as a programmer, but put it aside for now - C# will teach you most of what you need to learn about programming, and should last you a good long while. Go to C++ when you want to learn C++ - it can be a nightmare, especially if you're trying to get a working project out of it. Just getting something to compile can be a colossal pain in the butt, doubly so if you're trying to integrate other peoples' code to do the stuff you don't want to worry about (and I'm talking really low level stuff here, like polling the state of the keyboard or figuring out where the mouse is, or playing a sound that isn't a default windows beep, or loading an image into memory and displaying it on the screen).
There's a whole lot more to being a good programmer than knowing language X or Y. It's more about being able to figure out how to do things, being able to write easy-to-read, well commented, functional, efficient code, being able to squash bugs quickly and without causing new ones, and being able to familiarize yourself with a new codebase quickly, whatever the language. If you can do that, knowing the syntax and foibles of a particular language is secondary.
Post by: Andir on March 22, 2010, 09:12:00 pm
Post by: The Bismuth on March 22, 2010, 10:20:32 pm
The only problem with Taxicab Geometry with regards to stone distance is that the Z-axis is not as accessibly traversable as the X and Y are.
The Taxicab Geometry can easily be extended to 3 dimensions.
--edit-- (removed off topic screaming)
Post by: Akhier the Dragon hearted on March 23, 2010, 02:34:33 am
I'd actually argue that knowing a bit about C before diving into C++ is a great benefit. I'd actually recommend reading about data types, how they use memory and pointers from a C perspective, but from that point, branch into C++, reread how all that works in C++ and you'll start to understand it a little better than going straight to C++. All this will really cost you is a few chapters of reading about C. C++ makes C much more powerful IMHO (and people will disagree) in many ways.This sounds like a good idea. I probably have to take a class of C anyway so nothing lost in doing it anyway.
Post by: Shades on March 23, 2010, 03:34:51 am
The only problem with Taxicab Geometry with regards to stone distance is that the Z-axis is not as accessibly traversable as the X and Y are.The Taxicab Geometry can easily be extended to 3 dimensions.
I like how you quoted the reason why it's not good for the third dimension in DF as you posted about how easy it was to extend. It's almost like starting an idea with the perfect point why your idea is flawed :)
Post by: The Bismuth on March 23, 2010, 09:45:11 am
I like how you quoted the reason why it's not good for the third dimension in DF as you posted about how easy it was to extend. It's almost like starting an idea with the perfect point why your idea is flawed :)
You are right, I misread Draco's post. I might have been a little tired and emotional.
Post by: qwertyuiopas on March 25, 2010, 07:12:17 am
(and I'm talking really low level stuff here, like polling the state of the keyboard or figuring out where the mouse is, or playing a sound that isn't a default windows beep, or loading an image into memory and displaying it on the screen).
Polling the keyboard is hard? Doesn't the (win32) function GetKeyboardState (http://msdn.microsoft.com/en-us/library/ms646299(VS.85).aspx) do exactly that, in a single statement?
Post by: Teiwaz on March 25, 2010, 10:35:08 am
I'm on my way out the door right now, but compare the process in C++ (and I don't just mean "call function X" - I mean setting up the IDE and compiler, resolving cryptic link errors, including the header file, etc. And do you really think inflicting the Windows API on someone is a good idea? I mean anyone, never mind someone who is there to learn how to program.) to a more accessible platform like XNA. Inflicting the former on a beginning programmer just encourages frustration and abandonment of a project. People learning to code shouldn't go anywhere near C++ until, well, they've learned to code and are ready to focus on the technical issues of C++ specifically.
Post by: Richards on March 25, 2010, 02:22:06 pm
From a technical standpoint I think the performance issues of dwarf fortress is the greatest limiting factor in the game for the future. I like the idea of a new separately-threaded path-finding routine but all this talk won't go anywhere unless Toady gives some kind of foundation to build an path-finding routine on.
Has there been any talk along these lines? It's kind of hard to make any progress in this field until you know specifically what you need to start the "PathFinder Project". Has Toady ever been contacted seriously about this?
Post by: TiagoTiago on March 25, 2010, 08:51:49 pm
How about, after defining the regions in whatever way you think is better (convex shapes, rectangles, line of sight or whatever) you store for each region the best path to cross it coming from one neighboring region and exiting into another, and then unless the destination is inside the region, you just access the preprocessed connecting path?
edit: If you divide the map in squares/cubes (like, first you start with a single cube, then you get 8, then each of those 8 is divided into another 8 and so on until the smallest is a single tile wide), i think this could work for multiple resolutions, like you read the whole map at low resolution, then you keep increasing the resolution along the path until you get full resolution detail for the path that is gonna be used while keeping the unused areas at lower resolutions, or it might even be possible to just decrease the resolution away from the current position in a gradient so that even less data has to be loaded at any single time.
Post by: Shoku on March 26, 2010, 12:53:43 pm
Is this all reasonable?
Post by: Beeskee on March 26, 2010, 03:20:01 pm
Post by: qwertyuiopas on March 26, 2010, 06:32:38 pm
With all that wandering from the point of this discussion(should have used A*, so it would have been at least the shortest path), let me put this back to at least where my post derailed it:
Asking the player should never be required. Using what is there(like processing during pauses) is a good idea, but the design must focus around the idea that such events might never happen, so treat them as bonuses that help but if they aren't present then things MUST still run smoothly, or ignore them entirely since they would likely give little gain relative to the time to add them. (I strongly believe in the first, using when present but designing with their absence in mind, and some others are adamantly set on ingoring them outright, since they can't be guarenteed to ever happen or some other reasonable reason)
Post by: Shoku on March 27, 2010, 12:02:09 am
You could just have every unit go making their own tunnels for whatever was closest but I think that would be horrible. Having just a set number of units per squad go a digging would be a little better. For cases other than a bunch of prime entry points this would just mean a bunch of tunnels around the optimal entry position so this seems poor as well.
"I'm not moving" is usually a panic condition that is treated like a bad thing because a disorganized crowd shoving to get through a choke point is really ugly if it happens all the time. You don't want an army lining up to fight one guy when there's something they could be fighting two steps away, but at the same time but it is quite clear that taking a longer path instead of waiting is also really bad. Making waiting okay seems like a small weight lifted off of the processor while making the running around look much more purposeful, specifically because it would be.
Post by: tylor on March 28, 2010, 06:08:44 pm
Post by: Starver on March 29, 2010, 05:01:54 am
How hard is it to do pathfinding for creatures that are able to dig, or even construct bridges and ladders? For, like, some really resourceful siegers?It needs at least modifications to the base pathing[1], if not parallel pathing trees (as previously discussed) which treat solid earth, bridgable gaps or climbable walls as appropriately high-cost travel routes[2] or limited-accumulation[3] pathway management where it's a matter of resource-based resourcefulness.
Some might consider that a whole can of worms, of course. :)
[1] Possibly arbitrarily finding a tunelling/bridging/laddering point and assessing paths from origin to one side and from other side to destination, and seeing if that produces a better/more possible path than straight pathing. But unless the arbitrariness could be optimised to something rational (smallest gap between the otherwise unjoined 'inside' and 'outside' pathing trees, biased towards the gap-points most accessible to the agents' starting point/least dangerous for them to attempt) I could see high CPU pursuing, and it starts to look more and more like the other solution anyway...
[2] Which may be mitigated if the digging/bridging/laddering creatures are 'responsible' for creatures not of that ilk (e.g. under the command of a squad leader with a number of other subordinates, all of whom are currently sufficiently unable/unwilling to go another way... When such a squad-opinion is triggered, the sappers or other engineer-like units take on leadership role (or at least swap leader/led roles with the usual head honcho) and have their usual non-walking routing difficulties offset smewhat.
[3] Within the pathing algorithm, wall can be climbed (or other appropriate method used) for only nominal cost, but each level of wall that is climbed depletes a counter (on that routing algorithm's particular logical branch) that holds the finite amount relating to 'ladders that can be used'... Routes that run out of ladder (or bridge, or some measure of digging effort) can no longer branch over that kind of obstacle. Of course, one has to define what priority a short but 'effort using' traversal has over an equivalent long-way-round route, and whether one continues both routes onwards beyond the obstacle, in order to cater for later diversions or traversals... i.e.
Code: [Select]
A B C D E F
#################
>..++_+++++_+++++_++..>
#+#+###+#+###+#+#
#+#+# #+#+# #+#+#
<arbitrary continuation>
#+#+# #+#+# #+#+#
#+#+# #+++# #+#+#
#+#+# ##### #+#+#
#+#+# #+#+#
#+#+#########+#+#
#++++++___++++#+#
#################
G
Given 'x' amount of bridging ability, the routes from start to finish range from completely non-bridging (down A, back B, down C, back D, down E, back F) to bridging any or all of A_B, C_D and E_F or even going via the triple-length G-bridge, which might need one, two or three bridging elements, depending on how much based on the player's own bridging materials requirements it is, but also might involve a hefty diversion southwards as part of the assesment of routing efficiency.It might even be worthwhile for a suitably staffed unit to consider a route-finding branch which integrates the deployment down towards G with both its bridge engineers and some mining units (ignoring for now the possibility of mining across the other small gaps instead of bridging) in order to mine more bridging material to traverse even longer 'G' gaps, if known/discovered traps are found lying across the BC and DE stretches... But that's getting complicated. As may be the question of whether a bridging unit can bridge across a gap that has a bridge built upon it[4] but one that is currently retracted? That might need either a modification of the building code or 'temporary' bridging types of some form that can be laid across such gaps.[/3]
[4] Also note possible future 'grappling hook' possibilities, to 'grab' raised bridges (or retracted ones, presumably retracted into the opposite side), as another non-standard routing option. Or at least attempted, with the possibility of breaking the bridges or loosing the hooks...
Post by: Dark_Tundra on March 30, 2010, 09:25:40 am
I think a node graph of landmarks sounds like a good idea, especially if the graph contains distances, (perhaps checked one at a time?) and can reduce the search area.
The dwarfs would then have a rough idea of how to get where they are going and only need to do fine pathing between one node at a time.
If a dwarf then needs to get to an object on another floor; they could path to the stairs, knowing that they would eventually get to the object, even though they havent calculated a path all the way yet.
Then if an A→B→C path is the same , or a better fit than an A→C path, the A→C path could perhaps be culled.(there isn't really any point I can see for doubling up with a shortcut that takes more time)
Assuming this was used, I'd suggest stairs and doors as landmarks, for and landmarks be grouped when possible, a row of ramps could be one long ramp rather than several choosable ramps.
I apologise if this has already been covered, though I have read the thread, I don't understand everything that is talked about. (Exept that multithreading in no way improves pathfinding algorithms, which is the whole point of this project.)
Post by: ilsadir on April 06, 2010, 08:06:59 pm
All of these methods are simply additional connections in the pathfinding search space. If you can bridge a gap, then the otherwise impassable 'air' tile next to a ground tile is accessible. If you can only bridge in straight lines, the second 'air' tile is free, but after that the line is set. If you can only bridge X tiles, after X, air-tiles are impassable.
The trick is just getting the cost correct.
Post by: Starver on April 07, 2010, 07:26:17 am
If you can only bridge in straight lines, the second 'air' tile is free, but after that the line is set.As long as no supports exist to allow a new-bridge. (Although maybe weighted preference should be to a few longer bridges (with less dense material needs per distance) zig-zagging around an awkward gap, rather than single/double bridge-tiles being built hugging the wall and taking roughly 1:1 material:distance to do so... Really that needs a picture to explain properly.)
Quote
If you can only bridge X tiles, after X, air-tiles are impassable.And in two ways:
- The actual inability to create bridges of >10 length (save by connecting two end-supported <=10 bridges together to bridge a gap of 20, of course). This would be an upper limit.
- Materials available. But if materials can be harvested, then this might just be a weighting against, and associated with tasking an allied/squad-attached axe-unit to go and chop down a couple more trees.
Quote
The trick is just getting the cost correct.Agreed. Utterly.
(Might be simpler to allow siege units to have at hand a certain number of specific-length hurdles/ladders/rope bridges, upon spawning, and limit them to that, and one bridging-item per given allowable bridging length, non-reusably. Unless and until the AI can handle tree felling or rock mining than architecturally designing bridges or otherwise constructing their access points from locally obtained materials with a degree of sanity. Actually, I like the idea of rope bridges (or at least ropes). That would solve the whole idea of double-bridging, because you'd need to have them capable of throwing and grappling/pitoning one end of the rope onto the other side of the gap (or on top of the wall they wish to ascend), and it could even be considered a pathway only available to trained siege troops (maybe not even all units within a sieging squad) and so could leave awkward enemy-accessible routes all over the place without necessarily adding to the Fortress's own normal entrance and exit plan, to be 'cleaned up' a bit like a more resilient spider's web)... Maybe even add skill "climbing (rope)" and/or similar, as a military thing... Sorry, getting overly speculative there. Ignore most of that. :))
Post by: Draco18s on April 07, 2010, 08:15:29 am
If you can only bridge X tiles, after X, air-tiles are impassable.
How would that work? If you can only bridge 2 tiles, max then where does the pathfinder put its bridge?
###########
#+++2+++++#
#+#+#####+#
#+#+##++++#
#+#+##+####
#+#+##++++#
#+#+##3##+#
#+#+##++++#
#+#+##+####
S+1+45++++E
###+##+####
###+##++++#
###+#####+#
###+##++++#
###+##+####
###+##++++#
###+#####+#
###+++++++#
###########
If the pathfinder puts the first bridge at that 1 tile pit (#1) then it can't skip the 2 tile pit (#4, #5) for extra-short path.
And if it bridges #2, it still can't get across #45, but could skip over #3 (shorter than bridge over #1).
But if it skips over #1 and #2 and builds a bridge over #4 and #5, then it gets the shortest path.
Finding where to build your bridge is a non-trivial problem.
Post by: Shades on April 07, 2010, 08:29:12 am
Finding where to build your bridge is a non-trivial problem.
You just treat all air in a straight line as passable. It's a trivial problem assuming an algorithm such as A*, it just increases the size of the search space. Obvious cost for bridge squares are higher but that is just a detail. Digging means treating every square a passable which again increases the search space assuming the cost is higher than walking. (if the cost is the same as walking then the quickest path is a straight line)
Edit: Just realised you might have meant for the two square task, in which case it's similar but with limited air squares allowed, still fairly trivial.
If you meant the best location based on player defences then it becomes non-trivial.
Post by: tylor on April 07, 2010, 09:37:10 am
Thay can have unlimited nomber of rock on them? pick and masonry/mining skills each and just build floors and dig rock as they go. If thay die in numbers (to magma or some other trick) they can just mark that place as (too dangerous to be around) and try to dig to fortress entrails from some other side.
Only thing that can't be passed without some very obvious cheating, I think, is lava or deep water.
Post by: Draco18s on April 07, 2010, 09:43:00 am
Finding where to build your bridge is a non-trivial problem.Edit: Just realised you might have meant for the two square task, in which case it's similar but with limited air squares allowed, still fairly trivial.
I'd have to see an implementation.
Post by: Shades on April 07, 2010, 09:54:25 am
Edit: Just realised you might have meant for the two square task, in which case it's similar but with limited air squares allowed, still fairly trivial.
I'd have to see an implementation.
[/quote]
As well as the cost to a given square you'd need cost to a square for each size of bridge (assuming it's less than a smaller length bridge). The route itself would use it's current bridge cost when picking the next shortest path to check and seeing if the air square is passable or not.
Effectively this makes every air square within X from a land square potentially passable and so potentially on the network to path. There is no real complexity here at all as we are already checking cost to square as part of the A* search, we don't care if a square is sometimes passable and sometimes not all we care about is the current cost.
Post by: Starver on April 07, 2010, 10:30:09 am
But when bridges may or may not be made, I've already mused that you'd need to end up with a duplication of the possible pathing tree of some kind. A 'simple' method would be to keep the "zero bridging used" map much the same as your current standard one, with a subset of that map duplicated (on a per-node basis, thus not necessarily duplicating in a huge dumb array) for every relevent position that has a zero-bridging distance[3] and a non-zero bridging access with a lower path cost than all less-bridge routes to get there.
Initially, this would need some propagation of the new duplication of node details along existing paths (though it would essentially be making use of 'cached' pathing formthe previously explored routes along the path-tendrils already explored, so would be far less 'blind' in its search than the original exploration) but once it reached the pathing 'coal face', the usual ongoing feeling around for paths could continue with not just "it cost <foo> to get here, therefore try <foo+1> on any neighbours" but "it cost <foo> to get here with no bridgings used, <bar> to get here with one bridging used, <baz> to get here with two bridgings used [etc], so try <foo+1>, <bar+1>, <baz+1> [etc] on any neighbours".
And when a search tendril includes the possibility to bridge and the upper magnitude measurement (e.g. <baz>) has already used up the possible quota of bridgings, then the <baz+1> is quietly dropped and not propagated across the ignored in the bridged gap.
It does get complicated when a route from one direction featuring a certain number of bridgings meets and travels anti-parallel to a route from another with a different number (with one having the advantage on travel weight, the othr on bridge numbers), and maybe there could be a cut-off other than node-linking strictly per bridges behind them, but the aim is that a path discovery can reach almost to the target area with possibility of both low-cost paths having used up all bridgings (and then ends up rejected if it needs a bridge to take the final step) and at least one higher-cost path having spare bridgings that might end up being the only way to cross.
But I can see alternatives, such as noting each and every bridgable-from nodes in the tree, and when a either a successful path is found or everything ends up dead-ending (with, if available, each of these last-ditch bridgings from that spot also noted) the bridgable-from nodes are reactivated in bridging mode in priority of closeness to destination to search out succesful paths, terminating when those feelers encounter weighted path costs on potential neighbours less than the new weighted path cost it would have imposed to get there via the >= number of bridges on this new mode. If there's an easy route without any bridging, it should quickly discard all bridgings. If a short-cut with a single bridge is discovered, it should let that take over as the ideal and then fall back on a search for any two-bridge method (if more than one can be made) that further improves (with note made of potential three-bridge jump offs, if applicable). And so on while additional bridges are possible. Instead of a whole heap of "original node map array" + multiple*"N bridges used map node lists" you 'only' have the original node map and one simple list of nodes that you would want to try bridging off (in the following "bridge++" round... if applicable).
Although admittedly what you save on storage, though, you almost certainly lose on time if a quick three-bridge leapfrog ends up being the only solution to a problem, and effectively the entire map had been searched for a zero-bridge one, then a one-bridge one then a two-bridge one... Which goes a bit against the search for optimisation. (But then again, you could think up scenarios where all the alternatives are less optimal than all others, too. :))
(BTW, I am of course using "bridging" as a shortcut term for any 'consumable' movement, including ladders and trap-fouling with herded animals. But that does not including 'mining-movement', which would probably be represented by a high-weighting of search paths through the rock, soil or dismantlable barriers, unless the diggers have a finite digging strength/capability of some kind. e.g. sapper-goblins that exhaust themselves and die (or lie their exhausted), or using stone digging tools that wear out.)
[1] I'm trying to backtrack in my mind how A* deals with the specific example of path A having been pursued heavily due to it aiming advantageously 'closer' to the destination, despite heavy weightings against, and path B being an out-and-back-by-another-route path that is low cost but initially a low-prority in the nominally width-wise search tree, but ends up connecting to A (or, more likely, a not yet fully explored branch of A) with a low weight.
[2] And propogating the lower cost (with appropriate increments) back down the found path.
[3] If a square is so far only reached via a single route with one or more bridging options being exercised then it could be reduced to a magnitude-indicating flag, but as soon as multiple routes with competing "low cost but more bridges v.s. high cost but fewer bridges" routes tag that node, you need to consider a form of duplication.
Post by: Draco18s on April 07, 2010, 11:00:00 am
Effectively this makes every air square within X from a land square potentially passable and so potentially on the network to path. There is no real complexity here at all as we are already checking cost to square as part of the A* search, we don't care if a square is sometimes passable and sometimes not all we care about is the current cost.
Ah, but if you only save two numbers: shortest distance to here, and least bridge material used, then you have to figure out how to get to that square with the least materials if you need all of the material to get across the gap. A* only stores the shortest path distance to a given location.
So once you have shortest path and least materials you need to somehow workout the actual path that swaps in the middle from following "least materials" to "shortest distance." If you only follow least materials, then my maze (above) can be transversed using no materials at all (at the expense of distance). On the other hand, every tile can be pathed to using only the available materials, which could result in the unit attempting to make the shortest possible path (across 3 pits) with not enough materials (eg. the large pit can be pathed across with all my available materials, shortest route to there is across pit #1 -> error: can't cross pit, materials used).
Post by: Shades on April 08, 2010, 02:19:21 am
Ah, but if you only save two numbers: shortest distance to here, and least bridge material used, then you have to figure out how to get to that square with the least materials if you need all of the material to get across the gap. A* only stores the shortest path distance to a given location.
I said you store shortest distance for each size of bridge which is trivial with A* and means you have no problem with figuring out how to get there, you just follow the path.
Post by: Huggz on April 18, 2010, 07:29:59 pm
Post by: Shades on April 19, 2010, 03:25:44 am
A* is wrong in this situation due to memory usage. I would use the Dijkstra Algorithm.
You have the relevant information you need for A* so why would you want to increase your search space?
Post by: Huggz on June 14, 2010, 02:08:40 pm
A* is wrong in this situation due to memory usage. I would use the Dijkstra Algorithm.
^ Bullshitting. I have no idea what I am talking about, I just wanted to get in on the fun. I <3 Wikipedia! ^
Post by: Thief^ on June 14, 2010, 03:23:50 pm
Post by: Draco18s on June 14, 2010, 04:32:44 pm
Congrats, you've successfully been a waste of time :P
Mine, yours, and everyone else's.
Post by: HebaruSan on June 15, 2010, 09:01:01 am
QuoteThe trick is just getting the cost correct.Agreed. Utterly.
(I don't have the full conversational context here, but I assume you mean the pathing cost to build a bridge as compared to walking across empty tiles?)
I think that cost should vary according to the preferences of each goblin commander. This would reduce the monotony and solvability of gameplay--rather than the path engine generating a single (sub-)optimal path for all goblins to follow through each possible maze, you could have different squads exploring different branches, so more of your design would be used. Maybe Goblin Commander A prefers to stick to solid ground whenever possible, while Goblin Commander B knows that at least his own bridge tiles won't have traps. The distribution could be a nice bell curve centered around the real excess time it takes to build bridges.
Also, should fog of war factor in at all here? Everyone still seems to be assuming that newly arrived goblins would plan their route through your defenses based on a complete knowledge of the floor plan. I wouldn't mind seeing some of them track down a completely suboptimal route on their first try, only to be ambushed by crack marksdwarves (and presumably somehow warn the others, so their strategy can adapt over time).
Post by: madk on June 15, 2010, 12:29:14 pm
So you've got a dwarf or animal that suddenly decides it needs to get from point A to point B.
Step 1 is to use A* to find a path from A to B, if one exists. If a path does not exist, the dwarf cancels whatver action it needed the path for. The intelligence of the creature would have a weight on how strong the influence of the Steven van Dijk heuristic is upon the calculation. (The Steven van Dijk heuristic causes the path to follow a straight line upon the grid rather than an equally long movement along a more staggered path)
Step 2 involves finding which nodes along the generated path have an uninterrupted path between them and records only these "important" nodes. Fig. 1 illustrates the full A* path, fig. 2 demonstrates the result.
Figure 1:
Code: [Select]
#############
# xxx #
# x###x #
# x # # x ##########
# x # # xxxxxxxxB #
# A # ##############
#######Figure 2:
Code: [Select]
#############
# X X #
# ### #
# # # ##########
# # # X B #
# A # ##############
#######Step 3 sees the dwarf travelling a simple straight line from one node to the next.
Although this adds a bit of time for te initial computation, its effect becomes prominent when the path becomes somehow altered.
So let's suppose that the path has been interrupted - say a door locked or a bridge retracted, but for the sake of example, another, different path has been opened up. In any case, we end up with fig. 3.
Figure 3:
Code: [Select]
#############
# X#X #
# ### #
# # # ##########
# # # X B #
# A # ## ###########
#### # # #
# #### #
# #
#########For a small example such as this, the effect is less darastic, but here's what happens.
The original path is remembered, but a new path is construced. However, this path is no longer from A to B, but rather from the node the dwarf's current position to the next node on the path. The newly constructed A* path should look like this:
Figure 4:
Code: [Select]
#############
# ☻#X #
# x### #
# x # # ##########
# x # # X B #
# Ax # ##x###########
####x # #x#
# x####x#
# xxxx #
#########If the following node can't be reached, it does a check on B to ensure the target is still reachable, then goes down the line until it finds the next reachable node. The disadvantage is that the path ends up being longer - but the movement does become more natural and less artificial.
And, of course, the reduction is performed on this new path, as well, as portrayed in fig. 5.
Figure 5:
Code: [Select]
#############
# ☻# #
# ### #
# X # # ##########
# # # X B #
# A # ## ###########
####X # # #
# #### #
# X X #
#########
Post by: numerobis on June 19, 2010, 12:40:37 am
Your system will have arbitrarily suboptimal behaviour when the repath allows a shorter path that doesn't go through the precomputed waypoints. In figure 4, for example, I'm not sure why the path isn't hitting the second waypoint at the top. According to the algorithm you have described, it should try and succeed to path to it.
Post by: LShadow on March 06, 2011, 05:08:36 pm
Post by: j0nas on March 06, 2011, 05:56:20 pm
Post by: Urist McBusDriver on March 07, 2011, 07:06:08 am
Posting in epic (dead)thread to be able to find my way back to it later.You could just click on the 'Notify' button at the bottom of the thread to subscribe to it...
Post by: Shades on March 07, 2011, 07:35:02 am
You could just click on the 'Notify' button at the bottom of the thread to subscribe to it...
The interface for threads you've posted in is much nicer than the notify button version. Shame really because if they combined it it would be nice to un-notify from threads you posted in as well.
Post by: Assassinfox on March 08, 2011, 07:17:58 pm
Whatever happened to this project? Is it still ongoing?
http://www.bay12forums.com/smf/index.php?topic=58796.msg1310378#msg1310378 (http://www.bay12forums.com/smf/index.php?topic=58796.msg1310378#msg1310378)
Post by: Narmio on March 08, 2011, 08:08:23 pm
Which is a shame, as there's rather a lot of information in this thread that may be of use to Toady when the time comes to revisit pathfinding. One thing we could do is have some computer sciencey type go through this thread and pull out all the interesting and potentially useful suggestions, then present a summary in a new thread and restart the discussion from there. Excise the knowledge from its checkered past, as it were.