Bay 12 Games Forum

Putting the whole issue of soil pH not exactly being my idea in the first place aside

Irrelevant, you're supporting it now

It is the NPK and water aspect that requires constant upkeep.

I have been combining soil pH and some other aspects of your system under the label soil pH for the purpose of discussion, apologies if that caused confusion.

As for being "wrapped up in proving I'm right", I'll remind you that it's rather silly to accuse people of trying to advocate for the things they believe are best for the game.  Obviously I believe I am arguing for something I believe is right, or I wouldn't be doing it.  That, however, doesn't mean I'm somehow incapable of listening to or understanding what other people say.

You do not appear to approach discussions from a standpoint of objectively determining the truth or the best course of action. But rather with a preconceived notion of what the answer already is and vociferously defend that position. There is no hint of consideration or compromise in your language, merely the assumption that you are right and everyone else is wrong. That may not be entirely correct, it may merely be a function of phrasing but it's most certain the impression that comes across.

Look over the way that the Improved Farming thread has evolved

I have no interest in looking over the evolution of a 40+ page forum thread...

I have reversed myself on pests - I used to think it was just another labor sink for the simple sake of labor sinking, but now think it is perhaps the most dynamic feature that can be imposed upon a player.

The phrase "imposed upon a player" fairly well sums up my problems with parts of the system. It is designed to impose on the player as opposed to imposing on dwarves and allowing the player to remove those impositions through gameplay.

This notion that I ignore complaints is ill-founded.  The problem with many of the complaints that have been cropping up recently, however, is that they are not interested in what actually goes into the farming system, but rather about a mere design philosophy standpoint...  People argue that any work on the part of the player is "micromanagement" (so I work to automate what I can), but that if it is automated, it's "just watching a dwarf movie" (so I have to provide more complex and systematic problems for the player to tackle).  Not only are these goals mutually exclusive (and occasionally argued by the same person in the same post)

Which is pretty much an issue of interesting choices, hence why it was brought up and discussed. Too many choices (micromanagement) and the individual decisions do not carry enough consequence to be interesting. And too few (dwarf movie) and there aren't enough for the game to be fun. The fact that you're getting complaints from both sides is a good sign, there is no perfect spot where everyone will agree, only where total complaints are kept to a minimum. My impression however is that you are fielding more complaints about micromanagement than about too much automation which should tell you something. Especially if you take into account that the responders in the farming improvement thread are going to be a self selecting population that lean towards a heavier emphasis in farming than the general player base.

Even worse, this just creates an absurd semantic argument over is or isn't an "Interesting Decision", where, I will point out, Makaze, that you eventually landed on saying that "when Mario jumps" (and whether he hits a goomba or falls into a bottomless pit) is an "Interesting Decision"... if simply WHEN you press the button is an Interesting Decision, then by extension almost anything I make of the Farming system will be an Interesting Decision, since you can always just choose to grow what you want sooner or later.  (And of course, every single person had different definitions of the term, making it thoroughly useless as a means of conveying information - even if I match up direct comparisons of the things one person said were "Interesting Decisions" to something almost entirely like it, they could simply refuse to acknowledge the comparison (and move the goalposts again).)

Everything must be taken in context. Jumping to crush Goombas in Mario can be considered an interesting decision (though admittedly very loosely) due to the time constraints imposed on the player. The speed at which the player must make that decision factors in heavily. Playing Mario in slow motion is less entertaining than at full speed as a result. But speed constraints do not exist in the context of DF and so something as simple as choosing when to jump would not be (widely) considered interesting.

You're also missing the point of the concept. It's not a means of labeling. You don't say that in game X decision Y was interesting so we need decisions like that. And it's not binary, a choice it not interesting or not interesting, it's a sliding scale. It's just a guideline to encourage that you present your players with multiple meaningful and valid (Valid being reasonable to pick, not necessarily correct. In a multiple choice quiz game there are multiple valid answers but only 1 correct one) and that there be a different and understandable consequence for each of those choices. Agreed it's partially subjective, so is taste in food. But there is a large degree of overlap in what people find to taste good, it works the same way in what people find to be interesting decisions.

Then let me use a different term than "free stuff", let me call it "infinite resources".  You do not need to stop and consider what you use your stone for, because you will always have more of it than you will ever need.  You DO, however, need to consider what you use your steel for, because it is limited in supply, and it takes more management to set up.

Why is steel limited? Is it because there is a limited amount of iron ore/flux/fuel available? Because it took manual player interaction to dig/chop those resources (and notice how many people have wanted a dig vein designation or perma-chop this area for years now...)? Or because of the multitude of ingredients and amount of labor that goes into its production?

Obviously it's a mixture of all of those. But I would argue that it's primarily the later that both makes it limited and makes its production interesting. If iron ore and flux were effectively unlimited as wood or crops are you'd see very little change in the amount of steel produced in game.

The point of this is to make food and other farmable goods less like stone, where you can't get rid of it fast enough, and more like steel, where you have to consider how you manage it because you can't be absolutely certain that there will always be a surplus.

The difference of course being that steel is not (directly) required for living dwarves. I still have issues with the design philosophy of making a resource rare by piling burdens onto the player as opposed to in game limitations. But I am far less opposed to it in relation to optional resources than a requirement such as food.

This sort of thing simply isn't possible if the answer to any shortage is merely "make more dwarves farm", with a possible additional "designate a little more farm".  That's just a quick reflex, it doesn't make you consider

It does if the fundamental problem of too little cost of labor is solved. If there aren't 100 spare dwarves standing around then building an additional farm and allocating them to farming has a real cost that must be considered.

With that said, if you assume I am going to force you to repeatedly return to the farm tiles every 2 game weeks to punch the "water the crops" button, then you simply haven't been reading what I've written on the subject, because I've spent some significant time specifically on the subject of how to design automatable systems of exactly the sort you are arguing should be there.

My concern is not with the idea that the player will be forced to punch a button every 2 weeks (or ever). That's an obviously bad design decision and if ever implemented would quickly be yanked out as soon as it was tried. No, rather my concern is in the setting up of a purposefully unbalanced system that requires player interaction to function. Multiplied by the number of fields the player wishes to have. I question the possibility of automating the system you've proposed especially considering the current UI and balancing things so that they do not enter feedback loops that spiral out of control. Compound that with your earlier stated goal of preventing the player from easily making course corrections and you have the trappings of a system that can easily descend into a death spiral or rubberband unless setup in a very few perfect ways or constantly monitored.

(You can start reading here: http://www.bay12forums.com/smf/index.php?topic=22015.msg1481494#msg1481494)

Summarize, seriously. There should be a single post or a wiki page (wikis are free) that summarizes your current system, edit it as needed. I have read through several of your posts on the subject but it is difficult to determine exactly what your current thoughts are at any given moment. I am interested in reading a summary of your system and talking about it. However, as I mentioned before I am in no way interested in wading through a 40+ page thread and picking out the important bits from you ranting about people not understanding you and insisting that you're right and they're wrong.

A constant complaint of yours is that people do not understand what you've written. Perhaps the problem lies not with all of them but in your method of communication?

Likewise, if you think that I should include more than simply food in farms... CONGRATULATIONS!  I have been advocating this straight from the begining.

Something we agree on then. That was however not really the point. It was simply an example to solve what you consider the underlying problems with DF and therefore the impetus for your farming system in a different way. It was presented as an alternate form of complication that accomplishes your goal (more effectively I'd argue since it targets the root problem) but may be more palatable to other players. As an illustration that people can want more complication yet still dislike portions of your particular farming system, the two are not mutually exclusive as you so often imply.

More coming the less time I have to spend arguing this same argument with you all day every day.) 

I am not forcing you to do anything. If you don't find the discussion thought provoking or entertaining then don't participate.

edit: I am forced to repeatedly restate what, exactly, it is I am arguing for over and over again, and this is why I am so highly frustrated by this whole process, especially since it just seems to crop up all over again in another thread each week.

Again let me suggest that there is a fault in either your communications or in the message itself. If a multitude of people don't understand or disagree with what you're saying then there's probably a reason for that...

Page faults and cache thrashing are still problems for a single threaded implementation. There may be more when using more cores and especially if doing disparate tasks since generally L2 cache is shared between them, but that's simply part of the accepted overhead and why you don't see linear performance increases in relation to core count.

Obviously you'd want to profile things to look for slow spots but it should only be slightly more an issue than it already is for the single threaded version (and is probably simply an intrinsic part of pathfinding a large set of data with a complex heuristic). Cache thrashing is largely a red herring in relation to multithreading in this case. Time, knowledge and effort needed to set it up correctly are far larger barriers to implementation.

Multi-core programming works well for situations where most of the time burden is on calculations, not HD/Memory access

Actually one of the primary uses of multithreading is to use separate threads to load things from the HD (or any high latency I/O operation). If it takes one of your worker threads a long time to do or wait on something it's going to take your primary thread just as long or longer. The difference is that they can go to sleep and let other things do stuff while they wait.

Because you have the same exact problems using additional GPU threads as you do using additional CPU threads, only it's more of a pain in the ass since you have to write a customized CUDA A* implementation (someone has already written a generic one) and shoehorn all of your data in video memory. Just like core algorithms should be optimized before turning to multi threading, all free CPU cores should be utilized fully before turning to using CUDA on a GPU.

You only say DF hack because an above poster mentioned it. The point of dfhack is to hide the inner complexity of DF. The tool of choice to figure out what is going on is IDA.

True. If one cares enough I'm sure IDA is a fine tool for discovering the inner workings of DF, if one cares enough.

Well, damn. You should write a threading library. I wouldn't mind one that could make a million asynchronous calls a second to a single thread. That would be hard core and much better than this crap I got from Intel.

Why would I bother? The one I've got seems to do pretty well, just benchmarked it doing 10M Sets on a manual reset event in 4.060s, automatic reset took a slightly longer 4.773s. Set and a separate Reset (in case repeated sets were being ignored) on a manual reset event took an expected around double 8.483s. Those are the initial runs but repeated runs all come in within a tenth of a second.

Now, if you have any benchmarks that you believe contradict those findings or can explain to me what exactly my primary thread has to do other than set the wait handle and get back to business while another core's current process's quantum expires and schedules the now signaled thread to run I'm all ears...

Moreover, I find it amusing that you talk about 'refactoring' and 'poor algorithms' while you have never seen the sources and have no idea what algorithms does Toady use. You cannot provide ANY kind of estimates on worthiness of your 'refactoring' w/out analyzing sources first

One can make an educated guess based on the organic way that DF has grown, current performance, and past bugs and refactor efforts. That guess is that the internals of DF are understandably a friggin mess.

If you were a real programmer, you could take steps to answer these questions.

Or maybe I'm someone with a real job who doesn't have time to muck about with DFhack and just comes here during compiles. Who knows...

Ya, but you're having the discussion with people who think it takes a nanosecond to start or dispatch a thread. In their world, four threads will run something four times as fast as one thread.

NanosecondS in the plural. As in 700-950 of them. And I've yet to hear from you an actual argument as to why exactly it takes longer than that to do a kernel mode transition and have the scheduler set a flag.

And I never said that 4 cores would run anywhere close to 4 times faster. There are significant overheads to contend with. The cost of context switching, cache hits, and competing with the other ~400 threads windows is running at any given time all mean you're never going to see performance increase linearly with core count. But all of those costs (and they are big costs) I just mentioned are paid by those additional cores, not the one running DF that took <1 microsecond to lift a wait.

The fact remains that there would never be any performance gains from threaded pathfinding. There isn't a (established) pathfinding algorithm more expensive than a windows or linux thread.

Ignoring for the moment that that statement is completely pointless since you have not defined the set of pathing data or heuristic...

First of all I don't believe that, there are a lot of games out there that have multithreaded pathfinding. Do you really think they all implemented it without ever benchmarking to see if it was faster? But even if it were true in the case of DF I don't care. The issue here is not total work to be done (and I have no illusions, threading creates additional total work) but rather work that needs to be done by the main thread. It's perfectly fine even if we double our total workload for pathfinding once costs are factored in but are running them across 3 cores. It's still a net gain. Not to mention you can batch all the requests for each frame up and only make 1 thread per free core. If there are not that many path requests and/or they're as cheap as you say then why do we care about pathing as a performance issue at all?

There is nothing useful for him to read in this thread.

On the contrary, if you read the latest podcast transcript he pretty much implies that he's not all that up on threading. I'll say it again, Toady is not an infallible programming god, he's just a man with a disturbing level of obsession for simulating dwarves. Which I appreciate very much, but do not deify him for.

My point of refactoring in the context of Toady and DF is to improve previously written code which obviously suffers from atrocious scalability issues (e.g. it works fine when you have 1,000 items, but fails horribly with 100,000). The pathfinding algorithm is very likely only a small portion of the problem. You correctly said that the functionality of the game would remain the same, however, the performance could be significantly improved. I'm guessing that the majority of DF players would love for that to happen.

I think the point he was trying to make is that refactoring is a boring and generally unrewarding task. Adding a new feature has a lot of wow factor and a big sense of accomplishment at the end. Where as making something run 10% faster and have easier to read code is comparatively unfulfilling (for most people) to do. So regardless of whether or not it's logically the right thing to do it's not likely to get done by a one man team whose primary motivation is finding what he is doing interesting.

And that refactoring will tend to break core aspects of the game. Right now when a new feature is added the new bugs only tend to show up in relation to it, they're easier to track down or avoid as a result. Doing a refactor of something like how items are stored and looked up in memory may be a performance boost but is also likely to touch (and therefor break) large sections of the codebase. And the results tend to be underwhelming as they're more felt indirectly through the absence of negative experience as opposed to the active positive experience of a shiny new feature. So again, even if refactoring is logical, public response to it will not be as favorable as if the time was spend piling in more crap.

Multi-threading at this point would be a horrible mistake. It would be the same bad code with the added detriment of concurrency/syncing issues. I cannot emphasize enough how painful of a process it is to convert a pre-existing codebase from single to multi-threading. The number of bugs introduced with this would be an order of magnitude or two larger than simpy refactoring the code. I think Makaze's A* comment is relevant here, you can't just wave the magic multi-threading wand and gain performance improvements.

Depends. Threading out big chunks like trying to make drawing or input run in it's own thread, damn straight. Doing as I suggested above and having pathfinding run completely asynchronously, damn straight. But tossing off a few read only threads at points in the main loop when data is guaranteed to not be volatile to do something like pathfinding seems like a cheap way to pick up a little bit of performance. Just have to be careful that the threads never ever write anything anywhere for any reason and that you're in a section of the main loop that is either blocked while the threads run or not changing any world data (ie. pathing while doing the calcs for offscreen civs in the main thread).

You definitely can't wave a wand (my comment was more trying to express that, while there might be room for improvement, DF pathfinding is likely a lot slower than generic uniform cost A* because it's doing a lot more), you've got to plan it out carefully and you're not going to suddenly double your speed. But I think you can pick up some performance gains fairly easily by threading simple predictable tasks. Of course it's still largely a bandaid. If the performance on your algorithm is decaying at a geometric or exponential rate as more items are added then running it on linearly more cores is only going to temporarily stave off the inevitable.

And even if I am an advocate for threading if there's room for improvement in the core algorithm then it should be done there before threading is contemplated as an inefficient function now is still going to be inefficient when running on multiple threads.

Libtcod can manage the pathfinding of a thousand creatures with no apparent slowdown.

That's a fairly meaningless comment without including the size and complexity of the path map and the heuristic involved. No doubt DF path finding could probably be done better but it's not quite as simple as waving the generic A* wand and pretending that it does everything that needed and runs instantly.

One hundred thousand inert rocks should have no effect on FPS.  The game has no need to examine them each frame.

Except when a mason needs a rock the game paths to each and every rock out there to determine the nearest one of each type.

But even ignoring pathing running a lot of poorly optimized N² operations on the item list (or DwarfList*ItemList or TaskList*ItemList etc.) will swiftly bring things to a crawl as Items, Tasks, and Dwarves(who come with items and make tasks and items) increase. Pathing got a lot of discussion in this thread but it's far from the only thing slowing DF down as forts get bigger.

Right, but they aren't "inert"; they have material properties that cause them to respond to changes in the world.

Exactly, the temperature model is effecting each of these items every frame which means at very least an N crawl of the item list every frame.

And what is "objectively fun"?  Isn't fun, by definition, something subjective?

Partially. There are activities that no one considers fun and there are activities that some people consider fun. In terms of games by looking at the characteristics of universally unfun activities and common characteristics shared by fun activities you can develop a general idea of what is objectively fun. There is no single sweet spot where things are fun to everyone, but there is certainly lots and lots of design space where things are fun to no one. Interesting choices tend to be in the design space that is fun.

An interesting choice being a meaningful choice between 2 or more valid (though not necessarily correct) options. It's usually the meaningful part that people differ on. By ensuring that presented choices have multiple valid options rather than simply multiple options and by making each of those choices carry a consequence that is meaningful to the player you increase fun.

Although I probably shouldn't go off on MORE other games, let's talk about some things that "most players" play, like the incredibly popular Mario games or Sports games or First Person Shooters that dominate much more of the market.  Can you really talk about "Interesting Decisions" in terms of "Do I jump on the goomba now, or do I wait .5 seconds longer to jump?" or "Should I collect that power-up, or do I take advantage of being small, and dying in one hit?"  or "Should I shoot at the basket now, or when I'm actually close enough to have a chance at scoring?" or "What are the pros and cons of running over that grenade on the ground there?"

Sure. They're not everyone's cup of tea to be sure, but the concept still holds true. Here we see decisions where the player is presented with perhaps perfect knowledge but insufficient time to fully process and act on that knowledge. This leads to a quick decision among several choices with consequences being made. It is a different type of interesting choice but one none the less.

People don't play DF in terms of "Interesting Decisions" that weigh pros and cons.  They play in terms of "OK, what is my highest priority right now?"  If they need food badly, they're probably going to ignore everything else in favor of making more farms.  If they think they have vulnerabilities militarily, they shore it up.  If they want to go into caverns, they don't weigh pros and cons, they look at their military, and make an educated guess on whether it's strong enough to survive the caverns or not, and if they aren't, will simply bulk the military up.  It's not "interesting" to decide you don't have enough military might to move into a risky area when you don't have the strength to do so.  It's just a simple pass/fail analysis.

You just gave a perfect example of someone internally weighing pros and cons of various valid actions, choosing the one they thought was the best based of their limited knowledge, and acted with consequence. Those are interesting choices. It doesn't really matter if there is only one objectively correct answer only that there are multiple valid choices with significant enough meaning. Heck, even deciding what your highest priority is has the potential to qualify.

Ask people why they enjoy DF. I doubt you'll get many answers revolving around "Interesting Choices". What you ARE likely to get are answers revolving around the freedom the physics system gives them to do things like magma flood the map, or the difficulty of the game, or something along those lines. 

No doubt. It's a rather academic term used in design circles to attempt to describe an abstract game concept. Regular people don't use it, at least not in the context I'm using it here. The physics system (though I really have a hard time calling it that considering how little it actually does...) is liked because it gives people interesting choices. Whether they specifically call them that or not is irrelevant.

What Toady tends to talk about, though, is a little term called "Verisimilartude".  He wants to make the game have all this depth when it can, but detail even when it can't because he wants the game to LOOK like it's just a wide-open place with plenty to explore.  Consider a game like Oblivion, and it's massive, beautiful open environments and ability to just get on a horse and tramp around the forests and fields for a while.

Which to some degree is a worthwhile goal. Interesting choices are a game mechanic concept used to maximize the fun in that aspect of game design. Immersion is a separate concept and sometimes they knock heads. A more realistic and immersive system can limit choices or overwhelm with too many inconsequential ones. They both help to make a fun game overall and juggling between them (and other concepts) is a balancing act. In this case I believe that something like soil pH will do harm to pure gameplay aspects while providing very little in terms of immersion as most people are not familiar with the concepts involved and so will relate to it in pure abstract terms instead of as an immersive factor.

Or consider that The Sims is basically one of the highest-grossing games/series ever, and there's very little that counts as a difficult choice in that game.  "Fun" is sometimes just the ability to be able to understand and relate to a world, not having to figure out whether you need a +3 to tech instead of +4 to production.

True, The Sims does very well without many interesting choices, though there are some. It does so at the cost of interaction though. You don't so much play The Sims as watch it like you would TV (also very popular).