Bay 12 Games Forum

Dwarf Fortress => DF General Discussion => Topic started by: janglur on September 25, 2010, 06:39:31 pm

Title: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 25, 2010, 06:39:31 pm
Part 1:

So I have been doing a LOT of research into what DF seems to prefer in terms of hardware and software settings, and i've uncovered a few tips to squeeze that extra 1% out when you really need it.
I am only testing with Vanilla DF, the standard map generation settings, nothing changed or disabled.  This is purely a test for research on the PC and hardware itself, not modding DF.

The Threads Dwarves Weave:
As we all know, DF is largely single-threaded, which has led to a lot of myths to study regarding it's performance.  What i've discovered for tweaks are as follows:
DF enjoys being dedicated to a core.  Simply saying 'Stick to core 2' will improve performance a good 5-10%, though this is largely noticeable with pathfinding and only at already high (100+) FPSs.
This can further be enhanced by un-dedicating every other system processes and applications (with windows task manager or, better Process Explorer) to a different core, leaving DF it's onw dedicated CPU and cache.  This gives another 5-20%, depending wildly on how much crap you actually have running, how much cache your cores have dedicated, and the phase of the moon.

Stockpiling Data:
Based on the above, a logical theory is that having more dedicated cache would improve performance.  I tested this theory by setting the L3 cache allocation to 'All Cores' and 'DSP only'.  DSP Only is a setting meaning only the main core (core0) has access to using L3 cache.  Thoretically, in my system, this means an extra dedicated 6 MB of cache for DF.  However my testing showed quite the opposite result:  FPS actually dropped by a tiny 2-4% margin on initial embarks, and then became quite even after the '100,000 stone' test.  Thus suggesting allowing the L3 to be used by other cores is better.  I hypothesize this is because what can't fit into L1 and L2 goes to RAM if the L3 is not accessible, creating more cache competition in RAM.  More on this later.

Gangs and Undead Gangs:
I also tested whether Ganged and Unganged memory modes (on 2 sticks of Dual Channel memory) would work better.  First, an explanation.  Ganged and Unganged refer to CPUs with multiple onboard memory controllers. Namely, my Phenom II has two dedicated memory controllers.  These settings refer to how they are accessed.  Unganged means that your memory is effectively divided in half, and each of the controllers address it on a 64-bit lane.  So, 2x64bit.  Ganged means they work together, on a single 128-bit lane.  What this means in performance is that Unganged typically benefits multi-threaded tasks or extreme multitasking, while Ganged benefits single large transfers or single-threaded heavy duty apps like Dwarf Fortress.  My testing concluded almost no gain on initial embark, but a major gain in ultra-large (12x12+) maps and cluttered (100,000+ stones) tests.  This may be also due to RAM competition.  But Ganged is definitely better, by a 5-15% margin in large or older fortresses.

Lost but Not Forgotten:
Interestingly, although DF is single-threaded, it relies on hardware to perform many tasks.  And much of your hardware depends on the CPU.  As such, overclocking only the DF core is not the best solution:  Overclocking the not-in-use cores also helps!  Depending on the difference, it can be as much as a 60% FPS boost!  (This was testing with a 3.4 GHz core [DF dedicated] and two 100 MHz cores to save power and reduce heat.)  FPS suffered horribly with the other cores set to power save modes, so disabling Cool'n'Quiet or other features will help, as will synchronous overclocking.  This makes most effect on pathfinding and doesn't seem to affect clutter or map size speeds much.

RAM Competition:
The major killer of fort FPS for me has not been pathfinding, catsplosions, or even liquids, but in fact clutter!  Noticing the resource usage of DF over the course of a fort's 10 years, massive stone stockpiling (and crafts, and food, and waste, etc.) always crippled the system irrecoverably.  This is stemmed from simple RAM usage:  Not only does RAM usage increase from the much higher number of entries (though not as drastically as I expected), the number of read/writes, especially reads, on RAM increased astronomically.  It went from only a few tens of thousand a second (a few MHz) to a full bottleneck at well over 1 billion a second (Full 1066 MHz and up).  THIS is what has been behind the issue.  So I began testing RAM.  The first thing I noticed was that increasing the RAM's frequency, such as overclocking, made a fairly linear improvement.  A 5% increase in speed resulted in a 5% increase of reads, and about 1% in FPS.  However, I then clocked to 800MHz and dropped the timings from 5-5-5-15-30-2T to 4-4-4-12-26-2T, and the increase was much more noticeable.  While the reads/sec went down, the FPS went up!  I'm not entirely sure how this works, logically, and the drop from 1066 to 800 did cause overall damage to FPS.  So I went for a middleground.  Attempting to keep the FSB/NB, HT and CPU at 2.8 GHz, I clocked the RAM to 917 MHz at 4-4-4-14-28-2T timings, and got the best performance!  It beat both 800 MHz at 4 and 1066 MHz at 5.  This suggests that DF needs not only hefty througput, but moreso, rapid response time from memory and the memory controller!  Take this as you will.  The overall benefit was an increase from 20-24 to 22-28 FPS.  Not huge, but nice.
Reducing the number of services, programs, apps, etc. running will also reduce memory competition as less and less stuff takes up DF's insatiable consumption of accesses.

Give It A Break Already:
Although there does not seem to be a memory leak, DF gets a fair boost in FPS if you just save and exit (the client completely) every so often.  How frequently this makes a difference depends on your system's speed, as our P4 and Celeron test systems didn't benefit until many hours of play, while the Phenom II I use personally benefits after around 3-4 hours.  I'm not sure why this is.  Take it as you will.  The benefit is small, and proportional to how long since you closed it.  It increased FPS from 3-4 FPS to 15-30 FPS after I restarted from a 72-hour long nonstop Dwarfathon.  (There was also a lot of unattended Fun).  Readings were taken before and after the restart to ensure it wasn't from a sudden drop in pathfinding on dead dwarves or something trapped somewhere.

Goblin Processing Unit:
Another, albeit tiny, boost is in the videocard.  Using different driver versions on your system can make various improvements and detriments, and seems to vary from system to system and card to card.  I can't give a very solid prediction on this.
However the GPU is a key point.  A faster GPU of identical architecture gets a better FPS (and G_FPS) when operating at a higher frequency.  The test was done across an ATI 1300 and an AT 4830.  The 4830 was tested at 160 MHz, 300 MHz, 500 MHz, 575 MHz (default), 600 MHz, 65 0 MHz, and 695 MHz (max stable OC).  It increased linearly after 500 MHz, and drastically after 160 MHz.  We did the same test on the VRAM, and no frequency made any detectible difference good or bad, even at extreme OC and UC conditions.  This makes sense for the software rendering aspect being largely GPU based.

Go Flush Your Cold Buffer In You SCSI DASD:
Strangely, we tried enabling and disabling Virtual Memory/Swapfile/Pagefile/Disk Cache and the effects it had on DF.  I found that it is actually best to *enable* virtual memory and allocate a fair amount.  I assume this is the result of reducing the amount of clutter in the RAM and thus shaving off a bit more RAM Competition.  The effect was surprisingly substantial at a fairly steady 7-8% FPS improvement.  This has been the single best result short of CPU and RAM overclocking.



This concludes Part 1 of our tests.  Next, i'll be testing differences between equivocal Intel and AMD lineups.  This will cost more than the $80 on Part 1.  All for the love of dwarf.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Duelmaster409 on September 25, 2010, 07:00:10 pm
This thread reeks of science... I love it! This was an excellent read.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Urist Imiknorris on September 25, 2010, 07:06:52 pm
This is true Science. I tip my ☼adamantine helm☼ to thee.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Gearheart on September 25, 2010, 07:11:42 pm
Okay, so, what are the implications of this for those who are not quite technologically inept but fairly close to it?

I'm currently using a dual core 2.00GHz, with 4GB of RAM. The computer I recently ordered will have a 2.4GHz quad core with 8GB RAM.

How much impact will those two point alone have on the performance of DF, and what could average joe change if he wants to increase efficiency in a way which will not risk everything going FUBAR due to mistakes?
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 25, 2010, 07:44:40 pm
Quote from: Gearheart on September 25, 2010, 07:11:42 pm
Okay, so, what are the implications of this for those who are not quite technologically inept but fairly close to it?

I'm currently using a dual core 2.00GHz, with 4GB of RAM. The computer I recently ordered will have a 2.4GHz quad core with 8GB RAM.

How much impact will those two point alone have on the performance of DF, and what could average joe change if he wants to increase efficiency in a way which will not risk everything going FUBAR due to mistakes?

Simple:
Turn Pagefile on.  This is usually on by default.  Allocate a fairly good size (I reccomend 4 GB).  The minimum and maximum should be the same.  This way the pagefile doesn't get (as) fragmented.

Easiest:
Use ProcessExplorer to switch *everything* to cores 0-3, and dedicated DF to Core4.

Easier:
Close all unnecessary programs, and I mean ALL.  Turn off or stop any services or applications (especially antivirus) not in use.  (Since your AV is off, it's a good idea to disable your internet as well and temporarily stop the services.)

Easy:
In the CMOS/BIOS settings, turn memory to Ganged mode and lower your RAM timings as low as it can stand.  Try with the highest first, and move them down one by one.  Test using something like Memtest86+ or Prime95 for several hours, and then repeat until it becomes unstable.  Once you find the best possible, test it for 12+ hours to ensure perfect stability.  If it fails, move it back another bit.  Adjusting timings is safe for hardware (but can cause data corruption and bluescreens in windows), so Memtest86+ is reccomended, as it doesn't touch the harddrive.  Note that changing the *frequency*, however, can cause failure of hardware.  Same for voltage.

Not As Easy:
Disable Cool'n'Quiet, Spread Spectrum, and any other power-saving or heat-reduction options.  This can cause overheating and failure of components if not thoroughly cooled.  Not reccomended on stock heatsinks that come with the CPUs.

Hard:
Manually adjust the frequencies of the CPUs, core buses, RAM, and GPU using one of many available programs.


Extra tips:
Go for dual core at minimum.  Having 3 or 4 cores makes no bigger difference.  Phenom II are the best AMD options.
After that, go for frequency.  When the 975's come out, they are going to be quad-core 3.6 GHz CPUs, stock.  These will be the best AMDs for a while.  Remember, DF is single-threaded, meaning it depends on *one* core to be very, very fast.  Also, set it to a single core, or else it slows down a LOT from rapidly switching between cores while running.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 25, 2010, 08:15:46 pm
Oh, also, another note:

Using memory that is paired (sold together in one package as paired) equal to your memory controllers (2, in the case of Phenom II's) is ideal.

In the vast majority of cases, this means two sticks in dual-channel work best, in unganged mode.

Using three or more sticks, or non-matching memory will stress the memory controller on the motherboard (or CPU, in AMD's case) and can hurt performance, or worse, cause instability.

Remember, it takes twice the work to address 4 sticks than 2.

Also, smaller amounts of memory seem to work better, IF AND ONLY IF you still have adequate amounts of RAM.  Not enough RAM will KILL your FPS.  But having 24 GB is also counter-productive.  Think of it this way:
1 GB 800 MHz memory in single-channel reads at 3.2 GB/s.  So, it takes 1/3rd of a second, roughly.
2 GB 800 MHz memory in single-channel reads at about 3.2 GB/s.  So, it takes twice as long as before!  Realistically this means it takes a bit longer for the RAM to search through and find the requested data for DF.  (Not twice as long, more like 5-10%, because RAM can semi-non-sequentially read)
1 GB 800 MHz memory in dual-channel reads at 6.4 GB/s.  So, 1/6th of a second.
2 GB 800 MHz in dual channel is 6.4 GB/s.  So, 1/12th of a second.  See how that works?

It's a LOT more complex than that, but you get the idea.  Make sure you have plenty of RAM, but don't go overboard.  My 4 GB is plenty (2x2 GB)


!!SCIENCE!!:
The following diagram may help.  It uses 'made up' tliness, but are proportional to give you an idea of the differences in design and the impact they have on responsiveness, namely in how some setups can cause many more steps to access specific data.

http://207.177.39.129/Memory.JPG (http://207.177.39.129/Memory.JPG)
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Cyntrox on September 25, 2010, 08:35:38 pm
I applaud your research.

However, there is one thing that irks me: You measure performance in FPS, which isn't accurate. I'll try to explain why.

Say you are running at 600 FPS. This means that each frame will take 1.6667 milliseconds (ms) to perform, because 1/600=0.0006667. Now let's say you drop to 200 FPS. This probably seems like a huge amount, and it means that each frame takes5 ms to perform. So when going from 600 to 200 FPS, you go from 1.6667 to 5 ms frame time. The difference is 3.3333 ms.

Now let's say you are running at 30 FPS, which means 33.3333 ms per frame. You drop to 27 FPS, which is 37.037 ms per frame. The difference here is 3.7036.

So when going from 600 FPS to 200 FPS, you use 3.3333 ms more per frame.
When going from 30 to 27 FPS, you lose 3.7036 ms.

See what this means? You lose more performance when going from 30 to 27 FPS than when you are going from 600 to 200 FPS. This means that saying that you gain 5 FPS by changing some setting doesn't say anything.

It's a shame I can't find the article that explains this a lot better than me. It'd probably also help if I was sober, but I hope that's understandable.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 25, 2010, 08:53:22 pm
Oh no, I understand entirely.  It's just that some tweaks only make a difference at high FPS, and some only at low.  Sadly, FPS is the only benchmarking available that's relevent to DF.  Synthetic benchmarks can be DRASTICALLY inaccurate on the grounds that they behave nothing like the real application.

Rest assured however that each tweak was tested across no less (but maybe more) than ten different maps of varying size, age, clutter, and activity, at the exact same frames and time periods.  (IE, I didn't test, save, then load and test again.  They were all tested on neutral grounds.)

I tried to only post those tests that were relevent, in FPS, on a fixed scale.  IE, the restarting made the same maps jump by a factor of ten or more.

Others were more unilateral, such as the CPU cores clocks-  the FPS drop was proportional to the frequency.  IE, 1000 went to 1200, 100 went to 120, 10 went to 12.


Also, one interesting note, I could not embark on an area larger than 42x42 in any but a dedicated-core, ganged and optimized site.  It ran well below 1 FPS.  To say it was crippled was rediculous.  After an hour, they moved the twelve spaces to begin mining.  X.x



Edit:
This is also why I completely omitted what various driver versions did, because they had no clear and substantiated link, and no differences that were appreciable.

I also omitted any differences below 200 FPS.  Going from 150 to 200 isn't really pertinant, and 200 to 400 is as simple as designating mining.  I made a strong effort to only include findings that made a reasonably significant improvement.
VRAM, for example, made lots of G_FPS differences at extreme FPS (1000+), but noone plays at that realistically.  In real forts, it made absolutely no difference on FPS or G_FPS.  Certainly not enough to bother tweaking to accomplish.

I follow the '2% rule'.  It the FPS didn't, under heavy load, make an average of 2% of better improvement at a very severely lagging fort (10 FPS), I didn't bother to report it here.

So, take all my research with a grain of rock salt, but rest assured that although it may not be as solid at Granite, it's as valuable as Limestone.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: jei on September 25, 2010, 09:22:49 pm
Quote from: janglur on September 25, 2010, 06:39:31 pm
The major killer of fort FPS for me has not been pathfinding, catsplosions, or even liquids, but in fact clutter!  Noticing the resource usage of DF over the course of a fort's 10 years, massive stone stockpiling (and crafts, and food, and waste, etc.) always crippled the system irrecoverably. 

If you are interested, I have a savegame that lags from the embark start:

http://dffd.wimbli.com/file.php?id=3161

Extremely laggy savegame from the first Embark. FPS = 20-30.
Used same set to embark elsewhere, got 160 FPS last time.

Haven't dug or built much anything, just hauling stuff.

River & Volcano combination made with high variance and slow erosion cycles.
Year 500 due to elves having been all extinct in the previous worldgen, but it appears this is a NO-ELF world, ruled by titans and megabeasts.

No blood has been bled on this map yet.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Urist Imiknorris on September 25, 2010, 09:27:38 pm
My money is on the magma sea draining into HFS.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Mel_Vixen on September 25, 2010, 09:38:36 pm
Hmmm you mentioned a memory lea somewhere. Could you nail it down?
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: fivex on September 25, 2010, 09:39:59 pm
Quote from: Urist Imiknorris on September 25, 2010, 09:27:38 pm
My money is on the magma sea draining into HFS.
That can happen?  :o
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 25, 2010, 11:23:30 pm
Quote from: Heph on September 25, 2010, 09:38:36 pm
Hmmm you mentioned a memory lea somewhere. Could you nail it down?

You misread.  I could *not* find a memory leak.  Instead, it's legitimate memory allocation, but the reason why massive amounts of items (such as stone) causes lag is a sheer overload of read/writes.  I assume it polls the location of each item or something at an ultra-regular pace, causing severe memory thrashing.  It's the only theory that makes sense for why tighter timings would help so much compared to bus overclocking.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Vigilant on September 26, 2010, 12:08:26 am
Comp Sci major reporting in.

Do not increase the page file. In fact unless you have a really low amount of memory you're better off disabling virtual memory to make sure your computer uses it as little as possible. Windows will always use some HD space for paging programs even if you disable it. The problem is it's memory management can be bad sometimes and it'll start using virtual memory when it doesn't actually need to. And that's bad because the harddrive is a half dying snail compared to your cpu and memory speeds. If it starts trying to use the harddrive for memory, your speed will drop to hard drive access speeds since that's the weakest link.

Before matching up memory size, if you have an older computer it's better to check and see if your memory is even dual channel at all. If it isn't dual channel you receive no speed increase from having memory chips of the same size :(

The issues with memory are a LOT more complex that has been thus described since newer operating systems and lower level controllers are designed to use your memory and cache as best they can and have all sorts of fun algorithms and shortcuts. Unless you're a Computer Engineer I'd l

But memory is the place to consider as tracking items and doing pathfinding are extremely memory intensive. The faster (higher quality) memory you can get for your machine paired with the processor with the best-est cache you can find is where you'll really get performance. My laptop's CPU can match performance of heftier machines because it's got a larger cache and pretty new memory.

Oh. To contribute to the science rather than just picking on people that have posted so far : for any DFtermer's here's something to consider. I have a server with two dual core processors and 3 Gb of memory, so it can theoretically sustain multiple games of DF at once. But the thing most people don't realize is multi-core work is still limited by the fact that there's usually only one channel to access memory, and the bottleneck will hurt performance. I loaded up my 100+ dwarf fort with for a while was running 30 fps on the server, and made a copy of it and ran that too. While they bicker for resources instead of splitting evenly, the results are it averages out to a 2/3's performance for both. Dropping to around 21 fps. Although the cache on that server are rather small, i'd expect a newer multi-core setup could host multiple games with less performance degradation.

Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Cyntrox on September 26, 2010, 07:25:37 am
Quote from: janglur on September 25, 2010, 08:53:22 pm
Oh no, I understand entirely.  It's just that some tweaks only make a difference at high FPS, and some only at low.  Sadly, FPS is the only benchmarking available that's relevent to DF.  Synthetic benchmarks can be DRASTICALLY inaccurate on the grounds that they behave nothing like the real application.

*snip*
Although it's good that you're taking the measures you mentioned, it's very easy to convert between FPS and milliseconds per frame (ms/f): (1/FPS)*1000.

Now to try some of the stuff you mentioned...
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Gearheart on September 26, 2010, 07:30:27 am
So I gave DF it's own dedicated core, and I'm fairly shocked by the performance increase it provided.

I'd estimate about 15-20% increase in fps, which seems almost too high.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: nbonaparte on September 26, 2010, 07:49:09 am
nope. other stuff like the OS is distributing over the cores. It's sort of like the space gained from defragmenting.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: NKDietrich on September 26, 2010, 09:21:43 am
Hrm. I've been experimenting with the process affinity and Dwarf Fortress, specifically with worldgen. Here are my results.

System:
Core i7 930 @ 4.2GHz (HyperThreading On, so 4 physical, 8 logical cores)
6GB DDR3-1600
480 GTX SLI

Setup:
DF 0.31.14
STANDARD display mode.
Graphics on.

General Observations

With affinity set to all 8 logical cores, each of the four physical cores experience some usage during world gen. The 4 HyperThreading cores are not utilized at all, as you might expect.  However, none peak much above 50% utilization, and they basically jump between 0 and 50% usage.

With affinity set to just one logical core, that one core is pegged out at 100% the whole time. With affinity set to two logical cores (on different physical cores.) One does most of the heavy lifting while the other does a little.

GPU usage was steady around 13% on one card while actually playing (obviously the second did nothing), and 1% during world gen.

The Data

I generated a world from these parameters 3 times in each configuration:
Spoiler (click to show/hide)

Affinity To All Cores
Run 1: 62 seconds
Run 2: 63 seconds
Run 3: 62 seconds

Affinity To One Core
Run 1: 61 seconds
Run 2: 61 seconds
Run 3: 63 seconds

Affinity To Two Cores
Run 1: 62 seconds
Run 2: 62 seconds
Run 3: 62 seconds

Conclusion

Evidently affinity matters not for world generation purposes. Every world generated in the same amount of time. With more runs and a more accurate timer you could probably nail down a minor 1-2% differences between the configurations.

Actual Embark Framerate

I uncapped both FPS and G_FPS. And embarked on a site with a lot of cliffs with a Stream and a Volcano. I waited for the FPS counter to calm down and stabilize. General usage pattern was the same as during world gen. Affinity to 8 cores resulted in sporadic usage of the 4 physical cores only, and affinity to 1 core resulted in one pegged out core.

Affinity to All Cores
Dwarves standing around wagon:
350fps
Three Dwarves mining out a large area:
230fps

Affinity to One Core

Experienced identical framerates to the multi core affinity configuration

On my configuration it doesn't seem to matter at all.




Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 26, 2010, 11:20:04 am
Interesting, thanks for the !!SCIENCE!!  It shows some difference between Intel and AMD, and Hyperthreaded vs. Non-Hyperthreaded.  Hyperthreads sure complexify the fark out of these calculations because of the wierd way they work.

However the main area I tested was heavily loaded embarks and older forts with 100+ dwarves.  Can you do some tests when you reach around 50 dwarves, too?
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: NKDietrich on September 26, 2010, 11:22:44 am
Quote from: janglur on September 26, 2010, 11:20:04 am
Interesting, thanks for the !!SCIENCE!!  It shows some difference between Intel and AMD, and Hyperthreaded vs. Non-Hyperthreaded.  Hyperthreads sure complexify the fark out of these calculations because of the wierd way they work.

However the main area I tested was heavily loaded embarks and older forts with 100+ dwarves.  Can you do some tests when you reach around 50 dwarves, too?

Sure. I don't have any alive and kicking forts at the moment, but I'm currently trolling the world gen for a new map. I'll do some tests once I get a busy fort going.

One of the things I find interesting is how DF isn't really set up to take advantage of many cores. It seems, on the surface, like it would really benefit. Of course I know nothing about the underlying code. It seems like certain things could be asynchronously handled and threaded out like flows, but again, I'm just talking out of my arse here.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: shadow_slicer on September 26, 2010, 12:10:02 pm
Quote from: janglur on September 25, 2010, 06:39:31 pm
Give It A Break Already:
Although there does not seem to be a memory leak, DF gets a fair boost in FPS if you just save and exit (the client completely) every so often.  How frequently this makes a difference depends on your system's speed, as our P4 and Celeron test systems didn't benefit until many hours of play, while the Phenom II I use personally benefits after around 3-4 hours.  I'm not sure why this is.  Take it as you will.  The benefit is small, and proportional to how long since you closed it.  It increased FPS from 3-4 FPS to 15-30 FPS after I restarted from a 72-hour long nonstop Dwarfathon.  (There was also a lot of unattended Fun).  Readings were taken before and after the restart to ensure it wasn't from a sudden drop in pathfinding on dead dwarves or something trapped somewhere.

This is probably caused by heap fragmentation (http://en.wikipedia.org/wiki/Fragmentation_(computer)#External_fragmentation). This suggests Toady should implement object pooling to group similar sized objects together into single larger allocations. The most likely cause is text strings, because they come in a large variety of sizes. If Toady isn't already pooling them, he should definitely look into it.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: darkflagrance on September 26, 2010, 04:10:22 pm
Quote from: janglur on September 25, 2010, 06:39:31 pm
Give It A Break Already:
Although there does not seem to be a memory leak, DF gets a fair boost in FPS if you just save and exit (the client completely) every so often.  How frequently this makes a difference depends on your system's speed, as our P4 and Celeron test systems didn't benefit until many hours of play, while the Phenom II I use personally benefits after around 3-4 hours.  I'm not sure why this is.  Take it as you will.  The benefit is small, and proportional to how long since you closed it.  It increased FPS from 3-4 FPS to 15-30 FPS after I restarted from a 72-hour long nonstop Dwarfathon.  (There was also a lot of unattended Fun).  Readings were taken before and after the restart to ensure it wasn't from a sudden drop in pathfinding on dead dwarves or something trapped somewhere.

Thank you for posting the results of your tests, although I am probably not tech-savvy enough or in possession of the appropriate gaming equipment to act on most of them!

Regarding the specific phenomenon I quoted above, I had also noticed that my forts gained fps upon quitting and restarting, but I hadn't been sure what to make of it or why it was happening. Now I do know!
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Thief^ on September 28, 2010, 06:37:19 am
Quote from: janglur on September 25, 2010, 06:39:31 pm
Go Flush Your Cold Buffer In You SCSI DASD:
Strangely, we tried enabling and disabling Virtual Memory/Swapfile/Pagefile/Disk Cache and the effects it had on DF.  I found that it is actually best to *enable* virtual memory and allocate a fair amount.  I assume this is the result of reducing the amount of clutter in the RAM and thus shaving off a bit more RAM Competition.  The effect was surprisingly substantial at a fairly steady 7-8% FPS improvement.  This has been the single best result short of CPU and RAM overclocking.
I suspect I can explain this as Windows putting some programs to sleep and into the page file, which reduces their competition for CPU resources as well. Perhaps with the page file disabled it doesn't sleep processes like this?

I've also seen Windows page idle programs' data out and use the ram for file cache, which speeds up some programs (but I don't think would affect DF).
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 28, 2010, 01:22:10 pm
Yeah, it makes little sense.  I've ALWAYS gone without a pagefile and it improved performance, right until DF.  I can't figure it out.  But the results are there.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: NKDietrich on September 28, 2010, 01:30:01 pm
It's generally best to keep your pagefile enabled in Windows 7. Stuff that doesn't need to be put into memory generally won't be. But if you don't have a pagefile enabled, it's putting a lot of crap in memory that really doesn't need to be there. It's a lot smarter in that respect than XP, and even Vista.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: motorbitch on September 28, 2010, 04:45:57 pm
all praise the virtual memory, brought to us in 1956 by mr. Güntsch.
thou shall not disable it*.






*serious... its stupid.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Vigilant on September 28, 2010, 06:00:47 pm
Quote from: NKDietrich on September 28, 2010, 01:30:01 pm
It's generally best to keep your pagefile enabled in Windows 7. Stuff that doesn't need to be put into memory generally won't be. But if you don't have a pagefile enabled, it's putting a lot of crap in memory that really doesn't need to be there. It's a lot smarter in that respect than XP, and even Vista.

Re: virtual memory. Hard drive access speeds at their BEST run on the order of 2-7 miliseconds, you can get like 0.10ms if you're using a really new solid state drive. So that's 10^-3, pretty good right? ... Not exactly.

Memory access speed is between 0.6-10 nanoseconds in modern architecture. 10^-9. And if you get a cache hit with the processor memory it can be about ten times faster than that.

Granted still the 2-7 milisecond's is pretty fast and if you're only using the virtual memory for one or two things you'll probably not notice. But for something like DF... things like path-finding and reading information about items access memory a LOT. We're starting to come out with most of the time more memory that you'll need for most sane applications (this laptop has 4 GB, and i've heard of people that have as much as 16 on their machine). If you've got a low amount of memory i guess use it, but if you've got an abundance it's better to make the most use of it possible, and XP at the very least DOES not always make full use of your memory. If it used up every bit of your memory then switched to virtual, i'd be happy, but it doesn't. And i've noticed the difference in performance since i've turned it off, particularly on using Eclipse for big projects and Civ4.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 28, 2010, 06:35:39 pm
Vigilant is right.

As such, I have to conclude that the DF framerate improvement is the result of simply less memory fragmentation caused by not caching VM into RAM, and thusly reducing the accesses per second required for refreshing said data rather than reading or writing.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: darkflagrance on September 29, 2010, 03:24:33 am
Regarding clutter causing lag:

Does this only extend to loose clutter like barrels or stones? Or does clutter also include the blocks that have been built into megaprojects? Do immobile objects like that still constitute a drain on RAM?
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: motorbitch on September 29, 2010, 04:41:20 am
Quote from: janglur on September 28, 2010, 06:35:39 pm
Vigilant is right.
no he isnt. you both totaly did´nt understood how virtual memory is working, when data will be stored in virtual memory and when not.

Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: ungulateman on September 29, 2010, 06:09:41 am
And you haven't explained it, which makes you either a troll, inattentive or unwilling to admit that they're right for some reason.

So leave, explain yourself or explain yourself some more, respectively.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: NKDietrich on September 29, 2010, 06:28:01 am
Vigilant is correct, at least in terms of Windows XP. Windows XP has relatively poor memory management. It will leave a bunch of empty memory and use the pagefile when it isn't really necessary. Windows Vista and 7 are far better in that respect. Vista/7 actually have a very smart system for caching stuff in memory. Disabling the pagefile is an archaic concept that only ever applied to older operating systems, and even then it had some drawbacks. (Certain apps would crash and burn.) Anyone still recommending it for Vista and 7 are just holding on to an old habit.

But the best thing is to simply try it for yourself. It's not like it's going to hurt anything to turn it off, then back on if you don't see any benefit.



Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Socializator on September 29, 2010, 10:00:59 am
This post doesnt really contribute to virtual memory yes or no, but rather discuss"

Quote from: NKDietrich on September 29, 2010, 06:28:01 am
Vista/7 actually have a very smart system for caching stuff in memory.
Now I dont want to nitpick, but that smart caching system on my Vista64 was caching half downloaded  xGBs file into RAM. I can only guess, but his crystal ball assumed, that it will speed things up if uTorrent will have this in RAM right after the start, as it was used by uTorrent for several windows sessions... :-/
This wasnt sole problem, online game Battleforge checks its own integrity (like every online game) before letting you play. which is several gigs of files to check(sum). As this is only at game launch, it doesnt really matter that much. Yet, this Vista's precaching visioner decided to load it into memory right after the startup, just in case...
Both of this cases are nice example of complete lack of "timing"... Startup is one of the most critical times on my system - multiple apps competing for the resources already... why start this precaching at all? putting more strain on HDD...

Bah, I guess I did nitpick after all :)

But this is just to illustrate that REAL LIFE memory management is quite complicated thing and that "motorbitch" is just trolling some typical ideal world uni knowledge.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: ZhangC1459 on September 29, 2010, 01:03:05 pm
now now, just because he thinks he's right doesn't mean he's trolling.

Granted, I know nothing about this sort of thing, academic ideal or real life, so I can't join in this discussion.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 29, 2010, 01:29:47 pm
Quote from: darkflagrance on September 29, 2010, 03:24:33 am
Regarding clutter causing lag:

Does this only extend to loose clutter like barrels or stones? Or does clutter also include the blocks that have been built into megaprojects? Do immobile objects like that still constitute a drain on RAM?

It seems to count everything, including massive amounts of layers (water, blood, etc.)
Some testing in arena mode after a massive 5,000 vs. 5,000 killed framerate too after they painted the town red.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 29, 2010, 01:32:17 pm
Also, I should note my tests were on various versions of XP.  XP-64 Pro, XP-64 Pro, XP-32 Pro, and Xp-32 Media Center
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Vigilant on September 29, 2010, 03:56:23 pm
Quote from: NKDietrich on September 29, 2010, 06:28:01 am
Vigilant is correct, at least in terms of Windows XP. Windows XP has relatively poor memory management. It will leave a bunch of empty memory and use the pagefile when it isn't really necessary. Windows Vista and 7 are far better in that respect. Vista/7 actually have a very smart system for caching stuff in memory. Disabling the pagefile is an archaic concept that only ever applied to older operating systems, and even then it had some drawbacks. (Certain apps would crash and burn.) Anyone still recommending it for Vista and 7 are just holding on to an old habit.

But the best thing is to simply try it for yourself. It's not like it's going to hurt anything to turn it off, then back on if you don't see any benefit.

Vigilant uses XP ;)

However still, why would caching stuff in the page file ever be more useful even with smarter management? Even with a solid state drive you're not going to get anywhere near the access speeds being as fast as RAM. Virtual memory does seem like a great hardware solution for making more efficient use of resources... but I can't see how it improves performance, it looks like it just reduces the impact of having an insufficient amount of memory in your machine. I guess i could see it maybe extending RAM life, but that's about it.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: darkflagrance on September 29, 2010, 04:31:11 pm
Quote from: janglur on September 29, 2010, 01:29:47 pm
Quote from: darkflagrance on September 29, 2010, 03:24:33 am
Regarding clutter causing lag:

Does this only extend to loose clutter like barrels or stones? Or does clutter also include the blocks that have been built into megaprojects? Do immobile objects like that still constitute a drain on RAM?

It seems to count everything, including massive amounts of layers (water, blood, etc.)
Some testing in arena mode after a massive 5,000 vs. 5,000 killed framerate too after they painted the town red.

So in other words, Dwarf Fortress is guaranteed to grind in a halt if you in any way attempt to enjoy the game :(
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 29, 2010, 04:43:16 pm
Quote from: Vigilant on September 29, 2010, 03:56:23 pm
Quote from: NKDietrich on September 29, 2010, 06:28:01 am
Vigilant is correct, at least in terms of Windows XP. Windows XP has relatively poor memory management. It will leave a bunch of empty memory and use the pagefile when it isn't really necessary. Windows Vista and 7 are far better in that respect. Vista/7 actually have a very smart system for caching stuff in memory. Disabling the pagefile is an archaic concept that only ever applied to older operating systems, and even then it had some drawbacks. (Certain apps would crash and burn.) Anyone still recommending it for Vista and 7 are just holding on to an old habit.

But the best thing is to simply try it for yourself. It's not like it's going to hurt anything to turn it off, then back on if you don't see any benefit.

Vigilant uses XP ;)

However still, why would caching stuff in the page file ever be more useful even with smarter management? Even with a solid state drive you're not going to get anywhere near the access speeds being as fast as RAM. Virtual memory does seem like a great hardware solution for making more efficient use of resources... but I can't see how it improves performance, it looks like it just reduces the impact of having an insufficient amount of memory in your machine. I guess i could see it maybe extending RAM life, but that's about it.


Because pagefile is used for two primary reasons, assuming optimal use.  The first is to cache an 'original' copy while a loaded copy is altered in memory.  This mirroring lets it re-load the original if need be.
The second is that not all data in-use is used constantly.  DF is an example of something that must never, ever be paged off.  Whereas something like, say, my print spooler service?  I print once a year, maybe.  So paging that (rather large) mass of DLLs off to pagefile, where it will be accessible but not really accessed, is far more memory efficient.  It also reduces power consumption (less cells needing the approx. 2us refresh signal in RAM), reducing RAM competition (less crap to check on, which is a wasted cycle if it's not being read/written), less RAM fragmentation (a small gain at best, but..), and more RAM free for tasks that really need it.

Realistically, pagefiles tend to accumulate a lot of unnecessary crap that programs don't need at all, but would otherwise go to RAM.  Photoshop is an incredible masterpiece of unmentionable clutter for this reason.

Just open Task Manager and enable the 'Virtual Memory' column and see how much each program has allocated.  Notice that many of those allocations are unnecessarily large and not even in use.  With Pagefile turned off, this is *wasted* RAM.  Most people these days don't care since 4GB+ is the norm.  But in extreme system duress, that clutter makes a difference.

For example, RAM is the great barrier for how large an embark you can get.  (FPS is a different issue).  I recently had my friend run DF on his server and he can embark on a FULL SQUARE.  I beleive that's like 32x32 or 64x64, I forget.  But yeah, he takes the *entire* area up and can run it.  (FPS is, again, a whole different thing.)  He has 24 GB of RAM on a server board.  (Also dual quad opties).  Ironically, his still runs slower than mine.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Fedor on September 29, 2010, 05:32:54 pm
Good thread going on here.

About huge embarks:  I run a 15 750 with 4 GB of RAM and can embark without too much wait on a full square.  Playing's way laggy, but the embark itself goes well. Conclusion:  You don't need more than 4 GB to do full-square embarks, but you do need a fast, capable CPU.

About ganged and unganged memory:  I had never head of such a thing, so I looked it up and learned (I think) that it's an AMD-exclusive thing.  Intel doesn't have it.  If true, this should be mentioned in the OP.  Speaking more broadly, even if I mistake here, the advice given in the OP should be clearer about the fact that it only fully applies to specific CPUs and OSes and will not necessarily be valid for others.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Graebeard on September 29, 2010, 08:52:00 pm
Quote from: darkflagrance on September 29, 2010, 04:31:11 pm
Quote from: janglur on September 29, 2010, 01:29:47 pm
Quote from: darkflagrance on September 29, 2010, 03:24:33 am
Regarding clutter causing lag:

Does this only extend to loose clutter like barrels or stones? Or does clutter also include the blocks that have been built into megaprojects? Do immobile objects like that still constitute a drain on RAM?

It seems to count everything, including massive amounts of layers (water, blood, etc.)
Some testing in arena mode after a massive 5,000 vs. 5,000 killed framerate too after they painted the town red.

So in other words, Dwarf Fortress is guaranteed to grind in a halt if you in any way attempt to enjoy the game :(

Disclaimer:  I know nothing of CS.

That said, it seems like this issue is relatively easy to address.  Is there a reason to count constructed blocks and stone?  Is there a reason to query the status of objects not in use as frequently as objects currently being used?  Coverings that are easier to clean or that eventually decay would help.

If I had one wish for the future of DF it would be framerate improvement.  Lag has killed more forts than gobbos ever could.  It would be great to have succession games that could last more than 6-8 turns before becoming unplayable.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Vigilant on September 29, 2010, 08:56:40 pm
I know of CS. Unless the code base is particularly neat and pretty for this section of stuff (highly unlikely given what we've heard about most of the code base), there's probably no way around this without completely redoing those parts. So not easy :(
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: motorbitch on September 30, 2010, 06:57:30 am
Quote from: ungulateman on September 29, 2010, 06:09:41 am
And you haven't explained it, which makes you either a troll, inattentive or unwilling to admit that they're right for some reason.

So leave, explain yourself or explain yourself some more, respectively.
what makes you think you may tell me to go or leave anywhere? and what makes you think that  its my duty to explain anything?

however... i realy dont think one should try to tell the working principles of operation systems within a gaming forum.

1: there are *tons* of real good background informations regarding these techniques, even on wikipedia most informations are correct.

2: its obvious that some informations have been read -and misread - already... i dont belive telling it all agian would do the trick.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: ungulateman on September 30, 2010, 09:03:42 am
Quote from: motorbitch on September 30, 2010, 06:57:30 am
Quote from: ungulateman on September 29, 2010, 06:09:41 am
And you haven't explained it, which makes you either a troll, inattentive or unwilling to admit that they're right for some reason.

So leave, explain yourself or explain yourself some more, respectively.
what makes you think you may tell me to go or leave anywhere? and what makes you think that  its my duty to explain anything?

however... i realy dont think one should try to tell the working principles of operation systems within a gaming forum.

1: there are *tons* of real good background informations regarding these techniques, even on wikipedia most informations are correct.

2: its obvious that some informations have been read -and misread - already... i dont belive telling it all agian would do the trick.

I have the right to suggest for you to go or to leave because you're making the environment of the thread worse. Clearly I can't force you to, but that doesn't mean I can't suggest that you do.

The working principles of "operation systems" is the entire point of this thread. Acting like you shouldn't explain yourself after you say someone else is wrong without giving an alternative point of view is just insulting them.

Link, please, if it's that easy to find. I'm sure that if you want to explain what's going on providing me with information is the smartest thing to do, if not the easiest.

(P.S. Correct your grammar and spelling before you post please.)
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Socializator on September 30, 2010, 09:14:15 am
Quote from: motorbitch on September 30, 2010, 06:57:30 am
Quote from: ungulateman on September 29, 2010, 06:09:41 am
And you haven't explained it, which makes you either a troll, inattentive or unwilling to admit that they're right for some reason.

So leave, explain yourself or explain yourself some more, respectively.
what makes you think you may tell me to go or leave anywhere? and what makes you think that  its my duty to explain anything?

however... i realy dont think one should try to tell the working principles of operation systems within a gaming forum.

1: there are *tons* of real good background informations regarding these techniques, even on wikipedia most informations are correct.

2: its obvious that some informations have been read -and misread - already... i dont belive telling it all agian would do the trick.

Still some people manage to be quite informative... Also I wouldnt call this forum a typical gaming one.
If you have some exact information how the memory manager in XP and especially Vista/7 works, please share. Especially the later one is sometimes intriguing.
The thing is that most common operating system still is XP. This OS was designed some time (is it a decade now?) ago, where typical desktop, not even talking about laptop, had much smaller amounts of RAM available. Nowadays situation is quite different. Disks are still slow, but most of the folks around have lot of memory available.
How can this make a change? Well, ideally not, since "first use RAM, then swapfile" should be always right, but this simply is not a case. For example program (e.g. Dwarf Fortress, or windows update service) wants to allocate some additional memory. It may be hundreds of MB. RAM conserving manager will allocate it into swap, and moves it into RAM later, when the data starts pouring in, as these two moments might be quite distinctively different. But this inherently implies additional slowdown in the program run. Different manager might allocate it to RAM directly. But this may result in lots of allocated but unused RAM. The second one doesnt matter that much now, but it did much more few years ago.
What I experienced with XP was that when I was playing Civilization and was alttabing when waiting for turns, the manager always managed to move the background application's memory to swap. So every alttab was painful. He was actively cleaning RAM from unused memory. although I had enough (by my standards ofc) free of it.
Disabling the pagefile made huge difference. Point is, that the actual manager can be made in many different ways and saying "You are wrong" and "This forum is not worth it" can be indeed understand a bit trollish.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: Rysith on September 30, 2010, 10:08:25 am
Quote from: Graebeard on September 29, 2010, 08:52:00 pm
Disclaimer:  I know nothing of CS.

That said, it seems like this issue is relatively easy to address.  Is there a reason to count constructed blocks and stone?  Is there a reason to query the status of objects not in use as frequently as objects currently being used?  Coverings that are easier to clean or that eventually decay would help.

If I had one wish for the future of DF it would be framerate improvement.  Lag has killed more forts than gobbos ever could.  It would be great to have succession games that could last more than 6-8 turns before becoming unplayable.

If the primary reason behind the framerate decline really is volume of memory access, then I'd guess that it's things like "A dwarf walks past a building. The building must go check to see what material it is made out of, to determine what thought, if any, the dwarf gets out of it". So yes, there are definitely reasons to keep checking constructed blocks and stone.

However, profiling memory accesses is a reasonably easy thing to do (compared to, say, rewriting DF to support multiple threads), and it seems likely that with some well-planned caching there could be some significant improvements. Maybe a project similar to the graphics improvement project is in order?
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: motorbitch on September 30, 2010, 01:08:21 pm
Quote from: ungulateman on September 30, 2010, 09:03:42 am


I have the right to suggest for you to go or to leave because you're making the environment of the thread worse.
do i? pointing you onto bad but popular mistakes dosent make this thread worse. leaving them uncommented would ;)
[/quote]
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on September 30, 2010, 02:52:33 pm
Well, I do notice a .13/.14 vs. .12 speed change.  I beleive that the game may also track the amount of crap going on in the world too, which would explain why equal-sized embarks on different worlds are not usually linear in performance.  But this can also be related to underground geography, temperature, weather, and Z-heights.  It's a very, very hard area to nail down.


But there is an undeniable increase in RAM use and accesses when your fort has a metric crapton of crap.  I'm currently testing to see if this applies to specific objects.  Namely, i'm doing a megaproject fort entirely out of constructed stone blocks to eliminate as many items as possible, and see if walls contribute to this clutter, or only constructed walls, or only specific objects (like loose clutter, and maybe crafts)

I'm building a 100 crafts workshop long corridor to bench the before and after performance to try and see if RAM accesses increase from thought-checks or if it's just a pathfinding coincidence.

This is all fuzzy math.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: decius on October 04, 2010, 09:17:37 pm
Windows 7, AMD Phenom 2 X6   64 bit processor
ASUS MB, Radeon HD 5830 GPU (in case that matters)

RAM:DDR3 1333 8192MB (2x4096MB)

Test save: First winter, 9 dorfs, merchants, a page and a half of units, less than 1k items including stone in z-stocks. I think it's a 5x5 embark, but I could be wrong. I've dug to the first cavern and I'm playing around in there.


Intial settings (factory default): NB freq 2000.1 MHz DRAM freq 666.7
Timing 9-9-9-20-33, unganged. 27 FPS

First change: Ganged: 27 FPS. Other programs had a noticable effect.

Second change: unganged, 8-8-8-18-28. This was the lowest timing that passed POST. FPS 29.

Third (final) change: bumped the freq to 678.0 MHz, NB 2034.0 MHz, 9-9-9-24-30. This is the result of a BIOS auto-optimize, and I didn't expect the FSB and CPU changes it made. I don't have real before and after numbers, but processing performance showed a moderate gain. DF performace showed a remarkable gain: 44 FPS, roughly a 50% performance increase.

And yes, % change from original is a proper metric. 27-30 FPS is the same gain as 90-99 FPS,even though the MS/frame difference is not the same.

Current conclusion: I was limited by CPU cycles, not RAM speed. Anyone have a few benhmark forts, different sizes, number of creatures, items, and jobs? I suspect that things like large embarks, kittens, and designating 2k stones for dumping might stress the system in different ways.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: daishi5 on October 05, 2010, 11:31:41 am
I see a lot of talk on Virtual Memory, so I thought I should point everyone who wants to know more to http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx

Basic overview, yes removing the pagefile can help some programs that are having data paged out, but you are playing computer system chicken.  Your system can crash horribly if it runs out of RAM and has no pagefile.  With a pagefile, Dwarfortress slows down when it starts to run out of physical memory.  Without a pagefile, some random program crashes when you start to run out of physical memory, and it would probably be DF. 
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: janglur on October 05, 2010, 12:04:07 pm
here, decius.  This is one of my tests forts.  Download it HERE:  http://207.177.39.129/save.rar
Let it run for a full 2 minutes before taking the FPS reading.  Remember, it averages!
This is a .15 save (genned in .12)
All features ON.  No elves.  Flux+Steel+sand+magma+river.  Goblin ambushes (no invasions yet)
Year 13, 75 dwarves.  Over 50,000 clutter.
Compressed in WinRAR.  May not work in other clients, but feel free to try.  It gets better compression than zip, 7z, etc

Here's my specs:
AMD Phenom II X3 (2.8 GHz)
2000 MHz NB
4 GB (2x2GB) DDR2 1066 MHz (5-5-5-15-30-2T timings)
ASRock AOD970GX

Default:  31 FPS
Default with single-core DF:  40 FPS
Default with dedicated single-core DF:  44 FPS   [We'll abbreviate this as "DDSC"]
DDSC CPU overclocked to 1.6/1.6/3.4 GHz:  49 FPS
DDSC CPU overclocked to 2.6/2.4/3.4 GHz:  51 FPS
DDSC+CPU-OC plus FSB bumped from 200 to 212 (RAM is now 1129 MHz, CPU 3.6 GHz):  51 FPS
DDSC+CPU-OC with FSB set to 227 and RAM down to 800 (making 912 MHz RAM, CPU 3.42 GHz*) and 4-4-4-15-30-2T timings:  57 FPS
DDSC+CPU-OC as above, with HT set from 2000 MHz to 1600 MHz:  56 FPS
Same as above but 1200 MHz:  56 FPS
Setting pagefile to on (680 MB across 3 drives):  59 FPS
Setting pagefile to 3 GB (1GB per drive):  55 FPS

* At a much higher FSB, the max CPU core speed also dropped, forcing me to drop a couple multipliers.  Despite this, performance still raised considerable.  I kept the other cores as close to 2.6 and 2.4 GHz as possible)
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: helf on October 05, 2010, 12:25:09 pm
Download URL isn't working. Firefox keeps giving me timeout errors on it.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: NKDietrich on October 05, 2010, 12:30:28 pm
Quote from: helf on October 05, 2010, 12:25:09 pm
Download URL isn't working. Firefox keeps giving me timeout errors on it.

Yeah I was going to download and test it out FOAR SCIUNTZ too, but it stalled out on me.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: decius on October 05, 2010, 01:30:19 pm
janglur: I'm not getting the file either.

It seems that you are CPU limited as well... I'm gonna check out how much I can bump the speed and voltage on the CPU.

I suspect that my monitoring software is having a noticable efffect, but I'm not willing to lose the real-time temp monitoring because I'm paranoid. For the record, my current all-time CPU temp record is 24 C, and my current temp is 16 C- which is slightly below what the weather service has for outside air temp, and I'm not cooling the house. I think I might have a little room to play with here.

If I don't respond for a week or so, I might have invented the magma CPU.
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: tps12 on October 05, 2010, 02:59:38 pm
Quote from: Vigilant on September 28, 2010, 06:00:47 pm
If you've got a low amount of memory i guess use it, but if you've got an abundance it's better to make the most use of it possible, and XP at the very least DOES not always make full use of your memory. If it used up every bit of your memory then switched to virtual, i'd be happy, but it doesn't. And i've noticed the difference in performance since i've turned it off, particularly on using Eclipse for big projects and Civ4.

"Use all physical memory first" is a pretty brain-dead strategy, since when you approach the limits of your physical memory, any new allocation is going to require a swap before it can proceed. Whatever XP does might be even worse, of course.

Eclipse performance is interesting...possibly the JVM and OS working at cross purposes?
Title: Re: Advanced Tweaking (Warning: !!SCIENCE!!)
Post by: decius on October 05, 2010, 05:03:05 pm
Moderate overclocking (from 4x200=800MHz to 14x230=3220MHz) had roughly the expected effect: FPS up to about 70-77; the comparison isn't strictly fair, as during the test period an immigration wave arrived and population went from 9 to 31.

Any differences from changing priority from normal through realtime, or from setting affinity between 1,2 or 6 CPUs are less than the variation within each setting.

Multi-core processors have at least one major advantage over single-core. As I was writing this post, my armorer made "Aval Alek, The Loves of Humor" a billon chain leggings studded with billon.

Unless anyone has experience otherwise, I'm going to suggest that early-game slowdowns tend to be CPU related. Unless I see slowdown, combined with lowered cpu usage, I won't think RAM speed is a factor. Results might vary with slower RAM, of for anyone who has a more reasonable amout of RAM than I do.