A single savegame is not a proper indication of DF; the game is too varied. I'd rather have people work with their own savegames, which could provide a more broad approximation of DF performance.

The idea is for people who have access to two (or more) PC's to move their own savegame between them, and reporting any odd differences. I'd love for someone to confirm the AMD/Intel discrepancy.

Also limit gfps to a reasonable low value, turn off vsync, do not use graphics pack, etc.

Grab 7-Zip, compress the folder

and upload it for us to compare.

A single test fortress won't work, but a set of test fortresses seems necessary.  Letting people run their own fortresses introduces possible confounding factors, e.g. people on slower computers might compensate by running less demanding fortresses.

I don't have any AMD PCs.

Honestly, it might be hard to get any rigor in the test unless you can create a standardized benchmarking package (maybe using DFhack to capture FPS information?)  Other games have that (with a variety of preset scenarios), DF could too.  Otherwise, there are just confounding variables all over the place (if the methodology is just 1. run game 2. "quickly" write down your impression of average FPS).

Footkerchief: You do know that DF has a built-in FPS counter, right?

Yes, and it fluctuates over time.  It has to be collected for a period of time and then averaged.  How many samples over how long a period?  These questions matter, because unless your method for collecting that data is 100% consistent, you'll introduce bias.  For example, one might subconsciously pick a lower number as the "average" because they're already expecting the system to be slow.  Also, fortress events can change the FPS, which reinforces the need for a consistent measurement period.

Does the built-in FPS counter have accuracy issues?

When working on large, complicated data, one generally tries to minimize influence of unknown factors.
For example: when trying to figure out the side-effects of a new drug, you'd want to feed the drug to only healthy people. Ideally, you'd even test these people for conditions etc. they themselves might not be aware of. (sidenote; ethics do not normally allow this)
You'll get very clean data, and produce some very pretty clear pictures of what's going on.

The problem of this type of experimentation is that you'll only find results that are valid for the group of healthy individuals, which can mean your results don't apply to any other group. There've been multiple situations in which this has become quite clear.

In testing DF, the definition of 'healthy'... doesn't really apply in the dictionary way. Even worse, the variability between playstyles and the resulting fortresses (and the nature of the computational loads etc.) makes a single test fortress impossible, and would require a lot of test fortresses.

This is sometimes seen in medical research; variation between subjects is too big, and comparing them becomes impossible. In some cases there's a very easy way to compensate for this using difference scores. You expose a subject to two conditions, like a medication and a placebo (not at the same time), and note the differences between the two outcomes.

Since a computer usually doesn't display a placebo-effect, I would say this is a valid and easy way to track some big issues, like processor architecture, OS, RAM speed, etc. Simply said: you simply load the same save on both PC's, and note the FPS on both.

Yes, and it fluctuates over time.  It has to be collected for a period of time and then averaged.

Ideally, yes, but then you're introducing the time as a factor, which needn't be one. FPS doesn't fluctuate that much if a fort is stable (assuming you let the game run for a bit after loading), and big changes (breaching the caverns, releasing the clowns, etc) require a re-measurement anyway. I would even say that you can treat them as separate forts, for testing purposes, since the load is of a different variety.

However, I'm starting to think that setting up a set of 'volunteer testing fortresses' and 'volunteer testing systems' is starting to get inevitable, if I want to see issues like these brought to light...

A single savegame is not a proper indication of DF; the game is too varied. I'd rather have people work with their own savegames, which could provide a more broad approximation of DF performance.

Yes, but a higher FPS can indicate two things: a better CPU or more efficient fort design. Using a standard fortress is completely valid when measuring FPS. Then again, at that point you could basically compared the clock rates of a single core of your cpus. actually theres probably other factors that come into play too like cache size (and what other programs you are running in the background and your OS etc)