The AMA definitely implied it'll be changing only one enemy unit at a time, and that there would be a clear break with the old unit type, which would be retired before you see the next one. Since units are phased out one at a time, there will be continuity to the system.
If you always use flamethrowers, then one by one, the old enemies will be replaced by ones which handle the flamethrowers better, but they won't all magically gain flame-resistance at once. So, if no enemy is good against flamethrowers then at first they'll be very effective, but gradually less-so.
Also, because the game looks at performance-data then it won't necessarily have hard-coded ideas in it like "if player uses flame weapons, add flame resistance", because adding flame resistance isn't necessarily the best use of limited resources for dealing with flamethrower attacks. snipers would be more cost-effective, or leaping melee units. The enemy that's the most-useless against your flamethrowers will be phased out first, meaning that you'll still be able to use the flamethrowers against some enemies, but you'll have to be planning for new enemies to appear, at the same time.
As for metrics, I'd imagine that each unit class would have a different mission role, and thus have different metrics. You definitely don't want something naive like keeping all units that survived a long time. Cheap meat-shields and/or tanks should be there to draw fire, so even if all your "zombie" units died during a battle, that might be a great success, because they drew fire away from your heavy-dps wizard-type creature. So, for shield/tank units, the percentage of player fire that they drew during the battle should be one of the metrics.
For the metrics, hand-crafting them seems like it would be pretty iffy. If I was doing it, I'd set the AI to battling other AIs, collecting data about each unit's performance (this would include all actions taken by the unit, and also against the unit), then feeding that into a *randomized* set of metrics. Different AIs then optimize their armies based on their random metrics, and the dumbest AIs get weeded out and the best AIs get mutated. Thus, you can evolve AIs which are better at working out how to evolve their armies.