While I think that the current system of memorials for the dead is a good thing. I think that memorials shouldn't remove ghosts entirely, rather just appease them. Prevent you from getting ghosts that have lethal side effects rather than ones that cause unhappy thoughts.
This will require a proper religious tradition, but basically the idea is that dwarves are only properly lain to rest when the funerary rights of the civlization they're a part of are met.
For evil groups, this might automatically be set to 'die in battle' as an appropriate funerary right, thus preventing you from being flooded by goblin ghosts as long as they died during a siege. But some of the more prideful ones might rise up after being killed by cowardly traps, or simple magma. Thus requiring you to have special weapons to kill them or priests to banish them.
But other dwarves might require to be buried in a full coffin, their body as intact as possible, and not naked. The funerary rights of a culture might call for specific clothing. Perhaps a dwarf HAS to be buried with a pickaxe, helmet, and in chainmail. Nobles would require better funerals with more valuable materials as part of the funeral rights or they'd rise up as a ghost.
Things like that.
Religion would allow you to placate ghosts who simply could not be rescued for one reason or another. Further, ghosts would be stuck within X distance of their body parts, able to move the shortest distance possible between two such radiuses if they don't over lap (for cases of explosions of body parts and such).