There are several factors at play that lead to the current weird borders IIRC;
-Civs of the same type are never hostile to each other, so sites don't get traded between then.
-Armies can freely march (not that they're currently represented, they just launch blitz attacks from a home site) to an enemy site regardless of whats in between. Neutral territory does not impede them. Deserts, rivers, mountains or other geographic features that heavily influence borders IRL are also irrelevant to army movement.
-Civs that are hostile to eachother rarely occupy the same areas because they use different biomes.
-Civs aren't hostile that frequently anyway. During worldgen they just all fight goblins (with the occasional war between elves and humans/dwarfs).
So here are some things that should fix borders (some of them are in the next versions, idk about others);
-Armies cannot path through impassable obstacles (high mountains, long deserts, rivers).
-Armies do not path too close to a neutral site (if the PATROL_TERRITORY duty for nobles works, this can happen easily) and if marching close to a hostile site run the risk of being engaged by forces from that site.
-Settlers will not make a new site too close to a neutral (or even friendly) site (again, PATROL_TERRITORY). This could cause problems though; some civs could be hemmed in and rendered unable to expand. I think it would be offset by forcing populations to be more distributed though, as opposed to the current situation where civs cluster in very specific areas.
-Sites such as bandit hideouts also repel settlers. This could lead to problems though if bandit camps multiply uncontrollably, cutting off territory unless destroyed by civs in worldgen (not currently implemented I think) or by adventurers.
-Cultural variation might lead to wars based on one party's racism, or religious wars could be reimplemented to force otherwise friendly civs to have wars occasionally.
-On the other hand, if an enemy of a civ becomes the leader of their civ, it could cause a war to remove them from the throne.
-Site claims might also lead to internecine wars; say Dwarf Civ 1 looses a site to goblins. Then, Dwarf Civ 2 conquers the site from the goblins. DC1 now has a claim to a site occuppied by DC2, and might conceivably start a war to get it back.
If these were implemented, I think the following would happen;
a) Civs form 'natural' borders during peacetime to avoid ridiculous population density.
b) Border changes based on conquest will roughly follow natural borders due to geography.
c) Enclaves will be less common due to it being a lot harder for troops to reinforce them, making them easy prey for hostile conquest.
d) More wars will happen, and more sites will change hands, accelerating the rate that borders take a more reasonable shape. The only problem is that internecine wars might turn into continuous struggles as each side just gets more claims, more enemies of each civ have a chance of reaching a position causing another war, etc. This could lower population considerably.