I mean, there's mods like
StarNet AI and
StarTech AI that make the AI work pretty well, so it's not like Paradox couldn't easily take those mods as a basis and make the AI immediately human-competitive.
I think part of the issue and reason they can't just take what the mod does or contract out the mod developers is that mods like Starnet AI balance the AI so that it's competitive with humans who have played the game before even on the lowest difficulty that doesn't give the player buffs (player buffs risk teaching players bad habits, hence restricting them to the very lowest difficulty).
From a game design balance pov, the perspective they're taken for AI in Stellaris is balancing it to keep the game appealing to new players, so they'd ideally want an AI that on lower settings a new player can survive against, whilst is competitive on higher difficulties with more experienced players. Otherwise you risk players getting steamrolled when they don't know what they're doing and giving up rather than learning (think most people who try Dwarf Fortress).
The 'best' way to do that is completely AI behaviours for different difficulties, like Starnet-style being used for higher ones and 'current vanilla' for lower ones. That's a testing and maintenance nightmare though, which is why most games go the buff/debuff route for AI difficulties.
So it's definitely more complex than just "make AI good", because you can't make AI too good to always beat new players whilst also making AI good enough against older players. Hence the hack of giving a crapper AI buffs games use.
----
As for Planet/Sector AI, something they've expressed interest in before to reduce micromanagement is to let you create a 'template' for the different planet types it can follow. So you tell it what a Agri-World should look like, and it'll build each one up to look like that. The issue is just getting the time to design and test that, since it'd to be a UI/System able to take into account all the different sizes and modifiers a planet could have in some way.