Personally, I loved Morrowind. I would've loved Oblivion until I learned of an engine limitation that its predecessor did not have (in fact, I did love it 'til then).
In Morrowind, if you fortify an attribute or skill above 100, it works at that level. In Oblivion, the effect is capped at 100 (with few exceptions). I played for dozens of hours at maximum difficulty, slowly building my character up (with a homebrew mod similar to Galsiah's Character Development for MW) to the point that I would be able to contend with the high end beasties. At some 200-250 strength, I realized I wasn't making much progress. So I started testing, and discovered that bullshit hardcoded limitation. Pissed at this point at having wasted so much time, I stopped enjoying the rest of the game, rushed the main quest just to see the story, and uninstalled.
In short: Self-inflicted frustration left a bad taste in my mouth regarding Oblivion.
More objectively, Oblivion's character interaction is miserable compared to Morrowind's, and that wasn't exactly MW's strong suit, given the horrible dialogue interface (nice idea, dull implementation -- though the dialogue itself was fine). That and the "everything levels with you so exploration is meaningless" feature, which is easy enough to fix, are the only two major problems I see.
I'd like to add that Daggerfall, though far less polished/complete IMO than its successors, trounces the others in terms of mood. The main quest is not standard good vs. evil nonsense and has a healthy dose of intrigue. They did a fine job making the nobles seem like scheming bastards. I like that.
EDIT: Expanding on Oblivion's "everything levels with you" crap, it's important to note that, since the vast majority your skills/attributes provide no effect beyond 100 (which includes weapon damage), that once you've maxed the relevant skills it is actually detrimental to gain more levels. More leveling only increases the duration of battles since HP continues to grow for both you and your opponents.