Let's be clear first: we're comparing two ideal hypothetical scenarios. In both, you've just written a great novel.
The first scenario: Traditional publishing. Your book is published, you get a 5-15k advance (if you're lucky) and maybe, MAYBE, 5 or 6% royalties, and that's at the absolute best. (Maybe if you're George RR Martin you can negotiate up by a few points.) The book goes, makes 200,000 sales, and then is delisted. Your publisher doesn't print anymore copies, the bookstores don't ask for anymore, and that's it. The book's over.
Now the second scenario: Indie publishing. (For purposes of discussion, we're talking about an e-book here, as self-print-publishing comes with its own overhead.) Your book is published, you get no advance, but you DO get either 35% or 70% royalty on Amazon (depending on whether or not you enroll the book in the KDP Select program), or 100% if you publish on your own. Now for purposes of equanimity, let's assume the same sales figure: 200,000. (Optimistic, to say the least, but picking a big number so we have something to work with.)
Let's assume your book is priced the same at both outlets: $9.99.
On the first, your book would earn just shy of two million dollars. But your publisher would take their cut, and leave you with just about $100,000, plus your (probably tiny, but hell, let's go whole-hog) $15,000 advance. So about 115k. Not bad!
But now let's take the second scenario. Your book earns the same amount, almost two million. Now though, you take a lot more: almost 700k at the 35% royalty option, almost 1.4M at the 70% royalty option, or the whopping 2M at 100% royalty.
Now maybe you're thinking, "But my book will sell more if it's published traditionally and has a big company doing the PR for me." That's simply not true -- your book will live or die by its own merits. ESPECIALLY on online retailers like Amazon, where indie and traditionally-published books are not segregated from one another in any way, and 99/100 readers don't know the difference, and wouldn't understand it or care if they did.
And if you think the publisher would do the publicity for you? That's wrong too. Publishers don't do publicity for most writers anymore -- not unless you're a big earner like Stephen King or GRRM. They'll ask YOU what your plan is for publicity, what you're thinking about for book tours, etc. Shocking, but true.
And what if your book is in a niche market? What if you've written a novel about an auto-detailer who solves crimes, and the publisher doesn't think it has broad appeal? Book gets printed, gets almost no push from the publisher, goes to stores, sells for a little while (and by "little while" I mean "weeks"), then gets pulled off the shelves, and disappears. Publisher won't put it back on the lists, and it'll never see store shelves again.
But wait, what about the advance?
To put it simply: if your book is good enough to catch an agent's eye out of the slush-pile, to then catch a publisher's eye out of the agent's pitch, to then be signed for publishing and sent out for sale? You bet your ass that book will make AT LEAST the equivalent of the advance, if you publish it yourself.
There's also one major advantage to self-publishing: your books don't go away. Publish traditionally, and once your book goes off the shelves, that's it. Maybe, MAYBE it will go back on, if you write another book in the same series and they think it's worth the expenditure to list the book again -- or if you really hit the lottery and become the next Terry Pratchett, maybe they'll keep all your books stocked all the time. But if you're just the average nobody writer? Your book will be published, go on sale, and then vanish from existence.
But for self-published authors? They just keep earning revenue. Some have had breakout success on a title even years later. The book does poorly for months or years -- all the sudden whoosh, it takes off in sales. Then it dies down again. Then it breaks out again later. When you publish a new book in that series, or at all, your book will spike in sales. And it'll just keep earning money for years.
Maybe you have your own preconceptions about the quality of published vs self-published, but that's completely objective: you could show me shitty self-published books, and I could show you just as many shitty traditionally-published books.
The basic gist is this:
1) Traditional publishing is essentially a lottery. Having a good book increases your odds, but not by much. You lose the rights, your book gets one shot, and then it's delisted and vanishes.
2) Self-publishing succeeds or dies by the strength of your book, your ability to keep putting out new books, and your ability to make your presence known online. Your book stays online forever, and you build your audience gradually, earning revenue from a title indefinitely. And the moment you become successful, publishers WILL come knocking at your door. You wrote "I have a feeling most major publishers won't publish a book that's already been published in any other form." That's just not correct
If your book is already earning on its own, publishers will fall all over themselves to get a piece of it.
Finally, it should be understood that the ease of self-publishing (and e-books) has really blown open what's possible for a writer. With traditional publishing, you're a big name, or you're nothing. There's no middle-ground; publishers only want superstars. But with self-publishing, you can earn a living. There's a "middle class" for writers now. There are writers who sell enough to make their living and support families on their sales -- who you've never heard of, and never will.