Each planet adds 5% research penalty, each system is 1%. Generally planets produce way more than 5x research of a system, so planets are pretty much always worth colonizing if you're expanding at all.
Absolute worst case scenario (Which basically won't happen) on a 12 tile world with zero bonus tiles and your species has absolutely zero bonuses to research or energy. You could put a capital, one adjacent energy plant, one non-adjacent, three adjacent food buildings, and two non-adjacent ones - plus 4 labs. Leaving you with a net of 0 energy, +1 food, and 12 research points. Even that impossibly bad planet would be about the same or better than 5 average systems (many of which don't generate research at all). Throw in some basic bonus tiles and you would be able to easily produce more than double that in science, which would be like having 5 systems with +5 research mines on them. Add in useful population bonuses from happiness or good traits and you could produce way more.
It's pretty much always worth it to colonize planets in the long run. Once you get a bit of pop on them and some tech under your belt they'll produce way more science than the penalty applies. I find my large empires out-research my small ones, unless I'm just expanding haphazardly and not developing my colonies.