Eh, barbarian literally meant "not a roman / greek".
Even more literally, it meant someone whose language (non-Greek, at the origination of the term) just sounds like they're babbling "Bar bar bar bar barbar bar baaar bar!' when they talk. (Although later it also applied to badly-spoken Greek, usually as done by non-natives.) Or it
could have been a reference to the beards of the barbarians, with the same linguistic root as "Barber" has, regarding beard, but unless there's a stylistic distinction from the barbarian kind, classicist (and contemporaneously classical!) depictions of Greeks often included beards on their elders, at least, so that explanation seems a stranger and less likely etymology to me.
Of course this (whichever meaning) developed so that at one time there was the phrase "He who is not Greek is a Barbarian" (obviously originally said in Ancient Greek, not
our barbarious language), giving
your literal interpretation. Hence I'm not saying you're wrong, just that that's a derivative meaning.
The Romans, of course, pinched so much from the Greeks that (culturally, as well as in other ways) that they probably thought the same, even if the term was now a concrete one in its own right.