All of this trivia is fun. I knew ridiculous breast bouncing in anime is called "gainaxing" but I didn't know they literally made hentai, I just figured it was regular pervy anime stuff.
Well the term "literal hentai" would be confusing here, since it's not really well defined (and has no actual relevant meaning in Japanese). What is "literal" hentai here and where are we drawing the line between that and ecchi? Pretty much everything Gainax ever did is less ecchi that some of the current TV shows such as Highschool DxD, so no, they never made literal hentai.
EDIT: if we want an example of what are clearly literal hentai people who went legit, the example is clearly Hoods Entertainment, who make Seikon no Qwaser and similar animes. They were super-obviously a studio set up to give hentai artists a job in the legitimate TV anime world, and would pick up licenses too squicky for anyone else to touch (usually involving magical breasts, and/or bodily fluids as a plot device).
Then Hoods did Mysterious Girlfriend X (which is squicky but more mainsteam), got a hit series out of it, and after that they do more mainstream shows. To give you an idea how mainstream they now are, I'm watching 3D Kanojo Real Girl, a light hearted romcom shoujo anime with almost no fanservice, and just discovered (while looking up Hoods for this post) it's actually made by
Hoods, the same goddamn people who made Seikon no Qwaser, one of the most perverted TV anime of all time.