"Llanocetus denticrenatus is an ancient relative of our modern gentle giants, like humpback and blue whales," explained Felix Marx of the Royal Belgian Institute of Natural Sciences.
"Unlike them, however, it had teeth, and probably was a formidable predator."
The University of Otago's Professor Ewan Fordyce - who recently described one of the oldest known species of baleen whales - said it had been thought filter feeding first emerged when whales still had teeth.
"Llanocetus shows that this was not the case."
Like modern whales, Llanocetus had distinctive grooves on the roof of its mouth, the researchers explain, which usually contain blood vessels that supply the baleen.
But in Llanocetus, those grooves cluster around tooth sockets, where baleen would have been useless and at risk of being crushed.
"Instead of a filter, it seems that Llanocetus simply had large gums and, judging from the way its teeth are worn, mainly fed by biting large prey," Marx said.
"Even so, it was huge: at a total body length of around eight metres, it rivals some living whales in size."
The findings suggest that large gums in whales like Llanocetus gradually became more complex over evolutionary time and, ultimately, gave rise to baleen.
That transition probably happened only after the teeth had already been lost and whales had switched from biting to sucking in small prey - as many whales and dolphins now do.
Marx and Fordyce suggest that baleen most likely arose as a way to keep such small prey inside the mouth more effectively.
Soft tissues, including baleen, normally rot away, making it difficult to study their evolution.
As a result, researchers must rely on indicators preserved on the bones, such as tell-tale grooves or lumps indicating the position of a muscle, or holes for the passage of particular blood vessels and nerves.
"Llanocetus presents a lucky combination, where the shape of the bones, small features suggesting the course of soft tissues, and tooth wear all combine to tell a clear story," Fordyce said.
"Crucially, Llanocetus is also extremely old and lived at the very time when Mysticetes first appeared. As such, it provides a rare window into the earliest phase of their evolution."
They'd now like to sort out when filter feeding and baleen first evolved.
"The giants of our modern ocean may be gentle, but their ancestors were anything but," Marx said.
"Llanocetus was both large and a ferocious predator and probably had little in common with how modern whales behave."