Northern Fire Communications shift manager Scott Osmond said the house was part of a dairy farm on State Highway 5 in Rangitaiki, a tiny town on the Napier-Taupo Rd, about 30km southeast of Taupo.
Two ambulances and two helicopters were sent to the fire.
A Fire Service spokeswoman said the house was well alight when crews arrived.
It was under control and crews were dampening down the site by 5.30am, she said.
Rangitaiki Tavern owner Marion Klaus said the small, close-knit community was devastated by the news.
She and her team had been feeding and looking after dairy farm workers and emergency services staff members since the tragedy.
The tavern opened at 5am and Klaus heard the helicopter early this morning.
"It's tragic. It's very tragic," she said.
"There's a lot of young staff who have probably never experienced anything like this before and the fact that it's their mate from the farm... it will take them a long time to get over it."
Klaus said she knew who the woman and baby who had died were, but did not want to share that information as their family had not yet been told.
A large group of staff lived in the houses on the dairy farm and she understood a house near the cow shed caught fire.
The workers all knew each other well and the rest of the staff were devastated, Klaus said.
The community, including the Civil Defence officer, were supporting each other today.
"My God, the community here have all just come together and it's amazing."
Taupo Mayor David Trewavas says the settlement is quite a big producer of dairy.
It is made up of farming families, and will be reeling at the news, he said.
