The Olivers, both 37, have four children: Poppy Honey, ten; Daisy Boo, nine; Petal Blossom, three, and Buddy Bear, 21 months.
Her confession comes as a particular surprise as she and Oliver have previously spoken about their absolute trust in each other.
In fact, Mrs Oliver was once ridiculed for saying she was certain he would never cheat on her during his long absences filming cookery programmes.
In a 2008 interview, she said: "I am very secure. People say "Oh you can't trust a man 100 per cent," but I'm afraid I say I can.
"They say every man will have an affair, but I really don't think mine will.
"Actually, I know he won't."
The Olivers are worth an estimated £150million. They divide their lives between homes in London's Primrose Hill and Clavering, Essex. Mrs Oliver, who gets up at 6am every day to look after her children, juggles her family duties with careers as a children's clothes designer and author.
So it's little wonder the former model admitted to being angry with her husband for constantly referring to her as "the missus" and portraying her as unambitious in interviews.
She said: "I did mind. And he got a real b******ing. I thought it was funny when we started, but now, when he is on the phone to his friends and calls me "the missus", I say, "Don't call me that"."
Mrs Oliver also said she was riled when her husband suggested she did not understand his work projects.
But she admitted she had been naive to expect him to stay at home while he was filming Jamie's American Road Trip, in which he raised awareness of obesity and poor diets in the US.
She said she told him: 'How dare you say I don't get what you are doing? You don't get what I'm doing back here.'
She added: "I understand what he is doing a bit more now. Before, maybe I was a bit naive, and I thought it was all about me and the children.
"How could he leave me to go and help people in America?
"Now I do see. But there are still times when I say, 'I don't care, I just want you home'."
And despite her husband's fame, Mrs Oliver said she rarely accompanies him on high-profile engagements.
It also seems Oliver has adapted well to what was, until the birth of Buddy Bear, an all-female family.
"He's quite girly," Mrs Oliver said. "He chose the pink in the house. He does the girls' hair in the morning, plaits it, straightens it.
"He'll kill me, but yeah, he adores me. He'll do whatever I say, really."
Two years ago, Oliver confessed he had considered a vasectomy because he did not want any more children.
But it appears his wife may have persuaded him to change his mind, saying she might have another child.
Asked if she would concentrate more on work in the future, she said: "Or I'll have a fifth child - see how long I can eke this out. Maybe. Jamie said I could."
- Daily Mail
