Pilgrim has spoken with Hope Project NZ, talking in detail about what it was like to live at Gloriavale.
He said growing up in the isolated community was great and it "hummed along" like a well-oiled machine.
"All the residents that live there, they share everything. They have a common purse which means their finances are managed by people who manage finance and there are people that do all of the cooking.
"Everything is provided, kind of like a big family unit but with 600 people all working together to build something bigger than ourselves," Pilgrim said.
He married his wife at age 21 and started a family.
Pilgrim was busy, first working on the farm and later doing a variety of different tasks. His wife was a teacher and artist.
"At Gloriavale the people in charge identify the needs and try to find the right people for the job.
"What the leaders ask you to do, that's how you know what God wants so you always say yes."
While he was working through an accounting degree, an opportunity came up for him to become a pilot.
He went away to train for a year, came back and flew commercially for four years.
"That was quite hard on the family because I'd be away from 4.30 in the morning to 8.30 at night. It was pretty brutal.
"It was quite hard to juggle and balance that because there's always so much demand for work and so much need. You're not just supporting your family, you're supporting 600 people."
Pilgrim said being a young man, he thought it was the price he had to pay.
"You just listen to the preaching, listen to what you're told your whole life so you just don't really question that. You take that it is correct and it is true and right because that's all you've ever known.
"You think this big bad place called the world starts at the gates of Gloriavale and it's out there and not in here."
He said even in your own mind, you think Gloriavale is "special" and the people there are superior.
"Then it's almost like the lifestyle, the community becomes the most important thing.
"More important than faith and more important than family. That's just something to recognise in our own nature and to work against it, fight against it."
Pilgrim began to listen to audiobooks which he could access on Youtube through his work. He then went on to share them with his family.
He said together they read a book called Shepherding a Child's Heart by Tedd Trip which became a real blessing for the family.
He said because they wore the uniform, members of the community were branded as "sickos" by the public.
"I was thinking, I'm a parent, I'm a dad, when am I going to be a man and do something about this?"
He penned a letter to the leadership with suggestions to improve the community.
"The response from the leadership was we don't accept the letter, we reject it completely and if you're going to think that way, you can't stay here.
"It was a very crushing moment, the ultimatums that were given to us made it very clear if you're going to be here, you have to completely put aside you're own thinking, your own mind, your own thoughts on every issue in life and just accept what the leaders say."
The family went home to discuss it and decided they could not stay.
Pilgrim said what came next was disciplinary meetings and character assassination.
"Even after hours and hours of just getting ground down, I was at peace, knowing that we'd done the right thing."
That marked the start of their journey, which was a lot easier for Pilgrim than his family.
"I was used to interacting with people and wearing different clothes a lot more.
"These were new things for the family, just getting a bank account and the use of money. It's been so much to learn and so much to adjust to but we went into it knowing it would be that way."