"To be sitting in a tiny Arctic hut with ice and cold all around, and listen to the captain of the Apollo mission tell it like it was, was a thrill never to be forgotten," Mr Hillary said of the 1985 expedition to the geographic North Pole.
"We were incredibly fortunate, because Neil was well known as a very private man.
"He did a lot of things; he had an incredible career. But I don't think on the whole he liked sitting around talking about it.
"But in a way, being in an environment like that, we were all fellow expeditioners, and it was time to tell some ripping yarns."
Mr Hillary was part of a group that also included Steve Fossett, the first man to fly a balloon solo around the world.
Over two weeks, they travelled in small ski-equipped planes to the Pole.
But on the remote Ellesmere Island the group faced a whiteout, with temperatures outside of -40 degrees (Celsius and Fahrenheit are the same at this point), and they spent three days waiting for it to clear.
It was then that the usually private Armstrong opened up about becoming the first man to set foot on the moon on July 20, 1969.
Mr Hillary said that memory still gives him chills.
"I always remember [Armstrong] saying [that] as they were coming down to the surface of the moon, he had a pencil and a piece of cardboard, and he was re-calculating deceleration as they came down.
"Every minute they were changing and altering what they were doing. It was an incredible adventure; it wasn't just some sort of computer-operated event."
Mr Hillary, then aged in his late 20s, said he was visiting his father one day when Sir Ed asked him if he'd like to go to the North Pole.
The group had been invited by expedition leader Mike Dunn, an American who "was one of those guys who would just ring anybody up".
"The idea was to put this amazing group of people together, and that's what they did.
"And I was the youngster at the time who managed to go along with it and share some time with these extraordinary people."
He remembered Armstrong as an "amazing man", whose desire for privacy was evident when he was recognised as they went through Canadian airports.
"The whole business of having that star quality was not something ...he felt comfortable with.
"He felt what he was doing, obviously it was his ambition, but I think he did feel he was doing it for the US and Nasa, and the US Air Force."
Mr Hillary said he received a letter from Armstrong five years ago which warmly reminisced about the trip, and a copy of his book.
For both Sir Ed and Armstrong, the chance to meet and talk over their respective adventures meant a great deal, Mr Hillary said.
"They would come across each other from time to time, simply because in a way they had done similar things. They'd pushed the envelope.
"When [Sir Ed] and Tenzing got up there, it was extending the realm of possibility.
"And of course, Armstrong did that when they went to the moon."
