However, he was only two-years-old when the Sarjeant Gallery opened in 1919, an opening that also marked the start of his aunt and her new husband's regular trips to Europe - initially to acquire works of art for the Sarjeant collection.
Mr Stewart's memories of his Aunt Ellen from the time he was about 10 until her death in England in September 22, 1939, bring the shadowy figure of Ellen to life as an autocratic strong-willed woman who loved art, painting and socialising both at home and abroad.
She also retained strong links with family members in Whanganui and other parts of New Zealand during her lifetime.
Her first husband Henry Sarjeant, owned farms in the Mangamahu Valley and Fordell and was a close friend of Ellen's father, surveyor John Tiffin Stewart who was the first European to travel up the Whanganui River.
Ellen was the eldest of John Tiffin Stewart's 10 children and had four sisters and five brothers. Mr Stewart says it is likely that the friendship between Henry Sarjeant and John Stewart may have led to the marriage between Henry and Ellen despite their disparate ages. The couple were married in 1893 when Ellen was 25 and Henry was 63.
They moved into Whanganui after their marriage following a two year tour of Britain and the continent that included visits to many art galleries. On their return they built a grand two storey home in Bell Street. Prior to Henry's death in 1912 at the age of 82, the couple again travelled abroad in England and Europe.
Mr Stewart says Henry was known as a generous benefactor within the Wanganui community and it is very likely that Ellen played a big role in consolidating Henry's interest in the arts and his subsequent bequest for an art gallery and collection.
Within a year of Henry's death Ellen re-married John Armstrong Neame - a teacher at Wanganui Collegiate - who had come out from England to teach French. He was also a talented artist and Mr Stewart believes they must have met through their mutual love of art.
Mr Stewart says the couple retained the Bell Street home for several years and travelled extensively to Europe purchasing works for the Sarjeant Collection after the gallery opened in 1919.
He says Ellen owned cottages at Castlecliff and retained a strong involvement in the administration of the Sarjeant Gallery and the acquisition of paintings and sculpture for the Sarjeant art collection after Henry Sarjeant's death. Her direct involvement in purchasing works for the collection ceased in the late 1920s and early 30s due to the couple spending the majority of their time in England and Europe.
During their time abroad, Ellen and John spent a great deal of time in Alassio on the Italian Riviera and in England where Ellen also owned property.
"Ellen inherited a great deal of money from Uncle Henry and behaved in a way that made her appear very wealthy," says Mr Stewart.
'" I think it was a glamorous life in Italy. A large group of English who had retired there partied a lot and there were a lot of social events."
Mr Stewart says his Aunt Ellen was a "fairly domineering woman ". He says she had a great sense of style and was a huge fan of Queen Mary - wife of George V.
As well as acquiring works for the Sarjeant Gallery collection on their trips to Europe in the early to mid-1920s Ellen and her husband were both accomplished artists.
Mr Stewart says Ellen held regular two- yearly exhibitions in Leicester Square London. Queen Mary was rumoured to have purchased at least one of her paintings at each exhibition.
Mr Stewart says Ellen and her husband were living in Alassio in 1939 when war broke out. The pair were given safe passage through France by train to get on a boat to Britain.
"It was mid-winter and they spent two weeks on the train with hardly any food and no heating."
Mr Stewart says they eventually made it to England but Ellen collapsed with a heart attack and died on September 22, 1939 in Sussex.
Mr Stewart, who was 22 at the time Ellen died, had joined the RNZAF and was posted to Britain in 1941 to an RAF squadron and saw quite a lot of John Armstrong Neame after Ellen's death.
"He was working in London deciphering manuscripts at the war office and we would lunch together when I went on leave sometimes travelling down to a country house where he stayed that belonged to a cousin."
He says unfortunately John Neame - whom he always referred to as Uncle John - was left with very little money after Ellen's death.
In her will dated May 8 1937, Ellen bequeathed an annuity that she bought in the Australia Mutual Provident Society that was due to expire at the end of 15 years (1952) to her husband.
She stated in her will that the "political ideas of a socialist Government" had obliged her to purchase annuities "to provide for security in my and my husband's old age". Mr Stewart says the annuity would have enabled her to lead a very comfortable life while she was alive.
"It is clear that she believed her husband John would be left comfortably off in the event of her death, as he would have her annuity.
"However, the annuity expired on her death, and he was left with a very small annuity that she had taken out for him."
John Armstrong Neame died in Alassio. Mr Stewart was later told that some of his British friends in Alassio sold most of his remaining possessions to raise money for a headstone for his grave.
