"I learned enough trying to paint to know that I was no good at it."
Instead, he began collecting New Zealand art soon after graduating from Otago University in the early 60s.
In 1992, Sir James established the James Wallace Arts Trust to administer his collection - estimated at more than 5000 works - which has been available for public viewing as revolving exhibitions. The trust also runs annual awards worth more than $160,000 for emerging artists, and international residencies.
Its beneficiaries include the Auckland Writers Festival, the Auckland Philharmonia and the Auckland Theatre Company.
Sir James' annual contribution of $1.5 million is made possible through the success of his meat-rendering business, the Wallace Corporation.
But he said he worried that the next generation of those with wealth were not supporting the arts.
"There are a lot of very wealthy younger people now with huge incomes - CEOs with obscene salaries - and yet they hardly put any private money into the arts."
Although "embarrassed" about his award, Sir James said he accepted it in the hope it would encourage others to see the rewards that come through support of the arts.
"Even if it is helping with the administration or fundraising, it really has its own rewards; to see the results and be a part of it. My rewards are vicarious."
Last year, the trust and its collection moved to a new home in the restored historic Pah Homestead in Hillsborough.
Bought by Auckland City Council as part of its Monte Cecilia Park project, it will house the collection - worth about $50 million - for the next 30 years.
"We were given a target [by the council] of 15,000 visitors in the first year, and 35,000 by the fifth year. We've done 124,000 visitors in 11 months. All sorts of people come through."
Sir James said one girl who visited on a school trip brought her family back the following Sunday.
"The house is so beautiful it breaks down the barrier between people and seeing contemporary art. That's why they keep coming back."