"The role of Prime Minister is a tough gig at any time," Little told reporters.
"It is hugely challenging, not only in the role but for family members.
"I want to pay tribute to the contribution that John Key has made to New Zealand in his time as Prime Minister."
He added: "Today is a day to say to him about his service, thank you for serving New Zealand and for New Zealand well.
"There are still challenges, and we need to rise to those challenges as a country."
Little said he did not have a view on the next prime minister and the decision was "entirely up to National". He would not name a preferred successor to Key.
He also would not be drawn on whether an early election should be called, allowing voters to choose the next prime minister instead of the National caucus.