It gives the impression that he was siding with Mr Sutton and it gave an unfair platform for Mr Sutton when the complainant was not given that same platform, Mr Key said.
But he said Mr Rennie had done a lot of good over the past six years and should retain his position.
"I think it's fair to say that Iain is deeply remorseful for the fact that he was silly enough to hold that press conference on a joint basis - he should have held a press conference, just not one on a joint basis."
Mr Rennie had spent his working career dedicated to the service of others and the public service, he said.
"He will always have on his record now that in this particular instance, it could have been handled a lot better.
"But on the other side of the coin, do you simply say, if somebody makes one mistake like that, then that's the end of it?"
Mr Key did not know if State Services Minister Paula Bennett knew about the press conference.
When asked why Mr Rennie was not also publicly saying the press conference was a mistake, Mr Key told TVNZ's Breakfast show people would have to ask him that.
"I'm not his keeper in that regard." Mr Key told Newstalk ZB there had been "firm and frank" conversations between Mr Rennie and Ms Bennett.
Bennett: I didn't know press conference format
Meanwhile, Ms Bennett told Radio New Zealand in a statement today that her office knew about the press conference, but did not know the format the conference would take.
"Mr Sutton's attendance was not confirmed to me or my office until a media advisory was publicly released on Monday morning.
"The content and the structure of the press conference was not known by me or my office until the press conference was taking place.
"It would have been inappropriate for her or her office to intervene in the matter because it was an employment matter", Ms Bennett said.
Mr Sutton announced his resignation at the press conference last Monday after the investigation found he was guilty of "serious misconduct".
Media reported that the State Services Commission's director of communications, Lisa-Marie Richan, had advised the commissioner not to allow Mr Sutton to speak at the press conference but had been over-ruled. Ms Richan has been on sick leave since Wednesday and could not be reached for comment.
Mr Little said these new details showed Mr Rennie was "unfit for the job".
"The information that ... he was advised by an experienced, in-house communications adviser not to run a media conference, which has always been the issue, takes it beyond a lapse of judgment into a woeful lack of judgment."
Ms Bennett reiterated her confidence in Mr Rennie, though she was unhappy with his handling of Mr Sutton's resignation.
"[Mr Rennie] accepts that it should have been handled differently ... More importantly, he has expressed his regret at any hurt his actions last Monday may have caused the woman who has come forward."
Mr Rennie said yesterday he had a number of advisers who gave him free and frank advice.
"I am confident this is what I received in this case. Responsibility for all decisions rests with me," he said.
The decision to allow Mr Sutton to speak at the press conference has been widely criticised, in particular because he and the complainant were bound by a confidentiality agreement. It was also attended by several high-level officials, giving an impression of support for him.
Equal Opportunities Commissioner Jackie Blue said the handling of the investigation could have a "chilling effect" on victims of alleged sexual harassment speaking out.
- additional reporting NZ Herald
