The issue of confidence in Dr Brash's leadership has repeatedly resurfaced since he rolled former leader Rodney Hide in April.
It is an unwelcome distraction for Act at a critical point in the campaign.
Dr Brash will meet Mr Peters in TVNZ's minor leaders' debate tonight.
Yesterday Dr Brash did not rule out the possibility of taking up an overseas post midway through the next term.
"I'm not ruling out anything at this point ... I intend to remain Act Party leader, and whether I remain for the full three-year term depends on the wishes of the caucus.
"I'm not at all assuming that I'm going to walk away at any point. In fact my assumption is that I would not."
But he said having only one or two MPs would restrict how much influence Act would have.
"Clearly, if we have no leverage at all, the appeal of staying diminishes."
Party president Chris Simmons could not say if he expected Dr Brash to be the leader for the full term.
"I really don't know the answer to that. A lot depends on how long Don sees himself in the role."
But he said the board fully supported Dr Brash's leadership.
Mr Key would not say if he believed Act's leadership was stable, and batted away questions about whether his comments on the teacup tape would offend elderly NZ First supporters.
Asked if he remembered discussing Mr Peters or NZ First supporters, Mr Key said: "I genuinely can't recall."
Addressing a mainly elderly audience in Tauranga, Mr Peters indicated he knew what was on the tape.
"Some people think they're superior to you just because you're going grey," he said. "Doesn't show much gratitude, does it?"
He later told 3 News that Mr Key's comments were worse than callous. "It's contempt."
- Additional reporting: Claire Trevett, Jamie Morton