"It's been bittersweet to see this delightful creature take shape and come to the sad realisation that he simply doesn't have my voice."
The Pride and Prejudice star said while he'd decided to give up the part, he still felt "rather protective" of his furry alter ego.
"I'm pestering them all with suggestions for finding a voice worthy of him," he said.
Firth's departure comes after the first image from the film was released online, only to be turned into a mock-horror meme by internet pranksters.
They thought the somewhat menacing image of Paddington standing outside Buckingham Palace was spooky, and christened him "Creepy Paddington".
The hashtag gained traction on Twitter as users submitted photoshopped images implanting the bear into some of cinema's most famous horror films.
It's believed Firth's decision will not affect Paddington's release, slated for Christmas this year.
- AAP