Gregg said fans often approached him wanting to voice their concern about Coulson's death.
"More and more often people come over to me and talk to me as if we've just lost someone very close to us. Like, 'Dude, how could they do that?'
"Other people are like, 'I'm sitting here with my kids all crying'. I feel kinda terrible."
Gregg, who also has a long TV resume including stints on The West Wing and The New Adventures of Old Christine, said shooting Coulson's death scene was especially difficult.
"The day that we were shooting my farewell, they could tell it was hard on me and the Marvel guys brilliantly already had an early animated scene from near the end of the film where Hulk is pounding Loki to a pulp.
"They showed me an animated version of that, and it was vastly satisfying."
Despite Coulson's death, Gregg said he was hoping for a call from Marvel to resurrect the character in either a flashback role in one of the upcoming Iron Man, Thor, Captain America or Avengers sequels, or in the planned SHIELD TV show.
"I certainly pray for a flashback. I really came to embrace what had happened just because I could have done a lot more films and he might've only been a functionary (character) who showed up sometimes.
"But to have him step up and stand up for everything he believed in to motivate the rest of them was a really beautiful way to go out so I just try to accept it at that.
"I have Marvel's numbers on my phone, I fantasise those numbers will come up."
Despite all the big name actors on set - Robert Downey Jr, Scarlett Johansson, Samuel L Jackson and Mark Ruffalo were just some of the film's stars - Gregg said there was no attitude issue or problem with egos.
"There was something about it that the Avengers storyline seemed to be mirrored on set, and people were coming together for something that was more spectacular than anyone could do by themselves.
"It just cut right to, 'This is monumental, and monumentally difficult', and everyone checked their egos at the door."
Gregg said he couldn't choose a favourite Avengers superhero because he was close to all of them.
"I feel like I'm their sarcastic uncle who babysits sometimes, and has a lot of trouble keeping order. Because of that relationship, I'm never able to choose a favourite, and then have The Hulk hear about it.
"The longest-running relationship was between Coulson, Tony Stark and Pepper and those scenes always felt like coming back to the origins of it. Thor's the guy he really connects with, Captain America's my crush, Natasha and Clint are my brothers and sisters in arms.
"He has a very specific relationship with everyone involved."
* The Avengers is out now on DVD and Blu-ray.
- Herald online