But once you've made your bones it's actually a wonderful thing to be part of a legacy like I am."
In the 27 years since he made his screen debut as Darren in Jason Lives: Friday the 13th Part VI (he had one line of dialogue before being butchered), he has appeared in more than 60 TV shows and movies, two of which were shot in New Zealand - The Last Tattoo in 1994 and and The Last Samurai in 2003.
During that time he has developed a dual career as a director and credits include episodes of Dexter, Law & Order and Justified, which is what first brought him into the orbit of Scandal creator Shonda Rhimes.
"Shonda and I had worked together before when I'd directed two of her other shows, Grey's Anatomy and Private Practice," says Goldwyn. "I really had a lot of respect for her, so when she called me about whether I wanted to do Scandal I was already interested. And when she said the lead was Kerry Washington, who I'd wanted to work with for a long time, I knew the series was something I definitely wanted to be involved with."
Goldwyn had to take it on trust that President Fitzgerald Grant would evolve into an interesting character because he makes only a fleeting appearance in the pilot episode.
"And I'm very glad I said yes," he says. "I think most people will write Fitz off initially because he doesn't seem very likeable, but by the fourth or fifth episode you start to see things from his point of view. And that's what's really fun about the show - there are no good guys or bad guys. You will think someone is awful and all of a sudden they'll turn around and break your heart. Or you'll fall in love with a character and then they'll do something absolutely horrifying, as we do in life."
The cast don't know what's in store for their characters until they sit down together and read a new episode a couple of days before they start shooting it, says Goldwyn. "At first I was nervous about not having much time to prepare but it's turned out to be the most exciting thing - the fact we never know what's going to happen and we get to share it together means it's very much like the audience's experience of Scandal." Even though, as Goldwyn says, "the show just gets more and more crazy and off the charts and, frankly, addictive. I never find myself thinking, 'Oh boy, nobody's going to believe this, we gotta really sell this one'.
"What's amazing about Shonda's writing is that however outrageous the circumstances may be, every scene feels absolutely realistic in an emotional sense. That's because what she's really interested in is digging deep into human relationships, and she treats the things we all struggle with in our daily lives with great intelligence and respect."
Season one of Scandal screens Tuesdays and Wednesdays, 9.30pm on TV One, and is also available in its entirety on TVNZ Ondemand.