Gyllenhaal also revealed he hopes to one day get married.
"I think it's hard for everybody, but I also think at some point, you just decide to grow up, you know...?
"If you find the right person... I believe in monogamy. I believe when you meet somebody right it will be right and it will stay that way."
During the interview, Stern quizzed the actor further about his three-month romance with Swift, which ended in January 2011, and suggested Gyllenhaal shouldn't have let the singer get away.
"I feel - and this is me stepping in - you should have married Taylor Swift, and I'll tell you why ... What a power couple!" Stern said.
Stern went on to claim he would have advised Gyllenhaal to propose to the pop superstar had they been pals at the time, adding, "I would have said, 'Jake, this is The One'. Because to me, nothing is sexier than a woman who can play songs and write."
An amused Gyllenhaal, who agreed Swift is a "beautiful girl", quipped, "Why didn't you have me on this show sooner?"
- WENN