Fellowes said: 'We asked Dan if he'd come back at the beginning of series four and die in the first episode so that we could have Christmas sort of clean with the arrival of the new baby. But Dan was set on a clean break at the end of series three and he said, "I really feel it's right for me to go and finish now."
'Then you have a character who is happily married with a new baby and is heir to the estate but is not willing ever to be seen again.
'There is only one option - the Grim Reaper.'
Fellowes added: 'I was terribly disappointed at the time but Dan was totally within his rights.
'I don't want to pretend he did anything out of order, because he definitely did not. He gave his notice but it caught us on the hop.'
The death of Matthew Crawley provoked an angry backlash from viewers who believed that the Christmas Day episode should have had a more upbeat ending.
The character was killed just weeks after Lady Sybil, who was played by Jessica Brown Findlay, died in childbirth.
Fellowes said that Stevens only revealed he wanted to leave as filming was about to begin on the third series, which was screened last autumn.
By that stage Fellowes had already written five scripts for the series including an entire episode devoted to Sybil's death.
He recounted that: 'Jessica had said for ages, "I just want to do three years," so we had planned her departure a long time in advance.
'We knew she would die in episode five and we knew we would have another three episodes to get over it. But Dan didn't decide to leave until the point where we were beginning series three and we couldn't have another episode which was all about his death.'
Fellowes said he tried to create alternative storylines for Stevens and Brown Findlay.
'If Dan or Jessica had been prepared to do two out of the eight episodes on series four there wouldn't have been any problem.
'She would have been living in Dublin and he would have been sent to Washington and they would have come back.
'But they wouldn't do that.'
There was a further setback for the show when it was revealed that Siobhan Finneran, who plays the scheming but hugely popular housekeeper Sarah O'Brien, had also decided not to appear in series four.
Fellowes said: 'The advantage of a servant character leaving is that you don't have to kill them off. They go off to do another job and that's fine and you don't have to explain why they never came back.'
It is understood that there will be fewer departures and deaths in the new, fourth series, which is Britain's most successful TV export ever and has been shown all over the world, attracting an estimated global audience of some 120 million.
Key characters including Hugh Bonneville (Lord Grantham), Michelle Dockery (Lady Mary), Elizabeth McGovern (Lady Cora), Maggie Smith (the Dowager Countess of Grantham), Laura Carmichael (Lady Edith) and Lily James (Lady Rose) have all signed contracts that will bind them to the show at least until the end of series five.
Gareth Neame, the show's executive producer, said: 'I can tell you that we have now signed up the main cast members until series five, so there won't be any more shock exits for a while.'
Fellowes also eased the minds of fans by confirming that Maggie Smith, who plays the irresistible Dowager Countess, would be appearing in every episode of series four, including the Christmas Day episode.
He said: 'I am afraid that she is like Mark Twain: reports of her death have been greatly exaggerated.'
The first episode of the new series, which begins six months after Matthew Crawley's death, won rave reviews at a press screening last week.
- Mail On Sunday
