Writing in the Evening Standard, Mr Silva, who is of Sri Lankan descent, said that after 50 minutes of discussions "we were running out of things to talk about".
Mr Silva, 38, who was born in Yorkshire, went on: "Then I remembered my colleagues and I had been looking at ideas for reforming the government department for international trade, so I thought that might be a good subject to bring up. 'What about the trade department?' I asked. 'Could it be doing a better job'?"
Mr Silva recalled the Duke's reply. "'Well,' the Prince said with a smile, 'If you'll pardon the expression, that really is the n***** in the woodpile'."
At the time Mr Silva, now an entrepreneur and academic, was so taken aback but also so "overawed" he stayed silent. "I'm ashamed to say I didn't do anything about it," he wrote yesterday, "I felt overawed and extremely conscious of the gulf in status between the two of us."
The meeting ended and Mr Silva exited Buckingham Palace "blinking into the sunshine... reeling at the prince's use of language".
A year earlier, Mr Silva had attended a first meeting with the Duke, at the time a trade envoy, to discuss the tech economy. The conversation turned to the European Union and concern that Brussels was blocking entrepreneurial endeavour. Mr Silva said that Downing Street was trying to reform the EU. Mr Silva recalled: "Prince Andrew leaned across to interrupt me. 'What you have got to remember,' he said, 'is that you'll never get anywhere by playing the white man'."
Mr Silva writes that he was unsure what the Duke had meant and later looked up the phrase on the internet. "The definition flashed up on my screen: an old-fashioned saying, used during colonial times, meaning that only white people can be trusted to follow the rules, unlike dark-skinned natives," wrote Mr Silva.
The allegations - that Prince Andrew had used racist language - remained private for the past seven years. But prompted by the Newsnight interview, Mr Silva had decided to step into the limelight and recount the meetings, admitting: "I can't help but wish I'd said something when I had the chance."
Buckingham Palace insisted that the Duke had never used the phrase 'n***** in the woodpile" and categorically denied he would ever use such language.
The Duke's post-combat sweating condition was not recognised by government health professionals on Monday.
In the toe-curling interview, the Duke had denied suggestions of a night spent with Virginia Roberts Giuffre when she had been 17, saying her account that he had been "sweating profusely" could not be true on account of a condition brought on by his military service.
He said there was a "problem" with the suggestion he had been sweating profusely on the night in question because at the time been unable to sweat having served as a Royal Navy helicopter pilot in the Falklands War of 1982.
It is possible for the human body to develop the inability to sweat normally, a condition known as anhidrosis.
However, after contacting various government departments including the Department of Health and Social Care and the Ministry of Defence, the Telegraph could find no medical professional to support the claim such a reaction could be the result of a combat-induced adrenaline rush.
The Maitlis interview far from strengthening his position has weakened the prince and his standing in the Royal household. Buckingham Palace will inevitably fear more allegations in the coming days as the spotlight turns back on him.
Jo Swinson, the Liberal democrat leader, went down the highly unusual route of criticising a Royal even during an election campaign.
"I watched that interview, I'm sure many people did, and what struck me was that what wasn't really discussed was the people who are most important in all of this, which are the victims," she said, "Those girls and young women who were trafficked, who were sexually abused by Epstein.
"And how you could have a whole interview discussing that without him expressing his views, his sympathy, his feelings about what those young women had gone through I just thought was really disheartening."
Among Royal watchers, there is a concern that the grip on Palace business previously in the iron fist of the Queen and the Duke of Edinburgh has begun to loosen.
The Queen is 93 and Prince Philip is 98 and it may be no coincidence that both the Duke of York's ill-advised interview with the BBC and a statement released by the Duke of Sussex, in which he went to war with the British press, both went ahead without prior approval of the monarch.
Christopher Wilson, a respected, Royal biographer, said: "Prince Philip was the guiding hand, the disciplinarian, and now he's out of the picture, it's become a bit of a free for all.
Just look at Harry and Meghan freelancing out their PR and doing their own thing - there is no central command at Buckingham Palace anymore. You can see the disintegration."
Mr Wilson said the Epstein scandal threatened to impact on the wider Royal family - and could even damage the Queen's legacy.
"The Queen has been hugely compromised. She's coming towards the end of her reign and this will be seen as a colossal loss of judgment," said Mr Wilson, who urged the Prince of Wales to fill the power vacuum.
"Prince Charles really now needs to step up and crack the whip," he said, adding ominously of the Epstein scandal: "It will undoubtedly get worse before it gets better."
