What's more, Mr Silva said it wasn't the first time Prince Andrew had used foul language in his presence. In another meeting at Buckingham Palace the previous year, Mr Silva claimed the Duke said "What you have got to remember is that you’ll never get anywhere by playing the white man."
"I genuinely didn’t know what he meant, and the discussion moved on. But the phrase ‘playing the white man’ stuck in my head, as I’d never heard it before. So when I got back to my desk, I immediately googled it," Mr Silva told the London Evening Standard.
"The definition flashed up on my screen: an old-fashioned saying, used during colonial times, meaning that only white people can be trusted to follow the rules, unlike dark-skinned natives."
Recently, Prince Andrew addressed his relationship with disgraced financier Jeffrey Epstein in an interview with Newsnight.
The 59-year-old, who shares Princess Beatrice and Princess Eugenie with Sarah Ferguson, also admitted he disappointed people by staying at the home of the convicted sex offender.
Answering direct questions about his relationship to Epstein for the first time, Prince Andrew said he "let the side down" and his stay at Jeffrey's home was not "becoming of a member of the Royal Family."
In the wake of the interview, Prince Andrew has been met with a wave of backlash, with medical experts even challenging one of his alibis.