The Goodbye Yellow Brick Road singer said the bizarre incident involving the Duke took place shortly after he had been made chairman of the Watford Football Club.
“We were making polite conversation, when he suddenly changed the subject,” Elton wrote, referring to how the Duke then questioned him about living near Windsor Castle.
When Philip apparently asked Elton whether he had seen the “bloody idiot” who drove around in a “ghastly car”, the singer quipped that he believed the Duke was referring to him.
Elton said Philip was surprised to learn it was him driving around in his yellow Aston Martin that the Duke didn’t hold back from letting him know how he really felt.
“What the hell are you thinking? Ridiculous. Makes you look like a bloody fool. Get rid of it,” the Duke apparently snapped at Elton.
The singer said he was particularly shocked to hear Philip's comments because he’d had the car professionally painted to reflect his lifelong support of Watford.
“I didn’t realise how much attention it had attracted until I was introduced to Prince Philip,” Elton wrote.
The singer's claim comes after it was revealed the Queen is currently struggling to handle Prince Philip's continuous health concerns.
The Duke retired from a life of royal duties in 2017 when he was 96, and since then his health has been a serious cause for concern.