The royal expert also said the Queen began to reduce how often she would meet with Diana, as a result of the draining meetings.
Diana had always interpreted the Queen's distance as jealousy, due to the Princess of Wales' popularity amongst the people.
WATCH: Princess Diana discusses Camilla and Charles' affair in an emotional interview
Diana's beliefs were not completely unfounded.
According to author Penny Junor, the Queen did show brief moments of competitiveness towards Lady Diana.
In her 2005 book, The Firm, Junor wrote: “When the mania over Diana was at its height, one of the Queen’s courtiers said, ‘Ma’am, you will never have seen anything like the publicity Charles and Diana are having’.
“‘You were not around,’ she said witheringly, ‘when Margaret and I were having our future husbands talent-spotted for us’.”
Whether or not the Queen's distance was because of her jealousy of Diana or discomfort with her conversation, royal experts all agree that their relationship weakened significantly over time.