So much so that the pair would often clash.
"She's (Sarah) carriage-driving with Prince Phillip, she's out shooting with Prince Charles, she's just a bundle of energy," Andrew explains.
Adding further proof of tension between the duo, fellow biographer Katie Nicoll told the Royals at War documentary that "there were rumours of rivalry but I think actually those stem more from within the Royal family."
In fact, it is believed Diana and Sarah eventually had a huge falling out over a charity event and sadly, were unable to rekindle their friendship before Diana's untimely death in 1997.
"Diana was one of the quickest wits I knew," Sarah wrote in her book Finding Sarah in 2011.
"Nobody made me laugh like she did."
Fergie went on to share memories of their time together.
"We took vacations together with our children. Sadly at the end we hadn't spoken for a year, although I never knew the reason, except that once Diana got something in her head it stuck there for a while."