However, as they grew closer in age, the Spencer sisters became close.
Sarah, the eldest of the four Spencer children, was described as being “hero-worshipped” by Diana in Sarah Bradford's biography of the late royal, and was a close confidante.
In fact, Sarah dated Prince Charles before Diana for a brief moment in 1977 when she was 22-years-old and the Prince of Wales, six years her senior.
However, she reportedly hurt Charles' feelings when she told the press: "I can assure you that if there was to be any engagement between Prince Charles and myself, then it would have happened by now. I wouldn’t marry anyone I don't love, whether it was the dustman or the King of England. If he asked me I would turn him down. He doesn’t want to marry anyway. He’s not ready for marriage yet.”
Needless to say, there were no hard feelings when Charles and Diana's engagement was announced.
“I introduced them, I’m cupid,” she told The Guardian at the time.
In the lead-up to her wedding, Diana consulted her sisters for advice when rumours about Charles and Camilla Parker-Bowles emerged.
The bride-to-be had lunch with Sarah and Jane at Buckingham Palace and Andrew Morton wrote in the 1992 biography Diana - Her True Story that Diana was "confused, upset and bewildered by the train of events."
"At that moment, as she seriously considered calling off the wedding, they made light of her fears and premonitions of the disaster that lay ahead.
"'Bad luck Duch,' they said, using their nickname for their younger sister.
"'Your face is on the tea-towels so you're too late to chicken out now.'"
After Diana married Charles, the Spencer sisters still saw each other regularly.
Jane's husband Robert Fellowes was then the assistant private secretary to Queen Elizabeth and because of his role, the Fellowes family had a home at Kensington Palace.
However when Charles and Diana divorced, Jane's association with the Queen became especially difficult and when Robert's line of work ultimately came first, it caused the sisters to reportedly cut off communication for 18 months.
Sarah, meanwhile, would sometimes travel with Diana, serving as one of her ladies-in-waiting and though they didn't live near each other, they grew even closer towards the end of Diana's life.
At one point, the Princess of Wales called Sarah “the only person I know I can trust.”
When Diana tragically died in Paris, both Sarah and Jane went with Prince Charles to collect her body.
According to Earl Spencer and Lady Sarah in the BBC documentary Diana: 7 Days, it was Jane who broke the news to them about their sister's passing. Jane didn't take part in the documentary.
Sarah recalls feeling nothing but shock in the hours and days between Diana’s death and funeral, and branded the press “unacceptable” when they tried to obtain quotes or interviews from her so soon after her sister’s death.
Though Charles' eulogy is arguably the most famous one read at the funeral, Jane also did a reading of a poem by Henry Van Dyke Jr.
WATCH BELOW: Princess Diana's sister Lady Jane Fellowes gives a reading at the 2018 Royal Wedding. Post continues after video...
Since her death, both Sarah and Jane have maintained a relationship with Princes William and Harry.
The Spencer family attended Prince William's royal wedding to Catherine Middleton in 2011 as well as Prince Harry's to Meghan Markle in 2018. At the latter, Jane even gave a reading.
Jane's daughter, Laura Fellowes, is one of Princess Charlotte's godparents and both of Diana's sisters appear in Archie's official christening portraits.
Though Diana may be gone, the Spencer family will always remember her.