Harry and Meghan are due to move to Frogmore Cottage in the grounds of Windsor Castle this year and are expecting their first child in spring.
The couple have been living at Kensington Palace in London, near William and Kate and their three children.
The paper quoted a source saying: “William and Harry’s double act has naturally been supplanted by the two couples and their families.”
Kensington Palace declined to comment.