The source adds, "It is not a good look for the show which somehow gets huge interviews, bigger names than the weekday shows."
Lisa Wilkinson, who joined the show earlier this year after sensationally quitting Channel Nine's Today show, recently addressed the ratings drop.
"I cannot worry about that," she told the Daily Telegraph.
"I worry about putting the best possible show together every Sunday night.
"There are very few programs on Australian TV who can say that their numbers are up."
The Sydney Morning Herald previously reported that Lisa's first appearance on The Sunday Project "had a 7PM national average of 800,000."
However, the huge number was cited as a "curiosity bump" as people tuned in to see how Lisa was setting into her new channel.