James also said he felt Meghan was a 'hypocrite' for her lavish baby shower last month.
"The things we have seen about Meghan Markle where she took that private jet to fly to a baby shower in New York at great expense while she is preaching about climate change and so on.
"I think a lot of people are going to say 'you know, I think this woman is a bit of a hypocrite here' and she is probably - on the balance of probabilities - and not all that easy to work for."
Amy Pickering, the Duchess' latest assistant is the third employee to step down from her royal duties.
James described Meghan as a "pretty difficult character to deal with".