NSW Police said a member of the public discovered the woman's body in the harbour near Garden Island, just east of the Opera House, shortly after 9am on Monday.
A police spoksperson said: 'She was retrieved by a member of the public and transferred to a nearby NSW water police vessel.'
They were then joined by paramedics at the Man o' War steps near the Opera House but the woman was unable to be revived.
Tim Stephens, a witness to the event, said: 'The Manly Ferry just stopped near Fort Denison. We heard some shouting and then I went down the back. I heard a guy on the boat saying that a woman might have jumped or fell into the harbour.
'An announcement on the boat said there was a man overboard. We’ve just been sitting here.'
The water police rushed her to an ambulance but she was unable to be revived.
A NSW Ambulance spokesperson said she the woman, believed to be ages in her 30s, was 'unconscious and not breathing' upon arrival.
The woman is believed to have reportedly fallen overboard from a Sydney ferry nearby.
Her death is not being treated as suspicious.
This is a developing story.