Emergency services were called to a private wharf in Elizabeth Bay on Monday evening and found a woman, believed to be aged in her 20s, with "a serious injury" to her right leg, the police in New South Wales state said in a statement.
Police said they had been told the woman was swimming off the wharf when the shark attacked her. The incident happened about 20 metres (60 feet) off from the jetty, media reported, citing residents.
While shark sightings along Sydney's ocean-facing beaches are common, attacks in its iconic harbour are rare. Monday's incident took place near the Opera House and Harbour Bridge.
"Shark bites are really rare ... the last incident that occurred in Sydney Harbour was in 2009," Amy Smoothey, senior shark scientist at the New South Wales department of primary industries, told the Australian Broadcasting Corp.
Based on the bite patterns and images provided, the woman was likely attacked by a bull shark, Smoothey said.
