Police destroyed the shelters and possessions of hundreds of asylum seekers, who refuse to leave the camp, shut down on October 31, citing fears for their safety.
Rights groups say that asylum seekers have been attacked in the past, the BBC wrote.
Refugees, who barricaded themselves inside the camp, said police moved in during the early hours on Thursday, taking belongings from their rooms and ordering them to get on buses brought in to take them to accommodation centers elsewhere on the island.
“Police have started to break down the shelters, water tanks and are saying ‘move, move,’” Behrouz Boochani, an Iranian refugee, who has been acting as a spokesman for the detainees, wrote earlier on Twitter from inside the camp.
“Navy soldiers are outside the prison camp. We are on high alert right now. We are under attack,” he said, adding that two refugees needed urgent medical attention, The South China Morning Post wrote.
Australia has detained asylum seekers in camps on Manus Island and Nauru and is urging them to move to transit centers elsewhere on the island.
"They should obey the law and the lawful authorities of Papua New Guinea," Australian Prime Minister Malcolm Turnbull said.
He added that Canberra will not bend under UN pressure to accept the refugees for fear it could encourage human trafficking.
Even though Australian authorities claim that the alternative accommodation for the asylum seekers is ready, the UN's refugee agency says that it is still “under construction, is inadequately secured, and lacks medical care and “most basic services."
The asylum seekers had initially been offered to resettle in Papua New Guinea, apply for residence permits in Cambodia or a transfer to Nauru, but only a handful have agreed to do so.
Meanwhile, Canberra is resisting an offer by New Zealand to accept 150 refugees from the Manus Island center, arguing it could eventually become a "back door" to Australia.