Minister AKM Mozammel Huq accused some NGOs of having "ill motives" and of profligate spending on their accommodations in Cox's Bazar.
ARSA urged its followers on Wednesday to refrain from crime in Rohingya refugee camps in Bangladesh, following reports of killings and abductions attributed to the group.
Humanitarian groups have criticized the plan to relocate refugees to Bhasan Char, as the island is vulnerable to frequent cyclones, with some saying it may even be inhabitable.
Monday’s fire at the No Po refugee camp, where some 10,000 refugees from Myanmar live, destroyed at least 50 homes.
"Not a single Rohingya has volunteered to return to Rakhine due to the absence of conducive environment there," Bangladesh Foreign Secretary Shahidul Haque said.
At least 150 Rohingya believed to have been preparing for the journey, mostly women and children, have been caught since November, according to local authorities.
With no sign of the crisis ending and the looming monsoon and cyclone season starting from May, aid workers at Cox’s Bazar are rushing to secure the camps for the long haul.
The 31 Rohingya, fleeing a crackdown in India, were stranded on the border after being denied entry into Bangladesh and border officials failed to agree on what to do with them.
The camp leaders called a three-day strike on Monday to demand that they be identified as ethnic Rohingya on the ID cards the UN is issuing them.