Bangsamoro Islamic Freedom Fighters (BIFF) occupied a primary school on Wednesday and were still holding less than 10 civilian hostages at around 7 p.m., military officials said.
Hundreds of gunmen initially attacked a lightly guarded military outpost at dawn, with about 30 then taking over the school and using civilians as human shields, the military said.
"As of now, they are in the school holding the civilians. They are using them as human shields," Captain Arvin Encinas, spokesman for the Army division with responsibility for the area, told AFP by phone.
Encinas said the gunmen had planted improvised bombs around the school, a small building in a rural area, and that soldiers had surrounded it.
There were about 20 hostages taken from houses nearby, but no students, Antonio Maganto, the region's education spokesman, told AFP, although he said the exact numbers could not be confirmed.
The unrest occurred in Pigkawayan, North Cotabato, a farming town about 160 kilometres (100 miles) from Marawi city where fighters linked to the Islamic State (IS) group have been battling troops for a month in a conflict that has claimed hundreds of lives.
Armed Forces spokesman Restituto Padilla said the attackers at Pikgawayan belonged to the BIFF, one of four groups in the southern Philippine region of Mindanao that have pledged allegiance to IS.
Local police said the BIFF attack may have been intended to help the militants in Marawi by distracting the military.
Padilla said the gunmen attacked the outpost at daybreak, then exchanged fire with troops in the morning before retreating in a hit-and-run assault typical of BIFF fighters who are based in Muslim communities in the area.
"It's already resolved. The enemy has withdrawn... they failed," Padilla said late on Wednesday morning.
However, about six hours later, Encinas reported the hostage crisis at the school.
Padilla then confirmed militants had taken over the school, and said the military would seek to broker a solution.
"I hope this will be resolved peacefully and I hope there will be negotiations for the sake of the people they are holding hostage," he said.
Encinas and Padilla said there had also been skirmishes throughout the day outside of Pigkawayan, which is surrounded by marshlands, mountains and farmlands.
Those areas are largely lawless areas with mixed Muslim-Christian communities where the BIFF, other Muslim rebels and political warlords hold sway.
Pigkawayan mayor Eliseo Garsesa said about 200 gunmen were involved in the initial assault.
Muslim rebels have been fighting for more than four decades for an independent or autonomous region in the south of the mainly Catholic nation, with the conflict claiming more than 120,000 lives.
The major rebel organisations have signed, or are pursuing, peace deals with the government, but small hardline groups such as the BIFF have vowed to continue fighting.
Philippine President Rodrigo Duterte declared martial law across Mindanao, home to 20 million people, on May 23 immediately after fighters flying the IS flag rampaged through Marawi.
Their assault on Marawi ignited an unprecedented urban war that has claimed hundreds of lives and which Duterte has warned is part of an IS campaign to establish a base in Mindanao.
The fighting has left Marawi, the most important Muslim city in the Philippines, largely in ruins.
The militants involved in the Marawi fighting are mostly from the Maute and Abu Sayyaf organisations, which have united with the BIFF under the IS umbrella, according to the government.
The military has said foreign fighters, including those from Chechnya, Indonesia and Malaysia, have also joined the Marawi conflict.
IS has ambitions of setting up a caliphate in Southeast Asia -- home to largely Muslim nations like Indonesia and Malaysia -- as the group loses territory in Iraq and Syria.
The BIFF was blamed for attacking Christian communities in Mindanao in 2008, triggering a conflict that claimed about 400 lives and forced 600,000 people to flee their homes.