A battle in which at least five soldiers have died has been sparked by clashes between a Jihadist group and Libyan special forces, says the army.