The office of president Ibrahim Boubacar Keita announced the resignation of prime minister Soumeylou Boubeye Maiga and all other ministers late on Thursday.
Mr Keita did not reveal specifics behind the resignations, but added a new government would be set up after consultations with “all political forces”.
The administration had faced mounting criticism over the killings of 160 members of the Fulani community by suspected ethnic Dogon hunters in March.
The attack, which took place in the central Malian village of Ogassagou, shocked a nation already gripped by an ever-worsening climate of violence.
Authorities have detained five people suspected of taking part in the massacre. However, they have not yet succeeded in disarming the militia many believe organised it, despite pledges by Mr Maiga and Mr Keita to do so.
Legislators had discussed on Wednesday a possible motion of no confidence in the government due to both the recent killings and its failure to disarm militias or tackle Islamist militants.
The largely Saharan nation was thrown into turmoil by a rebellion of Tuareg people and allied jihadists, who took over half the country in 2012.
Mali, which had been seen as a bastion of democracy and stability in the region for more than 20 years before fighting broke out, has been mired in civil war ever since.
French forces intervened in the conflict by 2013 at the request of the Malian government, attempting to break militants’ stranglehold on the centre of the country.
A ceasefire was reached in 2015, but the frequency of rebel attacks began to increase again during 2018 and much of the northern half of the country remains outside government control.
Additional reporting by agencies