A report from Ofsted in October found the Muslim faith school was chaotic, dysfunctional and inadequate and it was placed in special measures.
The primary school will remain open, the Department for Education has said.
The schools minister Lord Nash has written to the school trust chairman to confirm the secondary school closure.
It will close at the end of the summer term with support promised to help the 137 pupils find new schools.
In a letter to Barry Day, the school's newly appointed chairman, Lord Nash said he was concerned about the poor quality of secondary teaching.
He said: "It was not in the interests of parents and pupils at the secondary school to continue to fund provision which has failed them."
Mr Day said in a letter to parents that he accepted the assessment and would find places for pupils in other local schools.
He confirmed that the primary school would remain open and improving it would be his main focus.
"Our vision remains to establish the Al-Madinah School as an outstanding all-through school for all members of the local community in and around Derby," he said.
"Every single member of staff wants this school to be successful but there is absolutely no doubt that many of the staff are inexperienced, some of the staff are not qualified and all of them need additional training to make this work."