Uttar Pradesh Director General of Police O.P. Singh said that Ansari was arrested while he was coming out of a mosque in Kanpur.
A missing person complaint had been filed by Ansari's family on Thursday in Mumbai.
The DGP said: "He had been out on parole. His arrest is a major achievement for the state police."
The 68-year-old convict of the 1993 Mumbai serial blasts case had been serving a life term. He was on parole for 21 days from the Ajmer Central Prison, Rajasthan, and was expected to surrender before prison authorities on Friday.
During the parole period, he was ordered to visit the Agripada Police Station everyday between 10.30 a.m. and 12 p.m. to mark his attendance.
However, Jalees Ansari did not visit the police station on Thursday during the designated time and his 35-year-old son Jaid Ansari approached the police station with a complaint about his "missing" father.
According to the complaint, Jalees Ansari woke up in the early hours and told family members he was going to offer namaaz, but did not return home.
On the son's complaint, the Agripada Police registered a complaint.
The Crime Branch of the Mumbai Police and the Maharashtra ATS had launched a massive manhunt to trace him. The Uttar Pradesh Police received a tip-off about his presence in Kanpur.
Jalees Ansari, known as Doctor Bomb in the underworld, was allegedly connected with terror outfits like SIMI and Indian Mujahideen and taught terror groups how to make bombs.
He was also questioned by the NIA in 2011 in connection with the 2008 bomb blast in Mumbai, the DGP said.
On Thursday, his family members complained to the Agripada Police Station that he had gone to offer 'namaz' at 5 a.m. and then went 'missing', his phone switched off and no trace anywhere.
The Mumbai Police alerted the Maharashtra ATS and others on Thursday and a massive manhunt was launched to trace and arrest Ansari.
A medico by profession, Ansari, who served with the BrihanMumbai Municipal Corporation at one point of time, had undergone terror and explosives training in Pakistan in the early 1990s.
He was finally convicted for the Jaipur serial blasts, Ajmer blast and Malegaon blast case and awarded a life sentence in jail.
Besides, Ansari was reportedly involved in several other blasts in Pune, Mumbai, Hyderabad and other places ostensibly as vendetta for the felling of the Babri Mosque in Ayodhya in 1992.
Influenced by a terror mastermind Abdul Karim Tunda, Ansari had developed expertise in timer devices and TNT bombs, besides creating terror modules for IM, HuJI, SIMI, indoctrinating more to their cause, etc.
( With inputs from IANS )