His comments are in stark contrast to Pakistan's official stand, as per which, it has denied any knowledge of Osama bin Laden's whereabouts until he was shot dead in a night-time raid by American special forces on May 2, 2011 in Abbottabad.
Khan, who is on a three-day visit to the US, made the remarks in an interview to Fox News when he was asked whether Pakistan would release jailed doctor Shakeel Afridi, who played a key role in the tracking down of the Al Qaeda chief.
He said that it was the ISI that had provided the initial location through a phone connection to the Central Intelligence Agency (CIA). "... It was ISI that gave the information which led to the location of Osama bin Laden. If you ask CIA, it was ISI which gave the initial location through the phone connection," the Pakistan Prime Minister said.
Khan did not say anything more on the release of Afridi even as US President Donald Trump has called for the release of the Pakistani doctor.
The Pakistani leader said the Afridi issue is considered an "emotive" one in Pakistan as he was a US spy.
"We in Pakistan always felt that we were an ally of the US and if we had been given the information about Osama, we should have taken him out," Khan said.
The killing of Osama right under Pakistan's nose was a major national embarrassment and deteriorated ties between the two countries.
Though Pakistan has officially denied knowing that Osama was living on its territory, Asad Durrani, a former spymaster, told Al Jazeera in 2015 that the ISI probably knew where he was hiding and hoped to use him as a bargaining chip before he was killed.
( With inputs from IANS )