Modi made the remarks in a telephonic conversation after Khan telephoned him to congratulate the Indian leader on his victory in the parliamentary elections.
Thanking Khan for his greetings, Modi recalled his initiatives in line with his government's "neighbourhood first policy" and referred to his earlier suggestion to Khan to battle poverty jointly, an External Affairs Ministry statement said.
"He stressed that creating trust and an environment free of violence and terrorism were essential to foster cooperation for peace, progress and prosperity in the region," said the statement.
Pakistan Foreign Ministry spokesperson Mohammad Faisal also tweeted that Khan spoke to Modi and conveyed that he looked forward to working with him for the betterment of the people of the two countries.
"PM spoke to PM Modi and congratulated him on his party's electoral victory in Lok Sabha elections. PM expressed his desire for both countries to work together for betterment of their peoples.
"Reiterating his vision for peace, progress and prosperity in South Asia, the Prime Minister said he looked forward to working with Prime Minister Modi to advance these objectives."
Tensions between India and Pakistan escalated after the Pulwama terror attack in February in which 40 CRPF personnel were killed. Pakistan-based terror outfit Jaish-e-Mohammed took responsibility for the attack.
India then bombed a terror camp in Pakistan which was followed by aerial engagement between fighters of the two countries.
Khan had earlier tweeted to congratulate Modi on the electoral victory of the BJP and its allies. Responding, Modi had written that he had always given primacy to peace and development in the region.
Khan had said in an interview in April that he believes there may be a better chance of peace talks with India and some settlement could be reached on the Kashmir issue if the BJP returned to power in the general elections.
( With inputs from IANS )