A former Chief Election Commissioner also said the proposal has both "strong merits and serious demerits".
A top Election Commission source said that if the idea was feasible, then the poll panel would have held simultaneous assembly polls this year only along with the Lok Sabha polls.
The Election Commission had conducted the general elections in seven phases for the 542 Lok Sabha seats between April 11 to May 19. Results of the election were declared on May 23.
The source also contended that taking case the law and order situation during simultaneous polls can be very challenging as the number of central paramilitary forces is limited.
He also said that there are over 90 crore voters across the country and to prepare the logistics for simultaneous polls can be very tough, as of now.
The remarks of the top Election Commission sources comes after Modi held meeting with the Presidents of several political parties during the day in Parliament to discuss the idea of 'one nation, one election'.
Former Chief Election Commissioner S.Y. Quraishi said: "This idea (simultaneous polls) has both strong merits and serious demerits."
He said that the merits include saving cost and saving disruption of normal governance.
"Additionally, it will reduce communalism and casteism and corruption which are rampant during elections" he said.
On the other hand, Quraishi said that the biggest issue in this is federalism.
"The national political considerations cannot subjugate the state's interests. Besides frequent elections increase accountability of elected representatives who are notorious for not showing their faces for five years after the elections," he said.
"Creation of work opportunities for the people is an advantage. The way out is a national political consensus," the former CEC added.
( With inputs from IANS )