He also stated that the decision of the Supreme Court, which is hearing the Ram Janambhoomi-Babri Masjid title suit, will be in favour of the Ram temple.
Swamy, who was in Ayodhya on a two-day visit, further said that the right to worship is a fundamental right and cannot be taken away by anyone.
"The temple which is at the spot of Lord Ram's birth cannot be removed," he said.
The remark by the BJP leader comes just days after NDA constituent Shiv Sena chief Uddhav Thackeray called for action by Prime Minister Narendra Modi-led central government to enable speedy construction of the grand Ram temple in Ayodhya.
The Shiv Sena chief had said that with abrogation of Article 370, which granted special status to Jammu and Kashmir, the Modi government had shown that it was decisive in nature.
He added that after the Article 370 move, it was now time for construction of Ram Mandir and introduction of uniform civil code.
"We had said before the elections that the Kashmir issue would be resolved. The opposition was saying we would not let Article 370 go. I am proud of you Modiji," Thackeray had said even as he advocated the cause of Ram temple.
( With inputs from IANS )