He noted that even when the UPA was in power, security cover of some politic was downgraded.
"It has always been a special committee under Home Ministry which decides (this) and if there is any issue about that one can always go to court and challenge it," Swamy said in the Rajya Sabha.
The Rajya Sabha MP said that the threat to the family arose mainly from the assassination of Rajiv Gandhi by the Liberation Tigers of Tamil Eelam (LTTE) and it has subsequently disappeared.
He cited two reasons for this - there is no LTTE and the attitude of protectees (Sonia Gandhi and other family members) towards those convicted by the Supreme Court.
"Sonia Gandhi herself wrote to the President that they (convicted and sentenced to be hanged) should not be hanged," Swamy said.
At this point, he was intervened by the House Chairman M Venkaiah Naidu with the latter advising him to not bring unrelated issue.
In the beginning of the House proceedings, the Chairman had allowed Congress leader Anand Sharma to raise the issue in response to the latter's notice under rule 267.
Protesting government's move to withdraw SPG security cover of Gandhi family and former Prime Minister Manmohan Singh, the party leader Anand Sharma demanded in Rajya Sabha that the highest security cover be restored.
Sharma said that restoration of the SPG cover to the four leaders was in national interest.
Sharma noted that there was a threat to the personal safety, security and life of the four leaders of the party and hence government needs to rise above "partisan politics" and restore the SPG cover.
( With inputs from IANS )