Carter said that Trump would not be in the White House if not for Russian interference in the 2016 presidential election.
"The President himself should condemn it, admit that it happened, which I think 16 intelligence agencies have already agreed to say," Carter said at a panel on human rights hosted by the Carter Centre in Leesburg, Virginia, on Friday.
"There's no doubt that the Russ did interfere in the election, and I think the interference, although not yet quantified, if fully investigated, would show that Trump didn't actually win the election in 2016.
"He lost the election and he was put into office because the Russ interfered on his behalf," Carter said.
Asked if he believes Trump is an "illegitimate President", Carter paused for a moment. "Based on what I just said, which I can't retract... I would say yes," he responded to laughs from the audience.
Responding to Carter's comments, Trump blasted him as "the forgotten President" who's not only "trashed" by his own party but also remembered only as being "terrible" for the country.
Trump made the remarks at a news conference after wrapping up the G-20 summit in Japan, saying that although Carter is "a nice man, he was a terrible President", before noting that "he's a Democrat and it's a typical talking point".
He added that Carter is "loyal to the Democrats" but "as everybody now understands, I won not because of Russia, not because of anybody but myself".
"I went out and campaigned better, smarter, harder than Hillary Clinton... I won Michigan, I won Wisconsin, I won Pennsylvania, I won states that traditionally haven't been won by Republicans for many years," Trump said.
"This had nothing to do with anybody but the fact that I worked harder and much smarter than Hillary Clinton did."
Trump said he was "surprised" Carter made such comments, but he also "felt bad" for him as he's been attacked even by his own party.
"He was not a good President. Look at what happened with Iran - that was a disaster. What Iran did to him, they tied him in knots...," he added.
Trump said that he did raise the meddling issue with Russian President Vladimir Putin during a private meeting on Friday. But publicly, Trump made light of Moscow's election meddling, jokingly telling Putin: "Don't meddle in the election, please."
US special counsel Robert Mueller's report found that Russia waged a "sweeping and systematic" influence campaign during the 2016 election with the goal of electing Trump, but did not establish a conspiracy between Trump's campaign and Moscow.
( With inputs from IANS )