Earlier in the day, Gandhi skipped the first session as the 17th Lok Sabha began for the first time after the general election.
He, however, tweeted after his absence was noticed in Parliament.
After the 542 MPs met for the first time, President Ram Nath Kovind started the process of administering of oath. The President administered the oath to Pro-tem Speaker Virendra Kumar, who in turn administered oaths to the other lawmakers.
Soon Prime Minister Narendra Modi took his oath as BJP MPs raised slogans of "Modi Modi".
BJP MPs also gave a loud welcome to Smriti Irani who defeated Gandhi from his family pocket borough of Amethi.
Minister of State for Social Justice and Empowerment and Rajya Sabha MP from Maharashtra Ramdas Athawale had questioned where Gandhi was. To his question, Congress members from the Opposition benches said that he was here and would come soon.
Gandhi arrived in Parliament around 2.50 p.m. and took his oath in English. However, after taking the oath he started walking back to his seat, when the officials and several lawmakers reminded him of the necessary signature in the Parliament register.
He was also reminded by Union Defence Minister Rajnath Singh to sign the register.
After signing the register he walked back to applause from his party members including his mother and Congress Parliamentary Party (CPP) leader Sonia Gandhi.
Earlier in the day, the Congress chief tweeted: "My fourth consecutive term as a Member of the Lok Sabha begins today. Representing Wayanad, Kerala, I begin my new innings in Parliament by taking my oath this afternoon, affirming that I will bear true faith and allegiance to the Constitution of India."
Gandhi has been elected to Parliament from Kerala's Wayanad. He had also contested from Amethi but was defeated by Irani by a margin of over 55,000 votes.
( With inputs from IANS )