The full Cabinet will take oath after Diwali.
The saffron party formed the government in alliance with the newly formed Jannayak Janta Party (JJP) led by Dushyant Chautala (31), who took oath as Deputy Chief Minister.
Governor Satyadeo Narain Arya administered the oath at a simple ceremony attended by a galaxy of opposition leaders and chief ministers.
Barring Khattar and Dushyant, no other Cabinet member took oath in the ceremony in Raj Bhawan that lasted less than 10 minutes.
After taking oath, the Chief Minister accompanying his deputy, paid obeisance at the popular Hindu shrine Mansa Devi on the outskirts of Chandigarh.
Khattar, 65, will be heading the second Bharatiya Janata Party (BJP) government of the state, which was formed on November 1, 1966.
A low-profile organisational man with an active RSS background, Khattar was chosen by the newly elected BJP legislators as their leader at a meeting here on Saturday.
"My government will be transparent," Khattar told the media ahead of taking the oath.
Five-time Punjab Chief Minister Parkash Singh Badal, whose party Shriomani Akali Dal (SAD) had an alliance with BJP's rival Indian National Lok Dal (INLD) during the assembly elections despite being an NDA partner, attended the ceremony along with son and Member of Parliament Sukhbir Badal.
Congress leader and two-time Chief Minister Bhupinder Hooda, who attended the ceremony, described the BJP-JJP alliance opportunist, unholy and against the mandate.
"Alliance has been forged in manner of 'vote kisi ki, support kisi ko'. This government is based on selfishness. The JJP disrespected people's mandate. We had less time after changes in our organisation. Had the changes been made earlier, results would've been different," Hooda told the media.
Dushyant's father Ajay Chautala, who was released on furlough from the Tihar Jail, attended the oath-taking ceremony along with wife and second-time legislator Naina Chautala.
"Dushyant asked me if he should go with the BJP or the Congress. I told him, whatever the circumstances, we shall never go with the Congress. I gave him the go-ahead to go with the BJP," Ajay Chautala said, adding "it will be beneficial for the state."
BJP working President J. P. Nadda, Himachal Pradesh Chief Minister Jai Ram Thakur and Uttarakhand Chief Minister Trivendra Singh Rawat, Punjab Governor V. P. Singh Badnore, Union ministers Krishan Pal Gurjar and R.L. Kataria, among others, were present during the swearing in ceremony.
Earlier, the BJP, which won 40 seats and was six short of a majority in the 90-member Assembly, announced an alliance with the JJP led by Dushyant Chautala, a great grandson of former Deputy Prime Minister Devi Lal.
Seven independent MLAs have also extended support to the BJP, helping it reach a tally of 57 seats.
In his congratulatory message, Prime Minister Narendra Modi tweeted, "Best wishes to them as they work to fulfil the aspirations of the people of Haryana."
Dushyant Chautala parted ways with the Indian National Lok Dal (INLD) in December 2018 after a bitter vertical split in the party and the Chautala family and formed the JJP.
( With inputs from IANS )