In an interview with ES Magazine, Javid said he put his hand out to the "Notting Hill" star and said "lovely to meet you" but Grant refused to shake it, reports bbc.com.
"I think that is incredibly rude," said Javid.
On Twitter, Grant put across his "side of the story", saying he declined the handshake because he felt Javid was dismissive to press abuse victims.
The 59-year-old actor, has been one of the leading voices in the campaign for stricter press regulation, after being one of the victims of the phone-hacking scandal.
He said his exact words to Javid were: "If you don't mind, I won't shake your hand because you were rude and dismissive to the victims of press abuse when you met them as culture secretary."
A spokesman for the actor said he was referring to a meeting when victims of the press said they felt Javid's "attitude in the meeting was 'borderline contemptuous'".
"Hugh would like to point out that the victims in question were not celebrities. They were people with personal family tragedies who had been abused by sections of the press," the spokesman added.
In his interview, Javid said he was at an event to promote filmmaker Martin Scorsese's new film "The Irishman" earlier this month when the handshake incident happened.
"I recognised him and put my hand out and said, 'Lovely to meet you', and you know what he does? He refuses to shake my hand. He says, 'I am not shaking your hand'. I am completely shocked.
"He said, 'When you were culture secretary you didn't support my friends in (anti-media intrusion campaign) Hacked Off'.
"I think that is incredibly rude. I wonder if people like Hugh Grant think they are part of the elite and they look down on working class people no matter what station they reach in life."
Javid was culture secretary for just over a year in 2014 and 2015. He later became home secretary before being appointed chancellor in July this year.
( With inputs from IANS )