"The honourable minister is terribly wrong. That's a perverted concept by him. He needs to look within... The first lady graces occasions when invited. When there was no invite, she was not there," Dhankhar said on Saturday.
The governor said the minister should think before he speaks.
"Since I happen to be the husband of the lady, I am not reacting more. If it had been any other lady, I would have reacted far more severely. Our culture doesn't allow it. And I will appeal personally, please think before you speak out. These are dangerous areas. And I am sure he will reflect and publicly apologise," said a visibly upset Dhankhar.
He also drew the attention of chief minister Mamata Banerjee to the minister's comment.
"The chief minister is a woman. She must learn the sensitive aspect of this," said the governor.
Chatterjee, also Trinamool Congress secretary general, referring to first lady Sudesh Dhankhar, said earlier this week that her presence at government meetings was "unconstitutional". Chatterjee also pulled up the media for not raising questions as to why "another person would be present during government meetings".
The minister said he has worked with three governors - Gopal Krishna Gandhi, M.K. Narayanan and K.N. Tripathi - but "none of them acted thus".
The verbal spat is only the latest in the continuous stream of confrontations between Dhankhar and the Mamata Banerjee government ever since he took up gubernatorial responsibilities in late July last year.
( With inputs from IANS )