Kanye, a guest on Letterman's "My Next Guest Needs No Introduction", said that liberals bully Trump supporters, reports deadline.com.
"This is like my thing with Trump - we don't have to feel the same way, but we have the right to feel what we feel," Kanye said.
His decision to wear a "Make American Great Again (MAGA)" hat is a way to show people they shouldn't be afraid to express opinions, Kanye said, adding that it is "not about politics".
Letterman challenged Kanye on allegations of voter suppression during mid-term elections.
"So if I see a person that I admire talking about Donald Trump can think whatever he does. I wonder if those thoughts, indirectly, aren't hurting people who are already being hurt," Letterman said.
Kanye countered by asking whether Letterman had ever been bullied in high school for wearing the wrong hat.
"Liberals bully people who are Trump supporters!" Kanye said.
He had visited the Oval Office and famously wore a MAGA hat on the stage at "Saturday Night Live" during a guest appearance.
While he said earlier this year that he would step away from politics, he did tweet his support in January.
He tweeted: "Trump all day. One of my favourite of many things about what the Trump hat represents to me is that people can't tell me what to do because I'm black."
( With inputs from IANS )