The problem comes from Democrats hanging the "racist" label on someone you agree is unlikely to be racist. This means the next time an actual racist comes along and gets called a racist, people remember all the people the Democrats have called racists without cause and aren't sure whether this person is just another guy who disagrees with their politics.

On the other side of this equation, arguing that they are being falsely accused of racism doesn't have the same problem. I seriously doubt you will find many Republicans claiming there is absolutely no racism in existence in the USA.

"To take from one, because it is thought his own industry and that of his fathers has acquired too much, in order to spare to others, who, or whose fathers, have not exercised equal industry and skill, is to violate arbitrarily the first principle of association, the guarantee to everyone the free exercise of his industry and the fruits acquired by it." --Thomas Jefferson