The boundaries of free speech?

I have been pondering the question of whether there should be limits or boundaries to free speech. The thing that bothers me about an absolute right to free speech is the possibility of using free speech to abuse the rights and freedoms of a minority. Free speech can only exist in a free and open society, and I believe a free and open society has to be a society that believes in pluralism.

A pluralistic society agrees that everyone has the right to be different, and to express different opinions, and to do so without fear of reprisal against their person, their life or their liberty. Speech which tends to foment prejudice or incite violence against others is not free speech, but hate speech.

In a pluralistic society, people can insult and offend me as much as they like, and I can insult and offend them in return, or I can ignore them. But individuals and groups that tend to a non-pluralistic outlook on the world believe that they and they alone know the absolute truth, and that they and they alone have the right to speak, and to control everyone else. Wherever they come from on the political or religious spectrum, it's totalitarianism, and that is (or should be) alien to what is known as the "western democratic" model. Some of these groups claim that "free speech" gives them the right to promulgate their "hate speech". I would disagree.