J.S. Mill is famous for setting the limits of individual liberty through his harm principle: "the only purpose for which power may be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others. His own good, either physical or moral, is not sufficient warrant".
Mill distinguished between 'other-regarding' actions, which fall within the public sphere of potential state interference, and 'self-regarding' actions, which fall within the private sphere and is no business of anyone else. The difficulty is defining where to draw the line, for it is clear that everything we do affects others in some (perhaps indirect) way. The decisive factor for Mill is not whether we merely affect others (for we always do), but rather, whether our actions affect the objective interests of anyone else. This appeal to objectivity mirrors Bentham's characterization of pleasure and pain as "real entities", rather than the purely subjective judgments of people (this move being necessary to exclude people's "offense" at others' opinions to count in the utilitarian calculus). For Mill, an objective interest was long term, observable to outsiders, and concerned with the relationship between means and ends. To thwart a fickle or arbitrary desire, then, would not count as a genuine 'harm'.
Still, it seems possible that one might have a long-term 'interest' which nevertheless does not justify impinging upon the liberty of others. A businessman, for example, would be right to consider the new competition down the road to be against his interests. Nevertheless, it is not a harm which he has any right to prevent. Mill suggested that a genuine harm involves the violation of "certain interests which... ought to be considered as rights". It must be emphasised that this is not an appeal to anything as absurd as natural rights, but rather, is simply an extension of utilitarianism.
Mill was a Rule Utilitarian. Rather than judging the morality of acts directly against the utility principle (the greatest happiness for the greatest number), he opted for a more subtle, indirect approach. Mill thought acts should be assessed against a set of secondary principles (or rules) which are in turn derived from the ultimate principle of utility. He recognised that if everyone tried to maximise happiness, with no concern for human rights or justice, the inevitable result would be much unhappiness. An indirect approach is much better: identify those general rules which would (if universally followed) tend to maximise utility, and get people to follow those rules instead. This, then, is what he meant by 'rights' - those "certain interests" whose protection would help maximise the general happiness. (Clearly, immunity from competition would not thus qualify!)
This raises the question of whether granting so much liberty would actually serve to maximise utility. Mill focused on two aspects in particular, freedom of speech, and freedom of action.
Freedom of Speech:
He advocated nearly limitless freedom of speech. Censorship, he argued, assumes infallibility - which history demonstrates is often mistaken. Furthermore, even if we suppose that the view of your opponents is false, Mill argued that society is still better off not to suppress it. This is because such challenges will help us to think critically and re-affirm/strengthen the foundations of our own ideas, avoiding their degradation into "a dead dogma". Mill pointed out that "He who knows only his own side of the case, knows little of that".
This argument is based on a sort of conceptual Darwinism: the conviction that in a 'free market of ideas', the best will come to the fore and survive. Some argue for censorship not because an idea is false, but because they say it will be harmful to society (atheism is a common target here). However, as Mill points out, this too assumes infallibility - the question of whether any specific view is helpful or harmful to society, is itself a proposition which might be true or false, and so, as argued above, should be open to debate: "the assumption of infallibility is merely shifted... The usefulness of an opinion is itself a matter of opinion".
There are some limits to free speech however. Mill was sensitive to the modern idea of a 'speech act' (that speech is action, and should be treated as such). If a man gives an inflammatory speech rousing a mob to violence, then he has clearly violated the harm principle. But as always, it is far from clear where to draw the line, and Mill tends to err on the side of liberty.
Freedom of Action:
It might seem that the harm principle is a flagrant violation of utilitarianism, for we can easily think of cases when the general happiness could be best served by paternalistically imposing our will on others 'for their own good' (e.g. limiting the availability of harmful drugs). So Mill's insistence that "his own good... is not sufficient warrant" (for intervention) will require some justification.
The crucial matter here is the way in which Mill's utilitarianism varies from Bentham's. As already discussed, he was a Rule, not Act, utilitarian. But he also went further, and made qualitative (not merely quantitative) distinctions about utility. That is, he considered some pleasures to be intrinsically better than others - better to be a discontented man than a happy pig, and all that. In a crucial passage at the start of On Liberty, Mill tells us: "I regard utility as the ultimate appeal on all ethical questions; but it must be utility in the largest sense, grounded on the permanent interests of man as a progressive being" (emphasis added). This can help us to understand how he came to the conclusion that a rule in favour of much personal liberty would tend to maximise utility.
Freedom, Mill thought, was beneficial both to individual and to society. Society would benefit from getting to observe various different "experiments of living". The diversity which results from freedom is the only sure way to guarantee long-term progress in society, as people can observe and learn from each other, in a semi-Darwinian process of trial & error.
From an individual's perspective, freedom is necessary to develop their individuality as a human being - an essential feature, for Mill, of a good (high-utility) life. To this, James Fitzjames Stephen objected that Mill was simply mistaken: people are lazy creatures, and given a choice they will tend to choose idleness and passivity. Their personal qualities and individuality could be better furthered by forcing people into activities which they otherwise wouldn't bother with.
Mill anticipated this objection, and rebutted it by appealing to the 'best judge' view, i.e. individuals are the best judges of what is best for them. Here Mill appeals to Bentham's notion of individuals having privileged information about what makes them happy. He also goes further, pointing out that individuals are the most motivated to look out for their own interests. So although there might be rare occasions when paternalism would indeed have the best results, in the vast majority of cases it would prove mistaken. As a general rule, we ought to respect individual liberty, for that is what will tend, overall, to maximise utility.
Being based on the principle of utility, Mill recognises that freedom should not be granted to those who are incapable of benefiting from it. So paternalism is well justified (indeed, morally required!) in the case of children and "barbarians". However, we should aim to impart to them the skills and knowledge necessary to become free agents in their own right.
Mill aimed to justify a limited sphere of personal liberty of action, and an almost limitless freedom of expression. His utilitarian justification focused on the benefits of individual spontaneity, both to the individual and society at large. It is to be valued for expressing the highest in human life itself (the qualitatively 'best' pleasures), and for promoting the development of civilization.