Good advice from others so far. Can I suggest that you ruthlessly focus on 'margin' not on turnover. That will mean that you will end up doing a number of niche things rather than one big thing. Some of the things you do might be seasonal.
Always make sure your service/product is better than average overall. (Becoming sucessful usually comes about after years of being better than average and gradually climbing up into the upper quartile and then into the top ten per cent.)
For example you can do high spec poultry all year with a decent margin but if you target only the Christmas market with direct selling your margin will be huge. Just to labour this point a little. If you provide high spec poultry all year you cannot then simply beef up your prices for Christmas as your customers will feel ripped off. The point is therefore, that you will need to focus your efforts not only on the product but on the market you are aiming at.
Another market, as has been mentioned is the livery one but here again, do not sell on price. You do not want customers who think of you as being 'cheap.' That kind of customer will always go to the next cheapest.
Specialist veg is a good bet but you must have outlets ready. It is never any good producing something and then wasting hours and hours looking for someone to buy it. In the old industries such as agriculture it doesn't work that way.
As to is it worth it? The satisfaction from being, to a large extent, master of your own destiny is worth it.
I bet as a proportion, there are more employed people wishing they had their own business, than there are self-employed wishing they were employed by someone else.