Hard to say whether it is a stalking situation or the guy is just totally inept socially, but I completely understand why the situation is creeping you out. Personally, I too do not particularly enjoy being rude to people but sometimes it is called for and this seems to be one of those times. For whatever reason your instincts seem to be telling you there is something not right about this situation and instincts I think should always be given attention. He obviously has made you feel quite uncomfortable and no one has the right to do that.

Sorry you are having to deal with this very unpleasant situation and I don't think you are being silly or overly sensitive at all. Good luck and hopefully he can just take no for an answer and go on his way. But if not, you should seek help from the local authorities.

If he accosts you again, I'd recommend telling him clearly that you aren't interested in having a drink with him now or ever. If it makes it easier, just say you are seeing someone or whatever. If he continues to persist after that, then it would probably be wise to contact the police.