Never experienced it and don't know what the 'proper' way would be. I'd discuss this with my Master and settle on a course of action. In my case I guess it would be him dealing with the situation.
I'd say if a polite, but firm explanation of how things are isn't sufficient for them to behave appropriately, then you might want to look for different people to socialize with.