I think Sir is overused as a "name" and is appropriate under any circumstances as a form of address for any male you have respect for or is in a superior position to you, such as a boss or respected elder (and elder being of superior/greater age.)

Master, Mistress, Lord, et.al. are excellent for those who directly hold power over you and should not be taken or given lightly.

So yes, there is a difference in my opinion. That said, Sir is a very acceptable form of address among non-lifestylers so it is an excellent substitute for the more formal Master when in public...