Let's ask the same question in english. What's the proper way to address a knight? Well, "sir" is the proper title of a knight. Sir Gawain. Sir Galahad. Sir Lancelot. Etc. And is Sir Gawain approaches Sir Galahad on the field, it would be proper for him to address him as such. "Well met, Sir Galahad. Does the battle go well?" On the other hand...what if a peasant were to speak to him? Would he blithely walk up and say "So, Sir Galahad...how are things?"
...probably not.
If a peasant were talking to another peasant, he might refer to the knight in third person as Sir Galahad. But if he were to address him personally he might be more likely to bow his head and say "My Lord?" and then politely wait for permission to speak further, without calling him "Sir" at all.
On the other hand, if the king addresses a knight, calling him by Sir is still appropriate. But, the knight can't address the king by sir. Not only is it inaccurate because "Sir" is a very specific title, it's also grossly disrespectful of his station. The king will be called "His Majesty." But...does the king of the neighboring kingdom call him that? Probably not. Doing so would be acknowledging superiority, and that would not be something he'd want to do. What about the queen? Would the queen call her husband the king "Your Majesty?" Well, she might if she were kissing up...but probably not in daily conversation. In person, to his face, "My Lord" would be plenty respectful enough. But, people other than the queen could call him "My Lord" or "Sire" and still get away with it, depending on the formality of the situation and the nature of their relationship. And...note that "My Lord" was the same form of address used earlier by the peasant to the knight. So...do you see how a simple question like "how do you refer to X" is not quite so simple?
This is all in english. The proper form of address varies by the relative levels of station of the two people involved. Japanese is
way worse. In japanese, not only do titles and addresses change...even verbs and verb conjugation changes.
For example: in english, if the knight gives his squire an apple, the squire might say "My liege, Sir Gawain gave me an apple." That would be totally respectful. And if the squire gave the knight an apple, the knight could just as easily say "My squire gave me an apple." And that would be appropriate. Or the knight could look at the squire and tell him directly "Give me the apple." And that would be fine too. And...if the squire then gave the knight the apple, and told somebody else about it after the fact, he could say "I gave my liege an apple." And that would also be correct.
In japanese...that's not the case. The verb "give/gave" would need to be different in each of these sentences. There are
seven different verbs for give, all depending all the relative status between the speakers and whether one is speaking to the person doing the giving, or speaking about it to a third person.
So...it can be complicated. In most cases we're likely to encounter, simply addressing a samurai, calling him with -san is probably sufficient. But if Joe Blow peasant walks up to Samurai Tom and says "Hey Tom san, how are things?" it's possible he might end up getting his head cut off.