Scott,
You're understanding of "Right of Way" is exactly point on. It's title is misleading. Rather it is a way for those navigating to reach an agreement of sorts on how to manuever when heading towards each other... to avoid confusion. Starboard holds position. Port gets out of the way. If port is a motorboat it can get out of the way in either direction. However, it is generally accepted that with sailing vessels that the only reasonable course of action is to give way to the leeward side. And it's just the courteous thing to do, with the understanding that starboard tack has "Right of Way" and you don't get in the way of their wind. An acceptance to the general rule would be for port to tack upwind given enough time and distance to make such a move reasonable.
I do however seem to recall coming across something online that says something else. That starboard tack should signal with plenty of advance notice as to whether the intention is to pass downwind, by flying the kite low, or upwind, by flying the kite high. I know it states this on the instructional video "The complete guide to Kiteboarding"... though I know this is not by any means an authoritative stance.
It'd be nice to have a followed rule... mainly starboard holds bearing and port gets the hell out of the way (after all this is already the "rule"), but I've spent to long on the water to know this is unlikely.
Probably the best course of action is simply what you stated.. to stay clear downwind of everyone else... or way upwind. Just out of the crowds in general
-Kevin