If someone is having to shift lanes behind you to pass you on the right, only to get right back into the left lane just cuz you refuse to get over for spenders, that means there are 2 lane shifts for every person you fail to get over for. That makes for a ton of unnecessary risk just so you can be in the left lane for no good reason (because like it or not, the left lane IS the passing lane).
This really isn't hard.
1. Go the speed limit in the lane adjacent to the left (passing) lane.
2. When necessary, use the left lane to pass, then get back into the non-passing lane. Some drivers will use the left lane to excessively speed. This does not effect you or anyone else if you are not in the left lane when you have no reason to be.
Choosing to go against these common sense rules for driving may not be illegal, strictly speaking. But it is stupid 100% of the time.