Blocking the right of way is actually illegal. Driving down the street is not. It’s actually what their intended use.
If it was a private street you’d be in the wrong, but if it’s a city owned street then they can pound sand.
Let the Karens fume, but as long as you’re keeping the speed limit and it’s a public street you are absolutely not a jerk or in any way morally, legally, or ethically in the wrong.