If you don't care about VMware nitty-gritty, don't read further. Seriously, I warned you.

I'm stuck on a call with a customer where they're getting some slow data transfer between systems. We confirmed that VM traffic bandwidth drops in half when they go from 2 VMs on the same host to 2 VMs on separate hosts. I needed to find a way to make sure my ESXi hosts weren't the problem. Network team is looking into NSX on their own.


Lo and behold! ESXi has iperf installed on it. I followed this dude's steps: https://vninjadfw.github.io/esxibandwidthtesting/ but William Lam has more detailed (and updated) instructions here: https://williamlam.com/2016/03/quick-tip-iperf-now-available-on-esxi.html

Got it set up, ran it, and discovered it wasn't my ESX servers causing the problem. An iperf to the mgmt IP on the second box ran at ~24.5Gbits/sec. No retries in 10 seconds. Next step is analyzing a tcpdump from when the customer tried a transfer. Fun times.