Yes, distributed clusters are limited by communicate costs, not computation power. Parallel computing is in general limited by communication costs, even on one node (as the amount of time it takes to service a cache miss). Minimizing communication is important in both cases.

DNN training is one problem where the GPU solution vastly outperforms the distributed HPC solution.