Why not data_parallel?

[LMescheder]

I wonder why you implemented the multi GPU training using a custom Event Loop instead of using torch.nn.DataParallel. I suppose it is for performance reasons?
If so, what is the main bottleneck in data_parallel that prevents you from using it? Do you have an estimate of how much the speed up compared to the (simpler) DataParallel solution is?

[myleott]

Yes, it's for performance reasons.

DataParallel relies on Python threading, which is slow due to the GIL [1][2]. When we tried nn.DataParallel initially, we saw negative speedup with multiple GPUs (e.g., one GPU training was faster than using four GPUs).

The custom event loop in fairseq-py uses multiprocessing (i.e., one Process per GPU), which gets around the GIL and gives much better multi-GPU performance. We typically see ~5.5-6x speedup with 8 GPUs.

[1] OpenNMT/OpenNMT-py#89 (comment)
[2] pytorch/pytorch#54

[LMescheder]

Okay, I see. Thanks for the prompt reply. Have you tried DistributedDataParallel which shouldn't have issues with the GIL? Amazing work by the way!

[myleott]

I haven't tried DistributedDataParallel yet, but it looks promising. I'll look into it when I get some time. This discussion also seems relevant (albeit a little discouraging):

• Single Node DistributedDataParallel Not Supported pytorch/pytorch#2363
• Allow more gpu/process configurations in distributed imagnet example pytorch/examples#202

[jekbradbury]

It's definitely worked for our use cases, including speech and MT. I think it's ultimately very similar to the implementation you built into fairseq, except that the user must explicitly launch N copies of the script, and each copy should have its own data loader or data loader shard.