Fix multi-argument allreduce in ProcessGroupGloo (#14688)
authorPieter Noordhuis <pietern@fb.com>
Mon, 3 Dec 2018 17:37:16 +0000 (09:37 -0800)
committerFacebook Github Bot <facebook-github-bot@users.noreply.github.com>
Mon, 3 Dec 2018 17:41:17 +0000 (09:41 -0800)
commit7da2448d6276370edde67ef22582613d819de2c3
treec76d0f0a30bcafd94819cac0d6ecbeb92409b51a
parentb15242f70cce10c0a0f8c36b690ac2481c51814f
Fix multi-argument allreduce in ProcessGroupGloo (#14688)

Summary:
If multiple arguments are specified to c10d allreduce, they are
interpreted as if they are expanding the ranks in the process group.
Therefore, not only is every argument to allreduce an input that must
be considered, it is also an output. The problem that this commit
fixes is that they were not correctly considered as outputs.

The upstream problem is tracked in facebookincubator/gloo#152. Once
this is fixed there we can remove the copies that this commit adds.

This fixes #14676.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/14688

Differential Revision: D13294405

Pulled By: pietern

fbshipit-source-id: 078a2a0a0ff12d051392461438f1496201ec3cb9
test/test_c10d.py
torch/lib/c10d/ProcessGroupGloo.cpp