Exceptions and failures when use MultiWorkerMirroredStrategy

[ma-siddiqui]

When I use tf.distribute.experimental.MultiWorkerMirroredStrategy to run training on multiple machine I face following errors. Please advise when other necessary changes are needed.

2020-11-16 12:03:50,968 (cross_device_ops:1130) INFO: Collective batch_all_reduce for IndexedSlices: 1 all-reduces, group_size = 2
2020-11-16 12:03:56.443402: W tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:439] error: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23
2020-11-16 12:03:56.443474: W tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:1121] error: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23
2020-11-16 12:03:56.443606: E tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:1138] ScopedAllocatorOptimizer: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23

[dathudeptrai]

When I use tf.distribute.experimental.MultiWorkerMirroredStrategy to run training on multiple machine I face following errors. Please advise when other necessary changes are needed.

2020-11-16 12:03:50,968 (cross_device_ops:1130) INFO: Collective batch_all_reduce for IndexedSlices: 1 all-reduces, group_size = 2
2020-11-16 12:03:56.443402: W tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:439] error: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23
2020-11-16 12:03:56.443474: W tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:1121] error: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23
2020-11-16 12:03:56.443606: E tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:1138] ScopedAllocatorOptimizer: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23

can you try to replace AdamWeightDecay by simple Adam first ?

[ma-siddiqui]

Yes. I did it but no luck and faced the same error again.

[ma-siddiqui]

When I use tf.distribute.experimental.MultiWorkerMirroredStrategy to run training on multiple machine I face following errors. Please advise when other necessary changes are needed.
2020-11-16 12:03:50,968 (cross_device_ops:1130) INFO: Collective batch_all_reduce for IndexedSlices: 1 all-reduces, group_size = 2
2020-11-16 12:03:56.443402: W tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:439] error: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23
2020-11-16 12:03:56.443474: W tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:1121] error: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23
2020-11-16 12:03:56.443606: E tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc:1138] ScopedAllocatorOptimizer: Internal: Complete shape not known for AdamWeightDecay/allreduce/CollectiveReduce_23

can you try to replace AdamWeightDecay by simple Adam first ?

Yes. I did it but no luck and faced the same error again.

[ma-siddiqui]

Hi, any update?

[ma-siddiqui]

Hi, just to confirm, the fix added (given below) will solve my problem? Please confirm if the fix is against this bug.

Support Multi-GPU gradient Accumulate for trainer. #377

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.