Print warning when ROCm memory leaking is detected in pytorch tests (#14151)
authorYour Name <bai@in.tum.de>
Sun, 18 Nov 2018 08:09:25 +0000 (00:09 -0800)
committerFacebook Github Bot <facebook-github-bot@users.noreply.github.com>
Sun, 18 Nov 2018 08:11:44 +0000 (00:11 -0800)
commit07a8a730af83df132bbb324535421c423e2a4783
tree23979d67802b39f607e4cdc515be98f223d91d1e
parenta5891e61242a070083e26c2527565dc0d0a09efa
Print warning when ROCm memory leaking is detected in pytorch tests (#14151)

Summary:
We keep seeing random failures in CI because of ROCm memory leaking, e.g:

https://ci.pytorch.org/jenkins/job/pytorch-builds/job/py2-clang7-rocmdeb-ubuntu16.04-test/3102//console
https://ci.pytorch.org/jenkins/job/pytorch-builds/job/py2-clang7-rocmdeb-ubuntu16.04-test/3080//console

To make the CI more stable, turn it to warning instead of failure.

iotamudelta please help investigating the memory leaking
Pull Request resolved: https://github.com/pytorch/pytorch/pull/14151

Differential Revision: D13115096

Pulled By: bddppq

fbshipit-source-id: a13b68274ecba363d9d8436aa6a62ac40a77d78c
test/common_utils.py
test/test_cuda.py