Use RDS for build size tracking (#64303)
authorDavid Riazati <driazati@users.noreply.github.com>
Tue, 14 Sep 2021 00:47:18 +0000 (17:47 -0700)
committerFacebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
Tue, 14 Sep 2021 00:48:44 +0000 (17:48 -0700)
commit1cd0252eed8ddb26e4599ef2b0fec4d8843b8828
treea3f8d634f019470f141713366e0bf0e3915321ef
parentc4073af61d4e530f8627dced870f42526d93dcaf
Use RDS for build size tracking (#64303)

Summary:
This adds 2 utilities: `register_rds_table` and `rds_write`. `register_rds_table` needs to be called once with the schema for the data that `rds_write` will write. These go to a lambda called `rds-proxy`, which will write to/read from the DB as necessary. This data can then be arbitrarily queried via `rds-proxy` (for use in CI) or on metrics.pytorch.org (for analysis).

It also hooks these up for build size tracking (which previously was not working on GHA)

TODO:
* verify output in logs + clean up prints

Pull Request resolved: https://github.com/pytorch/pytorch/pull/64303

Reviewed By: malfet, seemethere

Differential Revision: D30711934

Pulled By: driazati

fbshipit-source-id: 0af808ddf528a24875a378caeb1aa9cb0693f802
16 files changed:
.github/templates/linux_ci_workflow.yml.j2
.github/workflows/generated-libtorch-linux-xenial-cuda10.2-py3.6-gcc7.yml
.github/workflows/generated-libtorch-linux-xenial-cuda11.3-py3.6-gcc7.yml
.github/workflows/generated-linux-bionic-cuda10.2-py3.9-gcc7.yml
.github/workflows/generated-linux-bionic-py3.6-clang9.yml
.github/workflows/generated-linux-bionic-py3.8-gcc9-coverage.yml
.github/workflows/generated-linux-xenial-cuda10.2-py3.6-gcc7.yml
.github/workflows/generated-linux-xenial-cuda11.3-py3.6-gcc7.yml
.github/workflows/generated-linux-xenial-py3.6-gcc5.4.yml
.github/workflows/generated-parallelnative-linux-xenial-py3.6-gcc5.4.yml
.github/workflows/generated-paralleltbb-linux-xenial-py3.6-gcc5.4.yml
.github/workflows/generated-periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7.yml
.github/workflows/generated-periodic-linux-xenial-cuda11.1-py3.6-gcc7.yml
.github/workflows/generated-puretorch-linux-xenial-py3.6-gcc5.4.yml
tools/stats/scribe.py
tools/stats/upload_binary_size_to_scuba.py