Change workflow scheduled time to more "random" minute

[platform/core/system/upgrade-tools.git] / README

# Tizen System Update

This repository was previously known as platorm/core/system/tota-upg,
the "Tizen OTA Upgrade Package Generator".  From Tizen 8.0 upgrade is
handled by platform/core/system/upgrade-tools repository.


Tizen System Update is the one of Tizen feature. This implements
the functionality of firmware update base on OTA mechanism.

* Notice of Limitations
 This functionality does NOT support official signing & validation method.
This means it requires proper mechanism for binary validation offered by each
manufacturers.


## Docker environment

OS Upgrade tools have a dedicated Docker environment image, which can be used
for local OS Upgrade image creation, as well as for CI.

The image can be either built by executing `docker-build.sh` script or by
pulling the image from BART registry.

### Using BART registry (recommended)

The Docker image is built in
[`docker-build`](./.github/workflows/docker-build.yml) workflow in this
repository. It is stored in BART Docker registry as
`tizen-system-upgrade-tools-docker-local.bart.sec.samsung.net/upgrade-tools`.

For the default branch (`tizen`), an image with tag `latest` is generated. In
case a PR on GitHub is submitted, Docker image with tag name derived from the
git branch name is pushed to the registry in order to test the delta image
generation by the new image. For example, for a PR from branch `ghactions`,
there will be image called
`tizen-system-upgrade-tools-docker-local.bart.sec.samsung.net/upgrade-tools:ghactions`.

You can pull the image by executing the following commands:

```shell
# Needs to be executed only once:
$ docker login tizen-system-upgrade-tools-docker-local.bart.sec.samsung.net
Username: <BART ID>
Password: <BART Identity Key>
Login Succeeded
$ docker pull tizen-system-upgrade-tools-docker-local.bart.sec.samsung.net/upgrade-tools:<Docker tag>
```

- BART ID – username used as BART login.
- BART Identity Key – identity key for your account which can be generated
  [here](https://bart.sec.samsung.net/ui/user_profile).
- Docker tag – `latest` for the default image, or PR-specific tag name.

### Local image build

First, ensure that you have the latest Docker with BuildKit enabled (version 23
and above), as the image requires some extensions which were enabled recently.
It is advised to use the [official Docker
CE](https://docs.docker.com/engine/install/). It is not requried to have the
*Docker Desktop* – `docker-ce` package is enough. Distro-provided Docker package
may or may not work, and this setup is not directly supported. BuildKit is
enabled by default from version 23 of Docker Engine, but [can be explicitly
enabled](https://docs.docker.com/build/buildkit/#getting-started) for some
older versions.

By default, the FROM image is taken from the official Docker Hub (i.e.,
`library` namespace). If you want to source it from BART, export
`FROM_IMAGE_NAMESPACE=docker-remote.bart.sec.samsung.net` as environment
variable before executing the build script.

To build the Docker image locally, execute:

```shell
$ ./docker-build.sh
```

### Delta generation in Docker environment

Save input tar.gz images in `images/old` and `images/new`. Once you have the
Docker image pulled or built, you can execute delta image generation in the
following manner:

```shell
$ TARGET=rpi4
$ docker run --privileged=true \
    -v $(pwd)/images/old:/tota-upg/mk_delta/$TARGET/data/old_tar:ro \
    -v $(pwd)/images/new:/tota-upg/mk_delta/$TARGET/data/new_tar:ro \
    -v $(pwd)/images/result:/tota-upg/mk_delta/$TARGET/result \
    tizen-system-upgrade-tools-docker-local.bart.sec.samsung.net/upgrade-tools \
    /tota-upg/scripts/delta-generation.sh /tota-upg/ $TARGET
```

Generated delta images will be stored in `images/result`.


## GitHub Actions

### `mk_delta` action

`mk_delta` action creates delta OS Upgrade image for two sets of source images
(old and new), and stores the result on BART.

Before executing the delta script, the action checks if the delta needs to be
built – if input image checksums match with the previous build the delta image
is not built, thus reducing run time of the action (about 15 seconds for cache
check). This in turn allows to run the action frequently to check if new input
images are available.

Example use case of the action can be found in this repository in
[test-image-build](./.github/workflows/test-image-build.yml) workflow. For
external repositories this action should be referenced as
`srpol-syskern/upgrade-tools/mk_delta@tizen`.

**The action does the following things:**

1. Install JFrog tool – used for BART interaction.
2. Prepare environment – generate environment variables and pull cached input
   checksums from BART.
3. Download source images:
    1. Download `SHA256SUM` files if available.
    2. Check if new images need to be generated – **if not, no additional steps
       are executed**.
    3. Download input tar.gz images.
4. Log in to Docker registry.
5. Generate delta image – execute `delta-generation.sh` script in Docker
   environment.
6. Upload to artifactory:
    1. Compress resulting tar archive to `.tar.gz`.
    2. Upload delta archive to BART.
    3. Upload input file SHA256 checksums.
7. Add GitHub PR comment with links to generated delta image.

The tool uploads two copies of the archive to `${{ inputs.output_repository
}}`/`${{ inputs.output_directory }}` directory. One has a deterministic name,
which can be inferred from the upgraded system filesystem, and the second has
some additional information and is uploaded to `archive` subdirectory:

- deterministic name: `${device_type}--${model_name}--${image_type}--${arch}--${tz_build_date}@${{ inputs.docker_image_tag }}.tar.gz`,
- archival name: `archive/${device_type}--${model_name}--${image_type}--${arch}--${build_string.txt}--${delta_timestamp}.tar.gz`.

**Variables:**

- `device_type`, `model_name`, `tz_build_date` – taken from `update-info.ini`
  delta file.
- `image_type`:
  - `all` contains delta of all sub-images,
  - `platform` doesn't contain boot delta.
- `build_string.txt` – contents of `build_string.txt` delta file.
- `delta_timestamp` – delta build timestamp in format `%Y%m%d_%H%M%S`.
- `arch` – `${{ inputs.arch }}` for now, until `tz_build_arch` from
  `update-info.ini` is fixed.

In addition to the main `.tar.gz` archive, a `.sha256sums` file containing input
file checksums is uploaded along the delta files, and in `action_checksums/${{
inputs.run_name }}` directory.

**Example file tree on BART after a successful run:**

- `${{ inputs.output_repository }}`/`${{ inputs.output_directory }}/`
  - `IoT_Headless--rpi4--all--aarch64--20240124_061003@latest.tar.gz` –
    resulting delta archive after latest run on the main branch (`tizen`).
  - `IoT_Headless--rpi4--all--aarch64--20240124_061003@latest.tar.gz.sha256sums`
    – SHA256 checksums of input files for the run above.
  - `archive/`
    - `IoT_Headless--rpi4--all--aarch64--20240124_061003@20240123_164559--20240124_121048.tar.gz`
      – resulting delta archive after run executed on 2024-01-24 at 12:10:48
      UTC.
    - `IoT_Headless--rpi4--all--aarch64--20240124_061003@20240123_164559--20240124_121048.sha256sums`
      – SHA256 checksums of input files for the run above.
  - `action_checksums/${{ inputs.run_name }}/`
    - `latest.sha256sums` – copy of
    `IoT_Headless--rpi4--all--aarch64--20240124_061003@latest.tar.gz.sha256sums`
    (used for cache check in cache check in the second step).

#### Action inputs

For default values of some inputs please refer to [the action
file](./mk_delta/action.yml).

**Docker image configuration:**

For regular run it is not required to change these, but it can be useful for
checking if new images work properly.

- `docker_image_registry` – Docker image registry used for the delta build.
- `docker_image_name` – Docker image used for the delta build.
- `docker_image_tag` – Docker image tag used for the delta build.

**Secrets (BART and GitHub):**

It is advised to use GitHub secrets to store the BART credentials. Example on
how to use them, refer to the
[test-image-build](./.github/workflows/test-image-build.yml) workflow. It is the
safest to create service account especially for CI purpose and store the secrets
in GitHub repository (not organization-wide). You can access GitHub secrets in
project *Settings → Secrets and variables*.

- `bart_id` – user ID used for BART authentication.
- `bart_key` – BART Identity Token which can be generated
  [here](https://bart.sec.samsung.net/ui/user_profile).
- `github_token` – GitHub Token used for API access to fetch the job ID. Can be
  passed from `${{ secrets.GITHUB_TOKEN }}`.

**Delta tool configuration:**

Input image URL can be either directory or `tar.gz` file. Mind that previous
build detection doesn't currently work for `tar.gz` variant – only for
directories with `SHA256SUMS` file (as on https://download.tizen.org).

For multiple images, separate them with newline, e.g.:

```yaml
- name: Build delta images
  uses: srpol-syskern/upgrade-tools/mk_delta@tizen
  with:
    old_images: |
      https://download.tizen.org/snapshots/TIZEN/Tizen/Tizen-Unified/latest/images/standard/tizen-boot-arm64-rpi4/
      https://download.tizen.org/snapshots/TIZEN/Tizen/Tizen-Unified/latest/images/standard/tizen-headless-aarch64/
```

- `old_images` – Old image URLs.
- `new_images` – New image URLs.
- `arch` – Build architecture. Used for artifact naming.
- `target` – `delta-generation.sh` script target.
- `image_type` – Name of the image type for given input images.

**Output configuration:**

- `run_name` – Delta generation run name. Used in artifact path and messages.
- `output_repository` – Output BART repository.
- `output_directory` – Output directory in BART repository.
- `output_gzip_flags` – Flags given to gzip command used to compress the final
  delta tar.

#### Action outputs

- `artifact_path_latest` – Path to result delta image artifact in BART
  repository with `latest` tag.
- `artifact_path_archvie` – Path to result delta image artifact in BART
  repository with deterministic path.


### `mk_delta/summarize` action

`mk_delta/summarize` action processes job state and presents it in a concise
GitHub PR message.

Example use case of the action can be found in this repository in
[test-image-build](./.github/workflows/test-image-build.yml) workflow. For
external repositories this action should be referenced as
`srpol-syskern/upgrade-tools/mk_delta/summarize@tizen`.

#### Action inputs

**BART secrets:** Same as in `mk_delta` action.

**Output configuration:** `output_repository` and `output_directory` same as in `mk_delta` action.

#### Example output

> Here is the list of all jobs in run [8394854](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854):
> - [Tizen-7.0_Tizen-8.0_headed_rpi4_all_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478842): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen7_Unified/archive/IoT_Headed--rpi4--all--aarch64--20240205_160201@20240205_160245--20240205_171010.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen7_Unified/archive/IoT_Headed--rpi4--all--aarch64--20240205_160201@20240205_160245--20240205_171010.tar.gz)).
> - [Tizen-7.0_Tizen-8.0_headed_rpi4_all_armv7hl](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478838): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headed_rpi4_all_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478840): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headed_rpi4_platform_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478845): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen7_Unified/archive/IoT_Headed----platform--aarch64--20240205_160201@20240205_160245--20240205_171104.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen7_Unified/archive/IoT_Headed----platform--aarch64--20240205_160201@20240205_160245--20240205_171104.tar.gz)).
> - [Tizen-7.0_Tizen-8.0_headed_rpi4_platform_armv7hl](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478839): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headed_rpi4_platform_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478841): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headless_rpi4_all_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478848): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headless_rpi4_all_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478846): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headless_rpi4_platform_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478849): ✔️ Inputs are not different from the last run. Not generating deltas.
> - [Tizen-7.0_Tizen-8.0_headless_rpi4_platform_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478847): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen7_Unified/archive/IoT_Headless----platform--armv7l--20240205_155803@20240205_155843--20240205_165500.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen7_Unified/archive/IoT_Headless----platform--armv7l--20240205_155803@20240205_155843--20240205_165500.tar.gz)).
> - [Tizen-8.0_Tizen_headed_rpi4_all_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478855): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed--rpi4--all--aarch64--20240205_160245@20240205_161650--20240205_170156.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed--rpi4--all--aarch64--20240205_160245@20240205_161650--20240205_170156.tar.gz)).
> - [Tizen-8.0_Tizen_headed_rpi4_all_armv7hl](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478850): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed--rpi4--all--armv7hl--20240205_160219@20240205_161637--20240205_165902.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed--rpi4--all--armv7hl--20240205_160219@20240205_161637--20240205_165902.tar.gz)).
> - [Tizen-8.0_Tizen_headed_rpi4_all_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478853): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed--rpi4--all--armv7l--20240205_160233@20240205_161647--20240205_165952.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed--rpi4--all--armv7l--20240205_160233@20240205_161647--20240205_165952.tar.gz)).
> - [Tizen-8.0_Tizen_headed_rpi4_platform_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478856): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed----platform--aarch64--20240205_160245@20240205_161650--20240205_170306.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed----platform--aarch64--20240205_160245@20240205_161650--20240205_170306.tar.gz)).
> - [Tizen-8.0_Tizen_headed_rpi4_platform_armv7hl](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478851): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed----platform--armv7hl--20240205_160219@20240205_161637--20240205_170102.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed----platform--armv7hl--20240205_160219@20240205_161637--20240205_170102.tar.gz)).
> - [Tizen-8.0_Tizen_headed_rpi4_platform_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478854): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed----platform--armv7l--20240205_160233@20240205_161647--20240205_165955.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headed----platform--armv7l--20240205_160233@20240205_161647--20240205_165955.tar.gz)).
> - [Tizen-8.0_Tizen_headless_rpi4_all_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478859): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless--rpi4--all--aarch64--20240205_155854@20240205_161251--20240205_165319.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless--rpi4--all--aarch64--20240205_155854@20240205_161251--20240205_165319.tar.gz)).
> - [Tizen-8.0_Tizen_headless_rpi4_all_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478857): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless--rpi4--all--armv7l--20240205_155843@20240205_161300--20240205_165430.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless--rpi4--all--armv7l--20240205_155843@20240205_161300--20240205_165430.tar.gz)).
> - [Tizen-8.0_Tizen_headless_rpi4_platform_aarch64](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478860): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless----platform--aarch64--20240205_155854@20240205_161251--20240205_165319.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless----platform--aarch64--20240205_155854@20240205_161251--20240205_165319.tar.gz)).
> - [Tizen-8.0_Tizen_headless_rpi4_platform_armv7l](https://github.sec.samsung.net/srpol-syskern/upgrade-tools/actions/runs/8394854/job/21478858): ✔️ You can find the delta image [here](https://bart.sec.samsung.net/ui/repos/tree/General/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless----platform--armv7l--20240205_155843@20240205_161300--20240205_165224.tar.gz) ([direct](https://bart.sec.samsung.net/artifactory/tizen-system-upgrade-tools-generic-local/tests/Tizen8.0_Unified/archive/IoT_Headless----platform--armv7l--20240205_155843@20240205_161300--20240205_165224.tar.gz)).