From 7ee3421cad934199e5885b7a846c0fbb1eb11502 Mon Sep 17 00:00:00 2001 From: Thibault Saunier Date: Wed, 3 Jun 2020 14:29:22 -0400 Subject: [PATCH] validate: Update documentation now core plugins are integrated Part-of: --- docs/{plugins => }/fakesrc.simple.validatetest | 0 ...tetest.yaml => fakesrc.simple.validatetest.ini} | 0 .../flow-expectations/log-sink-sink-expected | 0 docs/gst-validate-config.md | 2 +- docs/gst-validate-flow.md | 196 +++++++++++++++++++++ docs/gst-validate-test-file.md | 2 +- docs/plugins/validateflow.md | 194 +------------------- docs/sitemap.txt | 1 + 8 files changed, 202 insertions(+), 193 deletions(-) rename docs/{plugins => }/fakesrc.simple.validatetest (100%) rename docs/{plugins/fakesrc.simple.validatetest.yaml => fakesrc.simple.validatetest.ini} (100%) rename docs/{plugins => }/fakesrc.simple/flow-expectations/log-sink-sink-expected (100%) create mode 100644 docs/gst-validate-flow.md diff --git a/docs/plugins/fakesrc.simple.validatetest b/docs/fakesrc.simple.validatetest similarity index 100% rename from docs/plugins/fakesrc.simple.validatetest rename to docs/fakesrc.simple.validatetest diff --git a/docs/plugins/fakesrc.simple.validatetest.yaml b/docs/fakesrc.simple.validatetest.ini similarity index 100% rename from docs/plugins/fakesrc.simple.validatetest.yaml rename to docs/fakesrc.simple.validatetest.ini diff --git a/docs/plugins/fakesrc.simple/flow-expectations/log-sink-sink-expected b/docs/fakesrc.simple/flow-expectations/log-sink-sink-expected similarity index 100% rename from docs/plugins/fakesrc.simple/flow-expectations/log-sink-sink-expected rename to docs/fakesrc.simple/flow-expectations/log-sink-sink-expected diff --git a/docs/gst-validate-config.md b/docs/gst-validate-config.md index a4a9c5a..90911d3 100644 --- a/docs/gst-validate-config.md +++ b/docs/gst-validate-config.md @@ -10,7 +10,7 @@ to setup its plugins and core behaviour. The config format is very similar to the [scenario](gst-validate-scenarios.md) file format. You can check the [ssim plugin](plugins/ssim.md) -and the [validate flow plugin](plugins/validateflow.md) +and the [validate flow plugin](gst-validate-flow.md) for examples. ## Core settings parameters diff --git a/docs/gst-validate-flow.md b/docs/gst-validate-flow.md new file mode 100644 index 0000000..6e67f0e --- /dev/null +++ b/docs/gst-validate-flow.md @@ -0,0 +1,196 @@ +--- +title: Validate Flow +short-description: Validate a pad data flow +... + +# Validate Flow + +Validate Flow — GStreamer validate component to record a log of buffers and +events flowing in a specified pad and compare it with an expectation file. + +## Description + +This component exists for the purpose of testing non-regular-playback use cases +where the test author specifies the full pipeline, a series of actions and needs +to check whether the generated buffers and events make sense. + +The testing procedure goes like this: + +1. The test author writes a [.validatetest](gst-validate-test-file.md) test + where validateflow is used. A pad where monitoring will occur is specified + and possibly a list of [actions](gst-validate-action-types.md) to run can + also be specified. + +2. The test author runs the test with the desired pipeline, the configuration + and the actions. Since an expectation file does not exist at + this point, validateflow will create one. The author should check its + contents for any missing or unwanted events. No actual checking is done by + validateflow in this step, since there is nothing to compare to yet. + +3. Further executions of the test will also record the produced buffers and + events, but now they will be compared to the previous log (expectation file). + Any difference will be reported as a test failure. The original expectation + file is never modified by validateflow. Any desired changes can be made by + editing the file manually or deleting it and running the test again. + +## Example + +### Simplest example + +The following is an example of a `fakesrc.simple.validatetest` file using +validateflow. + +{{ fakesrc.simple.validatetest.ini }} + +Then generate the expectation file with: + +``` bash +gst-validate-1.0 --set-test-file /path/to/fakesrc.simple.validatetest +``` + +This will generate the +`/path/to/fakesrc.simple/flow-expectations/log-sink-sink-expected` file +containing: + +{{ plugins/fakesrc.simple/flow-expectations/log-sink-sink-expected.log }} + +Note that the test will be marked as "SKIPPED" when we generate expectation +files. + +The test can now be run with: + +``` +gst-validate-1.0 --set-test-file /path/to/fakesrc.simple.validatetest +``` + +### Example controlling the source + +The following is an example of the `qtdemux_change_edit_list.validatetest` file using validateflow. + +``` ini +set-globals, media_dir="$(test_dir)/../../../medias/" +meta, + seek=false, + handles-states=false, + args = { + "appsrc ! qtdemux ! fakesink async=false", + }, + configs = { + "$(validateflow), pad=fakesink0:sink, record-buffers=false", + } + +# Scenario action types +appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-85.mp4/init.mp4" +appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-85.mp4/media1.mp4" +checkpoint, text="A moov with a different edit list is now pushed" +appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-86.mp4/init.mp4" +appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-86.mp4/media2.mp4" +stop +``` + +This example shows the elements of a typical validate flow test (a pipeline, a +config and a scenario). Some actions typically used together with validateflow +can also be seen. Notice variable interpolation is used to fill absolute paths +for media files in the scenario (`$(test_dir)`). In the configuration, +`$(validateflow)` is expanded to something like this, containing proper paths +for expectations and actual results (these values are interpolated from the +`.validatetest` file location): + +``` ini +validateflow, expectations-dir="/validate/test/file/path/validateqtdemux_change_edit_list/flow-expectations/", actual-results-dir="$(GST_VALIDATE_LOGSDIR)/logs/validate/launch_pipeline/qtdemux_change_edit_list" +``` + +The resulting log looks like this: + +``` ini +event stream-start: GstEventStreamStart, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1; +event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2.1, profile=(string)main, codec_data=(buffer)014d4015ffe10016674d4015d901b1fe4e1000003e90000bb800f162e48001000468eb8f20, width=(int)426, height=(int)240, pixel-aspect-ratio=(fraction)1/1; +event segment: format=TIME, start=0:00:00.000000000, offset=0:00:00.000000000, stop=none, time=0:00:00.000000000, base=0:00:00.000000000, position=0:00:00.000000000 +event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; +event tag: GstTagList-global, taglist=(taglist)"taglist\,\ datetime\=\(datetime\)2012-08-27T01:00:50Z\,\ container-format\=\(string\)\"ISO\\\ fMP4\"\;"; +event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; +event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2.1, profile=(string)main, codec_data=(buffer)014d4015ffe10016674d4015d901b1fe4e1000003e90000bb800f162e48001000468eb8f20, width=(int)426, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)24000/1001; + +CHECKPOINT: A moov with a different edit list is now pushed + +event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)main, codec_data=(buffer)014d401effe10016674d401ee8805017fcb0800001f480005dc0078b168901000468ebaf20, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1; +event segment: format=TIME, start=0:00:00.041711111, offset=0:00:00.000000000, stop=none, time=0:00:00.000000000, base=0:00:00.000000000, position=0:00:00.041711111 +event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; +event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; +event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)main, codec_data=(buffer)014d401effe10016674d401ee8805017fcb0800001f480005dc0078b168901000468ebaf20, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)24000/1001; +``` + +## Configuration + +In order to use the plugin a validate configuration must be provided, +containing a line starting by `validateflow` followed by a number of settings. +Every `validateflow` line creates a `ValidateFlowOverride`, which listens to a +given pad. A test may have several `validateflow` lines, therefore having +several overrides and listening to different pads with different settings. + +* `pad`: Required. Name of the pad that will be monitored. +* `record-buffers`: Default: false. Whether buffers will be logged. By default + only events are logged. +* `buffers-checksum`: Default: 'none'. Define the type of checksums to be used + valid values are: + * `none`: No checksum recorded + * `as-id`: Record checksum as 'ids' where the IDs are incremented on each new + checksum passed in + * `md5`: md5 checksum + * `sha1`: sha1 checksum + * `sha256`: sha256 checksum + * `sha512`: sha512 checksum + * *Note*: for backward compatibility reasons, this can be passed as a + boolean and it will default to 'sha1' if true, 'none' if false. +* `ignored-fields`: Default: `"stream-start={ stream-id }"` (as they are often + non reproducible). Key with a serialized GstValueList(str) of fields to not + record. +* `logged-fields`: Default: `NULL` Key with a serialized GstValueList(str) of + fields to record, eg. `logged-event-fields="stream-start={flags}, + caps={width, height, framerate}, buffer={pts}"`. Overrides + `ignored-event-fields` for specified event types. +* `ignored-event-types`: Default: `{ }`. List of event type names to not record +* `logged-event-types`: Default: `NULL`. List of event type names to not record, + if noone provided, all events are logged, except the ones defined in the + `ignored-event-types`. +* `expectations-dir`: Path to the directory where the expectations will be + written if they don't exist, relative to the current working directory. By + default the current working directory is used, but this setting is usually + set automatically as part of the `%(validateflow)s` expansion to a correct + path like `~/gst-validate/gst-integration-testsuites/flow-expectations/`. +* `actual-results-dir`: Path to the directory where the events will be recorded. + The expectation file will be compared to this. By default the current working + directory is used, but this setting is usually set automatically as part of + the `%(validateflow)s` expansion to the test log directory, i.e. + `~/gst-validate/logs/validate/launch_pipeline/`. +* `generate-expectations`: Default: unset. When set to `true` the expectation + file will be written and no testing will be done and if set to `false`, the + expectation file will be required. If a validateflow config is used without + specifying any other parametters, the validateflow plugin will consider that + all validateflow overrides will use that value. + + +## Scenario actions + +Scenarios with validateflow work in the same way as other tests. Often +validatetests will use appsrc in order to control the flow of data precisely, +possibly interleaving events in between. The following is a list of useful +actions. + + * `appsrc-push`: Pushes a buffer from an appsrc element and waits for the chain + operation to finish. A path to a file is provided, optionally with an offset + and/or size. + * `appsrc-eos`: Queues an EOS event from the appsrc. The action finishes + immediately at this point. + * `stop`: Tears down the pipeline and stops the test. + * `checkpoint`: Records a "checkpoint" message in all validateflow overrides, + with an optional explanation message. This is useful to check certain events + or buffers are sent at a specific moment in the scenario, and can also help + to the comprehension of the scenario. + +More details on these actions can be queried from the command line, like this: + +``` bash +gst-validate-1.0 --inspect-action-type appsrc-push +``` diff --git a/docs/gst-validate-test-file.md b/docs/gst-validate-test-file.md index 39ad4db..c743618 100644 --- a/docs/gst-validate-test-file.md +++ b/docs/gst-validate-test-file.md @@ -104,7 +104,7 @@ Validate testfile will define some variables to make those files relocable: * `$(validateflow)`: The validateflow structure name with the default/right values for the `expectations-dir` and `actual-results-dir` - fields. See [validateflow](plugins/validateflow.md) for more + fields. See [validateflow](gst-validate-flow.md) for more information. * `$(videosink)`: The GStreamer videosink to use if the test can work with diff --git a/docs/plugins/validateflow.md b/docs/plugins/validateflow.md index 4d51af2..4112028 100644 --- a/docs/plugins/validateflow.md +++ b/docs/plugins/validateflow.md @@ -1,191 +1,3 @@ -# Validate Flow plugin - -Validate Flow plugin — GstValidate plugin to record a log of buffers and -events and compare them to an expectation file. - -## Description - -This plugin exists for the purpose of testing non-regular-playback use cases -where the test author specifies the full pipeline, a series of actions and needs -to check whether the generated buffers and events make sense. - -The testing procedure goes like this: - -1. The test author writes a [.validatetest](gst-validate-test-file.md) test - where validateflow is used. A pad where monitoring will occur is specified - and possibly a list of [actions](gst-validate-action-types.md) to run can - also be specified. - -2. The test author runs the test with the desired pipeline, the configuration - and the actions. Since an expectation file does not exist at - this point, validateflow will create one. The author should check its - contents for any missing or unwanted events. No actual checking is done by - validateflow in this step, since there is nothing to compare to yet. - -3. Further executions of the test will also record the produced buffers and - events, but now they will be compared to the previous log (expectation file). - Any difference will be reported as a test failure. The original expectation - file is never modified by validateflow. Any desired changes can be made by - editing the file manually or deleting it and running the test again. - -## Example - -### Simplest example - -The following is an example of a `fakesrc.simple.validatetest` file using -validateflow. - -{{ plugins/fakesrc.simple.validatetest.yaml }} - -Then generate the expectation file with: - -``` bash -gst-validate-1.0 --set-test-file /path/to/fakesrc.simple.validatetest -``` - -This will generate the -`/path/to/fakesrc.simple/flow-expectations/log-sink-sink-expected` file -containing: - -{{ plugins/fakesrc.simple/flow-expectations/log-sink-sink-expected.log }} - -Note that the test will be marked as "SKIPPED" when we generate expectation -files. - -The test can now be run with: - -``` -gst-validate-1.0 --set-test-file /path/to/fakesrc.simple.validatetest -``` - -### Example controlling the source - -The following is an example of the `qtdemux_change_edit_list.validatetest` file using validateflow. - -``` yaml -set-globals, media_dir="$(test_dir)/../../../medias/" -meta, - seek=false, - handles-states=false, - args = { - "appsrc ! qtdemux ! fakesink async=false", - }, - configs = { - "$(validateflow), pad=fakesink0:sink, record-buffers=false", - } - -# Scenario action types -appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-85.mp4/init.mp4" -appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-85.mp4/media1.mp4" -checkpoint, text="A moov with a different edit list is now pushed" -appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-86.mp4/init.mp4" -appsrc-push, target-element-name=appsrc0, file-name="$(media_dir)/fragments/car-20120827-86.mp4/media2.mp4" -stop -``` - -This example shows the elements of a typical validate flow test (a pipeline, a -config and a scenario). Some actions typically used together with validateflow -can also be seen. Notice variable interpolation is used to fill absolute paths -for media files in the scenario (`$(test_dir)`). In the configuration, -`$(validateflow)` is expanded to something like this, containing proper paths -for expectations and actual results (these values are interpolated from the -`.validatetest` file location): - -``` yaml -validateflow, expectations-dir="/validate/test/file/path/validateqtdemux_change_edit_list/flow-expectations/", actual-results-dir="$(GST_VALIDATE_LOGSDIR)/logs/validate/launch_pipeline/qtdemux_change_edit_list" -``` - -The resulting log looks like this: - -``` yaml -event stream-start: GstEventStreamStart, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1; -event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2.1, profile=(string)main, codec_data=(buffer)014d4015ffe10016674d4015d901b1fe4e1000003e90000bb800f162e48001000468eb8f20, width=(int)426, height=(int)240, pixel-aspect-ratio=(fraction)1/1; -event segment: format=TIME, start=0:00:00.000000000, offset=0:00:00.000000000, stop=none, time=0:00:00.000000000, base=0:00:00.000000000, position=0:00:00.000000000 -event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; -event tag: GstTagList-global, taglist=(taglist)"taglist\,\ datetime\=\(datetime\)2012-08-27T01:00:50Z\,\ container-format\=\(string\)\"ISO\\\ fMP4\"\;"; -event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; -event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2.1, profile=(string)main, codec_data=(buffer)014d4015ffe10016674d4015d901b1fe4e1000003e90000bb800f162e48001000468eb8f20, width=(int)426, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)24000/1001; - -CHECKPOINT: A moov with a different edit list is now pushed - -event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)main, codec_data=(buffer)014d401effe10016674d401ee8805017fcb0800001f480005dc0078b168901000468ebaf20, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1; -event segment: format=TIME, start=0:00:00.041711111, offset=0:00:00.000000000, stop=none, time=0:00:00.000000000, base=0:00:00.000000000, position=0:00:00.041711111 -event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; -event tag: GstTagList-stream, taglist=(taglist)"taglist\,\ video-codec\=\(string\)\"H.264\\\ /\\\ AVC\"\;"; -event caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)main, codec_data=(buffer)014d401effe10016674d401ee8805017fcb0800001f480005dc0078b168901000468ebaf20, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)24000/1001; -``` - -## Configuration - -In order to use the plugin a validate configuration must be provided, -containing a line starting by `validateflow` followed by a number of settings. -Every `validateflow` line creates a `ValidateFlowOverride`, which listens to a -given pad. A test may have several `validateflow` lines, therefore having -several overrides and listening to different pads with different settings. - - * `pad`: Required. Name of the pad that will be monitored. - * `record-buffers`: Default: false. Whether buffers will be logged. By default - only events are logged. - * `buffers-checksum`: Default: 'none'. Define the type of checksums to be used - valid values are: - - none: No checksum recorded - - as-id: Record checksum as 'ids' IDs are incremented on each new checksum - passed in - - md5: md5 checksum - - sha1: sha1 checksum - - sha256: sha256 checksum - - sha512: sha512 checksum - Note that for backward compatibility reasons, this can be passed as a boolean - and it will default to 'sha1' if true, 'none' if false. - * `ignored-fields`: Default: `"stream-start={ stream-id }"` (as they are often - non reproducible). Key with a serialized GstValueList(str) of fields to not - record. - * `logged-fields`: Default: `NULL` Key with a serialized GstValueList(str) of - fields to record, eg. `logged-event-fields="stream-start={flags}, - caps={width, height, framerate}, buffer={pts}"`. Overrides - `ignored-event-fields` for specified event types. - * `ignored-event-types`: Default: `{ }`. List of event type names to not record - * `logged-event-types`: Default: `NULL`. List of event type names to not - record, if noone provided, all events are logged, except the ones defined in - the `ignored-event-types`. - * `expectations-dir`: Path to the directory where the expectations will be - written if they don't exist, relative to the current working directory. By - default the current working directory is used, but this setting is usually - set automatically as part of the `%(validateflow)s` expansion to a correct - path like `~/gst-validate/gst-integration-testsuites/flow-expectations/`. - * `actual-results-dir`: Path to the directory where the events will be - recorded. The expectation file will be compared to this. By default the - current working directory is used, but this setting is usually set - automatically as part of the `%(validateflow)s` expansion to the test log - directory, i.e. `~/gst-validate/logs/validate/launch_pipeline/`. - * `generate-expectations`: Default: unset. When set to `true` the expectation - file will be written and no testing will be done and if set to `false`, - the expectation file will be required. If a validateflow config is - used without specifying any other parametters, the validateflow plugin will - consider that all validateflow overrides will use that value. - - -## Scenario actions - -Scenarios with validateflow work in the same way as other tests. Often -validatetests will use appsrc in order to control the flow of data precisely, -possibly interleaving events in between. The following is a list of useful -actions. - - * `appsrc-push`: Pushes a buffer from an appsrc element and waits for the chain - operation to finish. A path to a file is provided, optionally with an offset - and/or size. - * `appsrc-eos`: Queues an EOS event from the appsrc. The action finishes - immediately at this point. - * `stop`: Tears down the pipeline and stops the test. - * `checkpoint`: Records a "checkpoint" message in all validateflow overrides, - with an optional explanation message. This is useful to check certain events - or buffers are sent at a specific moment in the scenario, and can also help - to the comprehension of the scenario. - -More details on these actions can be queried from the command line, like this: - -``` bash -gst-validate-1.0 --inspect-action-type appsrc-push -``` +--- +redirect: gst-validate-flow.md +... diff --git a/docs/sitemap.txt b/docs/sitemap.txt index 3474f82..091f2ae 100644 --- a/docs/sitemap.txt +++ b/docs/sitemap.txt @@ -9,6 +9,7 @@ index.md gst-validate-environment-variables.md gst-validate-action-types.md ges-validate-action-types.md + gst-validate-flow.md gi-index plugins/index.md plugins/ssim.md -- 2.7.4