validate: Use pad.last_flowret instead of trying to compute it ourselves

Which makes it more accurate

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172>

report: Add a way to force backtraces on reports

And stop report simple debug message

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172>

validate:launcher: Add a way to define test timeout from dicts

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/172>

Meson: Use pkg-config generator

gstvalidate.supp: update location of gst.supp

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/221>

docs: Update gst-validate-launcher documentation

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/193>

ci: include template from gst-ci master branch again

Back to development

Release 1.18.0

Revert "Revert "launcher: add webrtcbin datachannel tests to valgrind skip list""

This reverts commit ff79af843bc40a554795ee2a0682cfd12272dd24.

Looks like most of these are still problematic and flaky in valgrind,
so re-add to skiplist for now.

https://gitlab.freedesktop.org/thaytan/gst-plugins-base/-/jobs/4275045

Revert "launcher: add webrtcbin datachannel tests to valgrind skip list"

This reverts commit 369c74941f1607b421bc2f16edcaea0b887926a9.

Fixes https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1383

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/220>

Release 1.17.90

validate:check: Blacklist new ges valgrind test

I can't find what the issue is and the test is very long anyway

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/219>

validate: flow: Plug some leaks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/219>

validate:scenario: Plug some leaks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/219>

launcher: add webrtcbin datachannel tests to valgrind skip list

They were previously not run because the sctp plugin wasn't built
but they will be run now that we bundle libusrsctp.

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1465

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/218>

validate: silence g-ir-scanner warnings about GST_IS_VALIDATE_*

Which it complains about because we say our prefix is 'gst_validate'
so it should really be GST_VALIDATE_IS_* instead.

Hide the boilerplate defines from g-ir-scanner, it doesn't need
to process them.

Fixes #46

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/214>

Revert "validate: fix up gir namespace and symbol prefix"

This reverts commit b73e81614021a856fc7d3ff221a9bbf5ebf2ce6e.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/214>

validate: fix up gir namespace and symbol prefix

Change gir namespace and symbol prefix from
GstValidate / gst_validate to Gst / gst, same
as we do for other libs like GstVideo etc.

Helps with warnings about GST_IS_VALIDATE_*

Fixes #46, Closes !214

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/215>

validate: Fix media descriptor mp3 like formats

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/216>

validate-scenario: reflow #ifdef to work around bogus g-ir-scanner warning

Looks like a scanner bug.

The endif comments are the wrong way round too, but that's not it.

gst-validate-scenario.c:126: mismatched #endif  /* G_HAVE_GNUC_VARARGS */

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/213>

Back to development

Release 1.17.2

validate: Always check if scenario is done from the right thread

Action will be set_done from the right thread and we will check if the action is done from there

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/212>

validate: Print errors on action failures

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Plug some leaks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Move action finalization to _set_done where it belongs

gst_validate_action_set_done is the place where we should finalize the
action, not in `execute_next`, this way we better handle printing
interlaced action finalization too.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate:scenario: Replace the `sub-action` with a `foreach` action type

Sub-actions were really hard to use and conceptually weird. The
implementation was ugly and made the code complex for nothing.

Instead this commit introduces a `foreach` action type which allows
repeating actions passed in an `actions` array the number of time
specified by any `GstIntRange` value defined in the structure or its
`repeat` field.

This commit also makes sure that all action got through
gst_validate_action_set_done upon finalization.

+ Cleanup surrounding code
+ Add tests

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: scenario: Implement 'repeat' by copying actions

Instead of trying to reuse the same action structure and deal with
that in a complex way, copy the action the required number of times.

And add a simple test

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate:scenario: Round results of expressions in a sensible way

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

launcher: Keep running tests forever on KNOWN_ERROR

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Add private action type to check number of action type calls

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate:scenario: Avoid dereferencing NULL structure

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Fix marking expected issues as criticals

And never mark a repeat expected reports as repeated

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Add a way to use the expression parser in any field

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Allow variables to be set with other types than strings

And use value serialization from GStreamer to convert

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

scenario: Add an action to remove a feature/plugin from the registry

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

valiadate: Add a test for setting/checking (timed) properties

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate:scenario: Add action to set and check several properties at once

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

validate: Add an action type to set timed value properties

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/207>

debug-viewer: Add screenshot

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/210>

Back to development

Release 1.17.1

validate: flow: record GstRegionOfInterestMeta

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/206>

validate:launcher: Ensure that the main directory exists

Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/issues/53 part 1

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/209>

validate:tests: Cleanup the way set environment vars

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/209>

validate: Stop trying to support uninstalled autotools

Fix https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/issues/53 part 2

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/209>

validateflow: Plug leak

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>

scenario: Fix waiting for signal

The signal callback signature was just wrong and not generic leading to
crash if waiting for any signal that didn't match it. This commit fixes
it.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>

scenario: Add an option to set properties on all instances

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>

validate:flow: Add a way to dump buffer content as hex

Useful in unit tests with very small buffers

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>

validate: Cleanup implementation of appsrc-push

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/205>

validate: Remove useless condition

CID 1462652

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>

validate:ssim: Avoid dereferencing NULL pointer

CID 1462650

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>

validate: Give a proper argv[0] when running test files

Fixes CID 1462613

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>

validate:transcoding: Exit after printing the help

Fixes CID 1455575

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>

validate: Ensure Scenario isn't NULL when preparing action

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/192>

validate: Update documentation now core plugins are integrated

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/204>

validate: Make extracheck and flow part of core instead plugins

It makes things more complex and doesn't bring anything!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/204>

launcher: Limit copies of massive debug logs in markdown file

When debugging is activated, we could end up with log files ranging in the
multi-megabyte or even gigabyte range. Copying those is expensive from a cpu/io
point of view in addition to clobbering the storage.

Instead of always copying those files, check if they are smaller than 500kB. If
not, don't copy them and instead provide a link to their location.

Fixes #52

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/203>

validateflow: Allow specifying checksum type

And add an extra mode 'checksum-as-id' which basically numerate
buffers checksums as they are being received so that it is simpler
to compare expectations when you are tracking buffers from both
sinkpads and srcpads.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/202>

validate: Add a parameter to force waiting on the test clock

This allows to wait for a new buffer to reach the sink without
actually cranking that buffer, allowing to quite the test without
waiting for EOS in a 100% reproducible way

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/201>

validate: Error out when a config hasn't been used at all

It probably means a plugin is not available

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/201>

Ship validate as part of a gst-devtools tarball

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/199>

meson: make debug_viewer a feature option

... and disable by default.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/199>

validate: Fix mixup in quarks usages

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>

validate: Add missing return func when GLib < 2.50

CID 1463854

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>

validate: Update documentation

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>

validateflow: Add a way to configure when to generate expectations

By default, generate them whenever the file is missing but adding a way
to override that with `validateflow,generate-expectations=true` to force
regenerating them or setting `validateflow,generate-expectations=false`
to disallow generating them (on CI servers for example)

Also update the validateflow documentation to take that into account
and remove references to pipeline.json file which is now gone!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/200>

scenario: Minor cleanup around pipeline change state management

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/198>

scenario: Mark seek as done only when reaching next state

There is a race where following actions could generate a
flush-start/flush-stop dance but the state change resulting from the
seek hasn't been committed yet, leading to the ASYNC_START being
ignored by GstBin since its pending_state is not VOID when receiving
the ASYNC_START message.

Conceptually it is totally correct to consider an action done when
the state change of the pipeline is stabilized..

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/198>

validate:launcher: Stop wrongly erroring on testsuite loading failure

When the testsuite was actually already loaded as the .py
file was explicitly passed in

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

validate: Fix loading configs from a caps

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

flow: Use bat to color diffs when possible

Adding a function to check if can output colored logs

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

launcher: check: Properly set PLUGIN_PATH and registry when running in gst-build

This makes registry

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

launcher: Ensure that -j tests run in parallel when running forever

So that you can reproduce the issue you want faster!

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

docs: Update gst-validate-launcher documentation

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

launcher: Add support for running tests inside rr

Allowing us to easily run the tests forever and then replay the
failures!

https://rr-project.org/

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/197>

validate: add missing gir annotation

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/194>

validate-scenario: Initialize variable correctly

ret needs to be always reset to FALSE *before* checking attempting to load the
individual files. Otherwise there's the possibility it would silently accept an
invalid scenario name

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/196>

validate-scenario: Remove unused variable

We only need to check whether the field is present and of a given type

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/196>

validate: Stop using g_file_peek_path

It was introduced in 2.56 so is too recent

Fixes https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/issues/51

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/195>

validate: Add including support in the structure file parser

Adding proper error reporting support

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/191>

validate: utils: Do not try to replace vars in debug info fields

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/190>

validate: Fix parsing validate tests files with vars on windows

And ensure that we escape windows path in variables

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/190>

validate:launcher:check: Add support for gst-tester-1.0

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>

validate: Add support for known-issues in the .validatetest

And add some tests about remaining actions failures

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>

validate: Ensure a meta structure is found in test files

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>

validate:scenario: Check remaining action on stop, not EOS

When ignoring EOS, on addition action could end up not being executed
and no error was reported which was wrong.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/189>

validate: Fix rendering destination directory path creation

We were ending up creating file:/some/path in cwd

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>

validate: Plug some leaks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>

validate: Add details on all g_log message reports.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>

validate: Add a mechanism to mark tests as skipped

And use it when a plugin is missing and the user didn't ask for
failure when it happens

And use the TAP[0] synthax to report it

[0]: https://testanything.org

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>

validate: Enhance issue reporting from structures

And properly bail out when required
This is preparatory work for gst-test in core which will use the TAP
protocol

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/187>

validate:flow: Handle some more segment fields filtering

Those slept through when implementing filtering

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/188>

ssim: Minor improvements on the way we attach to pads

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>

validate-scenario: Refactor seek handling

* Store all seek values into a list of pending seeks instead
  of hardcoding some values
* Store all segments that sinks received
* Match segments to seeks when all sinks received segments with
  the same seqnum
* Detect when a seek did *not* result in segments with identical
  matching seqnums

Should allow checking for all types of seek handling, including
flush-less seeks

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>

validate-report: Add new check for top-level seek

This issue is to detect seeks that don't result in segments
with identical seqnums. This check can be done at the top-level
scenario

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>

validate-pad-monitor: Post GstBaseSink SEGMENT on the bus

Allows higher-level bin or app (like validate-scenario) to know
what each sink currently has in terms of SEGMENT.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>

validate-element-monitor: Detect basesink elements

And add a macro

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-devtools/-/merge_requests/174>