Workflows overview
Workflows surface valuable qualitative information about the tests in your test suite, which can be difficult to surmise from raw execution data. A workflow defines a process which allows you to create custom mappings between observations that Test Engine makes about your test suite, and the actions you'd like to take from them. This means that observations about the health and performance of your tests (for example, test is flaky) can generate automatic actions (for example, label the test as flaky, send a notification).
Workflows are composed of a single monitor, optional tag filters, and a number of different actions. The actionable insights generated by workflows can be used to improve your test suite, for example by reducing the number of flaky tests.
How they work
A single monitor watches over all the tests in your test suite (except for those excluded by filters) and generates individual alarm and recover events for each test, which then trigger the associated alarm and recover actions.
Alarm events are reported by the monitor when the alarm conditions are met for a given test. This could be an observation of a single special occurrence (for example, a test reports both a pass and fail result on the same commit SHA) or a cumulative score that is tracked over time being exceeded (for example, the transition count score for a test exceeds 0.05).
Recover events are hysteric, meaning that the recover event can only be reported on a test that has a previous alarm event. In such a situation, when the monitor detects that the test has met the recover conditions, a recover event is reported.
Depending on the monitor type, the alarm and recover conditions can be configured.
Actions are performed when the recover or alarm event is reported. Actions are automatically triggered user-defined operations, and can be for operations that happen within the Test Engine system (that is, changing a test's state or label), or externally to Test Engine (for example, sending a Slack notification about the test).
Repeated occurrences of the test meeting the alarm/recover conditions do not retrigger the corresponding actions.
