Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Revert "Merge pull request #30650 from teskje/disable-new-introspection-sources"

#10594

jkosh44:enable-new-introspection-sources-again/b7cdc352bf(#30692)

Canceled automatically after 7h 3m

Miri test (full)

Extended SSH connection tests

CRDB rolling restarts

PubSub disruption

Tests for balancerd

Legacy upgrade tests (last version from git)

CRDB / Persist backup and restore

Postgres / Persist backup and restore

Replica isolation

Zero downtime

Joe Koshakow

Created Mon 2nd Dec 2024 at 9:25 PM

Triggered from Web

Scalability benchmark (read & write) against merge base or 'latest' (#2) failed, main history:

Unknown error in Workload 'UpdateWorkload':

New regression against v0.127.0-dev.0 (e70c9d8d5)

Regression in workload 'UpdateWorkload' at concurrency 16 with MaterializeContainer (None specified as HEAD): 40.1 tps vs. 51.79 tps (-11.69 tps; -22.57%)

Unknown error in Workload 'UpdateWorkload':

New regression against v0.127.0-dev.0 (e70c9d8d5)

Regression in workload 'UpdateWorkload' at concurrency 32 with MaterializeContainer (None specified as HEAD): 38.45 tps vs. 51.1 tps (-12.66 tps; -24.77%)

Test details & reproducer

Benchmark for how various queries scale, compares against old Materialize versions.

bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256

🏎️ testdrive with SIZE 1 failed, main history:

Unknown error in kafka-recreate-topic.td:

kafka-recreate-topic.td:91:1: query succeeded, but expected error containing "topic was recreated: high watermark of partition 0 regressed from 1 to 0"
     |
  16 |     URL '${testdrive ... [rest of line truncated for security]
  90 | 
  91 | ! SELECT * FROM source1_tbl
     | ^

Test details & reproducer

Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result.

bin/mzcompose --find testdrive run default --default-size=1

Zippy w/ latest CRDB failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-f98604bf/materialize/nightly/test/zippy exec -T cockroach cockroach sql --insecure -e                  CREATE EXTERNAL CONNECTION backup_bucket                 AS 's3://persist/crdb-backup?AWS_ENDPOINT=http://minio:9000/&AWS_REGION=minio&AWS_ACCESS_KEY_ID=minioadmin&AWS_SECRET_ACCESS_KEY=minioadmin';                 BACKUP INTO 'external://backup_bucket';                 DROP EXTERNAL CONNECTION backup_bucket;

Test details & reproducer

Zippy generates a pseudo-random sequence of DDLs, DMLs, failures, data validation and other events and runs it sequentially. By keeping track of the expected state it can verify results for correctness.

bin/mzcompose --find zippy run default --scenario=KafkaSources --actions=10000 --cockroach-tag=latest --max-execution-time=30m

RQG subqueries workload failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-a1a091e6/materialize/nightly/test/rqg exec -T rqg perl gentest.pl --seed=01938943-5c2f-4ad3-a336-20edde49d7d3 --dsn1=dbi:Pg:dbname=materialize;host=mz_this;user=materialize;port=6875 --dsn2=dbi:Pg:dbname=postgres;host=postgres;user=postgres;password=postgres --grammar=conf/mz/subqueries.yy --queries=100000000 --threads=4 --duration=1800 --validator=ResultsetComparatorSimplify

Test details & reproducer

Test Materialize with the Random Query Generator (grammar-based): https://github.com/MaterializeInc/RQG/ Can find query errors and panics, but not correctness.

bin/mzcompose --find rqg run default subqueries --seed=01938943-5c2f-4ad3-a336-20edde49d7d3

SQLsmith explain succeeded with known error logs, main history:

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

sqlsmith-mz_2-1  | thread 'tokio:work-24' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Use SQLsmith to generate random queries (AST/code based) and run them against Materialize: https://github.com/MaterializeInc/sqlsmith The queries can be complex, but we can't verify correctness or performance.

bin/mzcompose --find sqlsmith run default --max-joins=5 --explain-only --runtime=1500

Output consistency (Postgres) failed, main history:

Unknown error in Query 409.1 (jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))):

Outcome differs

SUCCESS_MISMATCH: Outcome differs.
Expression: jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))
  Value 1 (Materialize evaluation): 'QueryResult' (type: <class 'str'>)
  Value 2 (Postgres evaluation): 'QueryFailure' (type: <class 'str'>)
  Error 2: 'invalid input syntax for type uuid: "invalid_value_123"'
  Query 1: SELECT jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123')) FROM t_dfr_horiz WHERE ((jsonb_val_1 ? 'z'));
  Query 2: SELECT jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123')) FROM t_pg_horiz WHERE ((jsonb_val_1 ? 'z'));
  Expression hash: 7284047682483061634

Code to reproduce

--- Minimal code for reproduction
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Setup for evaluation strategy 'Materialize evaluation':
DROP TABLE IF EXISTS mz_evaluation;
CREATE TABLE mz_evaluation (row_index INT, varchar_8_val_w_spaces VARCHAR(8), jsonb_val_1 JSONB);
INSERT INTO mz_evaluation VALUES (0, ' mAA m
	 '::VARCHAR(8), '{"a": 1}'::JSONB);
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Query using evaluation strategy 'Materialize evaluation':
SELECT
  jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))
FROM
  mz_evaluation
WHERE ((jsonb_val_1 ? 'z'));
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Setup for evaluation strategy 'Postgres evaluation':
DROP TABLE IF EXISTS postgres_evaluation;
CREATE TABLE postgres_evaluation (row_index INT, varchar_8_val_w_spaces VARCHAR(8), jsonb_val_1 JSONB);
INSERT INTO postgres_evaluation VALUES (0, ' mAA m
	 '::VARCHAR(8), '{"a": 1}'::JSONB);
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Query using evaluation strategy 'Postgres evaluation':
SELECT
  jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))
FROM
  postgres_evaluation
WHERE ((jsonb_val_1 ? 'z'));
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- All assumed directly or indirectly involved characteristics: STRING_WITH_SPECIAL_SPACE_CHARS, ENUM_INVALID

Test details & reproducer

Test the consistency of Materialize against Postgres as an oracle.

bin/mzcompose --find postgres-consistency run default --seed=01938943-5c2f-4ad3-a336-20edde49d7d3 --max-runtime-in-sec=1200

Output consistency (feature-flags for DFR) failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-cf614c4d/materialize/nightly/test/feature-flag-consistency up --detach --wait --quiet-pull mz_this mz_other

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

feature-flag-consistency-mz_this-1   | thread 'tokio:work-3' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Verify that Materialize has the same results with different sets of feature flags.

bin/mzcompose --find feature-flag-consistency run default --seed=01938943-5c2f-4ad3-a336-20edde49d7d3 --max-runtime-in-sec=1200 --evaluation-strategy=dataflow_rendering

aarch64 Docker tags produced in this build

materialize/ubuntu-base:mzbuild-ZDXXLIQ25S3GH73GW5Z2GUWHBSKRJM2H
materialize/prod-base:mzbuild-CHBE2FQ26CD4W4LYZJAVIYFIWDLCN7CV
materialize/balancerd:mzbuild-HLGD3IYQVOF2HS333DDY5XN2CQ6UOIRG
materialize/bazel:mzbuild-FZKRDOTGX32G2GYY3KA3XRK3IKSJ7MTC
materialize/chbenchmark:mzbuild-7NPNIYHRXT32FYNDEGMUYTETXLMU7QOE
materialize/ci-java-smoketest:mzbuild-MKZBE2SKCGNNPZG5MUOEEK6TXVTGKFT7
materialize/ci-metabase-smoketest:mzbuild-4N6WH4X4JFVLQSZWWK3QNNONEAK3XZH4
materialize/cli:mzbuild-UCRQHCMBLRIBHA6ECCKWN7NNZHXL25AM
materialize/clusterd:mzbuild-4DTJ7RZOW67JQ3RRODPL5TMVHGLKIC23
materialize/dbt-materialize:mzbuild-VP456JYHE6ADYWWUQXFOUBG6O2NV3S2W
materialize/debezium:mzbuild-RMYH5YYIXAC4K3VHNK7GUKZQHCAW75CY
materialize/environmentd:mzbuild-3PNOGCAK5N44EOQXOZHV3C3FQOU4IR4G
materialize/fivetran-destination:mzbuild-OBABMCSTTIPRFIZDKJGUONCCWEZEXLIT
materialize/fivetran-destination-tester:mzbuild-J23TS6XJNMWPL54XNIRTT2ZRYAP5YHC3
materialize/frontegg-mock:mzbuild-VUXKSERUNVC37EWFLQ7WXENHNTUSSHJO
materialize/jobs:mzbuild-ADPM5CW5QKGNXGEKHEQ4JJK2YH5BKNXK
materialize/kgen:mzbuild-BCVDPEBAHLK62WUO5M2Z6TV6VPTC2P47
materialize/maelstrom-persist-base:mzbuild-3CJBQQPQ4FKCQVSFEMGDVDZJNULAZVLX
materialize/maelstrom-persist:mzbuild-L3UTGT6FEHOYWJN2XH7KQJK5B2UNLZ75
materialize/materialized-base:mzbuild-AZCBTDMB3PWA55MEWSZTOVV5X4VHG45B
materialize/materialized:mzbuild-F4VV3FF3KHEEMLERCJI7Z5CDOMPAAN55
materialize/test-certs:mzbuild-DMYTEM6GOQFJPGBWRBN64J43QKHEGEDV
materialize/mysql:mzbuild-R4Y33YICQZZBXKY3YVLNFTCLGMUIXZ6L
materialize/mysql-client:mzbuild-SWMKXCOA7WKJWQVS3GXW3P5SA3CFARZ2
materialize/mz:mzbuild-K55FYDPOPS5JOTM7YMX5HGABA7HPTEUI
materialize/orchestratord:mzbuild-3D2CQB5FISO6IIGI4PSTTSN5KT2SDCD2
materialize/postgres:mzbuild-JDGCJUVYM3EY3QFMYDITXTFGXWBMDC2P
materialize/psql:mzbuild-3DKPCLCCR67L6GTXQP7AVLLOFTD2S2RB
materialize/sqlancer:mzbuild-WAVV6X6XEU2XZWMWRVXNHMLKB5GTFOHL
materialize/sqllogictest:mzbuild-DQ7VDSWINYZRQIQBQOQTTDS4YVV4A7ZU
materialize/sqlsmith:mzbuild-Q4VF54SVOAXL5XQIPXXA4YYV22APHPW7
materialize/testdrive-base:mzbuild-PHWSUEUATBIJUZCY62U3HCBUELMRUBVA
materialize/testdrive:mzbuild-XWW6HCJ7VMUNNGWMOVUX3L5DVNIPZGNV

x86_64 Docker tags produced in this build

materialize/ubuntu-base:mzbuild-PQQKF2CIMPLOQIF3EKEORR2MSRE45LH7
materialize/prod-base:mzbuild-SXSSD7ZVWJC26HY7DWHI5C2JEAGECCOM
materialize/balancerd:mzbuild-JPMKUQDOW4GZIU4FBRMROI2PFCYC5G2E
materialize/bazel:mzbuild-YY3NO65KI5YODSGNEENXXT7RT2PLDWIL
materialize/chbenchmark:mzbuild-QO5LD7ISGBEKYW4QVD75NK2PVSPFX65M
materialize/ci-java-smoketest:mzbuild-DEETHZ24HV3C4HE2POD2YYXZUG5MMRMI
materialize/ci-metabase-smoketest:mzbuild-LZ4N6XFPLYD4RAU56ILKSCBCYJ5EQY4G
materialize/cli:mzbuild-LCLQETIWXNZQWQEMTUT3TIYGVJPLVMVT
materialize/clusterd:mzbuild-GYXDA3JYK4PNXPERBNVPS6T6MRDBYF4F
materialize/dbt-materialize:mzbuild-BQHB4IE5DGZQG3D47QWOADQ3LGFNIXBB
materialize/debezium:mzbuild-DH3UATRUW63NX4NC45OHBETXJZNLKBQ5
materialize/environmentd:mzbuild-DMATWNJ5A2HN7MAWA2PQUZHM5HJRPYMF
materialize/fivetran-destination:mzbuild-Y5H7BVOKB4D4EM2V2RHDGGU6QQHY7Z5M
materialize/fivetran-destination-tester:mzbuild-GLJTUDVZTGL74MOWQ3FI3ZUNDKCJJKIF
materialize/frontegg-mock:mzbuild-DZXGONL7JBRDFNIFOUMTFJX3D24VZAZH
materialize/jobs:mzbuild-SSRG63HR7ZJEF5WXAUEJMFAQFA56KF2G
materialize/kgen:mzbuild-YBXKTSV5VZ4DJJDAWRCBTAGJKNA7VZ2Z
materialize/maelstrom-persist-base:mzbuild-NUGWOFCBNZI2NYE5N6D6XMQYPHIZIRDC
materialize/maelstrom-persist:mzbuild-ULASN4RIJRJ6RHFAKIMSENMHYZA4YAYQ
materialize/materialized-base:mzbuild-6KUIUKIMFXYYIFIEHCLJEMQTG25XRAHK
materialize/materialized:mzbuild-CN7LAWKUVIV2GJLESR7LKLJHSZBTL3TZ
materialize/test-certs:mzbuild-4X7BSRXRRUJQJ2BDOTRBCML4IFZKFT5B
materialize/mysql:mzbuild-3CTOQTRNEZ2WGT3Z6G5AF24W6P54HQBL
materialize/mysql-client:mzbuild-GIRUSXDV2UGNOSJOIOGWHGZDSLFOFD5G
materialize/mz:mzbuild-EDYQRTDMFFNYMBMPJRIILOQ3CUEVXKJE
materialize/orchestratord:mzbuild-XTUODFODYCSCLEFBLB2R6EJN33F5KBNZ
materialize/postgres:mzbuild-6UIZ6QCFEVDPTZS5MIIIPL5AHVQ2O7TI
materialize/psql:mzbuild-5ENPLRAVYVOA3F77VITI4TXI55SLUEO3
materialize/sqlancer:mzbuild-GMHU6QUASFPVVAZTSW4POUVKE4IXDPGG
materialize/sqllogictest:mzbuild-CBFWRXKNVMICGYZQZF465GD62JRMFNZK
materialize/sqlsmith:mzbuild-FATDMKSEQGHTZWSNRHJGMHDJZBFFFP2W
materialize/testdrive-base:mzbuild-ZYPHOO5YEMZIULYBNWTVT662433VFNZ2
materialize/testdrive:mzbuild-5IIDY4ETWTR3T7AXTRF6FRSPTVXMV4IS

bin/ci-builder run stable bin/pyactivate -m ci.mkpipeline nightly "$@"

Ran in 58s

Build x86_64bin/ci-builder run stable bin/pyactivate -m ci.test.build

Ran in 45s

Build aarch64bin/ci-builder run stable bin/pyactivate -m ci.test.build

Ran in 43s

Unused dependenciesbin/ci-builder run nightly bin/unused-deps

Ran in 24m 54s

Miri test (full)

Ran in 2h 14m

1/8

Feature benchmark against merge base or 'latest' 1

Ran in 5h 7m

2/8

Feature benchmark against merge base or 'latest' 2

Ran in 6h 42m

3/8

Feature benchmark against merge base or 'latest' 3

Ran in 4h 42m

4/8

Feature benchmark against merge base or 'latest' 4

Ran in 3h 23m

5/8

Feature benchmark against merge base or 'latest' 5

Ran in 2h 34m

6/8

Feature benchmark against merge base or 'latest' 6

Ran in 2h 24m

7/8

Feature benchmark against merge base or 'latest' 7

Ran in 3h 38m

8/8

Feature benchmark against merge base or 'latest' 8

Ran in 5h 43m

Scalability benchmark (read & write) against merge base or 'latest'

Ran in 29m 36s

Scalability benchmark (read & write) against merge base or 'latest'

Ran in 1h 49m

Scalability benchmark (read & write) against merge base or 'latest'

Ran in 5m 38s

Scalability benchmark (DDL) against merge base or 'latest'

Ran in 1h 23m

Scalability benchmark (connection) against merge base or 'latest'

Ran in 25m 31s

Parallel Benchmark

Ran in 1h 21m

1/2

Kafka smoke test against previous Kafka versions 1

Ran in 48m 5s

2/2

Kafka smoke test against previous Kafka versions 2

Ran in 17m 30s

Kafka multi-broker test

Ran in 7m 21s

🐼 resumption tests

Ran in 13m 37s

🐼 🏎️ testdrive

Ran in 1h 2m

🏎️ testdrive with old Upsert

Ran in 1h 3m

🏎️ testdrive with --kafka-default-partitions 5

Ran in 1h 5m

🏎️ testdrive 4 replicas

Ran in 1h 6m

🏎️ testdrive with SIZE 1

Ran in 1h 11m

🏎️ testdrive with SIZE 8

Ran in 1h 10m

🏎️ testdrive with Structured Persist (Only Structured)

Ran in 1h 4m

1/2

Full Testdrive in Cloudtest (K8s) 1

Ran in 1h 17m

2/2

Full Testdrive in Cloudtest (K8s) 2

Ran in 1h 12m

1/2

Product limits 1

Ran in 59m 2s

2/2

Product limits 2

Ran in 37m 40s

Instance size limits

Ran in 8m 10s

1/2

Bounded Memory 1

Ran in 33m 1s

2/2

Bounded Memory 2

Ran in 15m 3s

Upsert (compaction enabled)

Ran in 17m 21s

Upsert (compaction disabled)

Ran in 17m 23s

Extended SSH connection tests

Ran in 23m 59s

Zippy Kafka Sources

Ran in 38m 48s

Zippy Kafka Sources with old Upsert

Ran in 39m 10s

Zippy Kafka Parallel Insert

Ran in 38m 47s

Zippy User Tables

Ran in 38m 55s

Zippy Postgres CDC

Ran in 38m 14s

Zippy MySQL CDC

Ran in 38m 44s

Zippy Debezium Postgres

Ran in 39m 12s

Zippy Cluster Replicas

Ran in 40m 15s

Zippy w/ latest CRDB

Ran in 39m 34s

Zippy w/ alter connection

Ran in 38m 59s

MySQL CDC tests (before source versioning)

Ran in 20m 55s

MySQL CDC resumption tests (before source versioning)

Ran in 44m 49s

MySQL RTR tests (before source versioning)

Ran in 6m 11s

Postgres CDC tests (before source versioning)

Ran in 18m 25s

Postgres CDC resumption tests (before source versioning)

Ran in 26m 37s

Postgres RTR tests (before source versioning)

Ran in 6m 13s

Testdrive %N (before Kafka source versioning)

Ran in 41m 41s

AWS (Real)

Ran in 2m 46s

AWS (Localstack)

Ran in 7m 1s

Secrets Local File

Ran in 5m 17s

1/2

Checks + restart of the entire Mz 1

Ran in 26m 10s

2/2

Checks + restart of the entire Mz 2

Ran in 26m 23s

1/2

Checks + restart of the entire Mz with old Upsert 1

Ran in 21m 9s

2/2

Checks + restart of the entire Mz with old Upsert 2

Ran in 19m 18s

1/2

Checks + backup + rollback to previous 1

Ran in 18m 19s

2/2

Checks + backup + rollback to previous 2

Ran in 15m 58s

1/2

Checks parallel + DROP/CREATE replica 1

Ran in 12m 40s

2/2

Checks parallel + DROP/CREATE replica 2

Ran in 18m 50s

1/2

Checks parallel + restart compute clusterd 1

Ran in 16m 9s

2/2

Checks parallel + restart compute clusterd 2

Ran in 13m 30s

1/2

Checks parallel + restart of the entire Mz 1

Ran in 12m 42s

2/2

Checks parallel + restart of the entire Mz 2

Ran in 13m 58s

1/2

Checks parallel + restart of environmentd & storage clusterd 1

Ran in 13m 50s

2/2

Checks parallel + restart of environmentd & storage clusterd 2

Ran in 40m 12s

1/2

Checks parallel + kill storage clusterd 1

Ran in 14m 14s

2/2

Checks parallel + kill storage clusterd 2

Ran in 12m 22s

1/2

Checks upgrade, whole-Mz restart 1

Ran in 23m 21s

2/2

Checks upgrade, whole-Mz restart 2

Ran in 29m 38s

1/2

Checks preflight-check and roll back upgrade 1

Ran in 23m 59s

2/2

Checks preflight-check and roll back upgrade 2

Ran in 25m 11s

1/2

Checks upgrade across two versions 1

Ran in 26m 1s

2/2

Checks upgrade across two versions 2

Ran in 23m 28s

1/2

Checks upgrade across four versions 1

Ran in 28m 55s

2/2

Checks upgrade across four versions 2

Ran in 28m 15s

1/2

Checks 0dt restart of the entire Mz with forced migrations 1

Ran in 35m 2s

2/2

Checks 0dt restart of the entire Mz with forced migrations 2

Ran in 32m 8s

1/2

Checks 0dt upgrade, whole-Mz restart 1

Ran in 31m 41s

2/2

Checks 0dt upgrade, whole-Mz restart 2

Ran in 1h 4m

2/2

Checks 0dt upgrade, whole-Mz restart 2

1/2

Checks 0dt upgrade across two versions 1

Ran in 30m 33s

2/2

Checks 0dt upgrade across two versions 2

Ran in 25m 44s

1/2

Checks 0dt upgrade across four versions 1

Ran in 33m 44s

2/2

Checks 0dt upgrade across four versions 2

Ran in 32m 13s

Checks 0dt upgrade to a bumped version

Ran in 54m 31s

Platform checks upgrade in Cloudtest/K8s

Ran in 59m 44s

K8s recovery: storage on failing node

Ran in 21m 14s

K8s recovery: compute on failing node

Ran in 21m 8s

K8s recovery: replicated compute on failing node

Ran in 21m 59s

K8s recovery: envd on failing node

Ran in 25m 54s

Maelstrom coverage of persist

Ran in 5m 55s

Long single-node Maelstrom coverage of persist

Ran in 15m 0s

Long multi-node Maelstrom coverage of persist with postgres consensus

Ran in 10m 18s

Maelstrom coverage of txn-wal

Ran in 10m 10s

SQL-level feature flags

Ran in 5m 45s

LaunchDarkly

Ran in 5m 28s

Mz E2E Test

Ran in 5m 4s

Output consistency (internal)

Ran in 27m 9s

Output consistency (Postgres)

Ran in 20m 41s

Output consistency (version for DFR)

Ran in 25m 55s

Output consistency (version for CTF)

Ran in 17m 11s

Output consistency (feature-flags for DFR)

Ran in 12m 34s

SQLsmith

Ran in 30m 28s

SQLsmith explain

Ran in 32m 13s

SQLancer PQS

Ran in 32m 16s

SQLancer NoREC

Ran in 30m 25s

SQLancer QueryPartitioning

Ran in 30m 26s

SQLancer Having

Ran in 31m 41s

RQG dbt3-joins workload

Ran in 37m 5s

RQG subqueries workload

Ran in 37m 58s

RQG banking workload

Ran in 36m 38s

RQG left join stacks workload

Ran in 10m 16s

CRDB rolling restarts

Ran in 15m 14s

PubSub disruption

Ran in 21m 20s

1/2

Data Ingest 1

Ran in 48m 14s

2/2

Data Ingest 2

Ran in 27m 37s

Parallel Workload (DML)

Ran in 33m 38s

Parallel Workload (DDL)

Ran in 31m 12s

Parallel Workload (DDL Only)

Ran in 34m 2s

Parallel Workload (rename + naughty identifiers)

Ran in 35m 39s

Parallel Workload (rename)

Ran in 32m 30s

Parallel Workload (kill)

Ran in 30m 11s

Parallel Workload (0dt deploy)

Ran in 38m 34s

Tests for balancerd

Ran in 10m 17s

Legacy upgrade tests (last version from git)

Ran in 20m 39s

Cloudtest

Ran in 32m 8s

Slow Cloudtest

Ran in 26m 28s

1/2

Txn-wal fencing 1

Ran in 15m 50s

2/2

Txn-wal fencing 2

Ran in 21m 7s

Copy To S3 (1 replica)

Ran in 18m 6s

Copy To S3 (2 replicas)

Ran in 17m 18s

CRDB / Persist backup and restore

Ran in 21m 49s

Postgres / Persist backup and restore

Ran in 22m 40s

Replica isolation

Ran in 17m 11s

Zero downtime

Ran in 1h 29m

tests

Ran in 6m 43s

tests

Ran in 5m 30s

tests

Ran in 5m 22s

tests

Ran in 6m 56s

tests

Ran in 6m 27s

Total Job Run Time: 4d 7h