Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Refactor FetchedPart to avoid as much decoding as possible

#10920

bkirwi:structured-on-disk-2/43bc2c247b(#31080)

Passed in 7h 50m

Miri test (full)

Extended SSH connection tests

CRDB rolling restarts

PubSub disruption

Tests for balancerd

CRDB / Persist backup and restore

Postgres / Persist backup and restore

Ben Kirwin

Created Mon 20th Jan at 8:22 PM

Triggered from Web

Data Ingest 1 failed, main history:

Unknown error in workflow-default:

builtins.AssertionError:

Test details & reproducer

Test that ingests large amounts of data from Kafka/Postgres/MySQL and verifies that Materialize can handle it correctly by comparing the results.

BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find data-ingest run default --azurite

bin/ci-builder run stable bin/pyactivate -m ci.mkpipeline nightly "$@"

Ran in 1m 50s

Miri test (full)

Ran in 2h 43m

1/8

Feature benchmark against merge base or 'latest' 1

Ran in 2h 30m

2/8

Feature benchmark against merge base or 'latest' 2

Ran in 4h 36m

3/8

Feature benchmark against merge base or 'latest' 3

Ran in 4h 12m

4/8

Feature benchmark against merge base or 'latest' 4

Ran in 3h 3m

5/8

Feature benchmark against merge base or 'latest' 5

Ran in 2h 21m

6/8

Feature benchmark against merge base or 'latest' 6

Ran in 2h 21m

7/8

Feature benchmark against merge base or 'latest' 7

Ran in 3h 4m

8/8

Feature benchmark against merge base or 'latest' 8

Ran in 6h 41m

Scalability benchmark (read & write) against merge base or 'latest'

Ran in 2h 4m

Scalability benchmark (DDL) against merge base or 'latest'

Ran in 1h 33m

Scalability benchmark (connection) against merge base or 'latest'

Ran in 28m 4s

Parallel Benchmark

Ran in 1h 24m

1/2

Kafka smoke test against previous Kafka versions 1

Ran in 55m 50s

2/2

Kafka smoke test against previous Kafka versions 2

Ran in 13m 44s

Kafka multi-broker test

Ran in 8m 8s

🐼 resumption tests

Ran in 10m 17s

🐼 🏎️ testdrive

Ran in 1h 4m

🏎️ testdrive with old Upsert

Ran in 1h 6m

🏎️ testdrive with --kafka-default-partitions 5

Ran in 1h 19m

🏎️ testdrive 4 replicas

Ran in 1h 0m

🏎️ testdrive with SIZE 1

Ran in 58m 10s

🏎️ testdrive with SIZE 8

Ran in 54m 41s

🏎️ testdrive with Structured Persist (Only Structured)

Ran in 55m 12s

testdrive with

blob store

Ran in 1h 7m

🏎️ testdrive 4 replicas with

blob store

Ran in 1h 35m

🏎️ testdrive with SIZE 8 and

blob store

Ran in 1h 7m

1/2

Full Testdrive in Cloudtest (K8s) 1

Ran in 1h 6m

2/2

Full Testdrive in Cloudtest (K8s) 2

Ran in 1h 5m

1/2

Product limits 1

Ran in 1h 4m

2/2

Product limits 2

Ran in 38m 3s

Instance size limits

Ran in 9m 6s

1/2

Bounded Memory 1

Ran in 32m 19s

2/2

Bounded Memory 2

Ran in 15m 37s

Upsert (compaction enabled)

Ran in 17m 39s

Upsert (compaction disabled)

Ran in 17m 53s

Extended SSH connection tests

Ran in 16m 23s

Zippy Kafka Sources

Ran in 40m 22s

Zippy Kafka Sources with

blob store

Ran in 42m 13s

Zippy Kafka Sources with old Upsert

Ran in 41m 18s

Zippy Kafka Sources with old Upsert and

blob store

Ran in 42m 32s

Zippy Kafka Parallel Insert

Ran in 40m 26s

Zippy Kafka Parallel Insert with :azurite: blob store

Ran in 40m 11s

Zippy User Tables

Ran in 41m 34s

Zippy Postgres CDC

Ran in 39m 2s

Zippy MySQL CDC

Ran in 39m 5s

Zippy Debezium Postgres

Ran in 36m 10s

Zippy Cluster Replicas

Ran in 44m 38s

Zippy w/ latest CRDB

Ran in 42m 46s

Zippy w/ alter connection

Ran in 39m 33s

MySQL CDC tests (before source versioning)

Ran in 21m 26s

MySQL CDC resumption tests (before source versioning)

Ran in 46m 51s

MySQL RTR tests (before source versioning)

Ran in 6m 50s

Postgres CDC tests (before source versioning)

Ran in 21m 47s

Postgres CDC resumption tests (before source versioning)

Ran in 28m 45s

Postgres RTR tests (before source versioning)

Ran in 7m 12s

Testdrive (before Kafka source versioning)

Ran in 1h 5m

AWS (Real)

Ran in 5m 14s

AWS (Localstack)

Ran in 8m 0s

Secrets Local File

Ran in 5m 53s

1/2

Checks without restart or upgrade with

blob store 1 1

Ran in 20m 17s

2/2

Checks without restart or upgrade with

blob store 2 2

Ran in 25m 18s

1/2

Checks + restart of the entire Mz 1

Ran in 22m 12s

2/2

Checks + restart of the entire Mz 2

Ran in 22m 43s

1/2

Checks + restart of environmentd & storage clusterd with

blob store 1 1

Ran in 27m 46s

2/2

Checks + restart of environmentd & storage clusterd with

blob store 2 2

Ran in 27m 12s

1/2

Checks + restart of the entire Mz with old Upsert 1

Ran in 20m 48s

2/2

Checks + restart of the entire Mz with old Upsert 2

Ran in 20m 2s

1/2

Checks + backup + rollback to previous 1

Ran in 17m 39s

2/2

Checks + backup + rollback to previous 2

Ran in 18m 15s

1/2

Checks parallel + DROP/CREATE replica 1

Ran in 14m 2s

2/2

Checks parallel + DROP/CREATE replica 2

Ran in 13m 33s

1/2

Checks parallel + restart compute clusterd 1

Ran in 13m 29s

2/2

Checks parallel + restart compute clusterd 2

Ran in 13m 35s

1/2

Checks parallel + restart of the entire Mz 1

Ran in 13m 17s

2/2

Checks parallel + restart of the entire Mz 2

Ran in 14m 23s

1/2

Checks parallel + restart of the entire Mz with

blob store 1 1

Ran in 14m 43s

2/2

Checks parallel + restart of the entire Mz with

blob store 2 2

Ran in 20m 15s

1/2

Checks parallel + restart of environmentd & storage clusterd 1

Ran in 14m 0s

2/2

Checks parallel + restart of environmentd & storage clusterd 2

Ran in 14m 28s

1/2

Checks parallel + kill storage clusterd 1

Ran in 14m 41s

2/2

Checks parallel + kill storage clusterd 2

Ran in 13m 56s

1/2

Checks upgrade, whole-Mz restart 1

Ran in 25m 22s

2/2

Checks upgrade, whole-Mz restart 2

Ran in 27m 2s

1/2

Checks preflight-check and roll back upgrade 1

Ran in 28m 20s

2/2

Checks preflight-check and roll back upgrade 2

Ran in 26m 17s

1/2

Checks upgrade across two versions 1

Ran in 24m 51s

2/2

Checks upgrade across two versions 2

Ran in 29m 32s

1/2

Checks upgrade across four versions 1

Ran in 30m 16s

2/2

Checks upgrade across four versions 2

Ran in 28m 17s

1/2

Checks 0dt restart of the entire Mz with forced migrations with

blob store 1

Ran in 28m 11s

2/2

Checks 0dt restart of the entire Mz with forced migrations with

blob store 2

Ran in 41m 11s

1/2

Checks 0dt upgrade, whole-Mz restart 1

Ran in 25m 40s

2/2

Checks 0dt upgrade, whole-Mz restart 2

Ran in 27m 30s

1/2

Checks 0dt upgrade across two versions 1

Ran in 23m 20s

2/2

Checks 0dt upgrade across two versions 2

Ran in 24m 32s

1/2

Checks 0dt upgrade across four versions 1

Ran in 33m 27s

2/2

Checks 0dt upgrade across four versions 2

Ran in 33m 18s

Checks 0dt upgrade to a bumped version

Ran in 56m 53s

K8s recovery: storage on failing node

Ran in 21m 52s

K8s recovery: compute on failing node

Ran in 21m 56s

K8s recovery: replicated compute on failing node

Ran in 22m 20s

K8s recovery: envd on failing node

Ran in 26m 56s

Maelstrom coverage of persist

Ran in 7m 7s

Long single-node Maelstrom coverage of persist

Ran in 16m 1s

Long multi-node Maelstrom coverage of persist with postgres consensus

Ran in 10m 48s

Maelstrom coverage of txn-wal

Ran in 10m 49s

SQL-level feature flags

Ran in 7m 7s

LaunchDarkly

Ran in 5m 57s

Mz E2E Test

Ran in 17m 35s

SQLsmith

Ran in 31m 38s

SQLsmith explain

Ran in 31m 21s

SQLancer PQS

Ran in 31m 22s

SQLancer NoREC

Ran in 31m 46s

SQLancer QueryPartitioning

Ran in 31m 6s

SQLancer Having

Ran in 31m 37s

RQG dbt3-joins workload

Ran in 37m 45s

RQG subqueries workload

Ran in 37m 24s

RQG banking workload

Ran in 37m 48s

RQG left join stacks workload

Ran in 11m 24s

CRDB rolling restarts

Ran in 8m 24s

PubSub disruption

Ran in 19m 38s

1/2

Data Ingest 1

Ran in 1h 27m

1/2

Data Ingest 1

Ran in 1h 32m

1/2

Data Ingest 1

Ran in 59m 57s

2/2

Data Ingest 2

Ran in 23m 0s

Parallel Workload (DML)

Ran in 36m 19s

Parallel Workload (DDL) with

blob store

Ran in 34m 27s

Parallel Workload (DDL Only)

Ran in 34m 20s

Parallel Workload (rename + naughty identifiers)

Ran in 33m 39s

Parallel Workload (rename) with

blob store

Ran in 34m 44s

Parallel Workload (kill)

Ran in 34m 3s

Parallel Workload (0dt deploy)

Ran in 41m 50s

Tests for balancerd

Ran in 7m 40s

Legacy upgrade tests (last version from git)

Ran in 23m 25s

1/2

Legacy upgrade tests (last version from docs) 1

Ran in 19m 22s

2/2

Legacy upgrade tests (last version from docs) 2

Ran in 19m 9s

Cloudtest

Ran in 34m 57s

Slow Cloudtest

Ran in 24m 48s

1/2

Txn-wal fencing 1

Ran in 9m 50s

2/2

Txn-wal fencing 2

Ran in 17m 20s

Copy To S3 (1 replica)

Ran in 25m 55s

Copy To S3 (2 replicas)

Ran in 27m 3s

CRDB / Persist backup and restore

Ran in 17m 41s

Postgres / Persist backup and restore

Ran in 18m 5s

Replica isolation

Ran in 24m 40s

Zero downtime

Ran in 1h 36m

Materialize Emulator

Ran in 9m 15s

1/10

💡 SQL logic tests 1

Ran in 1h 38m

2/10

💡 SQL logic tests 2

Ran in 1h 37m

3/10

💡 SQL logic tests 3

Ran in 1h 54m

4/10

💡 SQL logic tests 4

Ran in 1h 50m

5/10

💡 SQL logic tests 5

Ran in 1h 39m

6/10

💡 SQL logic tests 6

Ran in 1h 58m

7/10

💡 SQL logic tests 7

Ran in 1h 51m

8/10

💡 SQL logic tests 8

Ran in 1h 48m

9/10

💡 SQL logic tests 9

Ran in 1h 50m

10/10

💡 SQL logic tests 10

Ran in 1h 30m

Cluster tests with

blob store

Ran in 52m 22s

Total Job Run Time: 5d 4h