Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#10909
/
main/4f132c8411
Passed in 7h 16m
:pipeline:
:rust: Miri test (full)
Extended SSH connection tests
CRDB rolling restarts
PubSub disruption
Tests for balancerd
CRDB / Persist backup and restore
Postgres / Persist backup and restore
Replica isolation
Zero downtime
Materialize Emulator
Cluster tests with :azure: blob store
Created
Triggered from Pipeline Schedule

Data Ingest 1 (#2) failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Jan 18 09:19:31 hetzner-aarch64-8cpu-16gb-a4d6717e kernel: Memory cgroup out of memory: Killed process 127833 (clusterd) total-vm:23979460kB, anon-rss:12874032kB, file-rss:98284kB, shmem-rss:0kB, UID:999 pgtables:42240kB oom_score_adj:0
Test details & reproducer Test that ingests large amounts of data from Kafka/Postgres/MySQL and verifies that Materialize can handle it correctly by comparing the results. 
BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find data-ingest run default --azurite

Scalability benchmark (read & write) against merge base or 'latest' failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

New regression against v0.130.0 (55aad73b1)

Regression in workload 'SelectLimitWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 574.94 tps vs. 902.27 tps (-327.33 tps; -36.28%)
Test details & reproducer Benchmark for how various queries scale, compares against old Materialize versions. 
bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256

Terraform + Helm Chart E2E on AWS failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve']' returned non-zero exit status 1.
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run default

Data Ingest 1 failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

psycopg.errors.ConnectionTimeout: connection timeout expired
Test details & reproducer Test that ingests large amounts of data from Kafka/Postgres/MySQL and verifies that Materialize can handle it correctly by comparing the results. 
BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find data-ingest run default --azurite
:pipeline:bin/ci-builder run stable bin/pyactivate -m ci.mkpipeline nightly "$@"
Waited 10s
·
Ran in 1m 47s
Agent
Detect references to already closed issuesbin/ci-builder run stable bin/ci-closed-issues-detect
Waited 8s
·
Ran in 1m 5s
Agent
Unused dependenciesbin/ci-builder run nightly bin/unused-deps
Waited 10s
·
Ran in 22m 25s
Agent
:rust: Miri test (full)
Waited 1m 51s
·
Ran in 2h 43m
Agent
1/8
Feature benchmark against merge base or 'latest' 1
Waited 1m 21s
·
Ran in 2h 27m
Agent
2/8
Feature benchmark against merge base or 'latest' 2
Waited 1m 21s
·
Ran in 4h 45m
Agent
3/8
Feature benchmark against merge base or 'latest' 3
Waited 1m 22s
·
Ran in 4h 5m
Agent
4/8
Feature benchmark against merge base or 'latest' 4
Waited 1m 22s
·
Ran in 2h 54m
Agent
5/8
Feature benchmark against merge base or 'latest' 5
Waited 1m 22s
·
Ran in 2h 15m
Agent
6/8
Feature benchmark against merge base or 'latest' 6
Waited 1m 24s
·
Ran in 2h 18m
Agent
7/8
Feature benchmark against merge base or 'latest' 7
Waited 1m 24s
·
Ran in 2h 36m
Agent
8/8
Feature benchmark against merge base or 'latest' 8
Waited 1m 25s
·
Ran in 4h 37m
Agent
Scalability benchmark (read & write) against merge base or 'latest'
Waited 1m 28s
·
Ran in 1h 43m
Agent
Scalability benchmark (read & write) against merge base or 'latest'
Waited 1m 26s
·
Ran in 1h 35m
Agent
Scalability benchmark (DDL) against merge base or 'latest'
Waited 10s
·
Ran in 1h 25m
Agent
Scalability benchmark (connection) against merge base or 'latest'
Waited 1m 20s
·
Ran in 26m 31s
Agent
Parallel Benchmark
Waited 1m 21s
·
Ran in 1h 23m
Agent
1/2
Kafka smoke test against previous Kafka versions 1
Waited 8s
·
Ran in 39m 3s
Agent
2/2
Kafka smoke test against previous Kafka versions 2
Waited 1m 14s
·
Ran in 12m 55s
Agent
Kafka multi-broker test
Waited 2m 21s
·
Ran in 8m 9s
Agent
🐼 resumption tests
Waited 2s
·
Ran in 10m 14s
Agent
🐼 🏎️ testdrive
Waited 2s
·
Ran in 55m 43s
Agent
🏎️ testdrive with old Upsert
Waited 3s
·
Ran in 56m 16s
Agent
🏎️ testdrive with --kafka-default-partitions 5
Waited 7s
·
Ran in 1h 4m
Agent
🏎️ testdrive 4 replicas
Waited 1m 51s
·
Ran in 1h 3m
Agent
🏎️ testdrive with SIZE 1
Waited 2m 21s
·
Ran in 1h 0m
Agent
🏎️ testdrive with SIZE 8
Waited 1m 53s
·
Ran in 1h 2m
Agent
🏎️ testdrive with Structured Persist (Only Structured)
Waited 1m 54s
·
Ran in 55m 55s
Agent
testdrive with :azure: blob store
Waited 1m 34s
·
Ran in 1h 21m
Agent
🏎️ testdrive 4 replicas with :azure: blob store
Waited 1m 56s
·
Ran in 1h 35m
Agent
🏎️ testdrive with SIZE 8 and :azure: blob store
Waited 1m 56s
·
Ran in 1h 12m
Agent
1/2
Full Testdrive in Cloudtest (K8s) 1
Waited 50s
·
Ran in 1h 6m
Agent
2/2
Full Testdrive in Cloudtest (K8s) 2
Waited 52s
·
Ran in 1h 6m
Agent
1/2
Product limits 1
Waited 1m 57s
·
Ran in 1h 21m
Agent
2/2
Product limits 2
Waited 1m 58s
·
Ran in 41m 6s
Agent
Instance size limits
Waited 2m 0s
·
Ran in 9m 34s
Agent
1/2
Bounded Memory 1
Waited 2m 18s
·
Ran in 30m 59s
Agent
2/2
Bounded Memory 2
Waited 2m 30s
·
Ran in 16m 26s
Agent
Upsert (compaction enabled)
Waited 2m 21s
·
Ran in 17m 49s
Agent
Upsert (compaction disabled)
Waited 2m 22s
·
Ran in 18m 13s
Agent
Extended SSH connection tests
Waited 11s
·
Ran in 17m 21s
Agent
Zippy Kafka Sources
Waited 2m 23s
·
Ran in 40m 33s
Agent
Zippy Kafka Sources with :azure: blob store
Waited 2m 25s
·
Ran in 40m 41s
Agent
Zippy Kafka Sources with old Upsert
Waited 2m 27s
·
Ran in 40m 51s
Agent
Zippy Kafka Sources with old Upsert and :azure: blob store
Waited 2m 28s
·
Ran in 44m 25s
Agent
Zippy Kafka Parallel Insert
Waited 2m 49s
·
Ran in 39m 44s
Agent
Zippy Kafka Parallel Insert with :azurite: blob store
Waited 2m 50s
·
Ran in 39m 21s
Agent
Zippy User Tables
Waited 2m 55s
·
Ran in 40m 43s
Agent
Zippy Postgres CDC
Waited 3m 12s
·
Ran in 40m 0s
Agent
Zippy MySQL CDC
Waited 7m 34s
·
Ran in 33m 11s
Agent
Zippy Debezium Postgres
Waited 10m 4s
·
Ran in 33m 48s
Agent
Zippy Cluster Replicas
Waited 10m 17s
·
Ran in 35m 47s
Agent
Zippy w/ latest CRDB
Waited 11m 44s
·
Ran in 34m 47s
Agent
Zippy w/ alter connection
Waited 2m 0s
·
Ran in 39m 28s
Agent
MySQL CDC tests (before source versioning)
Waited 2m 23s
·
Ran in 21m 2s
Agent
MySQL CDC resumption tests (before source versioning)
Waited 2m 23s
·
Ran in 46m 0s
Agent
MySQL RTR tests (before source versioning)
Waited 2m 24s
·
Ran in 2m 20s
Agent
Postgres CDC tests (before source versioning)
Waited 2m 25s
·
Ran in 21m 12s
Agent
Postgres CDC resumption tests (before source versioning)
Waited 2m 25s
·
Ran in 27m 31s
Agent
Postgres RTR tests (before source versioning)
Waited 2m 25s
·
Ran in 7m 7s
Agent
Testdrive (before Kafka source versioning)
Waited 11m 52s
·
Ran in 34m 36s
Agent
AWS (Real)
Waited 50s
·
Ran in 5m 13s
Agent
AWS (Localstack)
Waited 2m 25s
·
Ran in 7m 41s
Agent
Secrets Local File
Waited 2m 26s
·
Ran in 6m 10s
Agent
1/2
Checks without restart or upgrade with :azure: blob store 1
Waited 2m 10s
·
Ran in 29m 3s
Agent
2/2
Checks without restart or upgrade with :azure: blob store 2
Waited 2m 10s
·
Ran in 23m 23s
Agent
1/2
Checks + restart of the entire Mz 1
Waited 2m 3s
·
Ran in 22m 34s
Agent
2/2
Checks + restart of the entire Mz 2
Waited 2m 3s
·
Ran in 21m 6s
Agent
1/2
Checks + restart of environmentd & storage clusterd with :azure: blob store 1
Waited 2m 13s
·
Ran in 41m 33s
Agent
2/2
Checks + restart of environmentd & storage clusterd with :azure: blob store 2
Waited 2m 4s
·
Ran in 38m 16s
Agent
1/2
Checks + restart of the entire Mz with old Upsert 1
Waited 2m 4s
·
Ran in 21m 46s
Agent
2/2
Checks + restart of the entire Mz with old Upsert 2
Waited 2m 4s
·
Ran in 19m 44s
Agent
1/2
Checks + backup + rollback to previous 1
Waited 2m 5s
·
Ran in 19m 51s
Agent
2/2
Checks + backup + rollback to previous 2
Waited 2m 5s
·
Ran in 17m 33s
Agent
1/2
Checks parallel + DROP/CREATE replica 1
Waited 2m 5s
·
Ran in 20m 59s
Agent
2/2
Checks parallel + DROP/CREATE replica 2
Waited 2m 15s
·
Ran in 13m 40s
Agent
1/2
Checks parallel + restart compute clusterd 1
Waited 2m 5s
·
Ran in 14m 27s
Agent
2/2
Checks parallel + restart compute clusterd 2
Waited 2m 5s
·
Ran in 14m 9s
Agent
1/2
Checks parallel + restart of the entire Mz 1
Waited 2m 6s
·
Ran in 15m 12s
Agent
2/2
Checks parallel + restart of the entire Mz 2
Waited 2m 6s
·
Ran in 14m 8s
Agent
1/2
Checks parallel + restart of the entire Mz with :azure: blob store 1
Waited 2m 7s
·
Ran in 14m 18s
Agent
2/2
Checks parallel + restart of the entire Mz with :azure: blob store 2
Waited 2m 8s
·
Ran in 16m 50s
Agent
1/2
Checks parallel + restart of environmentd & storage clusterd 1
Waited 2m 8s
·
Ran in 15m 17s
Agent
2/2
Checks parallel + restart of environmentd & storage clusterd 2
Waited 2m 8s
·
Ran in 16m 37s
Agent
1/2
Checks parallel + kill storage clusterd 1
Waited 2m 9s
·
Ran in 13m 58s
Agent
2/2
Checks parallel + kill storage clusterd 2
Waited 2m 10s
·
Ran in 13m 33s
Agent
1/2
Checks upgrade, whole-Mz restart 1
Waited 2m 10s
·
Ran in 35m 20s
Agent
2/2
Checks upgrade, whole-Mz restart 2
Waited 2m 10s
·
Ran in 31m 23s
Agent
1/2
Checks preflight-check and roll back upgrade 1
Waited 2m 10s
·
Ran in 23m 56s
Agent
2/2
Checks preflight-check and roll back upgrade 2
Waited 2m 11s
·
Ran in 23m 11s
Agent
1/2
Checks upgrade across two versions 1
Waited 2m 11s
·
Ran in 22m 41s
Agent
2/2
Checks upgrade across two versions 2
Waited 2m 11s
·
Ran in 36m 32s
Agent
1/2
Checks upgrade across four versions 1
Waited 2m 11s
·
Ran in 25m 5s
Agent
2/2
Checks upgrade across four versions 2
Waited 2m 11s
·
Ran in 23m 51s
Agent
1/2
Checks 0dt restart of the entire Mz with forced migrations with :azure: blob store 1
Waited 2m 12s
·
Ran in 27m 16s
Agent
2/2
Checks 0dt restart of the entire Mz with forced migrations with :azure: blob store 2
Waited 2m 13s
·
Ran in 33m 2s
Agent
1/2
Checks 0dt upgrade, whole-Mz restart 1
Waited 2m 13s
·
Ran in 24m 56s
Agent
2/2
Checks 0dt upgrade, whole-Mz restart 2
Waited 2m 14s
·
Ran in 26m 10s
Agent
1/2
Checks 0dt upgrade across two versions 1
Waited 2m 15s
·
Ran in 38m 9s
Agent
2/2
Checks 0dt upgrade across two versions 2
Waited 2m 15s
·
Ran in 38m 8s
Agent
1/2
Checks 0dt upgrade across four versions 1
Waited 2m 16s
·
Ran in 29m 21s
Agent
2/2
Checks 0dt upgrade across four versions 2
Waited 2m 17s
·
Ran in 27m 49s
Agent
Checks 0dt upgrade to a bumped version
Waited 1m 23s
·
Ran in 56m 5s
Agent
K8s recovery: storage on failing node
Waited 50s
·
Ran in 22m 6s
Agent
K8s recovery: compute on failing node
Waited 51s
·
Ran in 21m 39s
Agent
K8s recovery: replicated compute on failing node
Waited 51s
·
Ran in 19m 52s
Agent
K8s recovery: envd on failing node
Waited 45s
·
Ran in 26m 39s
Agent
Maelstrom coverage of persist
Waited 2m 27s
·
Ran in 6m 35s
Agent
Long single-node Maelstrom coverage of persist
Waited 2m 29s
·
Ran in 15m 44s
Agent
Long multi-node Maelstrom coverage of persist with postgres consensus
Waited 2m 29s
·
Ran in 11m 3s
Agent
Maelstrom coverage of txn-wal
Waited 2m 30s
·
Ran in 11m 3s
Agent
SQL-level feature flags
Waited 2m 30s
·
Ran in 6m 16s
Agent
LaunchDarkly
Waited 55s
·
Ran in 5m 50s
Agent
Canary Deploy in Staging Cloud
Waited 56s
·
Ran in 1h 45m
Agent
Mz E2E Test
Waited 1m 3s
·
Ran in 17m 32s
Agent
Terraform + Helm Chart E2E on AWS
Waited 1m 6s
·
Ran in 47m 23s
Agent
Terraform + Helm Chart E2E on AWS
Waited 44s
·
Ran in 37m 48s
Agent
SQLsmith
Waited 2m 30s
·
Ran in 31m 3s
Agent
SQLsmith explain
Waited 2m 31s
·
Ran in 31m 1s
Agent
SQLancer PQS
Waited 2m 31s
·
Ran in 30m 56s
Agent
SQLancer NoREC
Waited 2m 32s
·
Ran in 31m 13s
Agent
SQLancer QueryPartitioning
Waited 2m 32s
·
Ran in 30m 54s
Agent
SQLancer Having
Waited 2m 32s
·
Ran in 30m 56s
Agent
RQG dbt3-joins workload
Waited 2m 18s
·
Ran in 43m 35s
Agent
RQG subqueries workload
Waited 2m 32s
·
Ran in 36m 53s
Agent
RQG banking workload
Waited 2m 32s
·
Ran in 37m 16s
Agent
RQG left join stacks workload
Waited 2m 33s
·
Ran in 10m 7s
Agent
CRDB rolling restarts
Waited 2s
·
Ran in 20m 15s
Agent
PubSub disruption
Waited 9s
·
Ran in 16m 39s
Agent
1/2
Data Ingest 1
Waited 11s
·
Ran in 27m 46s
Agent
1/2
Data Ingest 1
Waited 1m 54s
·
Ran in 58m 56s
Agent
1/2
Data Ingest 1
Waited 2m 10s
·
Ran in 48m 36s
Agent
2/2
Data Ingest 2
Waited 8s
·
Ran in 23m 8s
Agent
Parallel Workload (DML)
Waited 2m 18s
·
Ran in 33m 25s
Agent
Parallel Workload (DDL) with :azure: blob store
Waited 2m 18s
·
Ran in 34m 38s
Agent
Parallel Workload (DDL Only)
Waited 2m 19s
·
Ran in 33m 57s
Agent
Parallel Workload (rename + naughty identifiers)
Waited 2m 30s
·
Ran in 33m 11s
Agent
Parallel Workload (rename) with :azure: blob store
Waited 2m 20s
·
Ran in 33m 56s
Agent
Parallel Workload (kill)
Waited 2m 30s
·
Ran in 34m 4s
Agent
Parallel Workload (0dt deploy)
Waited 1m 42s
·
Ran in 35m 28s
Agent
Tests for balancerd
Waited 9s
·
Ran in 7m 16s
Agent
Legacy upgrade tests (last version from git)
Waited 2m 33s
·
Ran in 21m 5s
Agent
1/2
Legacy upgrade tests (last version from docs) 1
Waited 2m 33s
·
Ran in 13m 43s
Agent
2/2
Legacy upgrade tests (last version from docs) 2
Waited 2m 33s
·
Ran in 13m 21s
Agent
Cloudtest
Waited 1m 14s
·
Ran in 32m 41s
Agent
Slow Cloudtest
Waited 1m 48s
·
Ran in 24m 48s
Agent
1/2
Txn-wal fencing 1
Waited 9s
·
Ran in 9m 56s
Agent
2/2
Txn-wal fencing 2
Waited 2s
·
Ran in 16m 52s
Agent
Copy To S3 (1 replica)
Waited 2m 20s
·
Ran in 25m 35s
Agent
Copy To S3 (2 replicas)
Waited 2m 21s
·
Ran in 25m 36s
Agent
CRDB / Persist backup and restore
Waited 3s
·
Ran in 17m 23s
Agent
Postgres / Persist backup and restore
Waited 3s
·
Ran in 17m 9s
Agent
Replica isolation
Waited 1s
·
Ran in 11m 52s
Agent
Zero downtime
Waited 2m 23s
·
Ran in 1h 43m
Agent
Materialize Emulator
Waited 1s
·
Ran in 1m 33s
Agent
1/10
💡 SQL logic tests 1
Waited 2m 24s
·
Ran in 1h 39m
Agent
2/10
💡 SQL logic tests 2
Waited 2m 25s
·
Ran in 2h 5m
Agent
3/10
💡 SQL logic tests 3
Waited 2m 25s
·
Ran in 1h 59m
Agent
4/10
💡 SQL logic tests 4
Waited 2m 26s
·
Ran in 1h 59m
Agent
5/10
💡 SQL logic tests 5
Waited 2m 26s
·
Ran in 1h 51m
Agent
6/10
💡 SQL logic tests 6
Waited 2m 26s
·
Ran in 2h 5m
Agent
7/10
💡 SQL logic tests 7
Waited 2m 26s
·
Ran in 2h 0m
Agent
8/10
💡 SQL logic tests 8
Waited 2m 27s
·
Ran in 1h 40m
Agent
9/10
💡 SQL logic tests 9
Waited 2m 29s
·
Ran in 1h 45m
Agent
10/10
💡 SQL logic tests 10
Waited 9m 8s
·
Ran in 2h 9m
Agent
Cluster tests with :azure: blob store
Waited 9s
·
Ran in 51m 25s
Agent
:csharp: tests
Waited 2m 34s
·
Ran in 7m 23s
Agent
:javascript: tests
Waited 2m 44s
·
Ran in 6m 22s
Agent
:java: tests
Waited 2m 34s
·
Ran in 5m 52s
Agent
:python: tests
Waited 2m 36s
·
Ran in 6m 13s
Agent
:ruby: tests
Waited 2m 36s
·
Ran in 6m 41s
Agent
Total Job Run Time: 5d 6h