Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#11704
/
main/1d4d92a69c
Failed in 5h 20m
:pipeline:
:rust: Miri test (full)
Extended SSH connection tests
CRDB rolling restarts
PubSub disruption
Test for incident 70
Tests for balancerd
CRDB / Persist backup and restore
Postgres / Persist backup and restore
Replica isolation
Zero downtime
Materialize Emulator
Cluster tests with :azure: blob store
Created
Triggered from Pipeline Schedule

Scalability benchmark (read & write) against merge base or 'latest' failed, main history: :bk-status-passed::bk-status-passed::bk-status-failed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-x86-64-dedi-8cpu-32gb-b2f29476/materialize/nightly/test/scalability up --detach --wait --quiet-pull materialized
Test details & reproducer Benchmark for how various queries scale, compares against old Materialize versions. 
bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256

Terraform + Helm Chart E2E on GCP failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

builtins.ValueError: Never completed
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run gcp-temporary

Terraform + Helm Chart E2E on Azure failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

builtins.ValueError: Never completed
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run azure-temporary

Terraform + Helm Chart E2E on AWS failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

builtins.ValueError: Never completed
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run aws-temporary

Parallel Workload (DML) failed, main history: :bk-status-passed::bk-status-passed::bk-status-failed::bk-status-passed::bk-status-passed:

Apr 03 23:48:12 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 25605 (clusterd) total-vm:40138328kB, anon-rss:13376372kB, file-rss:109956kB, shmem-rss:0kB, UID:999 pgtables:50672kB oom_score_adj:0

Apr 03 23:48:47 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 25609 (clusterd) total-vm:42184472kB, anon-rss:12031916kB, file-rss:110436kB, shmem-rss:0kB, UID:999 pgtables:62516kB oom_score_adj:0

Apr 03 23:48:47 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 85869 (clusterd) total-vm:35904344kB, anon-rss:14480708kB, file-rss:108056kB, shmem-rss:0kB, UID:999 pgtables:43172kB oom_score_adj:0

Apr 03 23:49:27 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 91454 (clusterd) total-vm:35855640kB, anon-rss:16705672kB, file-rss:108444kB, shmem-rss:0kB, UID:999 pgtables:47572kB oom_score_adj:0

Apr 03 23:49:27 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 91196 (clusterd) total-vm:36826072kB, anon-rss:10279180kB, file-rss:107700kB, shmem-rss:0kB, UID:999 pgtables:51492kB oom_score_adj:0

Apr 03 23:50:04 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 97546 (clusterd) total-vm:35864984kB, anon-rss:14415240kB, file-rss:107820kB, shmem-rss:0kB, UID:999 pgtables:46160kB oom_score_adj:0

Apr 03 23:51:06 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 103246 (clusterd) total-vm:35968284kB, anon-rss:14457420kB, file-rss:108316kB, shmem-rss:0kB, UID:999 pgtables:43644kB oom_score_adj:0

Apr 03 23:51:49 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 111794 (clusterd) total-vm:35942232kB, anon-rss:14707096kB, file-rss:107960kB, shmem-rss:0kB, UID:999 pgtables:43568kB oom_score_adj:0

Apr 03 23:51:49 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 97673 (clusterd) total-vm:37960600kB, anon-rss:12436552kB, file-rss:107884kB, shmem-rss:0kB, UID:999 pgtables:55088kB oom_score_adj:0

Apr 03 23:52:45 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 118459 (clusterd) total-vm:36254040kB, anon-rss:14118820kB, file-rss:108320kB, shmem-rss:0kB, UID:999 pgtables:43824kB oom_score_adj:0

Apr 03 23:52:45 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 118270 (clusterd) total-vm:36282264kB, anon-rss:13512444kB, file-rss:108024kB, shmem-rss:0kB, UID:999 pgtables:43364kB oom_score_adj:0

Apr 03 23:53:30 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 126461 (clusterd) total-vm:36026328kB, anon-rss:15091024kB, file-rss:108240kB, shmem-rss:0kB, UID:999 pgtables:47808kB oom_score_adj:0

Apr 03 23:54:25 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 133254 (clusterd) total-vm:36074196kB, anon-rss:13070692kB, file-rss:108272kB, shmem-rss:0kB, UID:999 pgtables:43724kB oom_score_adj:0

Apr 03 23:55:05 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 141271 (clusterd) total-vm:36280664kB, anon-rss:14614212kB, file-rss:108512kB, shmem-rss:0kB, UID:999 pgtables:43776kB oom_score_adj:0

Apr 03 23:55:32 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 147426 (clusterd) total-vm:36040536kB, anon-rss:14574800kB, file-rss:108364kB, shmem-rss:0kB, UID:999 pgtables:43376kB oom_score_adj:0

Apr 03 23:55:32 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 126557 (clusterd) total-vm:38034264kB, anon-rss:12250372kB, file-rss:108244kB, shmem-rss:0kB, UID:999 pgtables:55316kB oom_score_adj:0

Apr 03 23:56:08 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 151773 (clusterd) total-vm:36056404kB, anon-rss:13858072kB, file-rss:108172kB, shmem-rss:0kB, UID:999 pgtables:44156kB oom_score_adj:0

Apr 03 23:56:46 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 157418 (clusterd) total-vm:36007320kB, anon-rss:14752112kB, file-rss:108316kB, shmem-rss:0kB, UID:999 pgtables:43796kB oom_score_adj:0

Apr 03 23:57:35 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 163355 (clusterd) total-vm:36020588kB, anon-rss:14426376kB, file-rss:108076kB, shmem-rss:0kB, UID:999 pgtables:43676kB oom_score_adj:0

Apr 03 23:58:16 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 170399 (clusterd) total-vm:36088280kB, anon-rss:14609660kB, file-rss:108356kB, shmem-rss:0kB, UID:999 pgtables:43612kB oom_score_adj:0

Apr 03 23:58:16 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 151942 (clusterd) total-vm:38131032kB, anon-rss:12500012kB, file-rss:108876kB, shmem-rss:0kB, UID:999 pgtables:55536kB oom_score_adj:0

Apr 03 23:58:59 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 176484 (clusterd) total-vm:35907864kB, anon-rss:13401048kB, file-rss:107444kB, shmem-rss:0kB, UID:999 pgtables:43016kB oom_score_adj:0

Apr 03 23:59:32 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 182920 (clusterd) total-vm:35973208kB, anon-rss:14471140kB, file-rss:106540kB, shmem-rss:0kB, UID:999 pgtables:42924kB oom_score_adj:0

Apr 04 00:00:33 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 188021 (clusterd) total-vm:36002648kB, anon-rss:13790052kB, file-rss:106684kB, shmem-rss:0kB, UID:999 pgtables:43012kB oom_score_adj:0

Apr 04 00:00:34 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 176296 (clusterd) total-vm:37926296kB, anon-rss:13343492kB, file-rss:110760kB, shmem-rss:0kB, UID:999 pgtables:55140kB oom_score_adj:0

Apr 04 00:01:14 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 196333 (clusterd) total-vm:36151252kB, anon-rss:14405980kB, file-rss:108404kB, shmem-rss:0kB, UID:999 pgtables:45208kB oom_score_adj:0

Apr 04 00:01:51 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 202596 (clusterd) total-vm:36041048kB, anon-rss:14386820kB, file-rss:107964kB, shmem-rss:0kB, UID:999 pgtables:43012kB oom_score_adj:0

Apr 04 00:02:34 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 196593 (clusterd) total-vm:38069336kB, anon-rss:12738344kB, file-rss:107552kB, shmem-rss:0kB, UID:999 pgtables:55352kB oom_score_adj:0

Apr 04 00:02:34 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 208301 (clusterd) total-vm:36474200kB, anon-rss:14323656kB, file-rss:108004kB, shmem-rss:0kB, UID:999 pgtables:43436kB oom_score_adj:0

Apr 04 00:03:11 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 214802 (clusterd) total-vm:36432664kB, anon-rss:13965076kB, file-rss:108764kB, shmem-rss:0kB, UID:999 pgtables:44592kB oom_score_adj:0

Apr 04 00:03:54 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 220552 (clusterd) total-vm:36424088kB, anon-rss:14356556kB, file-rss:108004kB, shmem-rss:0kB, UID:999 pgtables:43252kB oom_score_adj:0

Apr 04 00:04:42 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 214821 (clusterd) total-vm:38417624kB, anon-rss:12538672kB, file-rss:108972kB, shmem-rss:0kB, UID:999 pgtables:55788kB oom_score_adj:0

Apr 04 00:04:42 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 226988 (clusterd) total-vm:36570968kB, anon-rss:14330860kB, file-rss:110468kB, shmem-rss:0kB, UID:999 pgtables:43456kB oom_score_adj:0

Apr 04 00:05:23 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 234117 (clusterd) total-vm:36203992kB, anon-rss:13158192kB, file-rss:108640kB, shmem-rss:0kB, UID:999 pgtables:44508kB oom_score_adj:0

Apr 04 00:05:23 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 234239 (clusterd) total-vm:36261208kB, anon-rss:13828444kB, file-rss:107432kB, shmem-rss:0kB, UID:999 pgtables:42056kB oom_score_adj:0

Apr 04 00:06:04 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 240269 (clusterd) total-vm:36552024kB, anon-rss:13374736kB, file-rss:109048kB, shmem-rss:0kB, UID:999 pgtables:44432kB oom_score_adj:0

Apr 04 00:06:04 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 240374 (clusterd) total-vm:36039064kB, anon-rss:13532440kB, file-rss:107156kB, shmem-rss:0kB, UID:999 pgtables:42428kB oom_score_adj:0

Apr 04 00:06:43 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 246685 (clusterd) total-vm:36871000kB, anon-rss:14224080kB, file-rss:107936kB, shmem-rss:0kB, UID:999 pgtables:46132kB oom_score_adj:0

Apr 04 00:07:22 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 246527 (clusterd) total-vm:38854552kB, anon-rss:12615816kB, file-rss:109700kB, shmem-rss:0kB, UID:999 pgtables:56704kB oom_score_adj:0

Apr 04 00:07:22 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 252734 (clusterd) total-vm:37123864kB, anon-rss:14156884kB, file-rss:108768kB, shmem-rss:0kB, UID:999 pgtables:43864kB oom_score_adj:0

Apr 04 00:08:05 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 258711 (clusterd) total-vm:37027672kB, anon-rss:14975168kB, file-rss:109200kB, shmem-rss:0kB, UID:999 pgtables:45792kB oom_score_adj:0

Apr 04 00:08:06 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 258710 (clusterd) total-vm:37104856kB, anon-rss:11569964kB, file-rss:108828kB, shmem-rss:0kB, UID:999 pgtables:43124kB oom_score_adj:0

Apr 04 00:08:41 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 265185 (clusterd) total-vm:36711192kB, anon-rss:15125228kB, file-rss:106036kB, shmem-rss:0kB, UID:999 pgtables:47788kB oom_score_adj:0

Apr 04 00:09:15 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 265230 (clusterd) total-vm:38997272kB, anon-rss:12443072kB, file-rss:108480kB, shmem-rss:0kB, UID:999 pgtables:56856kB oom_score_adj:0

Apr 04 00:09:58 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 276113 (clusterd) total-vm:37225304kB, anon-rss:13007160kB, file-rss:107492kB, shmem-rss:0kB, UID:999 pgtables:44772kB oom_score_adj:0

Apr 04 00:10:33 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 282463 (clusterd) total-vm:37302552kB, anon-rss:14321612kB, file-rss:107652kB, shmem-rss:0kB, UID:999 pgtables:44768kB oom_score_adj:0

Apr 04 00:11:00 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 287951 (clusterd) total-vm:36770072kB, anon-rss:14292976kB, file-rss:107332kB, shmem-rss:0kB, UID:999 pgtables:43980kB oom_score_adj:0

Apr 04 00:11:28 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 292178 (clusterd) total-vm:37179096kB, anon-rss:14303216kB, file-rss:107512kB, shmem-rss:0kB, UID:999 pgtables:44392kB oom_score_adj:0

Apr 04 00:11:56 hetzner-aarch64-16cpu-32gb-6d5692d8 kernel: Memory cgroup out of memory: Killed process 296728 (clusterd) total-vm:36589784kB, anon-rss:14321040kB, file-rss:107400kB, shmem-rss:0kB, UID:999 pgtables:43720kB oom_score_adj:0
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness. 
bin/mzcompose --find parallel-workload run default --runtime=1500 --complexity=dml --threads=16
Parallel Workload (0dt deploy) succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
parallel-workload-materialized2-1    | 2025-04-03T23:59:18.898100Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:973:17: dependency since has advanced past dependent (u320) upper
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness. 
bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16

Postgres CDC resumption tests (before source versioning) failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

  • Unknown error in Scenario 'fix_pg_schema_while_mz_restarts' of workflow_disruptions:
Scenario 'fix_pg_schema_while_mz_restarts' of workflow_disruptions: Docker compose failed: docker compose -f/dev/fd/46 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-1c2c30ca/materialize/nightly/test/pg-cdc-resumption-old-syntax up --detach --wait --quiet-pull materialized postgres toxiproxy
Test details & reproducer Postgres source tests with interruptions, test that Materialize can recover. 
bin/mzcompose --find pg-cdc-resumption-old-syntax run default

Scalability benchmark (connection) against merge base or 'latest' failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/5 --project-directory /var/lib/buildkite-agent/builds/hetzner-x86-64-dedi-16cpu-64gb-9de1ae13/materialize/nightly/test/scalability up --detach --wait --quiet-pull materialized
Test details & reproducer Benchmark for how various queries scale, compares against old Materialize versions. 
bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker ConnectionWorkload --exponent-base 2.5 --max-concurrency 2048

Checks + restart of environmentd & storage clusterd with :azure: blob store 1 failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-d7857f53/materialize/nightly/test/platform-checks up --detach --wait --quiet-pull materialized
Test details & reproducer Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts. 
BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=4 bin/mzcompose --find platform-checks run default --scenario=RestartEnvironmentdClusterdStorage --seed=0195fdfd-84b7-42b1-b9d8-5f746b1e4292 --azurite
Checks 0dt restart of the entire Mz with forced migrations 1 succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
platform-checks-mz_4-1              | 2025-04-03T23:53:16.977471Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:973:17: dependency since has advanced past dependent (u518) upper
Test details & reproducer Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts. 
BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeRestartEntireMzForcedMigrations --seed=0195fdfd-84b7-42b1-b9d8-5f746b1e4292
Scalability benchmark (read & write) against merge base or 'latest'
Waited 3m 9s
·
Ran in 1h 28m
Agent
Scalability benchmark (connection) against merge base or 'latest'
Waited 3m 1s
·
Ran in 28m 17s
Agent
Upsert (compaction disabled)
Waited 1m 26s
·
Ran in 24m 15s
Agent
Postgres CDC resumption tests (before source versioning)
Waited 2m 57s
·
Ran in 30m 9s
Agent
1/4
Checks + restart of environmentd & storage clusterd with :azure: blob store 1
Waited 2m 12s
·
Ran in 26m 31s
Agent
1/2
Checks parallel + DROP/CREATE replica 1
Waited 2m 21s
·
Ran in 18m 13s
Agent
Canary Deploy in Staging Cloud
Waited 55s
·
Ran in 23m 47s
Agent
Terraform + Helm Chart E2E on AWS
Waited 45s
·
Ran in 44m 46s
Agent
Terraform + Helm Chart E2E on GCP
Waited 56s
·
Ran in 50m 33s
Agent
Terraform + Helm Chart E2E on Azure
Waited 1m 4s
·
Ran in 46m 51s
Agent
1/2
Data Ingest (1 replica) 1
Waited 3m 2s
·
Ran in 22m 6s
Agent
Parallel Workload (DML)
Waited 2m 32s
·
Ran in 40m 24s
Agent
Total Job Run Time: 5d 15h