Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#11680
/
main/1d7e6f2606
Failed in 5h 44m
:pipeline:
:rust: Miri test (full)
Extended SSH connection tests
CRDB rolling restarts
PubSub disruption
Test for incident 70
Tests for balancerd
CRDB / Persist backup and restore
Postgres / Persist backup and restore
Replica isolation
Zero downtime
Materialize Emulator
Cluster tests with :azure: blob store
Created
Triggered from Pipeline Schedule

Scalability benchmark (read & write) against merge base or 'latest' failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

New regression against v0.139.1 (b670dba46)

Regression in workload 'SelectLimitWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 524.47 tps vs. 735.24 tps (-210.77 tps; -28.67%)
Test details & reproducer Benchmark for how various queries scale, compares against old Materialize versions. 
bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256

Parallel Workload (DML) failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Apr 01 00:06:25 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 26953 (clusterd) total-vm:33981960kB, anon-rss:12679492kB, file-rss:109900kB, shmem-rss:0kB, UID:999 pgtables:31596kB oom_score_adj:0

Apr 01 00:07:05 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 208718 (clusterd) total-vm:33423752kB, anon-rss:14527672kB, file-rss:107172kB, shmem-rss:0kB, UID:999 pgtables:30584kB oom_score_adj:0

Apr 01 00:07:34 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 26954 (clusterd) total-vm:34964872kB, anon-rss:13779224kB, file-rss:109088kB, shmem-rss:0kB, UID:999 pgtables:46572kB oom_score_adj:0

Apr 01 00:07:35 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 214122 (clusterd) total-vm:22770952kB, anon-rss:12692828kB, file-rss:108548kB, shmem-rss:0kB, UID:999 pgtables:26816kB oom_score_adj:0

Apr 01 00:07:35 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 17330 (cockroach) total-vm:3760336kB, anon-rss:1202728kB, file-rss:149912kB, shmem-rss:0kB, UID:0 pgtables:4272kB oom_score_adj:0

Apr 01 00:08:15 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 218418 (clusterd) total-vm:33249544kB, anon-rss:14963652kB, file-rss:106664kB, shmem-rss:0kB, UID:999 pgtables:32156kB oom_score_adj:0

Apr 01 00:08:44 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 223583 (clusterd) total-vm:33250052kB, anon-rss:14800276kB, file-rss:106504kB, shmem-rss:0kB, UID:999 pgtables:30940kB oom_score_adj:0

Apr 01 00:09:19 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 218294 (clusterd) total-vm:34282248kB, anon-rss:14679612kB, file-rss:108184kB, shmem-rss:0kB, UID:999 pgtables:45260kB oom_score_adj:0

Apr 01 00:09:19 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 227765 (clusterd) total-vm:22770952kB, anon-rss:12735128kB, file-rss:106660kB, shmem-rss:0kB, UID:999 pgtables:26872kB oom_score_adj:0

Apr 01 00:10:03 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 232713 (clusterd) total-vm:33282332kB, anon-rss:14413228kB, file-rss:108664kB, shmem-rss:0kB, UID:999 pgtables:32064kB oom_score_adj:0

Apr 01 00:10:38 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 232820 (clusterd) total-vm:34232520kB, anon-rss:12524680kB, file-rss:107616kB, shmem-rss:0kB, UID:999 pgtables:45116kB oom_score_adj:0

Apr 01 00:10:38 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 238253 (clusterd) total-vm:33257736kB, anon-rss:14636000kB, file-rss:107184kB, shmem-rss:0kB, UID:999 pgtables:30588kB oom_score_adj:0

Apr 01 00:11:25 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 243431 (clusterd) total-vm:33272452kB, anon-rss:14825876kB, file-rss:108776kB, shmem-rss:0kB, UID:999 pgtables:31028kB oom_score_adj:0

Apr 01 00:11:25 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 243430 (clusterd) total-vm:33261512kB, anon-rss:12541152kB, file-rss:108420kB, shmem-rss:0kB, UID:999 pgtables:30408kB oom_score_adj:0

Apr 01 00:12:00 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 249355 (clusterd) total-vm:33271940kB, anon-rss:14659064kB, file-rss:107928kB, shmem-rss:0kB, UID:999 pgtables:32508kB oom_score_adj:0

Apr 01 00:12:00 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 249308 (clusterd) total-vm:33264968kB, anon-rss:12764624kB, file-rss:107836kB, shmem-rss:0kB, UID:999 pgtables:29132kB oom_score_adj:0

Apr 01 00:12:50 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 254434 (clusterd) total-vm:33263304kB, anon-rss:13985884kB, file-rss:107916kB, shmem-rss:0kB, UID:999 pgtables:31256kB oom_score_adj:0

Apr 01 00:12:50 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 254436 (clusterd) total-vm:33255368kB, anon-rss:13659004kB, file-rss:108048kB, shmem-rss:0kB, UID:999 pgtables:30312kB oom_score_adj:0

Apr 01 00:13:44 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 260417 (clusterd) total-vm:33725192kB, anon-rss:13357052kB, file-rss:107488kB, shmem-rss:0kB, UID:999 pgtables:37188kB oom_score_adj:0

Apr 01 00:13:45 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 260579 (clusterd) total-vm:33243400kB, anon-rss:14320620kB, file-rss:107168kB, shmem-rss:0kB, UID:999 pgtables:30228kB oom_score_adj:0

Apr 01 00:13:45 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 217584 (cockroach) total-vm:2773316kB, anon-rss:494276kB, file-rss:145340kB, shmem-rss:0kB, UID:0 pgtables:2712kB oom_score_adj:0

Apr 01 00:14:27 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 266787 (clusterd) total-vm:33720584kB, anon-rss:16314840kB, file-rss:107448kB, shmem-rss:0kB, UID:999 pgtables:42492kB oom_score_adj:0

Apr 01 00:14:27 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 266711 (clusterd) total-vm:33696456kB, anon-rss:11445456kB, file-rss:107468kB, shmem-rss:0kB, UID:999 pgtables:32732kB oom_score_adj:0

Apr 01 00:15:20 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 272301 (clusterd) total-vm:33236808kB, anon-rss:14479280kB, file-rss:107868kB, shmem-rss:0kB, UID:999 pgtables:31320kB oom_score_adj:0

Apr 01 00:15:21 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 272302 (clusterd) total-vm:33715464kB, anon-rss:13118812kB, file-rss:108092kB, shmem-rss:0kB, UID:999 pgtables:36520kB oom_score_adj:0

Apr 01 00:16:06 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 278893 (clusterd) total-vm:33253128kB, anon-rss:14398988kB, file-rss:108052kB, shmem-rss:0kB, UID:999 pgtables:31756kB oom_score_adj:0

Apr 01 00:16:43 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 278743 (clusterd) total-vm:34260104kB, anon-rss:12571460kB, file-rss:107848kB, shmem-rss:0kB, UID:999 pgtables:45156kB oom_score_adj:0

Apr 01 00:17:50 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 284291 (clusterd) total-vm:46838536kB, anon-rss:13854200kB, file-rss:107832kB, shmem-rss:0kB, UID:999 pgtables:45596kB oom_score_adj:0

Apr 01 00:17:51 hetzner-aarch64-16cpu-32gb-2d15d444 kernel: Memory cgroup out of memory: Killed process 289397 (clusterd) total-vm:33264392kB, anon-rss:13557740kB, file-rss:107752kB, shmem-rss:0kB, UID:999 pgtables:29644kB oom_score_adj:0
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness. 
bin/mzcompose --find parallel-workload run default --runtime=1500 --complexity=dml --threads=16
Parallel Workload (0dt deploy) succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
parallel-workload-materialized2-1    | 2025-03-31T23:53:52.148612Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:973:17: dependency since has advanced past dependent (u236) upper
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness. 
bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16
Checks 0dt upgrade, whole-Mz restart 2 succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
platform-checks-mz_3-1              | 2025-03-31T23:57:08.723152Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:973:17: dependency since has advanced past dependent (u448) upper
Test details & reproducer Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts. 
BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMz --seed=0195ee8a-70ed-41ab-aa68-5d0da4a7f5d1

Canary Deploy in Staging Cloud failed, main history: :bk-status-failed::bk-status-failed::bk-status-failed::bk-status-passed::bk-status-passed:

builtins.ValueError: Redpanda API call failed: 400 {"code":"INVALID_ARGUMENT", "message":"throughput tier tier-1-aws-v2-arm is not available in region us-east-1 for cluster type TYPE_DEDICATED", "details":[{"@type":"google.rpc.ErrorInfo", "reason":"REASON_THROUGHPUT_TIER_NOT_AVAILABLE_IN_REGION", "domain":"redpanda.com/controlplane", "metadata":{"cluster_type":"TYPE_DEDICATED", "region":"us-east-1", "throughput_tier_name":"tier-1-aws-v2-arm"}}]}
Test details & reproducer Deploy the current version on a real Staging Cloud, and run some basic verifications, like ingesting data from Kafka and Redpanda Cloud using AWS Privatelink. Runs only on main and release branches. 
bin/mzcompose --find cloud-canary run default
Scalability benchmark (read & write) against merge base or 'latest'
Waited 2m 39s
·
Ran in 1h 42m
Agent
Canary Deploy in Staging Cloud
Waited 50s
·
Ran in 23m 4s
Agent
Parallel Workload (DML)
Waited 2m 35s
·
Ran in 44m 19s
Agent
Total Job Run Time: 6d 4h