Nightly
PublicTests that are too slow or non-deterministic for the regular Test pipeline
Scheduled build
Failed in 9h 3m
Miri test (full)
blob storeTerraform + Helm Chart E2E on GCP (#2) failed, main history: 
- Unknown error in run.log:
[31mERROR[0m mz-debug: fatal: failed to read kubeconfig from '"/kube/config"': No such file or directory (os error 2): No such file or directory (os error 2)
- Unknown error in workflow-gcp-temporary:
subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve', '-var', 'operator_version=v25.2.0-beta.1', '-var', 'orchestratord_version=v0.141.0-dev.0--pr.g49c017b1dd7ee0517ebff6effb95bcde1d670e6c']' returned non-zero exit status 1.
Test details & reproducer
Tests the mz command line tool against a real Cloud instancebin/mzcompose --find terraform run gcp-temporary
Feature benchmark against merge base or 'latest' 2 failed, main history:
- Unknown error in Scenario 'ManyKafkaSourcesOnSameCluster':
New regression against v0.140.3
NAME | TYPE | THIS | OTHER | UNIT | THRESHOLD | Regression? | 'THIS' is
--------------------------------------------------------------------------------------------------------------------------------------------------------
ManyKafkaSourcesOnSameCluster | wallclock | 27.879 | 22.700 | s | 10% | !!YES!! | worse: 22.8% slower
ManyKafkaSourcesOnSameCluster | memory_mz | 2561.569 | 2577.782 | MB | 20% | no | better: 0.6% less
ManyKafkaSourcesOnSameCluster | memory_clusterd | 68.579 | 34.704 | MB | 50% | !!YES!! | worse: 97.6% more
Test details & reproducer
Simple benchmark of mostly individual queries using testdrive. Can find wallclock/memorys regressions in single-connection query executions, not suitable for concurrency.BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find feature-benchmark run default --other-tag common-ancestor
Feature benchmark against merge base or 'latest' 6 failed, main history:
- Unknown error in Scenario 'PgCdcInitialLoad':
New regression against v0.140.3
NAME | TYPE | THIS | OTHER | UNIT | THRESHOLD | Regression? | 'THIS' is
--------------------------------------------------------------------------------------------------------------------------------------------------------
PgCdcInitialLoad | wallclock | 1.278 | 1.158 | s | 10% | !!YES!! | worse: 10.3% slower
PgCdcInitialLoad | memory_mz | 787.544 | 785.923 | MB | 20% | no | worse: 0.2% more
PgCdcInitialLoad | memory_clusterd | 63.486 | 63.829 | MB | 50% | no | better: 0.5% less
Test details & reproducer
Simple benchmark of mostly individual queries using testdrive. Can find wallclock/memorys regressions in single-connection query executions, not suitable for concurrency.BUILDKITE_PARALLEL_JOB=5 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find feature-benchmark run default --other-tag common-ancestor
Scalability benchmark (read & write) against merge base or 'latest' failed, main history:
- Unknown error in Workload 'SelectStarWorkload':
New regression against v0.140.3 (284314f9d)
Regression in workload 'SelectStarWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 496.14 tps vs. 770.26 tps (-274.12 tps; -35.59%)
Test details & reproducer
Benchmark for how various queries scale, compares against old Materialize versions.bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256
Parallel Workload (0dt deploy) succeeded with known error logs, main history:
- Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:
parallel-workload-materialized2-1 | 2025-04-11T00:00:19.015112Z thread 'coordinator' panicked at src/storage-controller/src/lib.rs:974:17: dependency since has advanced past dependent (u355) upper
Test details & reproducer
Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16
Checks 0dt upgrade across four versions 1 succeeded with known error logs, main history:
platform-checks-mz_5-1 | 2025-04-10T23:57:37.640386Z thread 'coordinator' panicked at src/compute-client/src/as_of_selection.rs:392:25: failed to apply hard as-of constraint (id=u558, bounds=[[] .. []], constraint=Constraint { type_: Hard, bound_type: Upper, frontier: Antichain { elements: [1744329453199] }, reason: "storage export u558 write frontier" })
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMzFourVersions --seed=0196220a-046c-404c-8594-afbb9ad12f6a
Terraform + Helm Chart E2E on GCP failed, main history:
- Unknown error in run.log:
[31mERROR[0m mz-debug: fatal: failed to read kubeconfig from '"/kube/config"': No such file or directory (os error 2): No such file or directory (os error 2)
- Unknown error in workflow-gcp-temporary:
subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve', '-var', 'operator_version=v25.2.0-beta.1', '-var', 'orchestratord_version=v0.141.0-dev.0--pr.g49c017b1dd7ee0517ebff6effb95bcde1d670e6c']' returned non-zero exit status 1.
Test details & reproducer
Tests the mz command line tool against a real Cloud instancebin/mzcompose --find terraform run gcp-temporary
Total Job Run Time: 6d 3h