Nightly
PublicTests that are too slow or non-deterministic for the regular Test pipeline
Scheduled build
Failed in 8h 48m
Miri test (full)
blob storeScalability benchmark (read & write) against merge base or 'latest' failed, main history: 
- Unknown error in Workload 'SelectStarWorkload':
New regression against v0.141.2 (e783ed422)
Regression in workload 'SelectStarWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 566.84 tps vs. 908.55 tps (-341.71 tps; -37.61%)
Test details & reproducer
Benchmark for how various queries scale, compares against old Materialize versions.bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256
Checks 0dt upgrade, whole-Mz restart 1 failed, main history:
- Potential regression 0dt: Cluster with sink never hydrates (#8450, closed) in workflow-default:
materialize.ui.UIError: Timed out waiting for mz_3 to reach Mz deployment status ReadyToPromote, still in status Initializing
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in services.log:
platform-checks-mz_3-1 | 2025-04-15T23:48:34.821243Z thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result: [
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMz --seed=01963bc9-da9d-46d6-af47-3dc1e65248a5
Checks 0dt upgrade, whole-Mz restart 2 failed, main history:
- Potential regression 0dt: Cluster with sink never hydrates (#8450, closed) in workflow-default:
materialize.ui.UIError: Timed out waiting for mz_3 to reach Mz deployment status ReadyToPromote, still in status Initializing
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in services.log:
platform-checks-mz_3-1 | 2025-04-15T23:47:37.653974Z thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result: [
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMz --seed=01963bc9-da9d-46d6-af47-3dc1e65248a5
Parallel Workload (0dt deploy) succeeded with known error logs, main history:
- Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:
parallel-workload-materialized2-1 | 2025-04-15T23:42:58.093675Z thread 'coordinator' panicked at src/storage-controller/src/lib.rs:974:17: dependency since has advanced past dependent (u83) upper
Test details & reproducer
Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16
Legacy upgrade tests (last version from docs) 2 failed with known error logs, main history:
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in workflow-default:
Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-0c409f28/materialize/nightly/test/legacy-upgrade run -eCLUSTER_REPLICA_SIZES testdrive --no-reset --var=upgrade-from-version=v0.130.3 --var=default-storage-size=4-1 --var=created-cluster=quickstart --temp-dir=/share/tmp/upgrade-from-v0.130.3 --seed=1743012509 check-from-{any_version,v0.100.0,v0.100.0,v0.100.1,v0.101.0,v0.101.0,v0.101.1,v0.102.0,v0.103.0,v0.104.0,v0.105.0,v0.106.0,v0.106.0,v0.106.0,v0.106.1,v0.106.1,v0.106.2,v0.107.0,v0.107.0,v0.107.0,v0.107.0,v0.107.0,v0.107.0,v0.107.1,v0.107.1,v0.107.1,v0.107.1,v0.107.1,v0.107.2,v0.107.2,v0.107.2,v0.107.2,v0.107.3,v0.107.3,v0.107.3,v0.107.4,v0.107.4,v0.107.5,v0.108.0,v0.108.0,v0.108.0,v0.108.1,v0.108.1,v0.108.2,v0.109.0,v0.109.0,v0.109.1,v0.110.0,v0.110.0,v0.110.1,v0.111.0,v0.111.0,v0.111.0,v0.111.0,v0.111.0,v0.111.1,v0.111.1,v0.111.1,v0.111.1,v0.111.2,v0.111.2,v0.111.2,v0.111.3,v0.111.3,v0.111.4,v0.112.0,v0.112.0,v0.112.0,v0.112.1,v0.112.1,v0.112.2,v0.113.0,v0.113.0,v0.113.1,v0.113.2,v0.114.0,v0.114.0,v0.114.1,v0.115.0,v0.115.0,v0.115.1,v0.116.0,v0.117.0,v0.117.0,v0.117.0,v0.117.1,v0.117.1,v0.117.2,v0.118.0,v0.118.0,v0.118.0,v0.118.1,v0.118.1,v0.118.2,v0.120.0,v0.120.0,v0.120.0,v0.120.1,v0.120.1,v0.120.2,v0.121.0,v0.121.0,v0.121.0,v0.121.1,v0.121.1,v0.121.2,v0.122.0,v0.122.0,v0.122.0,v0.122.1,v0.122.1,v0.122.2,v0.123.0,v0.124.0,v0.124.0,v0.124.0,v0.124.1,v0.124.1,v0.124.2,v0.125.0,v0.125.0,v0.125.0,v0.125.0,v0.125.1,v0.125.1,v0.125.1,v0.125.2,v0.125.2,v0.125.3,v0.126.0,v0.126.0,v0.126.0,v0.126.1,v0.126.1,v0.126.2,v0.127.0,v0.127.0,v0.127.0,v0.127.1,v0.127.1,v0.127.2,v0.128.0,v0.129.0,v0.129.0,v0.129.0,v0.129.1,v0.129.1,v0.129.2,v0.130.0,v0.130.0,v0.130.0,v0.130.1,v0.130.1,v0.130.2,v0.27.0,v0.28.0,v0.28.0,v0.28.0,v0.28.0,v0.28.0,v0.28.1,v0.28.1,v0.28.1,v0.28.1,v0.28.2,v0.28.2,v0.28.2,v0.28.3,v0.28.3,v0.28.4,v0.29.0,v0.29.0,v0.29.0,v0.29.0,v0.29.1,v0.29.1,v0.29.1,v0.29.2,v0.29.2,v0.29.3,v0.30.0,v0.30.0,v0.30.0,v0.30.1,v0.30.1,v0.30.2,v0.31.0,v0.32.0,v0.32.0,v0.32.0,v0.32.0,v0.32.0,v0.32.1,v0.32.1,v0.32.1,v0.32.1,v0.32.2,v0.32.2,v0.32.2,v0.32.3,v0.32.3,v0.32.4,v0.33.0,v0.33.0,v0.33.0,v0.33.0,v0.33.1,v0.33.1,v0.33.1,v0.33.2,v0.33.2,v0.33.3,v0.36.0,v0.36.0,v0.36.0,v0.36.1,v0.36.1,v0.36.2,v0.37.0,v0.37.0,v0.37.0,v0.37.0,v0.37.1,v0.37.1,v0.37.1,v0.37.2,v0.37.2,v0.37.3,v0.38.0,v0.39.0,v0.39.0,v0.39.0,v0.39.0,v0.39.1,v0.39.1,v0.39.1,v0.39.2,v0.39.2,v0.39.3,v0.40.0,v0.41.0,v0.41.0,v0.41.1,v0.42.0,v0.42.0,v0.42.0,v0.42.0,v0.42.0,v0.42.1,v0.42.1,v0.42.1,v0.42.1,v0.42.2,v0.42.2,v0.42.2,v0.42.3,v0.42.3,v0.42.4,v0.43.0,v0.44.0,v0.44.0,v0.44.0,v0.44.0,v0.44.1,v0.44.1,v0.44.1,v0.44.2,v0.44.2,v0.44.3,v0.45.0,v0.46.0,v0.46.0,v0.46.1,v0.47.0,v0.47.0,v0.47.1,v0.48.0,v0.48.0,v0.48.0,v0.48.0,v0.48.0,v0.48.1,v0.48.1,v0.48.1,v0.48.1,v0.48.2,v0.48.2,v0.48.2,v0.48.3,v0.48.3,v0.48.4,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.1,v0.49.1,v0.49.1,v0.49.1,v0.49.1,v0.49.1,v0.49.2,v0.49.2,v0.49.2,v0.49.2,v0.49.2,v0.49.3,v0.49.3,v0.49.3,v0.49.3,v0.49.4,v0.49.4,v0.49.4,v0.49.5,v0.49.5,v0.49.6,v0.50.0,v0.50.0,v0.50.0,v0.50.0,v0.50.1,v0.50.1,v0.50.1,v0.50.2,v0.50.2,v0.50.3,v0.51.0,v0.51.0,v0.51.1,v0.52.0,v0.52.0,v0.52.0,v0.52.0,v0.52.1,v0.52.1,v0.52.1,v0.52.2,v0.52.2,v0.52.2,v0.52.3,v0.52.3,v0.52.4,v0.53.0,v0.54.0,v0.55.0,v0.56.0,v0.56.0,v0.56.0,v0.56.0,v0.56.1,v0.56.1,v0.56.1,v0.56.1,v0.56.2,v0.56.2,v0.56.2,v0.56.3,v0.56.3,v0.56.4,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.1,v0.57.1,v0.57.1,v0.57.1,v0.57.1,v0.57.1,v0.57.10,v0.57.2,v0.57.2,v0.57.2,v0.57.2,v0.57.2,v0.57.2,v0.57.3,v0.57.3,v0.57.3,v0.57.3,v0.57.3,v0.57.3,v0.57.4,v0.57.4,v0.57.4,v0.57.4,v0.57.4,v0.57.5,v0.57.5,v0.57.5,v0.57.5,v0.57.6,v0.57.6,v0.57.6,v0.57.6,v0.57.7,v0.57.7,v0.57.7,v0.57.7,v0.57.8,v0.57.8,v0.57.8,v0.57.9,v0.57.9,v0.58.0,v0.59.0,v0.59.0,v0.59.0,v0.59.1,v0.59.1,v0.59.2,v0.60.0,v0.60.0,v0.60.1,v0.61.0,v0.62.0,v0.62.0,v0.62.0,v0.62.0,v0.62.0,v0.62.1,v0.62.1,v0.62.1,v0.62.1,v0.62.2,v0.62.2,v0.62.2,v0.62.3,v0.62.3,v0.62.4,v0.63.0,v0.63.0,v0.63.0,v0.63.0,v0.63.0,v0.63.1,v0.63.1,v0.63.1,v0.63.1,v0.63.2,v0.63.2,v0.63.2,v0.63.3,v0.63.3,v0.63.4,v0.64.0,v0.65.0,v0.65.0,v0.65.0,v0.65.1,v0.65.1,v0.65.2,v0.66.0,v0.67.0,v0.67.0,v0.67.0,v0.67.0,v0.67.1,v0.67.1,v0.67.1,v0.67.2,v0.67.2,v0.67.3,v0.68.0,v0.68.0,v0.68.1,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.1,v0.69.1,v0.69.1,v0.69.1,v0.69.1,v0.69.1,v0.69.2,v0.69.2,v0.69.2,v0.69.2,v0.69.2,v0.69.3,v0.69.3,v0.69.3,v0.69.3,v0.69.4,v0.69.4,v0.69.4,v0.69.5,v0.69.5,v0.69.6,v0.70.0,v0.70.0,v0.70.0,v0.70.1,v0.70.1,v0.70.2,v0.71.0,v0.72.0,v0.72.0,v0.72.1,v0.73.0,v0.74.0,v0.74.0,v0.74.0,v0.74.1,v0.74.1,v0.74.2,v0.75.0,v0.75.0,v0.75.0,v0.75.0,v0.75.1,v0.75.1,v0.75.1,v0.75.2,v0.75.2,v0.75.3,v0.76.0,v0.77.0,v0.77.0,v0.77.1,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.10,v0.78.10,v0.78.10,v0.78.10,v0.78.10,v0.78.10,v0.78.11,v0.78.11,v0.78.11,v0.78.11,v0.78.11,v0.78.12,v0.78.12,v0.78.12,v0.78.12,v0.78.13,v0.78.13,v0.78.13,v0.78.14,v0.78.14,v0.78.15,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.79.0,v0.79.0,v0.79.1,v0.80.0,v0.80.0,v0.80.0,v0.80.1,v0.80.1,v0.80.2,v0.81.0,v0.81.1,v0.81.2,v0.81.3,v0.82.0,v0.82.0,v0.82.0,v0.82.1,v0.82.1,v0.82.2,v0.83.0,v0.83.0,v0.83.0,v0.83.0,v0.83.0,v0.83.1,v0.83.1,v0.83.1,v0.83.1,v0.83.2,v0.83.2,v0.83.2,v0.83.3,v0.83.3,v0.83.4,v0.84.0,v0.84.0,v0.84.0,v0.84.0,v0.84.0,v0.84.1,v0.84.1,v0.84.1,v0.84.1,v0.84.2,v0.84.2,v0.84.2,v0.84.3,v0.84.3,v0.84.4,v0.85.0,v0.85.0,v0.85.0,v0.85.1,v0.85.1,v0.85.2,v0.86.0,v0.86.0,v0.86.1,v0.87.0,v0.87.0,v0.87.0,v0.87.1,v0.87.1,v0.87.2,v0.88.0,v0.88.0,v0.88.1,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.1,v0.89.1,v0.89.1,v0.89.1,v0.89.1,v0.89.1,v0.89.2,v0.89.2,v0.89.2,v0.89.2,v0.89.2,v0.89.3,v0.89.3,v0.89.3,v0.89.3,v0.89.4,v0.89.4,v0.89.4,v0.89.5,v0.89.5,v0.89.6,v0.90.0,v0.90.0,v0.90.1,v0.91.0,v0.92.0,v0.92.1,v0.94.0,v0.94.0,v0.94.0,v0.94.1,v0.94.1,v0.94.2,v0.95.0,v0.95.0,v0.95.0,v0.95.1,v0.95.1,v0.95.2,v0.96.0,v0.96.0,v0.96.0,v0.96.1,v0.96.1,v0.96.2,v0.97.0,v0.97.0,v0.97.0,v0.97.0,v0.97.1,v0.97.1,v0.97.1,v0.97.2,v0.97.2,v0.97.3,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.2,v0.98.2,v0.98.2,v0.98.2,v0.98.2,v0.98.2,v0.98.3,v0.98.3,v0.98.3,v0.98.3,v0.98.3,v0.98.4,v0.98.4,v0.98.4,v0.98.4,v0.98.5,v0.98.5,v0.98.5,v0.98.6,v0.98.6,v0.98.7,v0.99.0,v0.99.0,v0.99.1,v0.99.2,v0.130.3}-*.td --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://mz_system@materialized:6877
Container legacy-upgrade-cockroach-1 Running
thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5:
snapshot_and_fetch guarantees a consolidated result: [
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"deploy_generation": Numeric(
OrderedDecimal(
0,
),
),
"epoch": Numeric(
OrderedDecimal(
4,
),
),
"kind": String(
"FenceToken",
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"user",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
90,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"oid",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
20338,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"schema",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
11,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"system",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
796,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"replica",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
4,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"auditlog",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
136,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744761276284,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"database",
),
},
),
[...]
Test details & reproducer
Verifies that objects created in previous versions of Materialize are still operational after an upgrade. See also the newer platform-checks' upgrade scenarios.BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find legacy-upgrade run default --versions-source=docs --lts-upgrade
Checks 0dt upgrade across four versions 2 succeeded with known error logs, main history:
- Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:
platform-checks-mz_4-1 | 2025-04-15T23:49:48.614409Z thread 'coordinator' panicked at src/storage-controller/src/lib.rs:974:17: dependency since has advanced past dependent (u407) upper
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMzFourVersions --seed=01963bc9-da9d-46d6-af47-3dc1e65248a5
Legacy upgrade tests (last version from docs) 1 failed with known error logs, main history:
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in workflow-default:
Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-4f97fb2d/materialize/nightly/test/legacy-upgrade run -eCLUSTER_REPLICA_SIZES testdrive --no-reset --var=upgrade-from-version=v0.130.3 --var=default-storage-size=4-1 --var=created-cluster=quickstart --temp-dir=/share/tmp/upgrade-from-v0.130.3 --seed=3466408395 check-from-{any_version,v0.100.0,v0.100.0,v0.100.1,v0.101.0,v0.101.0,v0.101.1,v0.102.0,v0.103.0,v0.104.0,v0.105.0,v0.106.0,v0.106.0,v0.106.0,v0.106.1,v0.106.1,v0.106.2,v0.107.0,v0.107.0,v0.107.0,v0.107.0,v0.107.0,v0.107.0,v0.107.1,v0.107.1,v0.107.1,v0.107.1,v0.107.1,v0.107.2,v0.107.2,v0.107.2,v0.107.2,v0.107.3,v0.107.3,v0.107.3,v0.107.4,v0.107.4,v0.107.5,v0.108.0,v0.108.0,v0.108.0,v0.108.1,v0.108.1,v0.108.2,v0.109.0,v0.109.0,v0.109.1,v0.110.0,v0.110.0,v0.110.1,v0.111.0,v0.111.0,v0.111.0,v0.111.0,v0.111.0,v0.111.1,v0.111.1,v0.111.1,v0.111.1,v0.111.2,v0.111.2,v0.111.2,v0.111.3,v0.111.3,v0.111.4,v0.112.0,v0.112.0,v0.112.0,v0.112.1,v0.112.1,v0.112.2,v0.113.0,v0.113.0,v0.113.1,v0.113.2,v0.114.0,v0.114.0,v0.114.1,v0.115.0,v0.115.0,v0.115.1,v0.116.0,v0.117.0,v0.117.0,v0.117.0,v0.117.1,v0.117.1,v0.117.2,v0.118.0,v0.118.0,v0.118.0,v0.118.1,v0.118.1,v0.118.2,v0.120.0,v0.120.0,v0.120.0,v0.120.1,v0.120.1,v0.120.2,v0.121.0,v0.121.0,v0.121.0,v0.121.1,v0.121.1,v0.121.2,v0.122.0,v0.122.0,v0.122.0,v0.122.1,v0.122.1,v0.122.2,v0.123.0,v0.124.0,v0.124.0,v0.124.0,v0.124.1,v0.124.1,v0.124.2,v0.125.0,v0.125.0,v0.125.0,v0.125.0,v0.125.1,v0.125.1,v0.125.1,v0.125.2,v0.125.2,v0.125.3,v0.126.0,v0.126.0,v0.126.0,v0.126.1,v0.126.1,v0.126.2,v0.127.0,v0.127.0,v0.127.0,v0.127.1,v0.127.1,v0.127.2,v0.128.0,v0.129.0,v0.129.0,v0.129.0,v0.129.1,v0.129.1,v0.129.2,v0.130.0,v0.130.0,v0.130.0,v0.130.1,v0.130.1,v0.130.2,v0.27.0,v0.28.0,v0.28.0,v0.28.0,v0.28.0,v0.28.0,v0.28.1,v0.28.1,v0.28.1,v0.28.1,v0.28.2,v0.28.2,v0.28.2,v0.28.3,v0.28.3,v0.28.4,v0.29.0,v0.29.0,v0.29.0,v0.29.0,v0.29.1,v0.29.1,v0.29.1,v0.29.2,v0.29.2,v0.29.3,v0.30.0,v0.30.0,v0.30.0,v0.30.1,v0.30.1,v0.30.2,v0.31.0,v0.32.0,v0.32.0,v0.32.0,v0.32.0,v0.32.0,v0.32.1,v0.32.1,v0.32.1,v0.32.1,v0.32.2,v0.32.2,v0.32.2,v0.32.3,v0.32.3,v0.32.4,v0.33.0,v0.33.0,v0.33.0,v0.33.0,v0.33.1,v0.33.1,v0.33.1,v0.33.2,v0.33.2,v0.33.3,v0.36.0,v0.36.0,v0.36.0,v0.36.1,v0.36.1,v0.36.2,v0.37.0,v0.37.0,v0.37.0,v0.37.0,v0.37.1,v0.37.1,v0.37.1,v0.37.2,v0.37.2,v0.37.3,v0.38.0,v0.39.0,v0.39.0,v0.39.0,v0.39.0,v0.39.1,v0.39.1,v0.39.1,v0.39.2,v0.39.2,v0.39.3,v0.40.0,v0.41.0,v0.41.0,v0.41.1,v0.42.0,v0.42.0,v0.42.0,v0.42.0,v0.42.0,v0.42.1,v0.42.1,v0.42.1,v0.42.1,v0.42.2,v0.42.2,v0.42.2,v0.42.3,v0.42.3,v0.42.4,v0.43.0,v0.44.0,v0.44.0,v0.44.0,v0.44.0,v0.44.1,v0.44.1,v0.44.1,v0.44.2,v0.44.2,v0.44.3,v0.45.0,v0.46.0,v0.46.0,v0.46.1,v0.47.0,v0.47.0,v0.47.1,v0.48.0,v0.48.0,v0.48.0,v0.48.0,v0.48.0,v0.48.1,v0.48.1,v0.48.1,v0.48.1,v0.48.2,v0.48.2,v0.48.2,v0.48.3,v0.48.3,v0.48.4,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.0,v0.49.1,v0.49.1,v0.49.1,v0.49.1,v0.49.1,v0.49.1,v0.49.2,v0.49.2,v0.49.2,v0.49.2,v0.49.2,v0.49.3,v0.49.3,v0.49.3,v0.49.3,v0.49.4,v0.49.4,v0.49.4,v0.49.5,v0.49.5,v0.49.6,v0.50.0,v0.50.0,v0.50.0,v0.50.0,v0.50.1,v0.50.1,v0.50.1,v0.50.2,v0.50.2,v0.50.3,v0.51.0,v0.51.0,v0.51.1,v0.52.0,v0.52.0,v0.52.0,v0.52.0,v0.52.1,v0.52.1,v0.52.1,v0.52.2,v0.52.2,v0.52.2,v0.52.3,v0.52.3,v0.52.4,v0.53.0,v0.54.0,v0.55.0,v0.56.0,v0.56.0,v0.56.0,v0.56.0,v0.56.1,v0.56.1,v0.56.1,v0.56.1,v0.56.2,v0.56.2,v0.56.2,v0.56.3,v0.56.3,v0.56.4,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.0,v0.57.1,v0.57.1,v0.57.1,v0.57.1,v0.57.1,v0.57.1,v0.57.10,v0.57.2,v0.57.2,v0.57.2,v0.57.2,v0.57.2,v0.57.2,v0.57.3,v0.57.3,v0.57.3,v0.57.3,v0.57.3,v0.57.3,v0.57.4,v0.57.4,v0.57.4,v0.57.4,v0.57.4,v0.57.5,v0.57.5,v0.57.5,v0.57.5,v0.57.6,v0.57.6,v0.57.6,v0.57.6,v0.57.7,v0.57.7,v0.57.7,v0.57.7,v0.57.8,v0.57.8,v0.57.8,v0.57.9,v0.57.9,v0.58.0,v0.59.0,v0.59.0,v0.59.0,v0.59.1,v0.59.1,v0.59.2,v0.60.0,v0.60.0,v0.60.1,v0.61.0,v0.62.0,v0.62.0,v0.62.0,v0.62.0,v0.62.0,v0.62.1,v0.62.1,v0.62.1,v0.62.1,v0.62.2,v0.62.2,v0.62.2,v0.62.3,v0.62.3,v0.62.4,v0.63.0,v0.63.0,v0.63.0,v0.63.0,v0.63.0,v0.63.1,v0.63.1,v0.63.1,v0.63.1,v0.63.2,v0.63.2,v0.63.2,v0.63.3,v0.63.3,v0.63.4,v0.64.0,v0.65.0,v0.65.0,v0.65.0,v0.65.1,v0.65.1,v0.65.2,v0.66.0,v0.67.0,v0.67.0,v0.67.0,v0.67.0,v0.67.1,v0.67.1,v0.67.1,v0.67.2,v0.67.2,v0.67.3,v0.68.0,v0.68.0,v0.68.1,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.0,v0.69.1,v0.69.1,v0.69.1,v0.69.1,v0.69.1,v0.69.1,v0.69.2,v0.69.2,v0.69.2,v0.69.2,v0.69.2,v0.69.3,v0.69.3,v0.69.3,v0.69.3,v0.69.4,v0.69.4,v0.69.4,v0.69.5,v0.69.5,v0.69.6,v0.70.0,v0.70.0,v0.70.0,v0.70.1,v0.70.1,v0.70.2,v0.71.0,v0.72.0,v0.72.0,v0.72.1,v0.73.0,v0.74.0,v0.74.0,v0.74.0,v0.74.1,v0.74.1,v0.74.2,v0.75.0,v0.75.0,v0.75.0,v0.75.0,v0.75.1,v0.75.1,v0.75.1,v0.75.2,v0.75.2,v0.75.3,v0.76.0,v0.77.0,v0.77.0,v0.77.1,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.0,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.1,v0.78.10,v0.78.10,v0.78.10,v0.78.10,v0.78.10,v0.78.10,v0.78.11,v0.78.11,v0.78.11,v0.78.11,v0.78.11,v0.78.12,v0.78.12,v0.78.12,v0.78.12,v0.78.13,v0.78.13,v0.78.13,v0.78.14,v0.78.14,v0.78.15,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.2,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.3,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.4,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.5,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.6,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.7,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.8,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.78.9,v0.79.0,v0.79.0,v0.79.1,v0.80.0,v0.80.0,v0.80.0,v0.80.1,v0.80.1,v0.80.2,v0.81.0,v0.81.1,v0.81.2,v0.81.3,v0.82.0,v0.82.0,v0.82.0,v0.82.1,v0.82.1,v0.82.2,v0.83.0,v0.83.0,v0.83.0,v0.83.0,v0.83.0,v0.83.1,v0.83.1,v0.83.1,v0.83.1,v0.83.2,v0.83.2,v0.83.2,v0.83.3,v0.83.3,v0.83.4,v0.84.0,v0.84.0,v0.84.0,v0.84.0,v0.84.0,v0.84.1,v0.84.1,v0.84.1,v0.84.1,v0.84.2,v0.84.2,v0.84.2,v0.84.3,v0.84.3,v0.84.4,v0.85.0,v0.85.0,v0.85.0,v0.85.1,v0.85.1,v0.85.2,v0.86.0,v0.86.0,v0.86.1,v0.87.0,v0.87.0,v0.87.0,v0.87.1,v0.87.1,v0.87.2,v0.88.0,v0.88.0,v0.88.1,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.0,v0.89.1,v0.89.1,v0.89.1,v0.89.1,v0.89.1,v0.89.1,v0.89.2,v0.89.2,v0.89.2,v0.89.2,v0.89.2,v0.89.3,v0.89.3,v0.89.3,v0.89.3,v0.89.4,v0.89.4,v0.89.4,v0.89.5,v0.89.5,v0.89.6,v0.90.0,v0.90.0,v0.90.1,v0.91.0,v0.92.0,v0.92.1,v0.94.0,v0.94.0,v0.94.0,v0.94.1,v0.94.1,v0.94.2,v0.95.0,v0.95.0,v0.95.0,v0.95.1,v0.95.1,v0.95.2,v0.96.0,v0.96.0,v0.96.0,v0.96.1,v0.96.1,v0.96.2,v0.97.0,v0.97.0,v0.97.0,v0.97.0,v0.97.1,v0.97.1,v0.97.1,v0.97.2,v0.97.2,v0.97.3,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.0,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.1,v0.98.2,v0.98.2,v0.98.2,v0.98.2,v0.98.2,v0.98.2,v0.98.3,v0.98.3,v0.98.3,v0.98.3,v0.98.3,v0.98.4,v0.98.4,v0.98.4,v0.98.4,v0.98.5,v0.98.5,v0.98.5,v0.98.6,v0.98.6,v0.98.7,v0.99.0,v0.99.0,v0.99.1,v0.99.2,v0.130.3}-*.td --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://mz_system@materialized:6877
Container legacy-upgrade-cockroach-1 Running
thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5:
snapshot_and_fetch guarantees a consolidated result: [
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"deploy_generation": Numeric(
OrderedDecimal(
0,
),
),
"epoch": Numeric(
OrderedDecimal(
4,
),
),
"kind": String(
"FenceToken",
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"user",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
96,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"oid",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
20344,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"schema",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
11,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"system",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
796,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"replica",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
4,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"auditlog",
),
},
),
"kind": String(
"IdAlloc",
),
"value": Map(
{
"next_id": Numeric(
OrderedDecimal(
142,
),
),
},
),
},
),
]},
),
),
),
Ok(
(),
),
),
1744760978990,
1,
),
(
(
Ok(
SourceData(
Ok(
Row{[
Map(
{
"key": Map(
{
"name": String(
"database",
),
},
),
[...]
Test details & reproducer
Verifies that objects created in previous versions of Materialize are still operational after an upgrade. See also the newer platform-checks' upgrade scenarios.BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find legacy-upgrade run default --versions-source=docs --lts-upgrade
Checks upgrade across two versions 1 failed, main history:
- Unknown error in workflow-default:
Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-bf2b0d81/materialize/nightly/test/platform-checks up --detach --wait --quiet-pull materialized
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in services.log:
platform-checks-materialized-1 | 2025-04-15T23:50:47.512022Z thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result: [
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=UpgradeEntireMzTwoVersions --seed=01963bc9-da9d-46d6-af47-3dc1e65248a5
Checks upgrade across two versions 2 failed, main history:
- Unknown error in workflow-default:
Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-51adabd1/materialize/nightly/test/platform-checks up --detach --wait --quiet-pull materialized
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in services.log:
platform-checks-materialized-1 | 2025-04-15T23:49:46.335085Z thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result: [
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=UpgradeEntireMzTwoVersions --seed=01963bc9-da9d-46d6-af47-3dc1e65248a5
Checks upgrade, whole-Mz restart 1 failed, main history:
- Unknown error in workflow-default:
Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-e1fc3dbd/materialize/nightly/test/platform-checks up --detach --wait --quiet-pull materialized
- Known issue thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result (#9179) in services.log:
platform-checks-materialized-1 | 2025-04-15T23:49:35.667125Z thread 'main' panicked at src/catalog/src/durable/persist.rs:1923:5: snapshot_and_fetch guarantees a consolidated result: [
Test details & reproducer
Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=UpgradeEntireMz --seed=01963bc9-da9d-46d6-af47-3dc1e65248a5
Total Job Run Time: 5d 22h