Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#11487

main/90346e11f1

Failed in 9h 13m

Miri test (full)

Extended SSH connection tests

CRDB rolling restarts

PubSub disruption

Test for incident 70

Tests for balancerd

CRDB / Persist backup and restore

Postgres / Persist backup and restore

Created Mon 17th Mar at 11:30 PM

Triggered from Pipeline Schedule

🏎️ testdrive 4 replicas (#2) failed, main history:

Unknown error in github-6744.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in materialized-view-refresh-options.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-1947.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-resolution-union-concrete.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-to-s3-minio.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-from-s3-minio.td:

copy-from-s3-minio.td:39:1: executing query failed: connection closed
     |
  15 | $ postgres-execute c ... [rest of line truncated for security]
  24 | $ s3-set-presigned-u ... [rest of line truncated for security]
  26 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
  37 | $ s3-set-presigned-u ... [rest of line truncated for security]
  38 | 
  39 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
     | ^

Unknown error in subquery-scalar-errors.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-avro-sinks-defaults.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-nonnull-record.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in privilege-checks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in blue-green.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in frontier-rounding.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in test-skip-if.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in session.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-resolution-types-map.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-json-sinks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-4315.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in alter-sink.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in controller-frontiers.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-5502.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in correctness-property-2.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Known issue thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2266:33: no sender for c1dad1a4-6d47-4d90-a269-7f2d9118ca6d! (#9095) in services.log:

testdrive-materialized-1     | 2025-03-18T14:05:03.616673Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2266:33: no sender for c9d1e7f4-3dd2-4b19-8e6d-db736679bf6f!

Test details & reproducer

Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result.

bin/mzcompose --find testdrive run default --replicas=4

🏎️ testdrive 4 replicas failed, main history:

Unknown error in services.log:

testdrive-materialized-1     | 2025-03-18T00:29:25.885566Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2266:33: no sender for c1dad1a4-6d47-4d90-a269-7f2d9118ca6d!

Unknown error in github-1947.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-from-s3-minio.td:

copy-from-s3-minio.td:54:1: executing query failed: connection closed
     |
  15 | $ postgres-execute c ... [rest of line truncated for security]
  24 | $ s3-set-presigned-u ... [rest of line truncated for security]
  26 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
  37 | $ s3-set-presigned-u ... [rest of line truncated for security]
  39 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
  52 | $ s3-set-presigned-u ... [rest of line truncated for security]
  53 | 
  54 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
     | ^

Unknown error in avro-resolution-union-concrete.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-5502.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-json-sinks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in materialized-view-refresh-options.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-resolution-types-map.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in controller-frontiers.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in test-skip-if.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-avro-sinks-defaults.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in alter-sink.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in blue-green.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in correctness-property-2.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in frontier-rounding.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in privilege-checks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-6744.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-4315.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in subquery-scalar-errors.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-to-s3-minio.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-nonnull-record.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in session.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Test details & reproducer

bin/mzcompose --find testdrive run default --replicas=4

Scalability benchmark (read & write) against merge base or 'latest' failed, main history:

Unknown error in Workload 'SelectLimitWorkload':

New regression against v0.137.1 (f216419df)

Regression in workload 'SelectLimitWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 482.52 tps vs. 795.71 tps (-313.19 tps; -39.36%)

Test details & reproducer

Benchmark for how various queries scale, compares against old Materialize versions.

bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256

Parallel Workload (0dt deploy) succeeded with known error logs, main history:

Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:

parallel-workload-materialized2-1    | 2025-03-18T00:00:36.963421Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:924:17: dependency since has advanced past dependent (u510) upper

Test details & reproducer

Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.

bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16

Terraform + Helm Chart E2E on Azure failed, main history:

Unknown error in run.log:

[31mERROR[0m [2mmz_self_managed_debug[0m[2m:[0m self-managed-debug: fatal: failed to read kubeconfig from '"/kube/config"': No such file or directory (os error 2): No such file or directory (os error 2)

Unknown error in workflow-azure-temporary:

subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve']' returned non-zero exit status 1.

Test details & reproducer

Tests the mz command line tool against a real Cloud instance

bin/mzcompose --find terraform run azure-temporary

Checks 0dt upgrade across four versions 2 succeeded with known error logs, main history:

Known issue 0dt: bootstrapping can get confused by concurrent DROP DDL (#8836) in services.log:

platform-checks-mz_5-1              | 2025-03-17T23:56:38.424964Z  thread 'coordinator' panicked at src/compute-client/src/as_of_selection.rs:392:25: failed to apply hard as-of constraint (id=u471, bounds=[[] .. []], constraint=Constraint { type_: Hard, bound_type: Upper, frontier: Antichain { elements: [1742255792852] }, reason: "storage export u471 write frontier" })

Test details & reproducer

Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.

BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMzFourVersions --seed=0195a671-6fb1-4e90-9e67-69083f4aab26

Checks parallel + restart compute clusterd 1 failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/15 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-19c1fd8a/materialize/nightly/test/platform-checks exec -T testdrive testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --materialize-param=statement_timeout='300s' --default-timeout=300s --seed=1 --persist-blob-url=s3://minioadmin:minioadmin@persist/persist?endpoint=http://minio:9000/&region=minio --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --var=replicas=1 --var=default-replica-size=4-4 --var=default-storage-size=4-1 --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-19c1fd8a/materialize/nightly/misc/python/materialize/checks/all_checks/sink.py:1240

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-19c1fd8a/materialize/nightly/misc/python/materialize/checks/all_checks/sink.py:1240

Test details & reproducer

Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.

BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=RestartClusterdCompute --execution-mode=parallel --seed=0195a671-6fb1-4e90-9e67-69083f4aab26

Parallel Workload (rename) with blob store failed, main history:

Unknown error in workflow-default:

materialize.parallel_workload.worker_exception.WorkerFailedException: ^^^ +++ Thread worker_14 failed, exiting

('terminating connection due to idle-in-transaction timeout', 'SET REAL_TIME_RECENCY TO FALSE;')

Test details & reproducer

Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.

bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=rename --threads=16 --azurite

Unused dependenciesbin/ci-builder run nightly bin/unused-deps

Ran in 20m 18s

🏎️ testdrive 4 replicas

Ran in 3h 4m

🏎️ testdrive 4 replicas

Ran in 3h 4m

Terraform + Helm Chart E2E on Azure

Ran in 33m 57s

Total Job Run Time: 5d 9h