Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#11549

main/31887a0267

Failed in 6h 2m

Miri test (full)

Extended SSH connection tests

CRDB rolling restarts

PubSub disruption

Test for incident 70

Tests for balancerd

CRDB / Persist backup and restore

Postgres / Persist backup and restore

Created Wed 19th Mar at 11:31 PM

Triggered from Pipeline Schedule

🏎️ testdrive 4 replicas failed, main history:

Unknown error in alter-sink.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in session.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in materialized-view-refresh-options.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-to-s3-minio.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-avro-sinks-defaults.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-1947.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in privilege-checks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in blue-green.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-decode-uuid.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in test-skip-if.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in correctness-property-2.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-5502.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-4315.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in controller-frontiers.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-6744.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-json-sinks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-resolution-types-map.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in frontier-rounding.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-resolution-union-concrete.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-from-s3-minio.td:

copy-from-s3-minio.td:41:1: preparing query failed: connection closed
     |
  15 | $ postgres-execute c ... [rest of line truncated for security]
  24 | $ s3-set-presigned-u ... [rest of line truncated for security]
  26 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
  37 | $ s3-set-presigned-u ... [rest of line truncated for security]
  39 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
  40 | 
  41 | > SELECT * FROM t1;
     | ^

Unknown error in avro-nonnull-record.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in subquery-scalar-errors.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Known issue thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2266:33: no sender for c1dad1a4-6d47-4d90-a269-7f2d9118ca6d! (#9095) in services.log:

testdrive-materialized-1     | 2025-03-20T01:33:08.253863Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2380:33: no sender for 06aa6783-fb9e-469c-aff6-a34f3599306c!

Test details & reproducer

Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result.

bin/mzcompose --find testdrive run default --replicas=4

Checks 0dt upgrade across four versions 1 succeeded with known error logs, main history:

Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:

platform-checks-mz_4-1              | 2025-03-20T01:02:58.308948Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:850:17: dependency since has advanced past dependent (u479) upper

Test details & reproducer

Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.

BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMzFourVersions --seed=0195b0bf-0a29-4551-b85c-cfae936dff4e

Checks 0dt upgrade across two versions 2 succeeded with known error logs, main history:

Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:

platform-checks-mz_4-1              | 2025-03-20T00:58:21.983498Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:976:17: dependency since has advanced past dependent (u416) upper

Test details & reproducer

Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.

BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find platform-checks run default --scenario=ZeroDowntimeUpgradeEntireMzTwoVersions --seed=0195b0bf-0a29-4551-b85c-cfae936dff4e

Parallel Workload (0dt deploy) succeeded with known error logs, main history:

Known issue parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper (#8425) in services.log:

parallel-workload-materialized2-1    | 2025-03-20T00:03:21.123689Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:976:17: dependency since has advanced past dependent (u345) upper

Test details & reproducer

Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.

bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16

Terraform + Helm Chart E2E on Azure failed, main history:

Unknown error in run.log:

[31mERROR[0m [2mmz_self_managed_debug[0m[2m:[0m self-managed-debug: fatal: failed to read kubeconfig from '"/kube/config"': No such file or directory (os error 2): No such file or directory (os error 2)

Unknown error in workflow-azure-temporary:

subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve']' returned non-zero exit status 1.

Test details & reproducer

Tests the mz command line tool against a real Cloud instance

bin/mzcompose --find terraform run azure-temporary

🏎️ testdrive 4 replicas

Ran in 3h 5m

Terraform + Helm Chart E2E on Azure

Ran in 1h 3m

Total Job Run Time: 4d 20h