Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#10595

main/ae9877708e

Failed in 8h 32m

Miri test (full)

Extended SSH connection tests

CRDB rolling restarts

PubSub disruption

Tests for balancerd

Legacy upgrade tests (last version from git)

CRDB / Persist backup and restore

Postgres / Persist backup and restore

Replica isolation

Zero downtime

Created Tue 3rd Dec 2024 at 12:30 AM

Triggered from Pipeline Schedule

Zippy w/ latest CRDB (#2) failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-426c31b3/materialize/nightly/test/zippy exec -T cockroach cockroach sql --insecure -e                  CREATE EXTERNAL CONNECTION backup_bucket                 AS 's3://persist/crdb-backup?AWS_ENDPOINT=http://minio:9000/&AWS_REGION=minio&AWS_ACCESS_KEY_ID=minioadmin&AWS_SECRET_ACCESS_KEY=minioadmin';                 BACKUP INTO 'external://backup_bucket';                 DROP EXTERNAL CONNECTION backup_bucket;

Test details & reproducer

Zippy generates a pseudo-random sequence of DDLs, DMLs, failures, data validation and other events and runs it sequentially. By keeping track of the expected state it can verify results for correctness.

bin/mzcompose --find zippy run default --scenario=KafkaSources --actions=10000 --cockroach-tag=latest --max-execution-time=30m

🏎️ testdrive 4 replicas failed, main history:

Unknown error in journalctl-merge.log:

Dec 03 01:12:41 hetzner-aarch64-16cpu-32gb-63f9f3fa audit[341672]: ANOM_ABEND auid=4294967295 uid=999 gid=999 ses=4294967295 subj=system_u:system_r:spc_t:s0 pid=341672 comm="rdk:main" exe="/usr/local/bin/materialized" sig=11 res=1

Unknown error in kafka-recreate-topic.td:

kafka-recreate-topic.td:91:1: connection closed
     |
  16 |     URL '${testdrive ... [rest of line truncated for security]
  90 | 
  91 | ! SELECT * FROM source1_tbl
     | ^

Unknown error in kafka-sink-empty-progress-topic.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-sink-errors.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-sink-headers.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-sink-partition-by.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-sink-statistics.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-sink-topic-config.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-sinks.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-source-errors.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-start-offset.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Test details & reproducer

Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result.

bin/mzcompose --find testdrive run default --replicas=4

Zippy w/ latest CRDB failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-884ba8f9/materialize/nightly/test/zippy exec -T cockroach cockroach sql --insecure -e                  CREATE EXTERNAL CONNECTION backup_bucket                 AS 's3://persist/crdb-backup?AWS_ENDPOINT=http://minio:9000/&AWS_REGION=minio&AWS_ACCESS_KEY_ID=minioadmin&AWS_SECRET_ACCESS_KEY=minioadmin';                 BACKUP INTO 'external://backup_bucket';                 DROP EXTERNAL CONNECTION backup_bucket;

Test details & reproducer

bin/mzcompose --find zippy run default --scenario=KafkaSources --actions=10000 --cockroach-tag=latest --max-execution-time=30m

SQLsmith explain succeeded with known error logs, main history:

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

sqlsmith-mz_1-1  | thread 'tokio:work-13' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Use SQLsmith to generate random queries (AST/code based) and run them against Materialize: https://github.com/MaterializeInc/sqlsmith The queries can be complex, but we can't verify correctness or performance.

bin/mzcompose --find sqlsmith run default --max-joins=5 --explain-only --runtime=1500

Output consistency (feature-flags for DFR) failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-6269003f/materialize/nightly/test/feature-flag-consistency up --detach --wait --quiet-pull mz_this mz_other

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

feature-flag-consistency-mz_this-1  | thread 'tokio:work-1' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Verify that Materialize has the same results with different sets of feature flags.

bin/mzcompose --find feature-flag-consistency run default --seed=019389ec-faf4-464f-9869-0bc2f92f73e1 --max-runtime-in-sec=1200 --evaluation-strategy=dataflow_rendering

Zippy w/ latest CRDB

Ran in 34m 23s

Zippy w/ latest CRDB

Ran in 35m 28s

Output consistency (feature-flags for DFR)

Ran in 12m 23s

Total Job Run Time: 4d 4h