Test

Public

Run fast unit and integration tests

DNM: Try 8 replicas

#102261

def-:pr-8repli/b3ec86ca80e1(#32202)

Failed in 1h 0m

Feature benchmark (Kafka only)

Persistence tests

Cluster isolation test

chbench smoke test

Metabase smoke test

dbt-materialize tests

Storage Usage Table Test

Tracing Fast Path

mz-debug tool

Dennis Felsing

Created Tue 15th Apr at 5:32 AM

Triggered from Webhook

Testdrive 6 failed, main history:

Unknown error in journalctl-merge.log:

Apr 15 05:58:49 hetzner-aarch64-16cpu-32gb-11b70d47 kernel: Memory cgroup out of memory: Killed process 13651 (environmentd) total-vm:1927684kB, anon-rss:673832kB, file-rss:136252kB, shmem-rss:0kB, UID:999 pgtables:3400kB oom_score_adj:0

Unknown error in materialization-lag.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in replica-expiration.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-time-offset.td:

kafka-time-offset.td:378:1: preparing query failed: connection closed
     |
  13 | $ postgres-execute c ... [rest of line truncated for security]
  29 |     URL '${testdrive ... [rest of line truncated for security]
 377 | 
 378 | > CREATE TABLE relative_time_offset_30_years_ago_tbl FROM SOURCE relative_time_offset_30_years_ago (REFERENCE "testdrive-t3-${testdrive.seed}")
     | ^

Unknown error in explain-pushdown.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in fetch-concurrent-same-source.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in nulls.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Test details & reproducer

Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result.

BUILDKITE_PARALLEL_JOB=5 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default

Testdrive 4 failed, main history:

Unknown error in journalctl-merge.log:

Apr 15 05:57:56 hetzner-aarch64-16cpu-32gb-763a004c kernel: Memory cgroup out of memory: Killed process 13549 (environmentd) total-vm:2044848kB, anon-rss:953024kB, file-rss:135424kB, shmem-rss:0kB, UID:999 pgtables:3592kB oom_score_adj:0

Unknown error in kafka-compression.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in index-source-stuck.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-exactly-once-sink.td:

kafka-exactly-once-sink.td:353:1: retrieving topic name: connection closed
     |
  13 | $ postgres-execute c ... [rest of line truncated for security]
 167 |     URL '${testdrive ... [rest of line truncated for security]
 352 | 
 353 | $ kafka-verify-topic sink=materialize.public.compaction_test_sink await-value-schema=true
     | ^

Unknown error in dataflow-dependencies.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in load-generator.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in rename.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Test details & reproducer

BUILDKITE_PARALLEL_JOB=3 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default

Testdrive 8 failed, main history:

Unknown error in functions.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in protobuf-corrupted.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in avro-decode-mismatched-columns.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in copy-from-s3-minio.td:

copy-from-s3-minio.td:26:1: executing query failed: connection closed
     |
  15 | $ postgres-execute c ... [rest of line truncated for security]
  24 | $ s3-set-presigned-u ... [rest of line truncated for security]
  25 | 
  26 | > COPY INTO t1 FROM  ... [rest of line truncated for security]
     | ^

Unknown error in tpch.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-1744.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in connection-create-drop.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Known issue thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2266:33: no sender for c1dad1a4-6d47-4d90-a269-7f2d9118ca6d! (#9095) in services.log:

testdrive-materialized-1     | 2025-04-15T05:50:57.797255Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:2380:33: no sender for 40de0265-a637-41b6-af1a-2bf998784c1d!

Test details & reproducer

BUILDKITE_PARALLEL_JOB=7 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default

Testdrive 1 failed, main history:

Unknown error in journalctl-merge.log:

Apr 15 06:04:48 hetzner-aarch64-16cpu-32gb-175ec622 kernel: Memory cgroup out of memory: Killed process 13707 (environmentd) total-vm:2310260kB, anon-rss:781496kB, file-rss:138880kB, shmem-rss:0kB, UID:999 pgtables:4176kB oom_score_adj:0

Unknown error in map.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-envelope-none.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in kafka-include-key-sources.td:

kafka-include-key-sources.td:333:1: executing query failed: connection closed
     |
  13 | $ postgres-execute c ... [rest of line truncated for security]
  55 |     URL '${testdrive ... [rest of line truncated for security]
 332 | 
 333 | > CREATE TABLE input_proto_tbl FROM SOURCE input_proto (REFERENCE "testdrive-proto-${testdrive.seed}")
     | ^

Unknown error in introspection-sources.td:

introspection-sources.td:32:1: executing query failed: db error: ERROR: log source reads must target a replica
DETAIL: The query references the following log sources:
    mz_compute_import_frontiers_per_worker
    mz_compute_exports_per_worker
HINT: Use `SET cluster_replica = <replica-name>` to target a specific replica in the active cluster. Note that subsequent queries will only be answered by the selected replica, which might reduce availability. To undo the replica selection, use `RESET cluster_replica`.: ERROR: log source reads must target a replica
DETAIL: The query references the following log sources:
    mz_compute_import_frontiers_per_worker
    mz_compute_exports_per_worker
HINT: Use `SET cluster_replica = <replica-name>` to target a specific replica in the active cluster. Note that subsequent queries will only be answered by the selected replica, which might reduce availability. To undo the replica selection, use `RESET cluster_replica`.
     |
  21 | $ postgres-execute c ... [rest of line truncated for security]
  31 | 
  32 | > SELECT COUNT(*)
     | ^

Unknown error in materializations.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Unknown error in github-6335.td:

error connecting to server: failed to lookup address information: Name or service not known: failed to lookup address information: Name or service not known

Test details & reproducer

BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default

Restart test failed, main history:

Unknown error in bound-size-mz-cluster-replica-metrics-history:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-8abae781/materialize/test/test/restart exec -T testdrive_no_reset testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --default-timeout=360s --persist-blob-url=file:///mzdata/persist/blob --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-8abae781/materialize/test/test/restart/mzcompose.py:718

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-8abae781/materialize/test/test/restart/mzcompose.py:718

Unknown error in bound-size-mz-status-history:

Docker compose failed: docker compose -f/dev/fd/12 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-8abae781/materialize/test/test/restart exec -T testdrive_no_reset testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --default-timeout=360s --persist-blob-url=file:///mzdata/persist/blob --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-8abae781/materialize/test/test/restart/mzcompose.py:645

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-8abae781/materialize/test/test/restart/mzcompose.py:645

Test details & reproducer

Testdrive-based tests involving restarting materialized (including its clusterd processes). See cluster tests for separate clusterds, see platform-checks for further restart scenarios.

bin/mzcompose --find restart run default

Testdrive 5 failed, main history:

Unknown error in audit-log.td:

audit-log.td:32:1: non-matching rows: expected:
[["create", "cluster-replica", "{\"cluster_id\":\"<GID>\",\"cluster_name\":\"audit_log_kafka_src\",\"disk\":true,\"logical_size\":\"4-1\",\"reason\":\"manual\",\"replica_name\":\"r1\"}", "materialize"], ["create", "source", "{\"cluster_id\":\"<GID>\",\"database\":\"materialize\",\"id\":\"<GID>\",\"item\":\"kafka_src\",\"schema\":\"public\",\"type\":\"kafka\"}", "materialize"], ["create", "source", "{\"cluster_id\":null,\"database\":\"materialize\",\"id\":\"<GID>\",\"item\":\"kafka_src_progress\",\"schema\":\"public\",\"type\":\"progress\"}", "materialize"], ["create", "table", "{\"database\":\"materialize\",\"id\":\"<GID>\",\"item\":\"kafka_src_tbl\",\"schema\":\"public\"}", "materialize"]]
got:
[["create", "cluster-replica", "{\"cluster_id\":\"<GID>\",\"cluster_name\":\"audit_log_kafka_src\",\"disk\":true,\"logical_size\":\"4-1\",\"reason\":\"manual\",\"replica_name\":\"r8\"}", "materialize"], ["create", "source", "{\"cluster_id\":\"<GID>\",\"database\":\"materialize\",\"id\":\"<GID>\",\"item\":\"kafka_src\",\"schema\":\"public\",\"type\":\"kafka\"}", "materialize"], ["create", "source", "{\"cluster_id\":null,\"database\":\"materialize\",\"id\":\"<GID>\",\"item\":\"kafka_src_progress\",\"schema\":\"public\",\"type\":\"progress\"}", "materialize"], ["create", "table", "{\"database\":\"materialize\",\"id\":\"<GID>\",\"item\":\"kafka_src_tbl\",\"schema\":\"public\"}", "materialize"]]
got raw rows:
[["create", "table", "{\"database\":\"materialize\",\"id\":\"u245\",\"item\":\"kafka_src_tbl\",\"schema\":\"public\"}", "materialize"], ["create", "source", "{\"cluster_id\":\"u53\",\"database\":\"materialize\",\"id\":\"u244\",\"item\":\"kafka_src\",\"schema\":\"public\",\"type\":\"kafka\"}", "materialize"], ["create", "source", "{\"cluster_id\":null,\"database\":\"materialize\",\"id\":\"u243\",\"item\":\"kafka_src_progress\",\"schema\":\"public\",\"type\":\"progress\"}", "materialize"], ["create", "cluster-replica", "{\"cluster_id\":\"u53\",\"cluster_name\":\"audit_log_kafka_src\",\"disk\":true,\"logical_size\":\"4-1\",\"reason\":\"manual\",\"replica_name\":\"r8\"}", "materialize"]]
Poor diff:
- create cluster-replica "{\"cluster_id\":\"<GID>\",\"cluster_name\":\"audit_log_kafka_src\",\"disk\":true,\"logical_size\":\"4-1\",\"reason\":\"manual\",\"replica_name\":\"r1\"}" materialize
+ create cluster-replica "{\"cluster_id\":\"<GID>\",\"cluster_name\":\"audit_log_kafka_src\",\"disk\":true,\"logical_size\":\"4-1\",\"reason\":\"manual\",\"replica_name\":\"r8\"}" materialize

     |
  31 | 
  32 | > SELECT event_type, object_type, details - 'replica_id' - 'billed_as' - 'internal', user FROM mz_audit_events ORDER BY id DESC LIMIT 4
     | ^

Test details & reproducer

BUILDKITE_PARALLEL_JOB=4 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default

Cluster tests 2 failed, main history:

Unknown error in test-refresh-mv-warmup:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-5c00a78f/materialize/test/test/cluster exec -T testdrive testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --default-timeout=360s --persist-blob-url=file:///mzdata/persist/blob --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-5c00a78f/materialize/test/test/cluster/mzcompose.py:3838

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-5c00a78f/materialize/test/test/cluster/mzcompose.py:3838

Unknown error in test-storage-controller-metrics:

builtins.AssertionError: got 8.0

Test details & reproducer

Functional tests which require separate clusterd containers (instead of the usual clusterd included in the materialized container).

BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=4 bin/mzcompose --find cluster run default

Testdrive 7 failed, main history:

Unknown error in source-statistics.td:

source-statistics.td:233:1: non-matching rows: expected:
[["upsert_tbl", "true", "4", "4", "true", "2"]]
got:
[["upsert_tbl", "true", "14", "4", "true", "2"]]
Poor diff:
+ upsert_tbl true 14 4 true 2
- upsert_tbl true 4 4 true 2

     |
  12 | $ postgres-execute c ... [rest of line truncated for security]
  57 |     URL '${testdrive ... [rest of line truncated for security]
 232 | 
 233 | > SELECT t.name,
     | ^

Unknown error in webhook.td:

webhook.td:545:1: executing query failed: db error: ERROR: cannot create webhook source in cluster with more than one replica: ERROR: cannot create webhook source in cluster with more than one replica
     |
 197 | > CREATE SECRET webh ... [rest of line truncated for security]
 199 | > CREATE SOURCE webh ... [rest of line truncated for security]
 202 |     WITH ( HEADERS,  ... [rest of line truncated for security]
 203 |     decode(headers-> ... [rest of line truncated for security]
 206 | $ webhook-append nam ... [rest of line truncated for security]
 207 | using an mz secret ... [rest of line truncated for security]
 209 | > SELECT * FROM webh ... [rest of line truncated for security]
 210 | "using an mz secret" ... [rest of line truncated for security]
 235 | # Can use SECRETs as ... [rest of line truncated for security]
 237 | > CREATE SECRET webh ... [rest of line truncated for security]
 245 |       SECRET webhook ... [rest of line truncated for security]
 246 |       SECRET webhook ... [rest of line truncated for security]
 248 |     decode(headers-> ... [rest of line truncated for security]
 250 |     decode(headers-> ... [rest of line truncated for security]
 429 |     decode(headers-> ... [rest of line truncated for security]
 544 | 
 545 | > CREATE SOURCE webhook_in_current FROM WEBHOOK BODY FORMAT JSON;
     | ^

Test details & reproducer

BUILDKITE_PARALLEL_JOB=6 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default

Cluster tests 3 failed, main history:

Unknown error in test-workload-class-in-metrics:

psycopg.errors.FeatureNotSupported: log source reads must target a replica DETAIL:  The query references the following log sources:     mz_dataflow_operators_per_worker HINT:  Use `SET cluster_replica = <replica-name>` to target a specific replica in the active cluster. Note that subsequent queries will only be answered by the selected replica, which might reduce availability. To undo the replica selection, use `RESET cluster_replica`.

Unknown error in test-zero-downtime-reconfigure:

builtins.AssertionError: Cluster should only have one replica prior to alter, found [('r1',), ('r2',), ('r3',), ('r4',), ('r5',), ('r6',), ('r7',), ('r8',)]

Test details & reproducer

Functional tests which require separate clusterd containers (instead of the usual clusterd included in the materialized container).

BUILDKITE_PARALLEL_JOB=2 BUILDKITE_PARALLEL_JOB_COUNT=4 bin/mzcompose --find cluster run default

Checks without restart or upgrade 3 failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-bb109937/materialize/test/test/platform-checks exec -T testdrive testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --materialize-param=statement_timeout='300s' --default-timeout=300s --seed=1 --persist-blob-url=s3://minioadmin:minioadmin@persist/persist?endpoint=http://minio:9000/&region=minio --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --var=replicas=1 --var=default-replica-size=4-4 --var=default-storage-size=4-1 --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-bb109937/materialize/test/misc/python/materialize/checks/all_checks/cluster.py:37

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-bb109937/materialize/test/misc/python/materialize/checks/all_checks/cluster.py:37

Test details & reproducer

Write a single set of .td fragments for a particular feature or functionality and then have Zippy execute them in upgrade, 0dt-upgrade, restart, recovery and failure contexts.

BUILDKITE_PARALLEL_JOB=2 BUILDKITE_PARALLEL_JOB_COUNT=6 bin/mzcompose --find platform-checks run default --scenario=NoRestartNoUpgrade --seed=019637ef-609f-4ab6-afd3-c9556c0d6e71

Cargo test

Ran in 22m 13s

1/8

Testdrive 1

Ran in 42m 36s

2/8

Testdrive 2

Ran in 30m 20s

3/8

Testdrive 3

Ran in 30m 6s

4/8

Testdrive 4

Ran in 43m 39s

5/8

Testdrive 5

Ran in 31m 29s

6/8

Testdrive 6

Ran in 43m 43s

7/8

Testdrive 7

Ran in 30m 42s

8/8

Testdrive 8

Ran in 43m 9s

1/4

Cluster tests 1

Ran in 29m 46s

2/4

Cluster tests 2

Ran in 31m 17s

3/4

Cluster tests 3

Ran in 30m 39s

1/6

Fast SQL logic tests 1

Ran in 4m 55s

2/6

Fast SQL logic tests 2

Ran in 5m 51s

3/6

Fast SQL logic tests 3

Ran in 4m 48s

4/6

Fast SQL logic tests 4

Ran in 9m 21s

5/6

Fast SQL logic tests 5

Ran in 9m 32s

6/6

Fast SQL logic tests 6

Ran in 9m 4s

Restart test

Ran in 31m 35s

1/2

Legacy upgrade tests (last version from docs, ignore missing) 1

Ran in 29m 47s

2/2

Legacy upgrade tests (last version from docs, ignore missing) 2

Ran in 29m 21s

2/2

MySQL CDC tests 2

Ran in 32m 31s

2/2

Postgres CDC tests 2

Ran in 32m 17s

Yugabyte CDC tests

Ran in 32m 47s

1/6

Checks + restart of environmentd & storage clusterd 1

Ran in 27m 12s

3/6

Checks + restart of environmentd & storage clusterd 3

Ran in 23m 43s

5/6

Checks + restart of environmentd & storage clusterd 5

Ran in 26m 55s

1/6

Checks without restart or upgrade 1

Ran in 9m 46s

3/6

Checks without restart or upgrade 3

Ran in 28m 31s

5/6

Checks without restart or upgrade 5

Ran in 19m 41s

Feature benchmark (Kafka only)

Ran in 15m 48s

dbt-materialize tests

Ran in 28m 27s

Total Job Run Time: 23h 22m