Test

Testdrive 5 failed, main history:

• Unknown error in kafka-progress.td:

kafka-progress.td:41:1: non-matching rows: expected:
[["                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nsource materialize.public.data_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"]]
got:
[["                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nconstraints (experimental):\nlower:\n  [<> <>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.data_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"]]
got raw rows:
[["                query timestamp: 1742828182068 (2025-03-24 14:56:22.068)\n          oracle read timestamp: 1742828182068 (2025-03-24 14:56:22.068)\nlargest not in advance of upper: 1742828183000 (2025-03-24 14:56:23.000)\n                          upper:[1742828183001 (2025-03-24 14:56:23.001)]\n                          since:[1742828182000 (2025-03-24 14:56:22.000)]\n        can respond immediately: true\n                       timeline: Some(EpochMilliseconds)\n              session wall time: 1742828183019 (2025-03-24 14:56:23.019)\n\nconstraints (experimental):\nlower:\n  [1742828182068 (2025-03-24 14:56:22.068)]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.data_progress (u39, storage):\n                  read frontier:[1742828182000 (2025-03-24 14:56:22.000)]\n                 write frontier:[1742828183001 (2025-03-24 14:56:23.001)]\n"]]
Poor diff:
+ "                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nconstraints (experimental):\nlower:\n  [<> <>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.data_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"
- "                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nsource materialize.public.data_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"

     |
  40 | $ set-regex match=(\s{12}0|\d{13,20}|u\d{1,5}|\(\d+-\d\d-\d\d\s\d\d:\d\d:\d\d\.\d\d\d\)|true|false) replacement=<>
  41 | > EXPLAIN TIMESTAMP FOR SELECT * FROM data_progress
     | ^

BUILDKITE_PARALLEL_JOB=4 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default 

Postgres CDC tests 2 failed, main history:

• Unknown error in cdc:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-a563d197/materialize/test/test/pg-cdc run -eCLUSTER_REPLICA_SIZES testdrive --var=ssl-ca=<CERTIFICATE>  --var=ssl-cert=<CERTIFICATE>  --var=ssl-key=<PRIVATE KEY>  --var=ssl-wrong-cert=<CERTIFICATE>  --var=ssl-wrong-key=<PRIVATE KEY>  --var=default-replica-size=4-4 --var=default-storage-size=4-1 pg-cdc.td

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
files involved: pg-cdc.td

BUILDKITE_PARALLEL_JOB=1 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find pg-cdc run default 

Checks without restart or upgrade 5 failed, main history:

• Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-475442d2/materialize/test/test/platform-checks exec -T testdrive testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --materialize-param=statement_timeout='300s' --default-timeout=300s --seed=1 --persist-blob-url=s3://minioadmin:minioadmin@persist/persist?endpoint=http://minio:9000/&region=minio --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --var=replicas=1 --var=default-replica-size=4-4 --var=default-storage-size=4-1 --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-475442d2/materialize/test/misc/python/materialize/checks/all_checks/materialized_views.py:255

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-475442d2/materialize/test/misc/python/materialize/checks/all_checks/materialized_views.py:255

BUILDKITE_PARALLEL_JOB=4 BUILDKITE_PARALLEL_JOB_COUNT=6 bin/mzcompose --find platform-checks run default --scenario=NoRestartNoUpgrade --seed=0195c888-028b-45c1-9c57-cc0bb4402e59 

Testdrive 7 failed, main history:

• Unknown error in explain-timestamps.td:

explain-timestamps.td:19:1: non-matching rows: expected:
[["                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nsource materialize.public.t1 (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"]]
got:
[["                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nconstraints (experimental):\nlower:\n  [<> <>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.t1 (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"]]
got raw rows:
[["                query timestamp: 1742828153125 (2025-03-24 14:55:53.125)\n          oracle read timestamp: 1742828153125 (2025-03-24 14:55:53.125)\nlargest not in advance of upper: 1742828153125 (2025-03-24 14:55:53.125)\n                          upper:[1742828153126 (2025-03-24 14:55:53.126)]\n                          since:[1742828152000 (2025-03-24 14:55:52.000)]\n        can respond immediately: true\n                       timeline: Some(EpochMilliseconds)\n              session wall time: 1742828153173 (2025-03-24 14:55:53.173)\n\nconstraints (experimental):\nlower:\n  [1742828153125 (2025-03-24 14:55:53.125)]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.t1 (u104, storage):\n                  read frontier:[1742828152000 (2025-03-24 14:55:52.000)]\n                 write frontier:[1742828153126 (2025-03-24 14:55:53.126)]\n"]]
Poor diff:
+ "                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nconstraints (experimental):\nlower:\n  [<> <>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.t1 (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"
- "                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nsource materialize.public.t1 (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"

     |
  10 | $ postgres-execute c ... [rest of line truncated for security]
  18 | > SET TRANSACTION_ISOLATION = 'STRICT SERIALIZABLE';
  19 | > EXPLAIN TIMESTAMP FOR SELECT * FROM t1
     | ^

BUILDKITE_PARALLEL_JOB=6 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default 

Checks + restart of environmentd & storage clusterd 5 failed, main history:

• Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-1ac4974e/materialize/test/test/platform-checks exec -T testdrive testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@materialized:6875 --materialize-internal-url=postgres://materialize@materialized:6877 --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --no-reset --materialize-param=statement_timeout='300s' --default-timeout=300s --seed=1 --persist-blob-url=s3://minioadmin:minioadmin@persist/persist?endpoint=http://minio:9000/&region=minio --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --var=replicas=1 --var=default-replica-size=4-4 --var=default-storage-size=4-1 --source=/var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-1ac4974e/materialize/test/misc/python/materialize/checks/all_checks/materialized_views.py:255

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-1ac4974e/materialize/test/misc/python/materialize/checks/all_checks/materialized_views.py:255

BUILDKITE_PARALLEL_JOB=4 BUILDKITE_PARALLEL_JOB_COUNT=6 bin/mzcompose --find platform-checks run default --scenario=RestartEnvironmentdClusterdStorage --seed=0195c888-028b-45c1-9c57-cc0bb4402e59 

Testdrive 4 failed, main history:

• Unknown error in load-generator.td:

load-generator.td:188:1: non-matching rows: expected:
[["                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nsource materialize.another.auction_house_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"]]
got:
[["                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nconstraints (experimental):\nlower:\n  [<> <>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.another.auction_house_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"]]
got raw rows:
[["                query timestamp: 1742828243132 (2025-03-24 14:57:23.132)\n          oracle read timestamp: 1742828243132 (2025-03-24 14:57:23.132)\nlargest not in advance of upper: 1742828244000 (2025-03-24 14:57:24.000)\n                          upper:[1742828244001 (2025-03-24 14:57:24.001)]\n                          since:[1742828243000 (2025-03-24 14:57:23.000)]\n        can respond immediately: true\n                       timeline: Some(EpochMilliseconds)\n              session wall time: 1742828244114 (2025-03-24 14:57:24.114)\n\nconstraints (experimental):\nlower:\n  [1742828243132 (2025-03-24 14:57:23.132)]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.another.auction_house_progress (u195, storage):\n                  read frontier:[1742828243000 (2025-03-24 14:57:23.000)]\n                 write frontier:[1742828244001 (2025-03-24 14:57:24.001)]\n"]]
Poor diff:
+ "                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nconstraints (experimental):\nlower:\n  [<> <>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.another.auction_house_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"
- "                query timestamp: <> <>\n          oracle read timestamp: <> <>\nlargest not in advance of upper: <> <>\n                          upper:[<> <>]\n                          since:[<> <>]\n        can respond immediately: <>\n                       timeline: Some(EpochMilliseconds)\n              session wall time: <> <>\n\nsource materialize.another.auction_house_progress (<>, storage):\n                  read frontier:[<> <>]\n                 write frontier:[<> <>]\n"

     |
  13 | $ postgres-execute c ... [rest of line truncated for security]
 187 | $ set-regex match=(\s{12}0|\d{13,20}|u\d{1,5}|\(\d+-\d\d-\d\d\s\d\d:\d\d:\d\d\.\d\d\d\)|true|false) replacement=<>
 188 | > EXPLAIN TIMESTAMP FOR SELECT * FROM another.auction_house_progress
     | ^

BUILDKITE_PARALLEL_JOB=3 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find testdrive run default 

Yugabyte CDC tests failed, main history:

• Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-986a9eb7/materialize/test/test/yugabyte-cdc run -eCLUSTER_REPLICA_SIZES testdrive --no-reset --max-errors=1 --seed=19872 --var=default-replica-size=4-4 yugabyte-cdc.td

^^^ +++
testdrive: maximum number of errors reached; giving up
+++ !!! Error Report
1 errors were encountered during execution
files involved: yugabyte-cdc.td

bin/mzcompose --find yugabyte-cdc run default 

Cargo test failed, main history:

• Unknown error in run.log:

        FAIL [  11.410s] mz-environmentd::sql test_explain_timestamp_table

(2 occurrences)

• Unknown error in workflow-default:

subprocess.CalledProcessError: Command '['cargo', 'nextest', 'run', '--workspace', '--all-features', '--profile=ci', '--cargo-profile=ci', '--partition=count:1/1']' returned non-zero exit status 100.

• Unknown error in test_explain_timestamp_table:

thread 'test_explain_timestamp_table' panicked at src/environmentd/tests/sql.rs:1246:5:
assertion `left == right` failed:                 query timestamp:<TIMESTAMP>
          oracle read timestamp:<TIMESTAMP>
largest not in advance of upper:<TIMESTAMP>
                          upper:[<TIMESTAMP>]
                          since:[<TIMESTAMP>]
        can respond immediately: true
                       timeline: Some(EpochMilliseconds)
              session wall time:<TIMESTAMP>

constraints (experimental):
lower:
  [<TIMESTAMP>]: (StorageInput([User(1)]))
  [<TIMESTAMP>]: (ComputeInput([]))
  [<TIMESTAMP>]: (IsolationLevel(StrictSerializable))


source materialize.public.t1 (u1, storage):
                  read frontier:[<TIMESTAMP>]
                 write frontier:[<TIMESTAMP>]


                query timestamp:<TIMESTAMP>
          oracle read timestamp:<TIMESTAMP>
largest not in advance of upper:<TIMESTAMP>
                          upper:[<TIMESTAMP>]
                          since:[<TIMESTAMP>]
        can respond immediately: true
                       timeline: Some(EpochMilliseconds)
              session wall time:<TIMESTAMP>

source materialize.public.t1 (u1, storage):
                  read frontier:[<TIMESTAMP>]
                 write frontier:[<TIMESTAMP>]

  left: "                query timestamp:<TIMESTAMP>\n          oracle read timestamp:<TIMESTAMP>\nlargest not in advance of upper:<TIMESTAMP>\n                          upper:[<TIMESTAMP>]\n                          since:[<TIMESTAMP>]\n        can respond immediately: true\n                       timeline: Some(EpochMilliseconds)\n              session wall time:<TIMESTAMP>\n\nconstraints (experimental):\nlower:\n  [<TIMESTAMP>]: (StorageInput([User(1)]))\n  [<TIMESTAMP>]: (ComputeInput([]))\n  [<TIMESTAMP>]: (IsolationLevel(StrictSerializable))\n\n\nsource materialize.public.t1 (u1, storage):\n                  read frontier:[<TIMESTAMP>]\n                 write frontier:[<TIMESTAMP>]\n"
 right: "                query timestamp:<TIMESTAMP>\n          oracle read timestamp:<TIMESTAMP>\nlargest not in advance of upper:<TIMESTAMP>\n                          upper:[<TIMESTAMP>]\n                          since:[<TIMESTAMP>]\n        can respond immediately: true\n                       timeline: Some(EpochMilliseconds)\n              session wall time:<TIMESTAMP>\n\nsource materialize.public.t1 (u1, storage):\n                  read frontier:[<TIMESTAMP>]\n                 write frontier:[<TIMESTAMP>]\n"
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed::<alloc::borrow::Cow<str>, &str>
             at /usr/local/lib/rustlib/src/rust/library/core/src/panicking.rs:367:5
   4: test_impl
             at ./tests/sql.rs:1246:5
   5: sql::test_explain_timestamp_table
             at ./tests/sql.rs:1220:1
   6: sql::test_explain_timestamp_table::{closure#0}
             at ./tests/sql.rs:1220:16
   7: <sql::test_explain_timestamp_table::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at /usr/local/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

bin/mzcompose --find cargo-test run default 

Kafka RTR tests 1 failed, main history:

• Unknown error in simple:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-16cpu-32gb-efebe5c0/materialize/test/test/kafka-rtr run -eCLUSTER_REPLICA_SIZES testdrive --max-errors=1 --seed=24481 --temp-dir=/share/tmp/kafka-resumption-24481 simple/toxiproxy-setup.td simple/mz-setup.td simple/verify-rtr.td

^^^ +++
testdrive: maximum number of errors reached; giving up
+++ !!! Error Report
1 errors were encountered during execution
files involved: simple/verify-rtr.td

BUILDKITE_PARALLEL_JOB=0 BUILDKITE_PARALLEL_JOB_COUNT=2 bin/mzcompose --find kafka-rtr run default