Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#11710
/
main/e95dfdaf25
Failed in 5h 59m
:pipeline:
:rust: Miri test (full)
Extended SSH connection tests
CRDB rolling restarts
PubSub disruption
Test for incident 70
Tests for balancerd
CRDB / Persist backup and restore
Postgres / Persist backup and restore
Replica isolation
Zero downtime
Materialize Emulator
Cluster tests with :azure: blob store
Created
Triggered from Pipeline Schedule

:rust: Miri test (full) failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

        FAIL [ 342.579s] mz-expr scalar::func::binary::test::test_fallible1

(2 occurrences)

        FAIL [ 352.722s] mz-expr scalar::func::binary::test::test_fallible2

(2 occurrences)

        FAIL [ 352.136s] mz-expr scalar::func::binary::test::test_fallible3

(2 occurrences)

        FAIL [ 346.428s] mz-expr scalar::func::binary::test::test_infallible1

(2 occurrences)

        FAIL [ 345.296s] mz-expr scalar::func::binary::test::test_infallible2

(2 occurrences)

        FAIL [ 341.563s] mz-expr scalar::func::binary::test::test_infallible3

(2 occurrences)

        FAIL [ 395.997s] mz-expr scalar::func::test_add_date_time

(2 occurrences)

        FAIL [ 402.241s] mz-expr scalar::func::test_add_date_interval

(2 occurrences)

        FAIL [ 394.160s] mz-expr scalar::func::test_add_float32

(2 occurrences)

        FAIL [ 394.530s] mz-expr scalar::func::test_add_float64

(2 occurrences)

        FAIL [ 373.980s] mz-expr scalar::func::test_add_int32

(2 occurrences)

        FAIL [ 374.569s] mz-expr scalar::func::test_add_int16

(2 occurrences)

        FAIL [ 373.044s] mz-expr scalar::func::test_add_int64

(2 occurrences)

        FAIL [ 376.306s] mz-expr scalar::func::test_add_uint16

(2 occurrences)

        FAIL [ 383.045s] mz-expr scalar::func::test_add_time_interval

(2 occurrences)

        FAIL [ 378.097s] mz-expr scalar::func::test_add_uint32

(2 occurrences)

        FAIL [ 383.218s] mz-expr scalar::func::test_add_uint64

(2 occurrences)

        FAIL [ 307.633s] mz-expr-derive-impl test::insta_test_binary_arena

(2 occurrences)

        FAIL [ 332.896s] mz-expr-derive-impl test::insta_test_add_int16

(2 occurrences)

        FAIL [ 328.447s] mz-expr-derive-impl test::insta_test_complex_output_type

(2 occurrences)

        FAIL [ 261.250s] mz-expr-derive-impl test::insta_test_unary_arena

(2 occurrences)

        FAIL [ 552.955s] mz-expr scalar::func::test_round_numeric_binary

(2 occurrences)

        FAIL [ 293.561s] mz-expr-derive-impl test::insta_test_unary

(2 occurrences)

        FAIL [ 290.042s] mz-expr-derive-impl test::insta_test_unary_ref

(2 occurrences)

subprocess.CalledProcessError: Command '['bin/ci-builder', 'run', 'nightly', 'ci/test/cargo-test-miri.sh']' returned non-zero exit status 100.
Test details & reproducer Runs the Rust-based unit tests in Debug mode. 
bin/mzcompose --find cargo-test run default --miri-full

Feature benchmark against merge base or 'latest' 4 failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-x86-64-dedi-8cpu-32gb-49c1c1f3/materialize/nightly/test/feature-benchmark up --detach --wait --quiet-pull cockroach materialized clusterd
Test details & reproducer Simple benchmark of mostly individual queries using testdrive. Can find wallclock/memorys regressions in single-connection query executions, not suitable for concurrency. 
BUILDKITE_PARALLEL_JOB=3 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find feature-benchmark run default --other-tag common-ancestor

Feature benchmark against merge base or 'latest' 8 failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-x86-64-dedi-8cpu-32gb-73c98a1e/materialize/nightly/test/feature-benchmark up --detach --wait --quiet-pull cockroach materialized clusterd
Test details & reproducer Simple benchmark of mostly individual queries using testdrive. Can find wallclock/memorys regressions in single-connection query executions, not suitable for concurrency. 
BUILDKITE_PARALLEL_JOB=7 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find feature-benchmark run default --other-tag common-ancestor

Full Testdrive in Cloudtest (K8s) 2 failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

materialize.ui.CommandFailureCausedUIError: Running testdrive --materialize-url=postgres://materialize:materialize@environmentd:6875/materialize --materialize-internal-url=postgres://mz_system@environmentd:6877/materialize --kafka-addr=redpanda:9092 --schema-registry-url=http://redpanda:8081 --default-timeout=300s --log-filter=off --var=replicas=1 --var=single-replica-cluster=quickstart --var=default-storage-size=1 --var=default-replica-size=1 --cluster-replica-sizes={"bootstrap": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "2-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 2, "workers": 4}, "free": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "0", "disabled": true, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 0, "workers": 0}, "1cc": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "1C": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "1-no-disk": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "2-no-disk": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4Gi", "scale": 1, "workers": 2}, "1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "1-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 1}, "1-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 1}, "1-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 1}, "1-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 1}, "1-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "mem-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "1 GiB", "scale": 1, "workers": 1}, "2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 2}, "2-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 2}, "2-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 2}, "2-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 2}, "2-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 2}, "2-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 2, "workers": 1}, "2-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 2, "workers": 2}, "mem-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "2 GiB", "scale": 1, "workers": 2}, "4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 4}, "4-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "4-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 4}, "4-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 4}, "4-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 4}, "4-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 4, "workers": 1}, "4-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 4, "workers": 4}, "mem-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 8}, "8-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 8}, "8-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "8-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 8}, "8-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 8}, "8-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 8, "workers": 1}, "8-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "64", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 8, "workers": 8}, "mem-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 16}, "16-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 16}, "16-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 16}, "16-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "16-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 16}, "16-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 16, "workers": 1}, "16-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "256", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 16, "workers": 16}, "mem-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "32": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 32}, "32-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 32}, "32-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 32}, "32-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 32}, "32-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 32}, "32-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 32, "workers":  [...]

Traceback (most recent call last):
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-04ad04096831d920b-1/materialize/nightly/misc/python/materialize/cloudtest/k8s/testdrive.py", line 167, in _run_internal
    self.kubectl(
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-04ad04096831d920b-1/materialize/nightly/misc/python/materialize/cloudtest/k8s/api/k8s_resource.py", line 47, in kubectl
    run_process_with_error_information(
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-04ad04096831d920b-1/materialize/nightly/misc/python/materialize/cloudtest/util/common.py", line 118, in run_process_with_error_information
    raise e
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-04ad04096831d920b-1/materialize/nightly/misc/python/materialize/cloudtest/util/common.py", line 113, in run_process_with_error_information
    subprocess.run(
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['kubectl', '--context', 'kind-mzcloud', '--namespace', 'default', 'exec', '-it', 'testdrive', '--', 'testdrive', '--materialize-url=postgres://materialize:materialize@environmentd:6875/materialize', '--materialize-internal-url=postgres://mz_system@environmentd:6877/materialize', '--kafka-addr=redpanda:9092', '--schema-registry-url=http://redpanda:8081', '--default-timeout=300s', '--log-filter=off', '--var=replicas=1', '--var=single-replica-cluster=quickstart', '--var=default-storage-size=1', '--var=default-replica-size=1', '--cluster-replica-sizes={"bootstrap": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "2-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 2, "workers": 4}, "free": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "0", "disabled": true, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 0, "workers": 0}, "1cc": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "1C": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "1-no-disk": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "2-no-disk": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4Gi", "scale": 1, "workers": 2}, "1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "1-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 1}, "1-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 1}, "1-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 1}, "1-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 1}, "1-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 1}, "mem-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "1 GiB", "scale": 1, "workers": 1}, "2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 2}, "2-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 2}, "2-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 2}, "2-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 2}, "2-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 2}, "2-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 2, "workers": 1}, "2-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 2, "workers": 2}, "mem-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "2 GiB", "scale": 1, "workers": 2}, "4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 4}, "4-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "4-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 4}, "4-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 4}, "4-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 4}, "4-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 4, "workers": 1}, "4-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 4, "workers": 4}, "mem-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 8}, "8-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 8}, "8-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "8-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 8}, "8-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 8}, "8-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 8, "workers": 1}, "8-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "64", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 8, "workers": 8}, "mem-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 1, "workers": 16}, "16-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4 GiB", "scale": 1, "workers": 16}, "16-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "8 GiB", "scale": 1, "workers": 16}, "16-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "16-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "32 GiB", "scale": 1, "workers": 16}, "16-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 16, "workers": 1}, "16-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "256", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": "4Gi", "scale": 16, "workers": 16}, "mem-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null [...]
Test details & reproducer 
bin/pytest --junitxml=junit_cloudtest_01960329-0a92-42b2-8e68-6ad2eae11f0e.xml -m=long test/cloudtest/test_full_testdrive.py

Terraform + Helm Chart E2E on GCP failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

[31mERROR[0m mz-debug: fatal: No SQL port forwarding info found

builtins.ValueError: Never completed
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run gcp-temporary

Terraform + Helm Chart E2E on Azure failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

[31mERROR[0m mz-debug: fatal: No SQL port forwarding info found

builtins.ValueError: Never completed
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run azure-temporary
Parallel Workload (0dt deploy) succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
parallel-workload-materialized2-1    | 2025-04-04T23:49:36.275879Z  thread 'coordinator' panicked at src/storage-controller/src/lib.rs:973:17: dependency since has advanced past dependent (u119) upper
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness. 
bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16

Terraform + Helm Chart E2E on AWS failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

[31mERROR[0m mz-debug: fatal: No SQL port forwarding info found

builtins.ValueError: Never completed
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run aws-temporary

Canary Deploy in Staging Cloud failed, main history: :bk-status-failed::bk-status-failed::bk-status-failed::bk-status-failed::bk-status-failed:

builtins.ValueError: Redpanda API call failed: 400 {"code":"INVALID_ARGUMENT", "message":"throughput tier tier-1-aws-v2-arm is not available in region us-east-1 for cluster type TYPE_DEDICATED", "details":[{"@type":"google.rpc.ErrorInfo", "reason":"REASON_THROUGHPUT_TIER_NOT_AVAILABLE_IN_REGION", "domain":"redpanda.com/controlplane", "metadata":{"cluster_type":"TYPE_DEDICATED", "region":"us-east-1", "throughput_tier_name":"tier-1-aws-v2-arm"}}]}
Test details & reproducer Deploy the current version on a real Staging Cloud, and run some basic verifications, like ingesting data from Kafka and Redpanda Cloud using AWS Privatelink. Runs only on main and release branches. 
bin/mzcompose --find cloud-canary run default
Security advisoriesbin/ci-builder run stable cargo deny check advisories
Waited 2m 28s
·
Ran in 4m 25s
Agent
:rust: Miri test (full)
Waited 2m 0s
·
Ran in 2h 55m
Agent
4/8
Feature benchmark against merge base or 'latest' 4
Waited 2m 29s
·
Ran in 2h 6m
Agent
8/8
Feature benchmark against merge base or 'latest' 8
Waited 2m 31s
·
Ran in 1h 56m
Agent
2/2
Full Testdrive in Cloudtest (K8s) 2
Waited 51s
·
Ran in 1h 11m
Agent
Platform checks upgrade in Cloudtest/K8s
Timed Out
Waited 51s
·
Ran in 4h 3m
Agent
Canary Deploy in Staging Cloud
Waited 46s
·
Ran in 22m 52s
Agent
Terraform + Helm Chart E2E on AWS
Waited 50s
·
Ran in 41m 31s
Agent
Terraform + Helm Chart E2E on GCP
Waited 56s
·
Ran in 50m 46s
Agent
Terraform + Helm Chart E2E on Azure
Waited 57s
·
Ran in 46m 5s
Agent
Total Job Run Time: 5d 18h