Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Scheduled build

#10647
/
main/53f93494cc
Failed in 7h 48m
:pipeline:
:rust: Miri test (full)
Extended SSH connection tests
CRDB rolling restarts
PubSub disruption
Tests for balancerd
Legacy upgrade tests (last version from git)
CRDB / Persist backup and restore
Postgres / Persist backup and restore
Replica isolation
Zero downtime
Created
Triggered from Pipeline Schedule

Testdrive %N (before Kafka source versioning) failed, main history: :bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/4 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-dd3aa81b/materialize/nightly/test/testdrive-old-kafka-src-syntax run -eCLUST_REPLICA_SIZES testdrive --junit-report=junit_testdrive-old-kafka-src-syntax_0193a3af-6be8-46f0-bfc2-62b63168e387.xml --var=default-replica-size=4-4 --var=default-storage-size=4-1 *.td

Container testdrive-old-kafka-src-syntax-fivetran-destination-1  Running
Container testdrive-old-kafka-src-syntax-minio-1  Running
importer.proto:2:1: warning: Import empty.proto is unused.
^^^ +++
importer.proto:2:1: warning: Import empty.proto is unused.
+++ !!! Error Report
1 errors were encountered during execution
files involved: indexes.td


indexes.td:282:1: non-matching rows: expected:
[["mz_active_peeks_per_worker_s2_primary_idx", "mz_active_peeks_per_worker", "mz_catalog_server", "{id,worker_id}", ""], ["mz_arrangement_batcher_allocations_raw_s2_primary_idx", "mz_arrangement_batcher_allocations_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_batcher_capacity_raw_s2_primary_idx", "mz_arrangement_batcher_capacity_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_batcher_records_raw_s2_primary_idx", "mz_arrangement_batcher_records_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_batcher_size_raw_s2_primary_idx", "mz_arrangement_batcher_size_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_batches_raw_s2_primary_idx", "mz_arrangement_batches_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_heap_allocations_raw_s2_primary_idx", "mz_arrangement_heap_allocations_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_heap_capacity_raw_s2_primary_idx", "mz_arrangement_heap_capacity_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_heap_size_raw_s2_primary_idx", "mz_arrangement_heap_size_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_records_raw_s2_primary_idx", "mz_arrangement_records_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_arrangement_sharing_raw_s2_primary_idx", "mz_arrangement_sharing_raw", "mz_catalog_server", "{operator_id,worker_id}", ""], ["mz_cluster_replica_history_ind", "mz_cluster_replica_history", "mz_catalog_server", "{dropped_at}", ""], ["mz_cluster_replica_metrics_history_ind", "mz_cluster_replica_metrics_history", "mz_catalog_server", "{replica_id}", ""], ["mz_cluster_replica_metrics_ind", "mz_cluster_replica_metrics", "mz_catalog_server", "{replica_id}", ""], ["mz_cluster_replica_name_history_ind", "mz_cluster_replica_name_history", "mz_catalog_server", "{id}", ""], ["mz_cluster_replica_sizes_ind", "mz_cluster_replica_sizes", "mz_catalog_server", "{size}", ""], ["mz_cluster_replica_status_history_ind", "mz_cluster_replica_status_history", "mz_catalog_server", "{replica_id}", ""], ["mz_cluster_replica_statuses_ind", "mz_cluster_replica_statuses", "mz_catalog_server", "{replica_id}", ""], ["mz_cluster_replicas_ind", "mz_cluster_replicas", "mz_catalog_server", "{id}", ""], ["mz_clusters_ind", "mz_clusters", "mz_catalog_server", "{id}", ""], ["mz_columns_ind", "mz_columns", "mz_catalog_server", "{name}", ""], ["mz_comments_ind", "mz_comments", "mz_catalog_server", "{id}", ""], ["mz_compute_dataflow_global_ids_per_worker_s2_primary_idx", "mz_compute_dataflow_global_ids_per_worker", "mz_catalog_server", "{id,worker_id}", ""], ["mz_compute_dependencies_ind", "mz_compute_dependencies", "mz_catalog_server", "{dependency_id}", ""], ["mz_compute_error_counts_raw_s2_primary_idx", "mz_compute_error_counts_raw", "mz_catalog_server", "{export_id,worker_id}", ""], ["mz_compute_exports_per_worker_s2_primary_idx", "mz_compute_exports_per_worker", "mz_catalog_server", "{export_id,worker_id}", ""], ["mz_compute_frontiers_per_worker_s2_primary_idx", "mz_compute_frontiers_per_worker", "mz_catalog_server", "{export_id,worker_id}", ""], ["mz_compute_hydration_times_per_worker_s2_primary_idx", "mz_compute_hydration_times_per_worker", "mz_catalog_server", "{export_id,worker_id}", ""], ["mz_compute_import_frontiers_per_worker_s2_primary_idx", "mz_compute_import_frontiers_per_worker", "mz_catalog_server", "{export_id,import_id,worker_id}", ""], ["mz_compute_lir_mapping_per_worker_s2_primary_idx", "mz_compute_lir_mapping_per_worker", "mz_catalog_server", "{global_id,lir_id,worker_id}", ""], ["mz_compute_operator_durations_histogram_raw_s2_primary_idx", "mz_compute_operator_durations_histogram_raw", "mz_catalog_server", "{id,worker_id,duration_ns}", ""], ["mz_connections_ind", "mz_connections", "mz_catalog_server", "{schema_id}", ""], ["mz_console_cluster_utilization_overview_ind", "mz_console_cluster_utilization_overview", "mz_catalog_server", "{cluster_id}", ""], ["mz_continual_tasks_ind", "mz_continual_tasks", "mz_catalog_server", "{id}", ""], ["mz_databases_ind", "mz_databases", "mz_catalog_server", "{name}", ""], ["mz_dataflow_addresses_per_worker_s2_primary_idx", "mz_dataflow_addresses_per_worker", "mz_catalog_server", "{id,worker_id}", ""], ["mz_dataflow_channels_per_worker_s2_primary_idx", "mz_dataflow_channels_per_worker", "mz_catalog_server", "{id,worker_id}", ""], ["mz_dataflow_operator_reachability_raw_s2_primary_idx", "mz_dataflow_operator_reachability_raw", "mz_catalog_server", "{address,port,worker_id,update_type,time}", ""], ["mz_dataflow_operators_per_worker_s2_primary_idx", "mz_dataflow_operators_per_worker", "mz_catalog_server", "{id,worker_id}", ""], ["mz_dataflow_shutdown_durations_histogram_raw_s2_primary_idx", "mz_dataflow_shutdown_durations_histogram_raw", "mz_catalog_server", "{worker_id,duration_ns}", ""], ["mz_frontiers_ind", "mz_frontiers", "mz_catalog_server", "{object_id}", ""], ["mz_indexes_ind", "mz_indexes", "mz_catalog_server", "{id}", ""], ["mz_kafka_sources_ind", "mz_kafka_sources", "mz_catalog_server", "{id}", ""], ["mz_materialized_views_ind", "mz_materialized_views", "mz_catalog_server", "{id}", ""], ["mz_message_batch_counts_received_raw_s2_primary_idx", "mz_message_batch_counts_received_raw", "mz_catalog_server", "{channel_id,from_worker_id,to_worker_id}", ""], ["mz_message_batch_counts_sent_raw_s2_primary_idx", "mz_message_batch_counts_sent_raw", "mz_catalog_server", "{channel_id,from_worker_id,to_worker_id}", ""], ["mz_message_counts_received_raw_s2_primary_idx", "mz_message_counts_received_raw", "mz_catalog_server", "{channel_id,from_worker_id,to_worker_id}", ""], ["mz_message_counts_sent_raw_s2_primary_idx", "mz_message_counts_sent_raw", "mz_catalog_server", "{channel_id,from_worker_id,to_worker_id}", ""], ["mz_notices_ind", "mz_notices", "mz_catalog_server", "{id}", ""], ["mz_object_dependencies_ind", "mz_object_dependencies", "mz_catalog_server", "{object_id}", ""], ["mz_object_history_ind", "mz_object_history", "mz_catalog_server", "{id}", ""], ["mz_object_lifetimes_ind", "mz_object_lifetimes", "mz_catalog_server", "{id}", ""], ["mz_object_transitive_dependencies_ind", "mz_object_transitive_dependencies", "mz_catalog_server", "{object_id}", ""], ["mz_objects_ind", "mz_objects", "mz_catalog_server", "{schema_id}", ""], ["mz_peek_durations_histogram_raw_s2_primary_idx", "mz_peek_durations_histogram_raw", "mz_catalog_server", "{worker_id,type,duration_ns}", ""], ["mz_recent_activity_log_thinned_ind", "mz_recent_activity_log_thinned", "mz_catalog_server", "{sql_hash}", ""], ["mz_recent_sql_text_ind", "mz_recent_sql_text", "mz_catalog_server", "{sql_hash}", ""], ["mz_recent_storage_usage_ind", "mz_recent_storage_usage", "mz_catalog_server", "{object_id}", ""], ["mz_roles_ind", "mz_roles", "mz_catalog_server", "{id}", ""], ["mz_scheduling_elapsed_raw_s2_primary_idx", "mz_scheduling_elapsed_raw", "mz_catalog_server", "{id,worker_id}", ""], ["mz_scheduling_parks_histogram_raw_s2_primary_idx", "mz_scheduling_parks_histogram_raw", "mz_catalog_server", "{worker_id,slept_for_ns,requested_ns}", ""], ["mz_schemas_ind", "mz_schemas", "mz_catalog_server", "{database_id}", ""], ["mz_secrets_ind", "mz_secrets", "mz_catalog_server", "{name}", ""], ["mz_show_all_objects_ind", "mz_show_all_objects", "mz_catalog_server", "{schema_id}", ""], ["mz_show_cluster_replicas_ind", "mz_show_cluster_replicas", "mz_catalog_server", "{cluster}", ""], ["mz_show_clusters_ind", "mz_show_clusters", "mz_catalog_server", "{name}", ""], ["mz_show_columns_ind", "mz_show_columns", "mz_catalog_server", "{id}", ""], ["mz_show_connections_ind", "mz_show_connections", "mz_catalog_server", "{schema_id}", ""], ["mz_show_databases_ind", "mz_show_databases", "mz_catalog_server", "{name}", ""], ["mz_show_indexes_ind", "mz_show_indexes", "mz_catalog_server", "{schema_id}", ""], ["mz_show_materialized_views_ind", "mz_show_materialized_views", "mz_catalog_server", "{schema_id}", ""], ["mz_show_roles_ind", "mz_show_roles", "mz_catalog_server", "{name}", ""], ["mz_show_schemas_ind", "mz_show_schemas", "mz_catalog_server", "{database_id}", ""], ["mz_show_secrets_ind", "mz_show_secrets", "mz_catalog_server", "{schema_id}", ""], ["mz_show_sinks_ind", "mz_show_sinks", "mz_catalog_server", "{schema_id}", ""], ["mz_show_sources_ind", "mz_show_sources", "mz_catalog_server", "{schema_id}", ""], ["mz_show_tables_ind", "mz_show_tables", "mz_catalog_server", "{schema_id}", ""], ["mz_show_types_ind", "mz_show_types", "mz_catalog_server", "{schema_id}", ""], ["mz_show_views_ind", "mz_show_views", "mz_catalog_server", "{schema_id}", ""], ["mz_sink_statistics_ind", "mz_sink_statistics", "mz_catalog_server", "{id}", ""], ["mz_sink_status_history_ind", "mz_sink_status_history", "mz_catalog_server", "{sink_id}", ""], ["mz_sink_statuses_ind", "mz_sink_statuses", "mz_catalog_server", "{id}", ""], ["mz_sinks_ind", "mz_sinks", "mz_catalog_server", "{id}", ""], ["mz_source_statistics_ind", "mz_source_statistics", "mz_catalog_server", "{id}", ""], ["mz_source_statistics_with_history_ind", "mz_source_statistics_with_history", "mz_catalog_server", "{id}", ""], ["mz_source_status_history_ind", "mz_source_status_history", "mz_catalog_server", "{source_id}", ""], ["mz_source_statuses_ind", "mz_source_statuses", "mz_catalog_server", "{id}", ""], ["mz_sources_ind", "mz_sources", "mz_catalog_server", "{id}", ""], ["mz_tables_ind", "mz_tables", "mz_catalog_server", "{schema_id}", ""], ["mz_types_ind", "mz_types", "mz_catalog_server", "{schema_id}", ""], ["mz_views_ind", "mz_views", "mz_catalog_server", "{schema_id}", ""], ["mz_wallclock_global_lag_recent_history_ind", "mz_wallclock_global_lag_recent_history", "mz_catalog_server", "{object_id}", ""], ["mz_webhook_sources_ind", "mz_webhook_sources", "mz_catalog_server", "{id}", ""], ["pg_attrdef_all_databases_ind", "pg_attrdef [...]
Test details & reproducer Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result. 
bin/mzcompose --find testdrive-old-kafka-src-syntax run default
Parallel Workload (0dt deploy) succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
parallel-workload-materialized-1     | thread 'timely:work-0' panicked at src/storage/src/upsert/types.rs:751:30: invalid upsert state: non 0/1 diff_sum: 2, state: Consolidating { len_sum: 122, checksum_sum: 8580904045956659995, diff_sum: 8580904045956659995, .. }
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness. 
bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16

Terraform + Helm Chart E2E on AWS failed, main history: :bk-status-failed::bk-status-failed::bk-status-failed::bk-status-passed::bk-status-passed:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-09a08043e2d360722-1/materialize/nightly/test/terraform exec -T testdrive testdrive --kafka-addr=kafka:9092 --schema-registry-url=http://schema-registry:8081 --materialize-url=postgres://materialize@127.0.0.1:6875/materialize --materialize-internal-url=postgres://mz_system:materialize@127.0.0.1:6877/materialize --materialize-use-https --aws-endpoint=http://minio:9000 --var=aws-endpoint=http://minio:9000 --aws-access-key-id=minioadmin --var=aws-access-key-id=minioadmin --aws-secret-access-key=minioadmin --var=aws-secret-access-key=minioadmin --default-timeout=360s --consistency-checks=disable --persist-blob-url=file:///mzdata/persist/blob --persist-consensus-url=postgres://root@materialized:26257?options=--search_path=consensus --source=/var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-09a08043e2d360722-1/materialize/nightly/test/terraform/mzcompose.py:392

^^^ +++
+++ !!! Error Report
1 errors were encountered during execution
source: /var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-09a08043e2d360722-1/materialize/nightly/test/terraform/mzcompose.py:392
Test details & reproducer Tests the mz command line tool against a real Cloud instance 
bin/mzcompose --find terraform run default
SQLsmith explain succeeded with known error logs, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed: 
sqlsmith-mz_1-1  | thread 'tokio:work-27' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation
Test details & reproducer Use SQLsmith to generate random queries (AST/code based) and run them against Materialize: https://github.com/MaterializeInc/sqlsmith The queries can be complex, but we can't verify correctness or performance. 
bin/mzcompose --find sqlsmith run default --max-joins=5 --explain-only --runtime=1500
Platform checks upgrade in Cloudtest/K8s failed with known error logs, main history: :bk-status-failed::bk-status-failed::bk-status-passed::bk-status-passed::bk-status-passed: 
materialize.ui.CommandFailureCausedUIError: Running testdrive --materialize-url=postgres://materialize:materialize@environmentd:6875/materialize --materialize-internal-url=postgres://mz_system@environmentd:6877/materialize --kafka-addr=redpanda:9092 --schema-registry-url=http://redpanda:8081 --default-timeout=300s --log-filter=off --var=replicas=1 --var=leaves-tombstones=false --var=single-replica-cluster=quickstart --var=default-storage-size=1 --var=default-replica-size=1 --cluster-replica-sizes={"bootstrap": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 1}, "2-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 2, "workers": 4}, "free": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "0", "disabled": true, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 0, "workers": 0}, "1cc": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": null, "scale": 1, "workers": 1}, "1C": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": null, "scale": 1, "workers": 1}, "1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 1}, "1-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 1}, "1-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 1}, "1-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 1}, "1-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 1}, "1-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 1}, "mem-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "1 GiB", "scale": 1, "workers": 1}, "2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 2}, "2-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 2}, "2-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 2}, "2-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 2}, "2-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 2}, "2-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 2, "workers": 1}, "2-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 2, "workers": 2}, "mem-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "2 GiB", "scale": 1, "workers": 2}, "4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 4}, "4-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "4-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 4}, "4-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 4}, "4-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 4}, "4-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 4, "workers": 1}, "4-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 4, "workers": 4}, "mem-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 8}, "8-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 8}, "8-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "8-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 8}, "8-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 8}, "8-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 8, "workers": 1}, "8-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "64", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 8, "workers": 8}, "mem-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 16}, "16-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 16}, "16-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 16}, "16-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "16-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 16}, "16-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 16, "workers": 1}, "16-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "256", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 16, "workers": 16}, "mem-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "32": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 32}, "32-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 32}, "32-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 32}, "32-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 32}, "32-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 32}, "32-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 32, "workers": 1}, "32-32": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1024", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 32, "workers": 32}, "mem-32": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc" [...]

Traceback (most recent call last):
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-07bdb265be282e6aa-1/materialize/nightly/misc/python/materialize/cloudtest/k8s/testdrive.py", line 176, in _run_internal
    self.kubectl(
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-07bdb265be282e6aa-1/materialize/nightly/misc/python/materialize/cloudtest/k8s/api/k8s_resource.py", line 47, in kubectl
    run_process_with_error_information(
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-07bdb265be282e6aa-1/materialize/nightly/misc/python/materialize/cloudtest/util/common.py", line 118, in run_process_with_error_information
    raise e
  File "/var/lib/buildkite-agent/builds/buildkite-medium-aarch64-f931a0c-i-07bdb265be282e6aa-1/materialize/nightly/misc/python/materialize/cloudtest/util/common.py", line 113, in run_process_with_error_information
    subprocess.run(
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['kubectl', '--context', 'kind-mzcloud', '--namespace', 'default', 'exec', '-it', 'testdrive', '--', 'testdrive', '--materialize-url=postgres://materialize:materialize@environmentd:6875/materialize', '--materialize-internal-url=postgres://mz_system@environmentd:6877/materialize', '--kafka-addr=redpanda:9092', '--schema-registry-url=http://redpanda:8081', '--default-timeout=300s', '--log-filter=off', '--var=replicas=1', '--var=leaves-tombstones=false', '--var=single-replica-cluster=quickstart', '--var=default-storage-size=1', '--var=default-replica-size=1', '--cluster-replica-sizes={"bootstrap": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 1}, "2-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 2, "workers": 4}, "free": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "0", "disabled": true, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 0, "workers": 0}, "1cc": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": null, "scale": 1, "workers": 1}, "1C": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": true, "memory_limit": null, "scale": 1, "workers": 1}, "1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 1}, "1-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 1}, "1-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 1}, "1-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 1}, "1-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 1}, "1-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 1}, "mem-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "1", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "1 GiB", "scale": 1, "workers": 1}, "2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 2}, "2-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 2}, "2-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 2}, "2-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 2}, "2-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 2}, "2-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 2, "workers": 1}, "2-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 2, "workers": 2}, "mem-2": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "2", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "2 GiB", "scale": 1, "workers": 2}, "4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 4}, "4-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "4-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 4}, "4-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 4}, "4-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 4}, "4-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 4, "workers": 1}, "4-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 4, "workers": 4}, "mem-4": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "4", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 4}, "8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 8}, "8-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 8}, "8-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "8-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 8}, "8-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 8}, "8-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 8, "workers": 1}, "8-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "64", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 8, "workers": 8}, "mem-8": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "8", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 8}, "16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 16}, "16-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "4 GiB", "scale": 1, "workers": 16}, "16-8G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "8 GiB", "scale": 1, "workers": 16}, "16-16G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "16-32G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "32 GiB", "scale": 1, "workers": 16}, "16-1": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 16, "workers": 1}, "16-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "256", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 16, "workers": 16}, "mem-16": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "16", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": "16 GiB", "scale": 1, "workers": 16}, "32": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_hour": "32", "disabled": false, "disk_limit": null, "is_cc": false, "memory_limit": null, "scale": 1, "workers": 32}, "32-4G": {"cpu_exclusive": false, "cpu_limit": null, "credits_per_ [...]
Test details & reproducer 
bin/pytest --junitxml=junit_cloudtest_0193a3af-6bfe-4316-a090-c97012c8221b.xml -m=long test/cloudtest/test_upgrade.py --splits=1 --group=1

Output consistency (feature-flags for DFR) failed, main history: :bk-status-passed::bk-status-failed::bk-status-failed::bk-status-failed::bk-status-failed:

Docker compose failed: docker compose -f/dev/fd/8 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-0d5adcec/materialize/nightly/test/feature-flag-consistency up --detach --wait --quiet-pull mz_this mz_other

feature-flag-consistency-mz_this-1   | thread 'tokio:work-5' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation
Test details & reproducer Verify that Materialize has the same results with different sets of feature flags. 
bin/mzcompose --find feature-flag-consistency run default --seed=0193a3ad-afbb-4704-afaf-cbbb1110e482 --max-runtime-in-sec=1200 --evaluation-strategy=dataflow_rendering
Testdrive %N (before Kafka source versioning)
Waited 2m 41s
·
Ran in 51m 52s
Agent
Platform checks upgrade in Cloudtest/K8s
Waited 1m 26s
·
Ran in 29m 28s
Agent
Terraform + Helm Chart E2E on AWS
Waited 59s
·
Ran in 34m 1s
Agent
Output consistency (feature-flags for DFR)
Waited 2m 42s
·
Ran in 13m 59s
Agent
Total Job Run Time: 4d 18h