Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Make decoding codec data optional in FetchedPart

#10906

bkirwi:structured-on-disk-2/60af8bbe91(#31080)

Ben Kirwin canceled after 4h 51m

Miri test (full)

Extended SSH connection tests

CRDB rolling restarts

PubSub disruption

Tests for balancerd

CRDB / Persist backup and restore

Postgres / Persist backup and restore

Ben Kirwin

Created Fri 17th Jan at 7:26 PM

Triggered from Web

Zippy w/ latest CRDB failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-d00e1ed7/materialize/nightly/test/zippy exec -T cockroach cockroach sql --insecure -e                  CREATE EXTERNAL CONNECTION backup_bucket                 AS 's3://persist/crdb-backup?AWS_ENDPOINT=http://minio:9000/&AWS_REGION=minio&AWS_ACCESS_KEY_ID=minioadmin&AWS_SECRET_ACCESS_KEY=minioadmin';                 BACKUP INTO 'external://backup_bucket';                 DROP EXTERNAL CONNECTION backup_bucket;

Test details & reproducer

Zippy generates a pseudo-random sequence of DDLs, DMLs, failures, data validation and other events and runs it sequentially. By keeping track of the expected state it can verify results for correctness.

bin/mzcompose --find zippy run default --scenario=KafkaSources --actions=10000 --cockroach-tag=latest --max-execution-time=30m

SQLsmith explain succeeded with known error logs, main history:

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

sqlsmith-mz_2-1  | thread 'tokio:work-36' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Use SQLsmith to generate random queries (AST/code based) and run them against Materialize: https://github.com/MaterializeInc/sqlsmith The queries can be complex, but we can't verify correctness or performance.

bin/mzcompose --find sqlsmith run default --max-joins=5 --explain-only --runtime=1500

Output consistency (feature-flags for DFR) failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-b92887b3/materialize/nightly/test/feature-flag-consistency up --detach --wait --quiet-pull mz_this mz_other

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

feature-flag-consistency-mz_this-1   | thread 'tokio:work-1' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Verify that Materialize has the same results with different sets of feature flags.

bin/mzcompose --find feature-flag-consistency run default --seed=01938d37-d234-4c7f-97f0-f663eb7e5d19 --max-runtime-in-sec=1200 --evaluation-strategy=dataflow_rendering

aarch64 Docker tags produced in this build

materialize/ubuntu-base:mzbuild-ZDXXLIQ25S3GH73GW5Z2GUWHBSKRJM2H
materialize/prod-base:mzbuild-CHBE2FQ26CD4W4LYZJAVIYFIWDLCN7CV
materialize/balancerd:mzbuild-ERTOOVQ4ZGQ7BRML5MHIBOBX3KYSYQ3K
materialize/bazel:mzbuild-FZKRDOTGX32G2GYY3KA3XRK3IKSJ7MTC
materialize/chbenchmark:mzbuild-7NPNIYHRXT32FYNDEGMUYTETXLMU7QOE
materialize/ci-java-smoketest:mzbuild-MKZBE2SKCGNNPZG5MUOEEK6TXVTGKFT7
materialize/ci-metabase-smoketest:mzbuild-4N6WH4X4JFVLQSZWWK3QNNONEAK3XZH4
materialize/cli:mzbuild-UCRQHCMBLRIBHA6ECCKWN7NNZHXL25AM
materialize/clusterd:mzbuild-42MCQTNCD5V7A5NRWJE3SQZSCZ73UF5D
materialize/dbt-materialize:mzbuild-VP456JYHE6ADYWWUQXFOUBG6O2NV3S2W
materialize/debezium:mzbuild-RMYH5YYIXAC4K3VHNK7GUKZQHCAW75CY
materialize/environmentd:mzbuild-XZEFHAE5OBZWKN2KQYF5LGTQPMA7B4AO
materialize/fivetran-destination:mzbuild-PFBIGKD6LM76MN7IWKK2KCGPVTFSG56J
materialize/fivetran-destination-tester:mzbuild-J23TS6XJNMWPL54XNIRTT2ZRYAP5YHC3
materialize/frontegg-mock:mzbuild-ZL754ND5F2BKQ65RWMIV3EKUYORUBY4U
materialize/jobs:mzbuild-4MRK3VL2GSFU2KS6NIC2XX56WSPF2XHD
materialize/kgen:mzbuild-BCVDPEBAHLK62WUO5M2Z6TV6VPTC2P47
materialize/maelstrom-persist-base:mzbuild-3CJBQQPQ4FKCQVSFEMGDVDZJNULAZVLX
materialize/maelstrom-persist:mzbuild-CR2PZQNZFFOVDZHZZFRTVLKLBZIHXRD2
materialize/materialized-base:mzbuild-AZCBTDMB3PWA55MEWSZTOVV5X4VHG45B
materialize/materialized:mzbuild-QNBZNAN5SJKNKRDEURM5RGRCTLG2SV7H
materialize/test-certs:mzbuild-DMYTEM6GOQFJPGBWRBN64J43QKHEGEDV
materialize/mysql:mzbuild-R4Y33YICQZZBXKY3YVLNFTCLGMUIXZ6L
materialize/mysql-client:mzbuild-SWMKXCOA7WKJWQVS3GXW3P5SA3CFARZ2
materialize/mz:mzbuild-SDQAX4BDV645FQFRJZPFOMGOXASVK4ZB
materialize/orchestratord:mzbuild-KLBQH3BKYTANSS6H5SQJDQ2JOQTYDBAS
materialize/postgres:mzbuild-JDGCJUVYM3EY3QFMYDITXTFGXWBMDC2P
materialize/psql:mzbuild-3DKPCLCCR67L6GTXQP7AVLLOFTD2S2RB
materialize/sqlancer:mzbuild-WAVV6X6XEU2XZWMWRVXNHMLKB5GTFOHL
materialize/sqllogictest:mzbuild-G47XXVIA2GFXNZATLCQN2V4KDHQWHHTA
materialize/sqlsmith:mzbuild-Q4VF54SVOAXL5XQIPXXA4YYV22APHPW7
materialize/testdrive-base:mzbuild-PHWSUEUATBIJUZCY62U3HCBUELMRUBVA
materialize/testdrive:mzbuild-XRUBK6NUR2OLGJ2JR3YXEDZIBZD45XBL

x86_64 Docker tags produced in this build

materialize/ubuntu-base:mzbuild-PQQKF2CIMPLOQIF3EKEORR2MSRE45LH7
materialize/prod-base:mzbuild-SXSSD7ZVWJC26HY7DWHI5C2JEAGECCOM
materialize/balancerd:mzbuild-5OI5XS4FEJSJ5E3IPYYGSTZLXRJQO7NF
materialize/bazel:mzbuild-YY3NO65KI5YODSGNEENXXT7RT2PLDWIL
materialize/chbenchmark:mzbuild-QO5LD7ISGBEKYW4QVD75NK2PVSPFX65M
materialize/ci-java-smoketest:mzbuild-DEETHZ24HV3C4HE2POD2YYXZUG5MMRMI
materialize/ci-metabase-smoketest:mzbuild-LZ4N6XFPLYD4RAU56ILKSCBCYJ5EQY4G
materialize/cli:mzbuild-LCLQETIWXNZQWQEMTUT3TIYGVJPLVMVT
materialize/clusterd:mzbuild-OAHLQSIECZTF4OWQ22VPQEYX5ZWNCRTH
materialize/dbt-materialize:mzbuild-BQHB4IE5DGZQG3D47QWOADQ3LGFNIXBB
materialize/debezium:mzbuild-DH3UATRUW63NX4NC45OHBETXJZNLKBQ5
materialize/environmentd:mzbuild-27HGRUBZHW57C6K73JY2D55H5I2HLZP4
materialize/fivetran-destination:mzbuild-GYHHAXGW7VDHGCUB7HGCBUC7WYLLU6WB
materialize/fivetran-destination-tester:mzbuild-GLJTUDVZTGL74MOWQ3FI3ZUNDKCJJKIF
materialize/frontegg-mock:mzbuild-XP6M44VEFRMAYMFPV7NWUKCIZLRF7ZAE
materialize/jobs:mzbuild-VACXYRQOOTKFPHSHH2YIXTNSNXN77KS2
materialize/kgen:mzbuild-YBXKTSV5VZ4DJJDAWRCBTAGJKNA7VZ2Z
materialize/maelstrom-persist-base:mzbuild-NUGWOFCBNZI2NYE5N6D6XMQYPHIZIRDC
materialize/maelstrom-persist:mzbuild-BS24ESEA4H3FVLGIVTLMAMBWL5TAXHS2
materialize/materialized-base:mzbuild-6KUIUKIMFXYYIFIEHCLJEMQTG25XRAHK
materialize/materialized:mzbuild-OHGHL2B3QNID46VYZPMH7NRUITF2XTS6
materialize/test-certs:mzbuild-4X7BSRXRRUJQJ2BDOTRBCML4IFZKFT5B
materialize/mysql:mzbuild-3CTOQTRNEZ2WGT3Z6G5AF24W6P54HQBL
materialize/mysql-client:mzbuild-GIRUSXDV2UGNOSJOIOGWHGZDSLFOFD5G
materialize/mz:mzbuild-YRE2DMO6CCETMZW2PP2CPWWVRPZTXYDD
materialize/orchestratord:mzbuild-MHMZ4JKIJ5DADBWREMCGWVVPXK7PRVMA
materialize/postgres:mzbuild-6UIZ6QCFEVDPTZS5MIIIPL5AHVQ2O7TI
materialize/psql:mzbuild-5ENPLRAVYVOA3F77VITI4TXI55SLUEO3
materialize/sqlancer:mzbuild-GMHU6QUASFPVVAZTSW4POUVKE4IXDPGG
materialize/sqllogictest:mzbuild-BCWN4ZONIQ56AUTGZ52PS5RJBPWVJL55
materialize/sqlsmith:mzbuild-FATDMKSEQGHTZWSNRHJGMHDJZBFFFP2W
materialize/testdrive-base:mzbuild-ZYPHOO5YEMZIULYBNWTVT662433VFNZ2
materialize/testdrive:mzbuild-3T44ZOCYHGMYUCIX6RR3B667XLMUDH6R

bin/ci-builder run stable bin/pyactivate -m ci.mkpipeline nightly "$@"

Ran in 2m 4s

Miri test (full)

Ran in 2h 53m

1/8

Feature benchmark against merge base or 'latest' 1

Ran in 2h 32m

2/8

Feature benchmark against merge base or 'latest' 2

Ran in 3h 56m

3/8

Feature benchmark against merge base or 'latest' 3

Ran in 3h 53m

4/8

Feature benchmark against merge base or 'latest' 4

Ran in 3h 3m

5/8

Feature benchmark against merge base or 'latest' 5

Ran in 2h 32m

5/8

Feature benchmark against merge base or 'latest' 5

Ran in 1h 15m

6/8

Feature benchmark against merge base or 'latest' 6

Ran in 2h 28m

7/8

Feature benchmark against merge base or 'latest' 7

Ran in 2h 19m

8/8

Feature benchmark against merge base or 'latest' 8

Ran in 3h 47m

Scalability benchmark (read & write) against merge base or 'latest'

Ran in 1h 25m

Scalability benchmark (DDL) against merge base or 'latest'

Ran in 1h 28m

Scalability benchmark (connection) against merge base or 'latest'

Ran in 20m 33s

Parallel Benchmark

Ran in 1h 17m

1/2

Kafka smoke test against previous Kafka versions 1

Ran in 40m 53s

2/2

Kafka smoke test against previous Kafka versions 2

Ran in 30m 39s

Kafka multi-broker test

Ran in 3m 12s

🐼 resumption tests

Ran in 10m 8s

🐼 🏎️ testdrive

Ran in 53m 30s

🏎️ testdrive with old Upsert

Ran in 54m 54s

🏎️ testdrive with --kafka-default-partitions 5

Ran in 55m 22s

🏎️ testdrive 4 replicas

Ran in 1h 11m

🏎️ testdrive with SIZE 1

Ran in 56m 22s

🏎️ testdrive with SIZE 8

Ran in 56m 17s

🏎️ testdrive with Structured Persist (Only Structured)

Ran in 54m 24s

testdrive with

blob store

Ran in 1h 2m

🏎️ testdrive 4 replicas with

blob store

Ran in 1h 29m

🏎️ testdrive with SIZE 8 and

blob store

Ran in 1h 10m

1/2

Full Testdrive in Cloudtest (K8s) 1

Ran in 1h 9m

2/2

Full Testdrive in Cloudtest (K8s) 2

Ran in 1h 6m

1/2

Product limits 1

Ran in 1h 4m

2/2

Product limits 2

Ran in 39m 35s

Instance size limits

Ran in 8m 45s

1/2

Bounded Memory 1

Ran in 32m 5s

2/2

Bounded Memory 2

Ran in 12m 45s

Upsert (compaction enabled)

Ran in 14m 57s

Upsert (compaction disabled)

Ran in 14m 6s

Extended SSH connection tests

Ran in 18m 39s

Zippy Kafka Sources

Ran in 36m 52s

Zippy Kafka Sources with

blob store

Ran in 35m 14s

Zippy Kafka Sources with old Upsert

Ran in 34m 45s

Zippy Kafka Sources with old Upsert and

blob store

Ran in 35m 49s

Zippy Kafka Parallel Insert

Ran in 34m 30s

Zippy Kafka Parallel Insert with :azurite: blob store

Ran in 33m 33s

Zippy User Tables

Ran in 35m 7s

Zippy Postgres CDC

Ran in 33m 6s

Zippy MySQL CDC

Ran in 33m 59s

Zippy Debezium Postgres

Ran in 33m 59s

Zippy Cluster Replicas

Ran in 35m 14s

Zippy w/ latest CRDB

Ran in 34m 43s

Zippy w/ alter connection

Ran in 39m 53s

MySQL CDC tests (before source versioning)

Ran in 17m 45s

MySQL CDC resumption tests (before source versioning)

Ran in 40m 1s

MySQL RTR tests (before source versioning)

Ran in 2m 20s

Postgres CDC tests (before source versioning)

Ran in 16m 55s

Postgres CDC resumption tests (before source versioning)

Ran in 27m 6s

Postgres RTR tests (before source versioning)

Ran in 7m 1s

Testdrive (before Kafka source versioning)

Ran in 32m 27s

AWS (Real)

Ran in 5m 11s

AWS (Localstack)

Ran in 7m 39s

Secrets Local File

Ran in 5m 59s

1/2

Checks without restart or upgrade with

blob store 1 1

Ran in 24m 16s

2/2

Checks without restart or upgrade with

blob store 2 2

Ran in 23m 3s

1/2

Checks + restart of the entire Mz 1

Ran in 34m 50s

2/2

Checks + restart of the entire Mz 2

Ran in 22m 8s

1/2

Checks + restart of environmentd & storage clusterd with

blob store 1 1

Ran in 35m 38s

2/2

Checks + restart of environmentd & storage clusterd with

blob store 2 2

Ran in 44m 11s

1/2

Checks + restart of the entire Mz with old Upsert 1

Ran in 27m 17s

2/2

Checks + restart of the entire Mz with old Upsert 2

Ran in 20m 12s

1/2

Checks + backup + rollback to previous 1

Ran in 20m 50s

2/2

Checks + backup + rollback to previous 2

Ran in 17m 2s

1/2

Checks parallel + DROP/CREATE replica 1

Ran in 14m 45s

2/2

Checks parallel + DROP/CREATE replica 2

Ran in 14m 30s

1/2

Checks parallel + restart compute clusterd 1

Ran in 14m 41s

2/2

Checks parallel + restart compute clusterd 2

Ran in 13m 50s

1/2

Checks parallel + restart of the entire Mz 1

Ran in 16m 1s

2/2

Checks parallel + restart of the entire Mz 2

Ran in 15m 20s

1/2

Checks parallel + restart of the entire Mz with

blob store 1 1

Ran in 16m 25s

2/2

Checks parallel + restart of the entire Mz with

blob store 2 2

Ran in 18m 3s

1/2

Checks parallel + restart of environmentd & storage clusterd 1

Ran in 21m 10s

2/2

Checks parallel + restart of environmentd & storage clusterd 2

Ran in 15m 14s

1/2

Checks parallel + kill storage clusterd 1

Ran in 13m 56s

2/2

Checks parallel + kill storage clusterd 2

Ran in 14m 54s

1/2

Checks upgrade, whole-Mz restart 1

Ran in 30m 3s

2/2

Checks upgrade, whole-Mz restart 2

Ran in 24m 22s

1/2

Checks preflight-check and roll back upgrade 1

Ran in 25m 17s

2/2

Checks preflight-check and roll back upgrade 2

Ran in 24m 37s

1/2

Checks upgrade across two versions 1

Ran in 27m 16s

2/2

Checks upgrade across two versions 2

Ran in 24m 3s

1/2

Checks upgrade across four versions 1

Ran in 28m 57s

2/2

Checks upgrade across four versions 2

Ran in 29m 59s

1/2

Checks 0dt restart of the entire Mz with forced migrations with

blob store 1

Ran in 46m 31s

2/2

Checks 0dt restart of the entire Mz with forced migrations with

blob store 2

Ran in 38m 56s

1/2

Checks 0dt upgrade, whole-Mz restart 1

Ran in 27m 40s

2/2

Checks 0dt upgrade, whole-Mz restart 2

Ran in 30m 30s

1/2

Checks 0dt upgrade across two versions 1

Ran in 36m 49s

2/2

Checks 0dt upgrade across two versions 2

Ran in 28m 35s

1/2

Checks 0dt upgrade across four versions 1

Ran in 29m 17s

2/2

Checks 0dt upgrade across four versions 2

Ran in 31m 43s

Checks 0dt upgrade to a bumped version

Ran in 49m 30s

K8s recovery: storage on failing node

Ran in 22m 15s

K8s recovery: compute on failing node

Ran in 21m 38s

K8s recovery: replicated compute on failing node

Ran in 20m 35s

K8s recovery: envd on failing node

Ran in 25m 46s

Maelstrom coverage of persist

Ran in 6m 20s

Long single-node Maelstrom coverage of persist

Ran in 15m 39s

Long multi-node Maelstrom coverage of persist with postgres consensus

Ran in 10m 46s

Maelstrom coverage of txn-wal

Ran in 10m 46s

SQL-level feature flags

Ran in 6m 30s

LaunchDarkly

Ran in 5m 58s

Mz E2E Test

Ran in 17m 40s

SQLsmith

Ran in 31m 11s

SQLsmith explain

Ran in 31m 20s

SQLancer PQS

Ran in 30m 57s

SQLancer NoREC

Ran in 30m 55s

SQLancer QueryPartitioning

Ran in 31m 26s

SQLancer Having

Ran in 31m 6s

RQG dbt3-joins workload

Ran in 37m 41s

RQG subqueries workload

Ran in 36m 55s

RQG banking workload

Ran in 37m 23s

RQG left join stacks workload

Ran in 11m 59s

CRDB rolling restarts

Ran in 10m 48s

PubSub disruption

Ran in 19m 40s

1/2

Data Ingest 1

Ran in 45m 33s

2/2

Data Ingest 2

Ran in 24m 9s

Parallel Workload (DML)

Ran in 34m 44s

Parallel Workload (DDL) with

blob store

Ran in 35m 59s

Parallel Workload (DDL Only)

Ran in 33m 22s

Parallel Workload (rename + naughty identifiers)

Ran in 33m 36s

Parallel Workload (rename) with

blob store

Ran in 33m 29s

Parallel Workload (kill)

Ran in 33m 29s

Parallel Workload (0dt deploy)

Ran in 28m 55s

Tests for balancerd

Ran in 7m 33s

Legacy upgrade tests (last version from git)

Ran in 22m 50s

1/2

Legacy upgrade tests (last version from docs) 1

Ran in 9m 39s

2/2

Legacy upgrade tests (last version from docs) 2

Ran in 17m 32s

Cloudtest

Ran in 32m 49s

Slow Cloudtest

Ran in 25m 54s

1/2

Txn-wal fencing 1

Ran in 10m 7s

2/2

Txn-wal fencing 2

Ran in 15m 44s

Copy To S3 (1 replica)

Ran in 33m 27s

Copy To S3 (2 replicas)

Ran in 32m 32s

CRDB / Persist backup and restore

Ran in 17m 32s

Postgres / Persist backup and restore

Ran in 17m 6s

Replica isolation

Ran in 11m 35s

Zero downtime

Ran in 1h 32m

Materialize Emulator

Ran in 1m 26s

1/10

💡 SQL logic tests 1

Ran in 1h 29m

2/10

💡 SQL logic tests 2

Ran in 2h 25m

3/10

💡 SQL logic tests 3

Ran in 2h 23m

4/10

💡 SQL logic tests 4

Ran in 1h 59m

5/10

💡 SQL logic tests 5

Ran in 1h 46m

6/10

💡 SQL logic tests 6

Ran in 2h 7m

7/10

💡 SQL logic tests 7

Ran in 1h 44m

8/10

💡 SQL logic tests 8

Ran in 2h 15m

9/10

💡 SQL logic tests 9

Ran in 2h 3m

10/10

💡 SQL logic tests 10

Ran in 2h 14m

Cluster tests with

blob store

Ran in 45m 50s

Total Job Run Time: 4d 22h