Nightly

Public

Tests that are too slow or non-deterministic for the regular Test pipeline

Normalize away the encoded part

Failed in 5h 10m
:pipeline:
:rust: Miri test (full)
Extended SSH connection tests
CRDB rolling restarts
PubSub disruption
Tests for balancerd
Legacy upgrade tests (last version from git)
CRDB / Persist backup and restore
Postgres / Persist backup and restore
Replica isolation
Zero downtime
Materialize Emulator

Feature benchmark against merge base or 'latest' 6 failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

New regression against v0.130.0-dev.0--pr.gf64c516140d1b840d6ab68d35d98f7a5b20fed7e
NAME                                | TYPE            |      THIS       |      OTHER      |  UNIT  | THRESHOLD  |  Regression?  | 'THIS' is
--------------------------------------------------------------------------------------------------------------------------------------------------------
OptbenchTPCHQ12                     | wallclock       |     8963010.000 |     6498509.000 |   ns   |    20%     |    !!YES!!    | worse:  37.9% slower
OptbenchTPCHQ12                     | memory_mz       |         464.725 |         457.478 |   MB   |    20%     |      no       | worse:   1.6% more
OptbenchTPCHQ12                     | memory_clusterd |          53.892 |          52.986 |   MB   |    50%     |      no       | worse:   1.7% more
Test details & reproducer Simple benchmark of mostly individual queries using testdrive. Can find wallclock/memorys regressions in single-connection query executions, not suitable for concurrency.
BUILDKITE_PARALLEL_JOB=5 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find feature-benchmark run default --other-tag common-ancestor 

Scalability benchmark (read & write) against merge base or 'latest' failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

New regression against v0.130.0-dev.0 (f64c51614)
Regression in workload 'SelectOneWorkload' at concurrency 1 with MaterializeContainer (None specified as HEAD): 461.43 tps vs. 655.62 tps (-194.19 tps; -29.62%)
New regression against v0.130.0-dev.0 (f64c51614)
Regression in workload 'SelectOneWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 1005.73 tps vs. 1296.43 tps (-290.7 tps; -22.42%)
Test details & reproducer Benchmark for how various queries scale, compares against old Materialize versions.
bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256 

Parallel Workload (DDL) failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

parallel-workload-materialized-1     | thread 'timely:work-0' panicked at src/repr/src/row/encode.rs:518:33: can't encode True into Time(FixedSizeBinaryBuilder { values_builder: BufferBuilder { buffer: MutableBuffer { data: 0xffff92804000, len: 0, layout: Layout { size: 8192, align: 64 (1 << 6) } }, len: 0, _marker: PhantomData<u8> }, null_buffer_builder: NullBufferBuilder { bitmap_builder: None, len: 0, capacity: 1024 }, value_length: 8 })
materialize.parallel_workload.worker_exception.WorkerFailedException: ^^^ +++ Thread worker_0 failed, exiting
('consuming input failed: server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.', "FETCH 817 c WITH (timeout='6s');")
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.
bin/mzcompose --find parallel-workload run default --runtime=1500 --threads=8 

Parallel Workload (0dt deploy) failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

+++ [worker_0] Query failed: SELECT "db-pw-1737073366-0"."s-2"."v-186"."key4-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key0-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key2-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key7-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key8-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key5-kafka_table0", MAX("db-pw-1737073366-0"."s-2"."v-186"."key5-kafka_table0") OVER (PARTITION BY "db-pw-1737073366-0"."s-2"."v-186"."key5-kafka_table0" ORDER BY "db-pw-1737073366-0"."s-2"."v-186"."key4-kafka_table0") FROM "db-pw-1737073366-0"."s-2"."v-186"  LIMIT 1; HTTP XX000: internal error: internal error: column 2 is not of expected type ColumnType { scalar_type: String, nullable: false }: Int32(0)
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.
bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16 

Parallel Workload (rename + naughty identifiers) failed, main history: :bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed::bk-status-passed:

parallel-workload-materialized-1     | thread 'timely:work-0' panicked at src/pgrepr/src/value.rs:216:18: can't serialize -32768::Float32
materialize.parallel_workload.worker_exception.WorkerFailedException: ^^^ +++ Thread worker_2 failed, exiting
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Test details & reproducer Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.
bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=rename --naughty-identifiers --threads=16 
aarch64 Docker tags produced in this build
  • materialize/ubuntu-base:mzbuild-QZA457JHLWJOUK6OLS6YEL34PIUK34TR
  • materialize/prod-base:mzbuild-BM3LRPOWX3JPTZQUEJ5ZBWXJ27UA2TOX
  • materialize/balancerd:mzbuild-Y65XJOZYNWHWRDN7QAC3VOXRMDIXZEPG
  • materialize/bazel:mzbuild-YO3L35N7JSJAQKF4V7IFSLA5OMPBKD4V
  • materialize/chbenchmark:mzbuild-D7Q76S7TV2IAXV3S4DBYRGJ42E53QXEM
  • materialize/ci-java-smoketest:mzbuild-HAXKEFGNUHDFSIN33W64HSX2SFOKHLLO
  • materialize/ci-metabase-smoketest:mzbuild-47LLRJDBM5XRR5FVH27ZA7VKWPS7ZXU2
  • materialize/cli:mzbuild-ZLVBX3FTXHGPEX2OXVKIVDWEH4T4IMKW
  • materialize/clusterd:mzbuild-3TYHRJBNX6UVKIYAGX5ZX3R4VWNLLYOL
  • materialize/dbt-materialize:mzbuild-J4J3DMT7L35C74LCJAXKD4IGH2CVWEZL
  • materialize/debezium:mzbuild-RMYH5YYIXAC4K3VHNK7GUKZQHCAW75CY
  • materialize/environmentd:mzbuild-MLKYTHTUTG7OYBJUBSSFBZ2WFZ45IJHQ
  • materialize/fivetran-destination:mzbuild-K3QW25OTQPRZ2BG46BL5JHD56Q5ZHHXO
  • materialize/fivetran-destination-tester:mzbuild-J23TS6XJNMWPL54XNIRTT2ZRYAP5YHC3
  • materialize/frontegg-mock:mzbuild-WJNQZ6CFHKRZJFMBGHOUFDELBPG4N5GU
  • materialize/jobs:mzbuild-6KLGKGBQCDLMJACBO7F5KX4HDDRO35VQ
  • materialize/kgen:mzbuild-FX6LXCY53R6HW6PEBHGVXRKRHCAEC62P
  • materialize/maelstrom-persist-base:mzbuild-VNTKKMSJHLSNFUUQMLVEVOEEPW2YXXQ2
  • materialize/maelstrom-persist:mzbuild-ZJDQAXVBA4UCXAN36BG2DAU4GN7OE74E
  • materialize/materialized-base:mzbuild-IPZ75KFA3KYBRBCLRDEWZDBPTQJMQPHF
  • materialize/materialized:mzbuild-S4XOQAZO7BRD5XRMOADNWWKNRBTZAPF5
  • materialize/test-certs:mzbuild-PRCZYXWYBEVJASXVU6RRHE3S6PLTRR2Y
  • materialize/mysql:mzbuild-OKRGOKZCI74PS3KLTUZHOZJOJV3K2RSW
  • materialize/mysql-client:mzbuild-KTTOKDVMC3TTY5SGNE5Y5LY67AXHAQLB
  • materialize/mz:mzbuild-VU6TEEMDPSJ3UO5V4KPX4OVCHHREL2LA
  • materialize/orchestratord:mzbuild-2JO67K66Y5T4I2N2LVSKQUUEYHUUPIS7
  • materialize/postgres:mzbuild-KFWC53JYINFN2RYUUS4HDSKIRHL4MAJ7
  • materialize/psql:mzbuild-LGSSUA6F4RQJJNUE4LKEM4DO2YLKMMMP
  • materialize/sqlancer:mzbuild-EPXZE4GDPB7XO4WRGTZO3XPMLZTIOROV
  • materialize/sqllogictest:mzbuild-IR4I6TG4JU24AIR7XLR4GPU4X3ZPYT6W
  • materialize/sqlsmith:mzbuild-IZWZLXVN6CPK65IHCM3M2Z4MNYCZJQ6Z
  • materialize/testdrive-base:mzbuild-TDIYNQ2JLK3RISFVZMWJOW6T2WHN25UV
  • materialize/testdrive:mzbuild-Y3GRGFLC6C77YY2T4MVQQMCEYSMESAIT
x86_64 Docker tags produced in this build
  • materialize/ubuntu-base:mzbuild-BROGB3WOXXKRZHNOL7KLPU4TL5TWLKXX
  • materialize/prod-base:mzbuild-QEQJNOGMT43J5XAKL5M6QOXVWUYXY66G
  • materialize/balancerd:mzbuild-BV6EGOOVK5UDIEOGHJ7TZEAPAKOMBY27
  • materialize/bazel:mzbuild-QBYFFA4JPVD75AU6MJLBLVJWILKZ6KSZ
  • materialize/chbenchmark:mzbuild-3ZSXLNZ6ZOFOWSXV5RAMGU53UGUXUKG3
  • materialize/ci-java-smoketest:mzbuild-VXPC3CEQEL5Z4X65UBP53FC3D3H5KQEL
  • materialize/ci-metabase-smoketest:mzbuild-PZ6BD5ITUKOGVEKGJW3D6RKFYG4NKDN6
  • materialize/cli:mzbuild-UBGGEOBW4CWBOIHFOXWVDDRA66EVK3QH
  • materialize/clusterd:mzbuild-KJCNG23IUTXNH5IAQHHP5OAQL3ILV4UO
  • materialize/dbt-materialize:mzbuild-FWRAGEFXEAE7YYTJ6MU3FYJAL3YPNG7W
  • materialize/debezium:mzbuild-DH3UATRUW63NX4NC45OHBETXJZNLKBQ5
  • materialize/environmentd:mzbuild-5A52VHM3SDX3NT4SNT2LB2A5VLEDUCJ2
  • materialize/fivetran-destination:mzbuild-IZMTT5PBRGY2667QWAL7ATLAFT5XUDSO
  • materialize/fivetran-destination-tester:mzbuild-GLJTUDVZTGL74MOWQ3FI3ZUNDKCJJKIF
  • materialize/frontegg-mock:mzbuild-QPOJCDPNBAGIHY4TPZRJ4M5ZRX7K52FD
  • materialize/jobs:mzbuild-2S64Y7OI5CDWBVL6PNM7FJGJISJ66Q7K
  • materialize/kgen:mzbuild-RFRKAFZLP4DJYQNLOHMKWI6WCXCOGCVG
  • materialize/maelstrom-persist-base:mzbuild-PUXN3KSSPREUAXB5YLX54AE3B4XACLBS
  • materialize/maelstrom-persist:mzbuild-ZCMDARIPHPNQXONCF6PLC3YYCB7M27UR
  • materialize/materialized-base:mzbuild-2PYZXDBUVHY5IMEW7H5NXCMHXKK67XZ3
  • materialize/materialized:mzbuild-W5EV4HUC4DIPKSSZ64KRIMNRNP7PYMCQ
  • materialize/test-certs:mzbuild-QQT3XC7KPZNJOMJTEC3JKDAWLL5TTNIJ
  • materialize/mysql:mzbuild-ZZ3ZRCQJ5X34NHHYVKTYOV2OTTZFQVV6
  • materialize/mysql-client:mzbuild-PL5AADHY2W4QJ5P5M73JAJRRIQQFCQ2L
  • materialize/mz:mzbuild-F72MVQGJ2MHNLVKYA2XRLNE4I7SBIAVX
  • materialize/orchestratord:mzbuild-OCI4KAAJ2ZFPDXYNSJUQAGDDX76B5ZFE
  • materialize/postgres:mzbuild-ORSA3DY6C63F2F7BOQ6LZLLHY27L7CZT
  • materialize/psql:mzbuild-VGTDKGODZS3AF7GEAGO6PCTPNLHO4RPL
  • materialize/sqlancer:mzbuild-I4OTD5FIZCOKBOP6GGUEXMWUNPPWRTGH
  • materialize/sqllogictest:mzbuild-M4NNUBWO2CPMM6ZTHAJRDLHB5L5FM3QV
  • materialize/sqlsmith:mzbuild-H5G74ASOX7NEKRVLJSH5WTUJ3K3FO6XH
  • materialize/testdrive-base:mzbuild-ZDEVMWGECPR32E7BKTYKI4FGWAB6IFZL
  • materialize/testdrive:mzbuild-23EFVJN3N5P7U7GTWEO3MVPLZPFAGUTU
6/8
Feature benchmark against merge base or 'latest' 6
Waited 1m 45s
·
Ran in 2h 19m
Scalability benchmark (read & write) against merge base or 'latest'
Waited 1m 48s
·
Ran in 1h 33m
Parallel Workload (DDL)
Waited 3m 17s
·
Ran in 26m 25s
Parallel Workload (rename + naughty identifiers)
Waited 3m 32s
·
Ran in 15m 7s
Parallel Workload (0dt deploy)
Waited 1m 44s
·
Ran in 25m 9s
Total Job Run Time: 4d 17h