Nightly
PublicTests that are too slow or non-deterministic for the regular Test pipeline
Normalize away the encoded part
Failed in 5h 10m
Feature benchmark against merge base or 'latest' 6 failed, main history:
- Unknown error in Scenario 'OptbenchTPCHQ12':
New regression against v0.130.0-dev.0--pr.gf64c516140d1b840d6ab68d35d98f7a5b20fed7e
NAME | TYPE | THIS | OTHER | UNIT | THRESHOLD | Regression? | 'THIS' is
--------------------------------------------------------------------------------------------------------------------------------------------------------
OptbenchTPCHQ12 | wallclock | 8963010.000 | 6498509.000 | ns | 20% | !!YES!! | worse: 37.9% slower
OptbenchTPCHQ12 | memory_mz | 464.725 | 457.478 | MB | 20% | no | worse: 1.6% more
OptbenchTPCHQ12 | memory_clusterd | 53.892 | 52.986 | MB | 50% | no | worse: 1.7% more
Test details & reproducer
Simple benchmark of mostly individual queries using testdrive. Can find wallclock/memorys regressions in single-connection query executions, not suitable for concurrency.BUILDKITE_PARALLEL_JOB=5 BUILDKITE_PARALLEL_JOB_COUNT=8 bin/mzcompose --find feature-benchmark run default --other-tag common-ancestor
Scalability benchmark (read & write) against merge base or 'latest' failed, main history:
- Unknown error in Workload 'SelectOneWorkload':
New regression against v0.130.0-dev.0 (f64c51614)
Regression in workload 'SelectOneWorkload' at concurrency 1 with MaterializeContainer (None specified as HEAD): 461.43 tps vs. 655.62 tps (-194.19 tps; -29.62%)
- Unknown error in Workload 'SelectOneWorkload':
New regression against v0.130.0-dev.0 (f64c51614)
Regression in workload 'SelectOneWorkload' at concurrency 2 with MaterializeContainer (None specified as HEAD): 1005.73 tps vs. 1296.43 tps (-290.7 tps; -22.42%)
Test details & reproducer
Benchmark for how various queries scale, compares against old Materialize versions.bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256
Parallel Workload (DDL) failed, main history:
parallel-workload-materialized-1 | thread 'timely:work-0' panicked at src/repr/src/row/encode.rs:518:33: can't encode True into Time(FixedSizeBinaryBuilder { values_builder: BufferBuilder { buffer: MutableBuffer { data: 0xffff92804000, len: 0, layout: Layout { size: 8192, align: 64 (1 << 6) } }, len: 0, _marker: PhantomData<u8> }, null_buffer_builder: NullBufferBuilder { bitmap_builder: None, len: 0, capacity: 1024 }, value_length: 8 })
- Unknown error in workflow-default:
materialize.parallel_workload.worker_exception.WorkerFailedException: ^^^ +++ Thread worker_0 failed, exiting
('consuming input failed: server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.', "FETCH 817 c WITH (timeout='6s');")
Test details & reproducer
Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.bin/mzcompose --find parallel-workload run default --runtime=1500 --threads=8
Parallel Workload (0dt deploy) failed, main history:
- Potential regression sqlsmith: ERROR: internal error: column 0 is not of expected type ColumnType { scalar_type: Range { element_type: Int64 }, nullable: false }: Int32(7700) (#6935, closed) in run.log:
+++ [worker_0] Query failed: SELECT "db-pw-1737073366-0"."s-2"."v-186"."key4-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key0-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key2-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key7-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key8-kafka_table0", "db-pw-1737073366-0"."s-2"."v-186"."key5-kafka_table0", MAX("db-pw-1737073366-0"."s-2"."v-186"."key5-kafka_table0") OVER (PARTITION BY "db-pw-1737073366-0"."s-2"."v-186"."key5-kafka_table0" ORDER BY "db-pw-1737073366-0"."s-2"."v-186"."key4-kafka_table0") FROM "db-pw-1737073366-0"."s-2"."v-186" LIMIT 1; HTTP XX000: internal error: internal error: column 2 is not of expected type ColumnType { scalar_type: String, nullable: false }: Int32(0)
Test details & reproducer
Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=0dt-deploy --threads=16
Parallel Workload (rename + naughty identifiers) failed, main history:
parallel-workload-materialized-1 | thread 'timely:work-0' panicked at src/pgrepr/src/value.rs:216:18: can't serialize -32768::Float32
- Unknown error in workflow-default:
materialize.parallel_workload.worker_exception.WorkerFailedException: ^^^ +++ Thread worker_2 failed, exiting
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Test details & reproducer
Runs a randomized parallel workload stressing all parts of Materialize, can mostly find panics and unexpected errors. See zippy for a sequential randomized tests which can verify correctness.bin/mzcompose --find parallel-workload run default --runtime=1500 --scenario=rename --naughty-identifiers --threads=16
aarch64 Docker tags produced in this build
materialize/ubuntu-base:mzbuild-QZA457JHLWJOUK6OLS6YEL34PIUK34TR
materialize/prod-base:mzbuild-BM3LRPOWX3JPTZQUEJ5ZBWXJ27UA2TOX
materialize/balancerd:mzbuild-Y65XJOZYNWHWRDN7QAC3VOXRMDIXZEPG
materialize/bazel:mzbuild-YO3L35N7JSJAQKF4V7IFSLA5OMPBKD4V
materialize/chbenchmark:mzbuild-D7Q76S7TV2IAXV3S4DBYRGJ42E53QXEM
materialize/ci-java-smoketest:mzbuild-HAXKEFGNUHDFSIN33W64HSX2SFOKHLLO
materialize/ci-metabase-smoketest:mzbuild-47LLRJDBM5XRR5FVH27ZA7VKWPS7ZXU2
materialize/cli:mzbuild-ZLVBX3FTXHGPEX2OXVKIVDWEH4T4IMKW
materialize/clusterd:mzbuild-3TYHRJBNX6UVKIYAGX5ZX3R4VWNLLYOL
materialize/dbt-materialize:mzbuild-J4J3DMT7L35C74LCJAXKD4IGH2CVWEZL
materialize/debezium:mzbuild-RMYH5YYIXAC4K3VHNK7GUKZQHCAW75CY
materialize/environmentd:mzbuild-MLKYTHTUTG7OYBJUBSSFBZ2WFZ45IJHQ
materialize/fivetran-destination:mzbuild-K3QW25OTQPRZ2BG46BL5JHD56Q5ZHHXO
materialize/fivetran-destination-tester:mzbuild-J23TS6XJNMWPL54XNIRTT2ZRYAP5YHC3
materialize/frontegg-mock:mzbuild-WJNQZ6CFHKRZJFMBGHOUFDELBPG4N5GU
materialize/jobs:mzbuild-6KLGKGBQCDLMJACBO7F5KX4HDDRO35VQ
materialize/kgen:mzbuild-FX6LXCY53R6HW6PEBHGVXRKRHCAEC62P
materialize/maelstrom-persist-base:mzbuild-VNTKKMSJHLSNFUUQMLVEVOEEPW2YXXQ2
materialize/maelstrom-persist:mzbuild-ZJDQAXVBA4UCXAN36BG2DAU4GN7OE74E
materialize/materialized-base:mzbuild-IPZ75KFA3KYBRBCLRDEWZDBPTQJMQPHF
materialize/materialized:mzbuild-S4XOQAZO7BRD5XRMOADNWWKNRBTZAPF5
materialize/test-certs:mzbuild-PRCZYXWYBEVJASXVU6RRHE3S6PLTRR2Y
materialize/mysql:mzbuild-OKRGOKZCI74PS3KLTUZHOZJOJV3K2RSW
materialize/mysql-client:mzbuild-KTTOKDVMC3TTY5SGNE5Y5LY67AXHAQLB
materialize/mz:mzbuild-VU6TEEMDPSJ3UO5V4KPX4OVCHHREL2LA
materialize/orchestratord:mzbuild-2JO67K66Y5T4I2N2LVSKQUUEYHUUPIS7
materialize/postgres:mzbuild-KFWC53JYINFN2RYUUS4HDSKIRHL4MAJ7
materialize/psql:mzbuild-LGSSUA6F4RQJJNUE4LKEM4DO2YLKMMMP
materialize/sqlancer:mzbuild-EPXZE4GDPB7XO4WRGTZO3XPMLZTIOROV
materialize/sqllogictest:mzbuild-IR4I6TG4JU24AIR7XLR4GPU4X3ZPYT6W
materialize/sqlsmith:mzbuild-IZWZLXVN6CPK65IHCM3M2Z4MNYCZJQ6Z
materialize/testdrive-base:mzbuild-TDIYNQ2JLK3RISFVZMWJOW6T2WHN25UV
materialize/testdrive:mzbuild-Y3GRGFLC6C77YY2T4MVQQMCEYSMESAIT
x86_64 Docker tags produced in this build
materialize/ubuntu-base:mzbuild-BROGB3WOXXKRZHNOL7KLPU4TL5TWLKXX
materialize/prod-base:mzbuild-QEQJNOGMT43J5XAKL5M6QOXVWUYXY66G
materialize/balancerd:mzbuild-BV6EGOOVK5UDIEOGHJ7TZEAPAKOMBY27
materialize/bazel:mzbuild-QBYFFA4JPVD75AU6MJLBLVJWILKZ6KSZ
materialize/chbenchmark:mzbuild-3ZSXLNZ6ZOFOWSXV5RAMGU53UGUXUKG3
materialize/ci-java-smoketest:mzbuild-VXPC3CEQEL5Z4X65UBP53FC3D3H5KQEL
materialize/ci-metabase-smoketest:mzbuild-PZ6BD5ITUKOGVEKGJW3D6RKFYG4NKDN6
materialize/cli:mzbuild-UBGGEOBW4CWBOIHFOXWVDDRA66EVK3QH
materialize/clusterd:mzbuild-KJCNG23IUTXNH5IAQHHP5OAQL3ILV4UO
materialize/dbt-materialize:mzbuild-FWRAGEFXEAE7YYTJ6MU3FYJAL3YPNG7W
materialize/debezium:mzbuild-DH3UATRUW63NX4NC45OHBETXJZNLKBQ5
materialize/environmentd:mzbuild-5A52VHM3SDX3NT4SNT2LB2A5VLEDUCJ2
materialize/fivetran-destination:mzbuild-IZMTT5PBRGY2667QWAL7ATLAFT5XUDSO
materialize/fivetran-destination-tester:mzbuild-GLJTUDVZTGL74MOWQ3FI3ZUNDKCJJKIF
materialize/frontegg-mock:mzbuild-QPOJCDPNBAGIHY4TPZRJ4M5ZRX7K52FD
materialize/jobs:mzbuild-2S64Y7OI5CDWBVL6PNM7FJGJISJ66Q7K
materialize/kgen:mzbuild-RFRKAFZLP4DJYQNLOHMKWI6WCXCOGCVG
materialize/maelstrom-persist-base:mzbuild-PUXN3KSSPREUAXB5YLX54AE3B4XACLBS
materialize/maelstrom-persist:mzbuild-ZCMDARIPHPNQXONCF6PLC3YYCB7M27UR
materialize/materialized-base:mzbuild-2PYZXDBUVHY5IMEW7H5NXCMHXKK67XZ3
materialize/materialized:mzbuild-W5EV4HUC4DIPKSSZ64KRIMNRNP7PYMCQ
materialize/test-certs:mzbuild-QQT3XC7KPZNJOMJTEC3JKDAWLL5TTNIJ
materialize/mysql:mzbuild-ZZ3ZRCQJ5X34NHHYVKTYOV2OTTZFQVV6
materialize/mysql-client:mzbuild-PL5AADHY2W4QJ5P5M73JAJRRIQQFCQ2L
materialize/mz:mzbuild-F72MVQGJ2MHNLVKYA2XRLNE4I7SBIAVX
materialize/orchestratord:mzbuild-OCI4KAAJ2ZFPDXYNSJUQAGDDX76B5ZFE
materialize/postgres:mzbuild-ORSA3DY6C63F2F7BOQ6LZLLHY27L7CZT
materialize/psql:mzbuild-VGTDKGODZS3AF7GEAGO6PCTPNLHO4RPL
materialize/sqlancer:mzbuild-I4OTD5FIZCOKBOP6GGUEXMWUNPPWRTGH
materialize/sqllogictest:mzbuild-M4NNUBWO2CPMM6ZTHAJRDLHB5L5FM3QV
materialize/sqlsmith:mzbuild-H5G74ASOX7NEKRVLJSH5WTUJ3K3FO6XH
materialize/testdrive-base:mzbuild-ZDEVMWGECPR32E7BKTYKI4FGWAB6IFZL
materialize/testdrive:mzbuild-23EFVJN3N5P7U7GTWEO3MVPLZPFAGUTU
Total Job Run Time: 4d 17h