Prysm

Public

Ethereum Serenity Client!

Revert Change

#90435

triggerExecutionRequests/a28b94f1f0(#14971)

Failed in 21m 45s

Build & Test Everything

Linter rules

ARM64 builds beacon-chain and validator

AMD64 builds beacon-chain and validator

Build docker images

End to end tests (presubmit)

Nishant Das

Created Thursday at 1:26 PM

Triggered from Webhook

Scalability benchmark (read & write) against merge base or 'latest' (#2) failed, main history:

Unknown error in Workload 'UpdateWorkload':

New regression against v0.127.0-dev.0 (e70c9d8d5)

Regression in workload 'UpdateWorkload' at concurrency 16 with MaterializeContainer (None specified as HEAD): 40.1 tps vs. 51.79 tps (-11.69 tps; -22.57%)

Unknown error in Workload 'UpdateWorkload':

New regression against v0.127.0-dev.0 (e70c9d8d5)

Regression in workload 'UpdateWorkload' at concurrency 32 with MaterializeContainer (None specified as HEAD): 38.45 tps vs. 51.1 tps (-12.66 tps; -24.77%)

Test details & reproducer

Benchmark for how various queries scale, compares against old Materialize versions.

bin/mzcompose --find scalability run default --target HEAD --target common-ancestor --regression-against common-ancestor --workload-group-marker DmlDqlWorkload --max-concurrency 256

🏎️ testdrive with SIZE 1 failed, main history:

Unknown error in kafka-recreate-topic.td:

kafka-recreate-topic.td:91:1: query succeeded, but expected error containing "topic was recreated: high watermark of partition 0 regressed from 1 to 0"
     |
  16 |     URL '${testdrive ... [rest of line truncated for security]
  90 | 
  91 | ! SELECT * FROM source1_tbl
     | ^

Test details & reproducer

Testdrive is the basic framework and language for defining product tests under the expected-result/actual-result (aka golden testing) paradigm. A query is retried until it produces the desired result.

bin/mzcompose --find testdrive run default --default-size=1

Zippy w/ latest CRDB failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-8cpu-16gb-f98604bf/materialize/nightly/test/zippy exec -T cockroach cockroach sql --insecure -e                  CREATE EXTERNAL CONNECTION backup_bucket                 AS 's3://persist/crdb-backup?AWS_ENDPOINT=http://minio:9000/&AWS_REGION=minio&AWS_ACCESS_KEY_ID=minioadmin&AWS_SECRET_ACCESS_KEY=minioadmin';                 BACKUP INTO 'external://backup_bucket';                 DROP EXTERNAL CONNECTION backup_bucket;

Test details & reproducer

Zippy generates a pseudo-random sequence of DDLs, DMLs, failures, data validation and other events and runs it sequentially. By keeping track of the expected state it can verify results for correctness.

bin/mzcompose --find zippy run default --scenario=KafkaSources --actions=10000 --cockroach-tag=latest --max-execution-time=30m

RQG subqueries workload failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-a1a091e6/materialize/nightly/test/rqg exec -T rqg perl gentest.pl --seed=01938943-5c2f-4ad3-a336-20edde49d7d3 --dsn1=dbi:Pg:dbname=materialize;host=mz_this;user=materialize;port=6875 --dsn2=dbi:Pg:dbname=postgres;host=postgres;user=postgres;password=postgres --grammar=conf/mz/subqueries.yy --queries=100000000 --threads=4 --duration=1800 --validator=ResultsetComparatorSimplify

Test details & reproducer

Test Materialize with the Random Query Generator (grammar-based): https://github.com/MaterializeInc/RQG/ Can find query errors and panics, but not correctness.

bin/mzcompose --find rqg run default subqueries --seed=01938943-5c2f-4ad3-a336-20edde49d7d3

SQLsmith explain succeeded with known error logs, main history:

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

sqlsmith-mz_2-1  | thread 'tokio:work-24' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Use SQLsmith to generate random queries (AST/code based) and run them against Materialize: https://github.com/MaterializeInc/sqlsmith The queries can be complex, but we can't verify correctness or performance.

bin/mzcompose --find sqlsmith run default --max-joins=5 --explain-only --runtime=1500

Output consistency (Postgres) failed, main history:

Unknown error in Query 409.1 (jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))):

Outcome differs

SUCCESS_MISMATCH: Outcome differs.
Expression: jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))
  Value 1 (Materialize evaluation): 'QueryResult' (type: <class 'str'>)
  Value 2 (Postgres evaluation): 'QueryFailure' (type: <class 'str'>)
  Error 2: 'invalid input syntax for type uuid: "invalid_value_123"'
  Query 1: SELECT jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123')) FROM t_dfr_horiz WHERE ((jsonb_val_1 ? 'z'));
  Query 2: SELECT jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123')) FROM t_pg_horiz WHERE ((jsonb_val_1 ? 'z'));
  Expression hash: 7284047682483061634

Code to reproduce

--- Minimal code for reproduction
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Setup for evaluation strategy 'Materialize evaluation':
DROP TABLE IF EXISTS mz_evaluation;
CREATE TABLE mz_evaluation (row_index INT, varchar_8_val_w_spaces VARCHAR(8), jsonb_val_1 JSONB);
INSERT INTO mz_evaluation VALUES (0, ' mAA m
	 '::VARCHAR(8), '{"a": 1}'::JSONB);
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Query using evaluation strategy 'Materialize evaluation':
SELECT
  jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))
FROM
  mz_evaluation
WHERE ((jsonb_val_1 ? 'z'));
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Setup for evaluation strategy 'Postgres evaluation':
DROP TABLE IF EXISTS postgres_evaluation;
CREATE TABLE postgres_evaluation (row_index INT, varchar_8_val_w_spaces VARCHAR(8), jsonb_val_1 JSONB);
INSERT INTO postgres_evaluation VALUES (0, ' mAA m
	 '::VARCHAR(8), '{"a": 1}'::JSONB);
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- Query using evaluation strategy 'Postgres evaluation':
SELECT
  jsonb_object_agg(varchar_8_val_w_spaces, (UUID 'invalid_value_123') ORDER BY ARRAY[row_index]::INT[], 0, varchar_8_val_w_spaces, (UUID 'invalid_value_123'))
FROM
  postgres_evaluation
WHERE ((jsonb_val_1 ? 'z'));
-- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-- All assumed directly or indirectly involved characteristics: STRING_WITH_SPECIAL_SPACE_CHARS, ENUM_INVALID

Test details & reproducer

Test the consistency of Materialize against Postgres as an oracle.

bin/mzcompose --find postgres-consistency run default --seed=01938943-5c2f-4ad3-a336-20edde49d7d3 --max-runtime-in-sec=1200

Output consistency (feature-flags for DFR) failed, main history:

Unknown error in workflow-default:

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/hetzner-aarch64-4cpu-8gb-cf614c4d/materialize/nightly/test/feature-flag-consistency up --detach --wait --quiet-pull mz_this mz_other

Known issue thread 'tokio:work-108' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation (#8797) in services.log:

feature-flag-consistency-mz_this-1   | thread 'tokio:work-3' panicked at src/transform/src/equivalence_propagation.rs:76:13: ColumnKnowledge performed work after EquivalencePropagation

Test details & reproducer

Verify that Materialize has the same results with different sets of feature flags.

bin/mzcompose --find feature-flag-consistency run default --seed=01938943-5c2f-4ad3-a336-20edde49d7d3 --max-runtime-in-sec=1200 --evaluation-strategy=dataflow_rendering

aarch64 Docker tags produced in this build

materialize/ubuntu-base:mzbuild-ZDXXLIQ25S3GH73GW5Z2GUWHBSKRJM2H
materialize/prod-base:mzbuild-CHBE2FQ26CD4W4LYZJAVIYFIWDLCN7CV
materialize/balancerd:mzbuild-HLGD3IYQVOF2HS333DDY5XN2CQ6UOIRG
materialize/bazel:mzbuild-FZKRDOTGX32G2GYY3KA3XRK3IKSJ7MTC
materialize/chbenchmark:mzbuild-7NPNIYHRXT32FYNDEGMUYTETXLMU7QOE
materialize/ci-java-smoketest:mzbuild-MKZBE2SKCGNNPZG5MUOEEK6TXVTGKFT7
materialize/ci-metabase-smoketest:mzbuild-4N6WH4X4JFVLQSZWWK3QNNONEAK3XZH4
materialize/cli:mzbuild-UCRQHCMBLRIBHA6ECCKWN7NNZHXL25AM
materialize/clusterd:mzbuild-4DTJ7RZOW67JQ3RRODPL5TMVHGLKIC23
materialize/dbt-materialize:mzbuild-VP456JYHE6ADYWWUQXFOUBG6O2NV3S2W
materialize/debezium:mzbuild-RMYH5YYIXAC4K3VHNK7GUKZQHCAW75CY
materialize/environmentd:mzbuild-3PNOGCAK5N44EOQXOZHV3C3FQOU4IR4G
materialize/fivetran-destination:mzbuild-OBABMCSTTIPRFIZDKJGUONCCWEZEXLIT
materialize/fivetran-destination-tester:mzbuild-J23TS6XJNMWPL54XNIRTT2ZRYAP5YHC3
materialize/frontegg-mock:mzbuild-VUXKSERUNVC37EWFLQ7WXENHNTUSSHJO
materialize/jobs:mzbuild-ADPM5CW5QKGNXGEKHEQ4JJK2YH5BKNXK
materialize/kgen:mzbuild-BCVDPEBAHLK62WUO5M2Z6TV6VPTC2P47
materialize/maelstrom-persist-base:mzbuild-3CJBQQPQ4FKCQVSFEMGDVDZJNULAZVLX
materialize/maelstrom-persist:mzbuild-L3UTGT6FEHOYWJN2XH7KQJK5B2UNLZ75
materialize/materialized-base:mzbuild-AZCBTDMB3PWA55MEWSZTOVV5X4VHG45B
materialize/materialized:mzbuild-F4VV3FF3KHEEMLERCJI7Z5CDOMPAAN55
materialize/test-certs:mzbuild-DMYTEM6GOQFJPGBWRBN64J43QKHEGEDV
materialize/mysql:mzbuild-R4Y33YICQZZBXKY3YVLNFTCLGMUIXZ6L
materialize/mysql-client:mzbuild-SWMKXCOA7WKJWQVS3GXW3P5SA3CFARZ2
materialize/mz:mzbuild-K55FYDPOPS5JOTM7YMX5HGABA7HPTEUI
materialize/orchestratord:mzbuild-3D2CQB5FISO6IIGI4PSTTSN5KT2SDCD2
materialize/postgres:mzbuild-JDGCJUVYM3EY3QFMYDITXTFGXWBMDC2P
materialize/psql:mzbuild-3DKPCLCCR67L6GTXQP7AVLLOFTD2S2RB
materialize/sqlancer:mzbuild-WAVV6X6XEU2XZWMWRVXNHMLKB5GTFOHL
materialize/sqllogictest:mzbuild-DQ7VDSWINYZRQIQBQOQTTDS4YVV4A7ZU
materialize/sqlsmith:mzbuild-Q4VF54SVOAXL5XQIPXXA4YYV22APHPW7
materialize/testdrive-base:mzbuild-PHWSUEUATBIJUZCY62U3HCBUELMRUBVA
materialize/testdrive:mzbuild-XWW6HCJ7VMUNNGWMOVUX3L5DVNIPZGNV

x86_64 Docker tags produced in this build

materialize/ubuntu-base:mzbuild-PQQKF2CIMPLOQIF3EKEORR2MSRE45LH7
materialize/prod-base:mzbuild-SXSSD7ZVWJC26HY7DWHI5C2JEAGECCOM
materialize/balancerd:mzbuild-JPMKUQDOW4GZIU4FBRMROI2PFCYC5G2E
materialize/bazel:mzbuild-YY3NO65KI5YODSGNEENXXT7RT2PLDWIL
materialize/chbenchmark:mzbuild-QO5LD7ISGBEKYW4QVD75NK2PVSPFX65M
materialize/ci-java-smoketest:mzbuild-DEETHZ24HV3C4HE2POD2YYXZUG5MMRMI
materialize/ci-metabase-smoketest:mzbuild-LZ4N6XFPLYD4RAU56ILKSCBCYJ5EQY4G
materialize/cli:mzbuild-LCLQETIWXNZQWQEMTUT3TIYGVJPLVMVT
materialize/clusterd:mzbuild-GYXDA3JYK4PNXPERBNVPS6T6MRDBYF4F
materialize/dbt-materialize:mzbuild-BQHB4IE5DGZQG3D47QWOADQ3LGFNIXBB
materialize/debezium:mzbuild-DH3UATRUW63NX4NC45OHBETXJZNLKBQ5
materialize/environmentd:mzbuild-DMATWNJ5A2HN7MAWA2PQUZHM5HJRPYMF
materialize/fivetran-destination:mzbuild-Y5H7BVOKB4D4EM2V2RHDGGU6QQHY7Z5M
materialize/fivetran-destination-tester:mzbuild-GLJTUDVZTGL74MOWQ3FI3ZUNDKCJJKIF
materialize/frontegg-mock:mzbuild-DZXGONL7JBRDFNIFOUMTFJX3D24VZAZH
materialize/jobs:mzbuild-SSRG63HR7ZJEF5WXAUEJMFAQFA56KF2G
materialize/kgen:mzbuild-YBXKTSV5VZ4DJJDAWRCBTAGJKNA7VZ2Z
materialize/maelstrom-persist-base:mzbuild-NUGWOFCBNZI2NYE5N6D6XMQYPHIZIRDC
materialize/maelstrom-persist:mzbuild-ULASN4RIJRJ6RHFAKIMSENMHYZA4YAYQ
materialize/materialized-base:mzbuild-6KUIUKIMFXYYIFIEHCLJEMQTG25XRAHK
materialize/materialized:mzbuild-CN7LAWKUVIV2GJLESR7LKLJHSZBTL3TZ
materialize/test-certs:mzbuild-4X7BSRXRRUJQJ2BDOTRBCML4IFZKFT5B
materialize/mysql:mzbuild-3CTOQTRNEZ2WGT3Z6G5AF24W6P54HQBL
materialize/mysql-client:mzbuild-GIRUSXDV2UGNOSJOIOGWHGZDSLFOFD5G
materialize/mz:mzbuild-EDYQRTDMFFNYMBMPJRIILOQ3CUEVXKJE
materialize/orchestratord:mzbuild-XTUODFODYCSCLEFBLB2R6EJN33F5KBNZ
materialize/postgres:mzbuild-6UIZ6QCFEVDPTZS5MIIIPL5AHVQ2O7TI
materialize/psql:mzbuild-5ENPLRAVYVOA3F77VITI4TXI55SLUEO3
materialize/sqlancer:mzbuild-GMHU6QUASFPVVAZTSW4POUVKE4IXDPGG
materialize/sqllogictest:mzbuild-CBFWRXKNVMICGYZQZF465GD62JRMFNZK
materialize/sqlsmith:mzbuild-FATDMKSEQGHTZWSNRHJGMHDJZBFFFP2W
materialize/testdrive-base:mzbuild-ZYPHOO5YEMZIULYBNWTVT662433VFNZ2
materialize/testdrive:mzbuild-5IIDY4ETWTR3T7AXTRF6FRSPTVXMV4IS

End to end tests (presubmit)./bazel.sh --bazelrc=.buildkite-bazelrc clean --async && ./bazel.sh --bazelrc=.buildkite-bazelrc test //testing/endtoend:presubmit --remote_download_toplevel --remote_cache=grpc://rbe.prylabs.net --remote_local_fallback --config=nostamp --test_timeout=10000 --local_test_jobs=2 --local_ram_resources=HOST_RAM --flaky_test_attempts=1 --remote_header=$BUILDBUDDY_TOKEN --build_metadata=BRANCH_NAME=$BUILDKITE_BRANCH

Ran in 21m 40s

Total Job Run Time: 26m 35s