🧪

CI

Public

CI Testing of the vLLM Repo

Queue Paused

tpu_queue•Kevin LuuKevin Luu•

Testing IBM Z (s390x) platform support2

#16786
/
s390x/buildkite-ci-community/8f98d5d5b1
Passed in 58m 18s and blocked
bootstrap
:docker: build image
Build CUDA 12.1 image
:docker: build image CUDA 12.1
Build CUDA 11.8 image
:docker: build image CUDA 11.8
Documentation Build
Run Async Engine, Inputs, Utils, Worker Test
Async Engine, Inputs, Utils, Worker Test
Run Python-only Installation Test
Python-only Installation Test
Run Basic Correctness Test
Basic Correctness Test
Run Chunked Prefill Test
Chunked Prefill Test
Run Core Test
Core Test
Run Entrypoints Test
Entrypoints Test
Run Distributed Tests (4 GPUs)
Distributed Tests (4 GPUs)
Run Metrics, Tracing Test
Metrics, Tracing Test
Run Regression Test
Regression Test
Run Engine Test
Engine Test
Run V1 Test
V1 Test
Run Examples Test
Examples Test
Run Prefix Caching Test
Prefix Caching Test
Run Samplers Test
Samplers Test
Run LogitsProcessor Test
LogitsProcessor Test
Run Speculative decoding tests
Speculative decoding tests
Run LoRA Test %N
Run PyTorch Fullgraph Smoke Test
PyTorch Fullgraph Smoke Test
Run PyTorch Fullgraph Test
PyTorch Fullgraph Test
Run Kernels Test %N
Run Tensorizer Test
Tensorizer Test
Run Benchmarks
Benchmarks
Run Quantization Test
Quantization Test
Run LM Eval Small Models
LM Eval Small Models
Run OpenAI API correctness
OpenAI API correctness
Run Encoder Decoder tests
Encoder Decoder tests
Run OpenAI-Compatible Tool Use
OpenAI-Compatible Tool Use
Run Basic Models Test
Basic Models Test
Run Language Models Test (Standard)
Language Models Test (Standard)
Run Language Models Test (Extended)
Language Models Test (Extended)
Run Multi-Modal Models Test (Standard)
Multi-Modal Models Test (Standard)
Run Multi-Modal Models Test (Extended) 1
Multi-Modal Models Test (Extended) 1
Run Multi-Modal Models Test (Extended) 2
Multi-Modal Models Test (Extended) 2
Run Custom Models Test
Custom Models Test
Run Distributed Comm Ops Test
Distributed Comm Ops Test
Run 2 Node Tests (4 GPUs in total)
2 Node Tests (4 GPUs in total)
Run Distributed Tests (2 GPUs)
Distributed Tests (2 GPUs)
Run Plugin Tests (2 GPUs)
Plugin Tests (2 GPUs)
Run Multi-step Tests (4 GPUs)
Multi-step Tests (4 GPUs)
Run Pipeline Parallelism Test
Pipeline Parallelism Test
Run LoRA TP Test (Distributed)
LoRA TP Test (Distributed)
Run Weight Loading Multiple GPU Test
Weight Loading Multiple GPU Test
Run Weight Loading Multiple GPU Test - Large Models
Weight Loading Multiple GPU Test - Large Models
Run Distributed Tests (A100)
Distributed Tests (A100)
Run LM Eval Large Models
LM Eval Large Models
Neuron Test
Run Intel CPU test
Intel CPU Test
Intel HPU Test
Intel GPU Test
IBM Z (s390x) CPU Test
TPU V0 Test
TPU V1 Test
Nishan Acharya
Nishan Acharya
Created
Triggered from Web
bootstrapif [[ -n "s390x/ibm-buildkite-support" ]]; then VLLM_CI_BRANCH=s390x/ibm-buildkite-support curl -sSL "https://raw.githubusercontent.com/vllm-project/buildkite-ci/s390x/ibm-buildkite-support/scripts/ci_aws_bootstrap.sh" | bash && exit 0; fi && curl -sSL "https://raw.githubusercontent.com/vllm-project/buildkite-ci/main/scripts/ci_aws_bootstrap.sh" | bash
Waited 39s
·
Ran in 14s
Agent
:docker: build imageaws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 && #!/bin/bash && if [[ -z $(docker manifest inspect public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:8f98d5d5b170a2372c463f311963e235ceedc94f) ]]; then && echo "Image not found, proceeding with build..." && else && echo "Image found" && exit 0 && fi && docker build --file docker/Dockerfile --build-arg max_jobs=16 --build-arg buildkite_commit=8f98d5d5b170a2372c463f311963e235ceedc94f --build-arg USE_SCCACHE=1 --tag public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:8f98d5d5b170a2372c463f311963e235ceedc94f --target test --progress plain . && docker push public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:8f98d5d5b170a2372c463f311963e235ceedc94f
Waited 32s
·
Ran in 8s
Agent
:docker: build image CUDA 12.1aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 && #!/bin/bash && if [[ -z $(docker manifest inspect ) ]]; then && echo "Image not found, proceeding with build..." && else && echo "Image found" && exit 0 && fi && docker build --file docker/Dockerfile --build-arg max_jobs=16 --build-arg buildkite_commit=8f98d5d5b170a2372c463f311963e235ceedc94f --build-arg USE_SCCACHE=1 --build-arg CUDA_VERSION=12.1.0 --tag --target test --progress plain . && docker push
:docker: build image CUDA 11.8aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 && #!/bin/bash && if [[ -z $(docker manifest inspect ) ]]; then && echo "Image not found, proceeding with build..." && else && echo "Image found" && exit 0 && fi && docker build --file docker/Dockerfile --build-arg max_jobs=16 --build-arg buildkite_commit=8f98d5d5b170a2372c463f311963e235ceedc94f --build-arg USE_SCCACHE=1 --build-arg CUDA_VERSION=11.8.0 --tag --target test --progress plain . && docker push
Documentation Build
Waited 38s
·
Ran in 6m 25s
Agent
Async Engine, Inputs, Utils, Worker Test
Python-only Installation Test
Basic Correctness Test
Chunked Prefill Test
Core Test
Entrypoints Test
Distributed Tests (4 GPUs)
Metrics, Tracing Test
Regression Test
Engine Test
V1 Test
Examples Test
Prefix Caching Test
Samplers Test
LogitsProcessor Test
Speculative decoding tests
1/4
LoRA Test 1
2/4
LoRA Test 2
3/4
LoRA Test 3
4/4
LoRA Test 4
PyTorch Fullgraph Smoke Test
PyTorch Fullgraph Test
1/4
Kernels Test 1
2/4
Kernels Test 2
3/4
Kernels Test 3
4/4
Kernels Test 4
Tensorizer Test
Benchmarks
Quantization Test
LM Eval Small Models
OpenAI API correctness
Encoder Decoder tests
OpenAI-Compatible Tool Use
Basic Models Test
Language Models Test (Standard)
Language Models Test (Extended)
Multi-Modal Models Test (Standard)
Multi-Modal Models Test (Extended) 1
Multi-Modal Models Test (Extended) 2
Custom Models Test
Distributed Comm Ops Test
2 Node Tests (4 GPUs in total)./.buildkite/run-multi-node-test.sh /vllm-workspace/tests 2 2 public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:8f98d5d5b170a2372c463f311963e235ceedc94f "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_multi_node_assignment.py && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed'"
Distributed Tests (2 GPUs)
Plugin Tests (2 GPUs)
Multi-step Tests (4 GPUs)
Pipeline Parallelism Test
LoRA TP Test (Distributed)
Weight Loading Multiple GPU Test
Weight Loading Multiple GPU Test - Large Models
Distributed Tests (A100)
LM Eval Large Models
AMD: :docker: build imagegrep -i 'from base as test' docker/Dockerfile.rocm && docker build --build-arg max_jobs=16 --tag rocm/vllm-ci:8f98d5d5b170a2372c463f311963e235ceedc94f -f docker/Dockerfile.rocm --target test --progress plain . || docker build --build-arg max_jobs=16 --tag rocm/vllm-ci:8f98d5d5b170a2372c463f311963e235ceedc94f -f docker/Dockerfile.rocm --progress plain . && docker push rocm/vllm-ci:8f98d5d5b170a2372c463f311963e235ceedc94f
AMD: Core Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s core"
AMD: Entrypoints Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; export VLLM_WORKER_MULTIPROC_METHOD=spawn && pytest -v -s entrypoints/llm --ignore=entrypoints/llm/test_lazy_outlines.py --ignore=entrypoints/llm/test_generate.py --ignore=entrypoints/llm/test_generate_multiple_loras.py --ignore=entrypoints/llm/test_guided_generate.py --ignore=entrypoints/llm/test_collective_rpc.py && pytest -v -s entrypoints/llm/test_lazy_outlines.py && pytest -v -s entrypoints/llm/test_generate.py && pytest -v -s entrypoints/llm/test_generate_multiple_loras.py && VLLM_USE_V1=0 pytest -v -s entrypoints/llm/test_guided_generate.py && pytest -v -s entrypoints/openai --ignore=entrypoints/openai/test_oot_registration.py --ignore=entrypoints/openai/test_chat_with_tool_reasoning.py --ignore=entrypoints/openai/correctness/ && pytest -v -s entrypoints/test_chat_utils.py && VLLM_USE_V1=0 pytest -v -s entrypoints/offline_mode"
AMD: Regression Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pip install modelscope && pytest -v -s test_regression.py"
AMD: Engine Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s engine test_sequence.py test_config.py test_logger.py && pytest -v -s tokenization"
AMD: Prefix Caching Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s prefix_caching"
AMD: LogitsProcessor Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s test_logits_processor.py && pytest -v -s model_executor/test_guided_processors.py"
AMD: LoRA Test %Nbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s lora --shard-id=$BUILDKITE_PARALLEL_JOB --num-shards=$BUILDKITE_PARALLEL_JOB_COUNT --ignore=lora/test_chatglm3_tp.py --ignore=lora/test_llama_tp.py --ignore=lora/test_minicpmv_tp.py --ignore=lora/test_transfomers_model.py"
AMD: Kernels Test %Nbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s kernels --shard-id=$BUILDKITE_PARALLEL_JOB --num-shards=$BUILDKITE_PARALLEL_JOB_COUNT"
AMD: Tensorizer Testbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; apt-get update && apt-get install -y curl libsodium23 && export VLLM_WORKER_MULTIPROC_METHOD=spawn && pytest -v -s tensorizer_loader"
AMD: Benchmarksbash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/.buildkite ; bash run-benchmarks.sh"
AMD: OpenAI-Compatible Tool Usebash .buildkite/run-amd-test.sh "(command rocm-smi || true) && export VLLM_LOGGING_LEVEL=DEBUG && export VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 && cd /vllm-workspace/tests ; pytest -v -s tool_use"
Neuron Testbash .buildkite/run-neuron-test.sh
Waited 10s
·
Ran in 3m 58s
Agent
Intel CPU Testbash .buildkite/run-cpu-test.sh
Intel HPU Testbash .buildkite/run-hpu-test.sh
Waited 6s
·
Ran in 1m 30s
Agent
Intel GPU Testbash .buildkite/run-xpu-test.sh
Waited 4s
·
Ran in 4m 37s
Agent
IBM Z (s390x) CPU Testbash .buildkite/run-cpu-test-s390x.sh
Waited 1s
·
Ran in 58m 3s
Agent
TPU V0 Testyes | docker system prune -a && if [[ -f ".buildkite/run-tpu-test.sh" ]]; then bash .buildkite/run-tpu-test.sh; fi
Waited 9s
·
Ran in 10s
Agent
TPU V1 Testif [[ -f ".buildkite/run-tpu-v1-test.sh" ]]; then bash .buildkite/run-tpu-v1-test.sh; fi && yes | docker system prune -a
Waited 5s
·
Ran in 8m 26s
Agent
Total Job Run Time: 1h 23m