Latency tests

Input length: 32 tokens.
Output length: 128 tokens.
Batch size: fixed (8).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: end-to-end latency (mean, median, p99).

Test name	GPU	Mean latency (ms)	Median latency (ms)	P99 latency (ms)
latency_llama8B_tp1	H200	835.803	835.823	836.494

Throughput tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm to achieve maximum throughput.
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: throughput.

Test name	GPU	Tput (req/s)
throughput_llama8B_tp1	H200	22.6483

Serving tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm and the arrival pattern of the requests.
Average QPS (query per second): 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
We also added a speculative decoding test for llama-3 70B, under QPS 2
Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

Test name	GPU	Tput (req/s)	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean ITL (ms)	Median ITL (ms)	P99 ITL (ms)
serving_llama8B_tp1_sharegpt_qps_1	H200	1.00724	23.9425	22.1272	42.3729	6.2677	6.18246	6.71632
serving_llama8B_tp1_sharegpt_qps_4	H200	3.86697	26.0825	24.0827	42.4271	6.67531	6.39688	20.7187
serving_llama8B_tp1_sharegpt_qps_16	H200	12.1593	36.7253	30.8242	96.4576	9.15765	7.92175	22.6109
serving_llama8B_tp1_sharegpt_qps_inf	H200	22.0921	1340.95	1297.49	2326.37	12.2868	11.3552	23.3757

json version of the benchmarking tables

This section contains the data of the markdown tables above in JSON format. You can load the benchmarking tables into pandas dataframes as follows:

import json
import pandas as pd

benchmarking_results_json = """The json string"""
benchmarking_results = json.loads(benchmarking_results_json)
latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])

The json string for all benchmarking tables:

{"latency": {"Test name": {"0": "latency_llama8B_tp1"}, "GPU": {"0": "H200"}, "Mean latency (ms)": {"0": 835.8030850766227}, "Median latency (ms)": {"0": 835.8232399914414}, "P99 latency (ms)": {"0": 836.4935506531037}}, "throughput": {"Test name": {"0": "throughput_llama8B_tp1"}, "GPU": {"0": "H200"}, "Tput (req/s)": {"0": 22.648335944225522}}, "serving": {"Test name": {"0": "serving_llama8B_tp1_sharegpt_qps_1", "1": "serving_llama8B_tp1_sharegpt_qps_4", "2": "serving_llama8B_tp1_sharegpt_qps_16", "3": "serving_llama8B_tp1_sharegpt_qps_inf"}, "GPU": {"0": "H200", "1": "H200", "2": "H200", "3": "H200"}, "Tput (req/s)": {"0": 1.0072357053803496, "1": 3.8669743803061287, "2": 12.159321087984523, "3": 22.092071735227822}, "Mean TTFT (ms)": {"0": 23.942484236322343, "1": 26.082463303464465, "2": 36.72528339549899, "3": 1340.9540325473063}, "Median TTFT (ms)": {"0": 22.12722651893273, "1": 24.08269647276029, "2": 30.824234010651708, "3": 1297.4901639972813}, "P99 TTFT (ms)": {"0": 42.372874387074255, "1": 42.42708148434748, "2": 96.4576136635148, "3": 2326.3715022196993}, "Mean ITL (ms)": {"0": 6.267703567255004, "1": 6.675313546601709, "2": 9.157652455733187, "3": 12.286796663411865}, "Median ITL (ms)": {"0": 6.182460987474769, "1": 6.396879500243813, "2": 7.921750482637435, "3": 11.355184018611908}, "P99 ITL (ms)": {"0": 6.7163177009206265, "1": 20.718654856318608, "2": 22.610900370636955, "3": 23.375684348866343}}}

You can also check the raw experiment data in the Artifact tab of the Buildkite page.

Performance Benchmark

fix scripts