Latency tests

Input length: 32 tokens.
Output length: 128 tokens.
Batch size: fixed (8).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: end-to-end latency (mean, median, p99).

Test name	GPU	Mean latency (ms)	Median latency (ms)	P99 latency (ms)
latency_llama8B_tp1	A100-SXM4-80GB	1617.28	1616.47	1622.04
latency_llama70B_tp4	A100-SXM4-80GB	4037.62	4036.7	4041.88
latency_mixtral8x7B_tp2	A100-SXM4-80GB	3626.66	3628.5	3660.52

Throughput tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm to achieve maximum throughput.
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: throughput.

Test name	GPU	Tput (req/s)
throughput_llama8B_tp1	A100-SXM4-80GB	10.574
throughput_llama70B_tp4	A100-SXM4-80GB	4.84393
throughput_mixtral8x7B_tp2	A100-SXM4-80GB	5.0045

Serving tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm and the arrival pattern of the requests.
Average QPS (query per second): 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
We also added a speculative decoding test for llama-3 70B, under QPS 2
Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

Test name	GPU	Tput (req/s)	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean ITL (ms)	Median ITL (ms)	P99 ITL (ms)
serving_llama8B_tp1_sharegpt_qps_1	A100-SXM4-80GB	0.995214	41.0538	35.6511	90.1374	13.656	13.2625	26.9675
serving_llama8B_tp1_sharegpt_qps_4	A100-SXM4-80GB	3.52499	47.1808	38.8993	103.948	16.5601	15.3253	42.5954
serving_llama8B_tp1_sharegpt_qps_16	A100-SXM4-80GB	7.54346	85.0759	63.683	271.986	28.9588	27.5382	57.1875
serving_llama8B_tp1_sharegpt_qps_inf	A100-SXM4-80GB	8.88395	2546.21	2322.35	5208.82	35.1656	30.7634	96.594
serving_llama70B_tp4_sharegpt_qps_1	A100-SXM4-80GB	0.947304	109.763	89.2504	248.975	35.9519	33.751	87.0247
serving_llama70B_tp4_sharegpt_qps_4	A100-SXM4-80GB	2.80893	135.376	108.895	312.081	48.9115	41.8344	123.802
serving_llama70B_tp4_sharegpt_qps_16	A100-SXM4-80GB	4.27	594.271	601.971	1304.89	69.0364	55.3433	253.689
serving_llama70B_tp4_sharegpt_qps_inf	A100-SXM4-80GB	4.2166	6314.31	6336.93	12584.1	70.5416	54.8834	253.217
serving_mixtral8x7B_tp2_sharegpt_qps_1	A100-SXM4-80GB	0.948341	275.529	62.3878	2585.94	34.3988	29.9793	79.8977
serving_mixtral8x7B_tp2_sharegpt_qps_4	A100-SXM4-80GB	2.76413	102.659	63.8425	1418.03	45.2685	38.8579	138.652
serving_mixtral8x7B_tp2_sharegpt_qps_16	A100-SXM4-80GB	4.19408	84.5289	76.7746	177.817	55.2194	45.924	299.78
serving_mixtral8x7B_tp2_sharegpt_qps_inf	A100-SXM4-80GB	4.56234	4432.25	4277.85	5202.67	50.6703	48.0288	66.7545
serving_llama70B_tp4_sharegpt_specdecode_qps_2	A100-SXM4-80GB	1.84767	100.753	101.708	243.941	67.8051	56.0745	254.302

json version of the benchmarking tables

This section contains the data of the markdown tables above in JSON format. You can load the benchmarking tables into pandas dataframes as follows:

import json
import pandas as pd

benchmarking_results_json = """The json string"""
benchmarking_results = json.loads(benchmarking_results_json)
latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])

The json string for all benchmarking tables:

{"latency": {"Test name": {"0": "latency_llama8B_tp1", "1": "latency_llama70B_tp4", "2": "latency_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB"}, "Mean latency (ms)": {"0": 1617.2771843150258, "1": 4037.622473171601, "2": 3626.6630801061788}, "Median latency (ms)": {"0": 1616.469708736986, "1": 4036.7008089087903, "2": 3628.502029925585}, "P99 latency (ms)": {"0": 1622.0402523502707, "1": 4041.8770485557616, "2": 3660.51890344359}}, "throughput": {"Test name": {"0": "throughput_llama8B_tp1", "1": "throughput_llama70B_tp4", "2": "throughput_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB"}, "Tput (req/s)": {"0": 10.574022438443377, "1": 4.843926492038328, "2": 5.004497987087634}}, "serving": {"Test name": {"0": "serving_llama8B_tp1_sharegpt_qps_1", "1": "serving_llama8B_tp1_sharegpt_qps_4", "2": "serving_llama8B_tp1_sharegpt_qps_16", "3": "serving_llama8B_tp1_sharegpt_qps_inf", "4": "serving_llama70B_tp4_sharegpt_qps_1", "5": "serving_llama70B_tp4_sharegpt_qps_4", "6": "serving_llama70B_tp4_sharegpt_qps_16", "7": "serving_llama70B_tp4_sharegpt_qps_inf", "8": "serving_mixtral8x7B_tp2_sharegpt_qps_1", "9": "serving_mixtral8x7B_tp2_sharegpt_qps_4", "10": "serving_mixtral8x7B_tp2_sharegpt_qps_16", "11": "serving_mixtral8x7B_tp2_sharegpt_qps_inf", "12": "serving_llama70B_tp4_sharegpt_specdecode_qps_2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB", "3": "A100-SXM4-80GB", "4": "A100-SXM4-80GB", "5": "A100-SXM4-80GB", "6": "A100-SXM4-80GB", "7": "A100-SXM4-80GB", "8": "A100-SXM4-80GB", "9": "A100-SXM4-80GB", "10": "A100-SXM4-80GB", "11": "A100-SXM4-80GB", "12": "A100-SXM4-80GB"}, "Tput (req/s)": {"0": 0.9952138565398158, "1": 3.524989902624624, "2": 7.543463182939955, "3": 8.883946853127913, "4": 0.9473037921739562, "5": 2.8089264013807798, "6": 4.269996513142713, "7": 4.216597507373386, "8": 0.9483414707098156, "9": 2.7641276023263224, "10": 4.194075229374556, "11": 4.562342923870557, "12": 1.847671562707932}, "Mean TTFT (ms)": {"0": 41.05375174200162, "1": 47.1808249829337, "2": 85.07591066416353, "3": 2546.206761582289, "4": 109.76287236902863, "5": 135.37619267590344, "6": 594.2707019392401, "7": 6314.308426815551, "8": 275.528943259269, "9": 102.65943526756018, "10": 84.52894428279251, "11": 4432.245984103065, "12": 100.75261078774929}, "Median TTFT (ms)": {"0": 35.65106447786093, "1": 38.89929782599211, "2": 63.68295173160732, "3": 2322.3459464497864, "4": 89.25037388689816, "5": 108.89450181275606, "6": 601.9711957778782, "7": 6336.933511076495, "8": 62.387789599597454, "9": 63.84247844107449, "10": 76.77459949627519, "11": 4277.8530875220895, "12": 101.70763009227812}, "P99 TTFT (ms)": {"0": 90.1373714674264, "1": 103.9479166455557, "2": 271.98611097875977, "3": 5208.816420561633, "4": 248.97455110214634, "5": 312.08118633832737, "6": 1304.8873715335499, "7": 12584.133333461357, "8": 2585.943753621539, "9": 1418.0258338944986, "10": 177.81658177264032, "11": 5202.66991937533, "12": 243.94096343778045}, "Mean ITL (ms)": {"0": 13.655986791112367, "1": 16.560109870233497, "2": 28.958777859872445, "3": 35.16564375812524, "4": 35.9519015759552, "5": 48.91151052609081, "6": 69.03637773508281, "7": 70.54159337282339, "8": 34.398813092414365, "9": 45.26854334227806, "10": 55.21941989068943, "11": 50.670299539490635, "12": 67.8050746881394}, "Median ITL (ms)": {"0": 13.262526830658317, "1": 15.325346263125539, "2": 27.53820107318461, "3": 30.763356015086174, "4": 33.75103580765426, "5": 41.83437908068299, "6": 55.34327286295593, "7": 54.88343117758632, "8": 29.97931418940425, "9": 38.85794896632433, "10": 45.924048870801926, "11": 48.02882997319102, "12": 56.07449868693948}, "P99 ITL (ms)": {"0": 26.96745497174561, "1": 42.595356479287155, "2": 57.18748763669286, "3": 96.59399007447064, "4": 87.02465936075896, "5": 123.8016093429178, "6": 253.6886413162574, "7": 253.2170176366344, "8": 79.89774275571104, "9": 138.65248357877067, "10": 299.77985393255943, "11": 66.75454512238493, "12": 254.3018987914547}}}

You can also check the raw experiment data in the Artifact tab of the Buildkite page.

Performance Benchmark

adjust the command to kill vllm