Latency tests

Input length: 32 tokens.
Output length: 128 tokens.
Batch size: fixed (8).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: end-to-end latency (mean, median, p99).

Test name	GPU	Mean latency (ms)	Median latency (ms)	P99 latency (ms)
latency_llama70B_tp4	A100-SXM4-80GB	3990.59	3990.39	3991.87
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
latency_llama8B_tp1	A100-SXM4-80GB	1579.27	1579.34	1579.87
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
latency_mixtral8x7B_tp2	A100-SXM4-80GB	3643.84	3648.91	3680.16
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB

Throughput tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm to achieve maximum throughput.
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: throughput.

Test name	GPU	Tput (req/s)
throughput_llama70B_tp4	A100-SXM4-80GB	4.78859
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
throughput_llama8B_tp1	A100-SXM4-80GB	11.0567
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
throughput_mixtral8x7B_tp2	A100-SXM4-80GB	5.2096
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB

Serving tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm and the arrival pattern of the requests.
Average QPS (query per second): 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
We also added a speculative decoding test for llama-3 70B, under QPS 2
Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

Test name	GPU	Tput (req/s)	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean ITL (ms)	Median ITL (ms)	P99 ITL (ms)
serving_llama70B_tp4_sharegpt_qps_1	A100-SXM4-80GB	0.952471	115.311	91.0313	265.056	33.5852	30.7201	97.3908
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama70B_tp4_sharegpt_qps_16	A100-SXM4-80GB	4.80791	739.389	763.067	1486.83	58.6988	44.9026	247.291
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama70B_tp4_sharegpt_qps_4	A100-SXM4-80GB	2.91433	150.67	128.708	380.441	42.4353	35.2233	111.432
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama70B_tp4_sharegpt_qps_inf	A100-SXM4-80GB	4.83096	6647.28	6483.58	13063.6	59.6783	45.1585	238.631
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama70B_tp4_sharegpt_specdecode_qps_2	A100-SXM4-80GB	1.56388	119.466	104.25	253.872	60.8368	50.1386	240.037
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama8B_tp1_sharegpt_qps_1	A100-SXM4-80GB	0.99686	41.2431	33.9182	86.1282	12.3817	11.9783	30.4699
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama8B_tp1_sharegpt_qps_16	A100-SXM4-80GB	8.78881	85.7946	67.627	341.346	21.2062	18.8506	44.4968
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama8B_tp1_sharegpt_qps_4	A100-SXM4-80GB	3.59462	44.0776	36.9963	88.3449	13.9754	12.9534	32.9243
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_llama8B_tp1_sharegpt_qps_inf	A100-SXM4-80GB	11.1166	2343.4	2269.15	4494.72	26.1787	21.8317	50.6188
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_mixtral8x7B_tp2_sharegpt_qps_1	A100-SXM4-80GB	0.950567	322.814	63.0845	3074.9	32.3384	28.2235	63.3446
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_mixtral8x7B_tp2_sharegpt_qps_16	A100-SXM4-80GB	4.65562	100.619	96.0579	206.358	47.6891	36.4703	295.411
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_mixtral8x7B_tp2_sharegpt_qps_4	A100-SXM4-80GB	2.88023	73.6468	67.235	140.869	38.7074	33.7848	110.65
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
serving_mixtral8x7B_tp2_sharegpt_qps_inf	A100-SXM4-80GB	5.47425	2974.18	2614.49	3604.41	40.8073	38.1229	254.489
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB
	A100-SXM4-80GB

json version of the benchmarking tables

This section contains the data of the markdown tables above in JSON format. You can load the benchmarking tables into pandas dataframes as follows:

import json
import pandas as pd

benchmarking_results_json = """The json string"""
benchmarking_results = json.loads(benchmarking_results_json)
latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])

The json string for all benchmarking tables:

{"latency": {"Test name": {"0": "latency_llama8B_tp1", "1": "latency_llama70B_tp4", "2": "latency_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "1": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "2": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB"}, "Mean latency (ms)": {"0": 1579.2741344310343, "1": 3990.5927983112633, "2": 3643.844443745911}, "Median latency (ms)": {"0": 1579.3387778103352, "1": 3990.386940073222, "2": 3648.910060059279}, "P99 latency (ms)": {"0": 1579.8685044329613, "1": 3991.868680436164, "2": 3680.155389076099}}, "throughput": {"Test name": {"0": "throughput_llama8B_tp1", "1": "throughput_llama70B_tp4", "2": "throughput_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "1": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "2": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB"}, "Tput (req/s)": {"0": 11.056650025301797, "1": 4.788594455549321, "2": 5.209599610769483}}, "serving": {"Test name": {"0": "serving_llama8B_tp1_sharegpt_qps_1", "1": "serving_llama8B_tp1_sharegpt_qps_4", "2": "serving_llama8B_tp1_sharegpt_qps_16", "3": "serving_llama8B_tp1_sharegpt_qps_inf", "4": "serving_llama70B_tp4_sharegpt_qps_1", "5": "serving_llama70B_tp4_sharegpt_qps_4", "6": "serving_llama70B_tp4_sharegpt_qps_16", "7": "serving_llama70B_tp4_sharegpt_qps_inf", "8": "serving_mixtral8x7B_tp2_sharegpt_qps_1", "9": "serving_mixtral8x7B_tp2_sharegpt_qps_4", "10": "serving_mixtral8x7B_tp2_sharegpt_qps_16", "11": "serving_mixtral8x7B_tp2_sharegpt_qps_inf", "12": "serving_llama70B_tp4_sharegpt_specdecode_qps_2"}, "GPU": {"0": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "1": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "2": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "3": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "4": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "5": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "6": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "7": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "8": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "9": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "10": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "11": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB", "12": "A100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB\nA100-SXM4-80GB"}, "Tput (req/s)": {"0": 0.996860105131725, "1": 3.594616274181651, "2": 8.788809990815022, "3": 11.116569511102174, "4": 0.9524712318177595, "5": 2.9143259428791333, "6": 4.807908351065189, "7": 4.830957409318412, "8": 0.9505671381170304, "9": 2.880227262737138, "10": 4.655617720115071, "11": 5.474248795427025, "12": 1.5638805341960242}, "Mean TTFT (ms)": {"0": 41.243121719453484, "1": 44.077623966149986, "2": 85.79455303959548, "3": 2343.400535401888, "4": 115.31116942409426, "5": 150.67011817125604, "6": 739.3892058706842, "7": 6647.2774839098565, "8": 322.8140198113397, "9": 73.64680503029376, "10": 100.61861591646448, "11": 2974.1790991905145, "12": 119.46559221690107}, "Median TTFT (ms)": {"0": 33.91816699877381, "1": 36.99628356844187, "2": 67.62702250853181, "3": 2269.1522729583085, "4": 91.0313033964485, "5": 128.70792299509048, "6": 763.0671088118106, "7": 6483.580605825409, "8": 63.08454996906221, "9": 67.23504257388413, "10": 96.05789557099342, "11": 2614.4928943831474, "12": 104.24970323219895}, "P99 TTFT (ms)": {"0": 86.12816051114349, "1": 88.34485004656015, "2": 341.3459550682454, "3": 4494.72184590064, "4": 265.0556325260538, "5": 380.4409689595922, "6": 1486.8260128656395, "7": 13063.584973793477, "8": 3074.8974635964246, "9": 140.8694759849459, "10": 206.35831077583092, "11": 3604.4086371082813, "12": 253.8721384108068}, "Mean ITL (ms)": {"0": 12.381713669474479, "1": 13.97535663168982, "2": 21.206160098427702, "3": 26.17872029052179, "4": 33.585217624787845, "5": 42.435285622622, "6": 58.698767704667674, "7": 59.67830900113209, "8": 32.33838742374971, "9": 38.70738570247557, "10": 47.689148928676595, "11": 40.80733728618981, "12": 60.83681516647543}, "Median ITL (ms)": {"0": 11.978331953287125, "1": 12.953380355611444, "2": 18.850606866180897, "3": 21.831671707332134, "4": 30.72014870122075, "5": 35.22330219857395, "6": 44.90255401469767, "7": 45.15848867595196, "8": 28.22347404435277, "9": 33.78476295620203, "10": 36.47025674581528, "11": 38.12288399785757, "12": 50.13856524601579}, "P99 ITL (ms)": {"0": 30.469923452474195, "1": 32.9242795240134, "2": 44.4967800006271, "3": 50.61877420172099, "4": 97.39084912464023, "5": 111.43196043092756, "6": 247.29101412929384, "7": 238.63066593185067, "8": 63.34455063566565, "9": 110.65045634284614, "10": 295.41056528687477, "11": 254.4893297180533, "12": 240.03740200772882}}}

You can also check the raw experiment data in the Artifact tab of the Buildkite page.

Performance Benchmark

restore other changes

Buildkite had a problem