Latency tests

Input length: 32 tokens.
Output length: 128 tokens.
Batch size: fixed (8).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: end-to-end latency (mean, median, p99).

Test name	GPU	Mean latency (ms)	Median latency (ms)	P99 latency (ms)
latency_llama70B_tp4	8xH100	2439.99	2439.81	2443.71
latency_llama8B_tp1	8xH100	994.5	994.366	995.876
latency_mixtral8x7B_tp2	8xH100	2323.69	2326.92	2346.61

Throughput tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm to achieve maximum throughput.
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: throughput.

Test name	GPU	Tput (req/s)
throughput_llama70B_tp4	8xH100	8.86568
throughput_llama8B_tp1	8xH100	19.4585
throughput_mixtral8x7B_tp2	8xH100	8.09512

Serving tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm and the arrival pattern of the requests.
Average QPS (query per second): 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
We also added a speculative decoding test for llama-3 70B, under QPS 2
Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

Test name	GPU	Tput (req/s)	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean ITL (ms)	Median ITL (ms)	P99 ITL (ms)
serving_llama70B_tp4_sharegpt_qps_1	8xH100	0.984489	61.4781	55.9601	110.383	19.2868	18.6875	52.309
serving_llama70B_tp4_sharegpt_qps_16	8xH100	7.32158	119.756	107.21	312.069	28.0966	23.6723	67.3224
serving_llama70B_tp4_sharegpt_qps_4	8xH100	3.3427	71.0293	60.3191	146.415	21.9831	20.2686	63.0629
serving_llama70B_tp4_sharegpt_qps_inf	8xH100	8.94641	2803.53	2754.62	5281.13	30.3769	25.3143	176.022
serving_llama70B_tp4_sharegpt_specdecode_qps_2	8xH100	1.63609	66.0252	63.0563	115.782	35.2526	32.4523	101.565
serving_llama8B_tp1_sharegpt_qps_1	8xH100	1.00488	24.2363	21.254	41.9327	7.61503	7.54647	8.76526
serving_llama8B_tp1_sharegpt_qps_16	8xH100	11.5553	35.1213	29.6817	218.156	10.1962	9.41167	21.5632
serving_llama8B_tp1_sharegpt_qps_4	8xH100	3.80433	24.5499	22.3632	43.1358	8.07315	7.84253	19.604
serving_llama8B_tp1_sharegpt_qps_inf	8xH100	19.3955	1199.18	1149.88	2190.49	14.3041	12.4498	24.3135
serving_mixtral8x7B_tp2_sharegpt_qps_1	8xH100	0.9873	343.462	40.7963	3278.56	18.4631	16.1571	38.6695
serving_mixtral8x7B_tp2_sharegpt_qps_16	8xH100	6.56843	233.919	57.6394	1852.83	30.9854	24.0808	179.758
serving_mixtral8x7B_tp2_sharegpt_qps_4	8xH100	3.25041	46.5903	43.9872	80.2607	22.6957	20.4888	47.9203
serving_mixtral8x7B_tp2_sharegpt_qps_inf	8xH100	8.70984	1235.6	1122	1478.41	26.2625	24.5172	193.916

json version of the benchmarking tables

This section contains the data of the markdown tables above in JSON format. You can load the benchmarking tables into pandas dataframes as follows:

import json
import pandas as pd

benchmarking_results_json = """The json string"""
benchmarking_results = json.loads(benchmarking_results_json)
latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])

The json string for all benchmarking tables:

{"latency": {"Test name": {"0": "latency_mixtral8x7B_tp2", "1": "latency_llama70B_tp4", "2": "latency_llama8B_tp1"}, "GPU": {"0": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "1": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "2": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100"}, "Mean latency (ms)": {"0": 2323.691133332856, "1": 2439.987666599336, "2": 994.5002218662314}, "Median latency (ms)": {"0": 2326.9154469999194, "1": 2439.811969998118, "2": 994.3655380011478}, "P99 latency (ms)": {"0": 2346.6061855579756, "1": 2443.714076419783, "2": 995.8756385972811}}, "throughput": {"Test name": {"0": "throughput_llama8B_tp1", "1": "throughput_llama70B_tp4", "2": "throughput_mixtral8x7B_tp2"}, "GPU": {"0": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "1": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "2": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100"}, "Tput (req/s)": {"0": 19.458540696165873, "1": 8.865678855686525, "2": 8.095120387426817}}, "serving": {"Test name": {"0": "serving_llama8B_tp1_sharegpt_qps_inf", "1": "serving_mixtral8x7B_tp2_sharegpt_qps_4", "2": "serving_llama70B_tp4_sharegpt_qps_inf", "3": "serving_llama70B_tp4_sharegpt_specdecode_qps_2", "4": "serving_llama70B_tp4_sharegpt_qps_16", "5": "serving_mixtral8x7B_tp2_sharegpt_qps_16", "6": "serving_llama70B_tp4_sharegpt_qps_1", "7": "serving_mixtral8x7B_tp2_sharegpt_qps_inf", "8": "serving_llama8B_tp1_sharegpt_qps_16", "9": "serving_mixtral8x7B_tp2_sharegpt_qps_1", "10": "serving_llama70B_tp4_sharegpt_qps_4", "11": "serving_llama8B_tp1_sharegpt_qps_4", "12": "serving_llama8B_tp1_sharegpt_qps_1"}, "GPU": {"0": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "1": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "2": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "3": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "4": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "5": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "6": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "7": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "8": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "9": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "10": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "11": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100", "12": "H100\nH100\nH100\nH100\nH100\nH100\nH100\nH100"}, "Tput (req/s)": {"0": 19.39547342025632, "1": 3.2504107798731248, "2": 8.946408659937337, "3": 1.6360925833489208, "4": 7.321583348307787, "5": 6.568426897746511, "6": 0.9844889231612682, "7": 8.709839356312672, "8": 11.555300088413059, "9": 0.9873002072655824, "10": 3.342701977705153, "11": 3.8043288612903066, "12": 1.004880003655527}, "Mean TTFT (ms)": {"0": 1199.1794605448376, "1": 46.59033128515148, "2": 2803.5311091650146, "3": 66.02521472612872, "4": 119.75594840489066, "5": 233.918814050121, "6": 61.47811349501353, "7": 1235.6008808949628, "8": 35.1213088950135, "9": 343.46179797010336, "10": 71.02933013515212, "11": 24.549927424850466, "12": 24.236332524997124}, "Median TTFT (ms)": {"0": 1149.8816895000346, "1": 43.98723900158075, "2": 2754.62286399852, "3": 63.05625700042583, "4": 107.20957949888543, "5": 57.63944300088042, "6": 55.960136000066996, "7": 1121.9970264992298, "8": 29.681739000807283, "9": 40.79632299908553, "10": 60.31914750201395, "11": 22.363176500221016, "12": 21.253961000184063}, "P99 TTFT (ms)": {"0": 2190.4866110999137, "1": 80.26066061305754, "2": 5281.130582102014, "3": 115.78247012810608, "4": 312.06855842781795, "5": 1852.8282021323678, "6": 110.38250380679523, "7": 1478.4071801575915, "8": 218.1564169913326, "9": 3278.564036661773, "10": 146.41501043908644, "11": 43.13582603062966, "12": 41.93274982248111}, "Mean ITL (ms)": {"0": 14.30410569383408, "1": 22.69574103438627, "2": 30.3769058502509, "3": 35.252630533106746, "4": 28.09658355709754, "5": 30.985417160922612, "6": 19.286785884456307, "7": 26.262527887220163, "8": 10.196201753715183, "9": 18.463092159938817, "10": 21.983147218905497, "11": 8.073154728193522, "12": 7.615030542735504}, "Median ITL (ms)": {"0": 12.449808999008383, "1": 20.488764999754494, "2": 25.31431199895451, "3": 32.452288000058616, "4": 23.67233050063078, "5": 24.08084200033045, "6": 18.687542500629206, "7": 24.517202999049914, "8": 9.411667499080068, "9": 16.157070000190288, "10": 20.26860599835345, "11": 7.842528000765014, "12": 7.546472999820253}, "P99 ITL (ms)": {"0": 24.3134814014411, "1": 47.92028530049719, "2": 176.02200865869236, "3": 101.5653808391653, "4": 67.32243038157321, "5": 179.7578364215953, "6": 52.30898935122243, "7": 193.9160450973941, "8": 21.563168099964972, "9": 38.6694669813005, "10": 63.06288476906058, "11": 19.604046852145984, "12": 8.765259550818906}}}

You can also check the raw experiment data in the Artifact tab of the Buildkite page.

Performance Benchmark

newline