Latency tests

Input length: 32 tokens.
Output length: 128 tokens.
Batch size: fixed (8).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: end-to-end latency (mean, median, p99).

Test name	GPU	Mean latency (ms)	Median latency (ms)	P99 latency (ms)
latency_llama8B_tp1	A100-SXM4-80GB	1581.7	1581.79	1582.08
latency_llama70B_tp4	A100-SXM4-80GB	4012.19	4008.98	4028.96
latency_mixtral8x7B_tp2	A100-SXM4-80GB	3601.3	3603.77	3637.19

Throughput tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm to achieve maximum throughput.
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: throughput.

Test name	GPU	Tput (req/s)
throughput_llama8B_tp1	A100-SXM4-80GB	11.6035
throughput_llama70B_tp4	A100-SXM4-80GB	5.04438
throughput_mixtral8x7B_tp2	A100-SXM4-80GB	5.23671

Serving tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm and the arrival pattern of the requests.
Average QPS (query per second): 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
We also added a speculative decoding test for llama-3 70B, under QPS 2
Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

Test name	GPU	Tput (req/s)	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean ITL (ms)	Median ITL (ms)	P99 ITL (ms)
serving_llama8B_tp1_sharegpt_qps_1	A100-SXM4-80GB	0.995749	53.8569	47.7288	96.3587	13.1487	12.7547	27.0356
serving_llama8B_tp1_sharegpt_qps_4	A100-SXM4-80GB	3.56167	58.0361	51.1632	119.318	15.0603	13.9719	43.022
serving_llama8B_tp1_sharegpt_qps_16	A100-SXM4-80GB	8.37996	94.3836	82.0153	320.156	23.5769	21.214	52.9051
serving_llama8B_tp1_sharegpt_qps_inf	A100-SXM4-80GB	10.1384	2579.95	2486.31	5108.82	28.9158	24.7253	52.7236
serving_llama70B_tp4_sharegpt_qps_1	A100-SXM4-80GB	0.948938	140.849	122.325	272.463	34.8706	31.858	96.6512
serving_llama70B_tp4_sharegpt_qps_4	A100-SXM4-80GB	2.85294	179.434	158.159	360.732	45.3849	38.4727	143.368
serving_llama70B_tp4_sharegpt_qps_16	A100-SXM4-80GB	4.57559	724.476	764.861	1489.17	61.4163	47.9866	185.708
serving_llama70B_tp4_sharegpt_qps_inf	A100-SXM4-80GB	4.65647	6246.38	6185.52	12266.6	61.8238	47.7323	129.399
serving_mixtral8x7B_tp2_sharegpt_qps_1	A100-SXM4-80GB	0.947476	347.171	91.5827	3035.76	33.8104	29.4533	65.2048
serving_mixtral8x7B_tp2_sharegpt_qps_4	A100-SXM4-80GB	2.82347	101.6	96.9633	187.093	40.3959	35.0257	121.157
serving_mixtral8x7B_tp2_sharegpt_qps_16	A100-SXM4-80GB	4.45764	177.817	119.87	1698.32	52.0731	39.5495	353.185
serving_mixtral8x7B_tp2_sharegpt_qps_inf	A100-SXM4-80GB	5.22987	3516.31	3581.48	3614.13	41.4171	40.5092	81.7683
serving_llama70B_tp4_sharegpt_specdecode_qps_2	A100-SXM4-80GB	1.85003	101.834	100.26	238.701	65.3016	53.7865	254.811

json version of the benchmarking tables

This section contains the data of the markdown tables above in JSON format. You can load the benchmarking tables into pandas dataframes as follows:

import json
import pandas as pd

benchmarking_results_json = """The json string"""
benchmarking_results = json.loads(benchmarking_results_json)
latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])

The json string for all benchmarking tables:

{"latency": {"Test name": {"0": "latency_llama8B_tp1", "1": "latency_llama70B_tp4", "2": "latency_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB"}, "Mean latency (ms)": {"0": 1581.6977575886995, "1": 4012.1949183987454, "2": 3601.2994588663178}, "Median latency (ms)": {"0": 1581.7918629618362, "1": 4008.9805519673973, "2": 3603.7654490210116}, "P99 latency (ms)": {"0": 1582.080804805737, "1": 4028.959271318745, "2": 3637.188006446231}}, "throughput": {"Test name": {"0": "throughput_llama8B_tp1", "1": "throughput_llama70B_tp4", "2": "throughput_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB"}, "Tput (req/s)": {"0": 11.603484214082911, "1": 5.0443831822286445, "2": 5.236709504900982}}, "serving": {"Test name": {"0": "serving_llama8B_tp1_sharegpt_qps_1", "1": "serving_llama8B_tp1_sharegpt_qps_4", "2": "serving_llama8B_tp1_sharegpt_qps_16", "3": "serving_llama8B_tp1_sharegpt_qps_inf", "4": "serving_llama70B_tp4_sharegpt_qps_1", "5": "serving_llama70B_tp4_sharegpt_qps_4", "6": "serving_llama70B_tp4_sharegpt_qps_16", "7": "serving_llama70B_tp4_sharegpt_qps_inf", "8": "serving_mixtral8x7B_tp2_sharegpt_qps_1", "9": "serving_mixtral8x7B_tp2_sharegpt_qps_4", "10": "serving_mixtral8x7B_tp2_sharegpt_qps_16", "11": "serving_mixtral8x7B_tp2_sharegpt_qps_inf", "12": "serving_llama70B_tp4_sharegpt_specdecode_qps_2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB", "3": "A100-SXM4-80GB", "4": "A100-SXM4-80GB", "5": "A100-SXM4-80GB", "6": "A100-SXM4-80GB", "7": "A100-SXM4-80GB", "8": "A100-SXM4-80GB", "9": "A100-SXM4-80GB", "10": "A100-SXM4-80GB", "11": "A100-SXM4-80GB", "12": "A100-SXM4-80GB"}, "Tput (req/s)": {"0": 0.9957487992761351, "1": 3.561670819551393, "2": 8.379956963090715, "3": 10.13839268153034, "4": 0.9489381851334644, "5": 2.8529402262732266, "6": 4.5755874387378785, "7": 4.656469785340788, "8": 0.947475513901232, "9": 2.8234694299294407, "10": 4.4576386152210095, "11": 5.2298722485442495, "12": 1.8500292103535443}, "Mean TTFT (ms)": {"0": 53.8568948613829, "1": 58.03607420180924, "2": 94.38364682893734, "3": 2579.9519847342162, "4": 140.84919153043302, "5": 179.433954419801, "6": 724.4763798743952, "7": 6246.378558813012, "8": 347.17137419153005, "9": 101.59950747271068, "10": 177.8165889083175, "11": 3516.3125867338385, "12": 101.83409780205693}, "Median TTFT (ms)": {"0": 47.728776989970356, "1": 51.16323800757527, "2": 82.01528448262252, "3": 2486.3077935006004, "4": 122.32488000881858, "5": 158.15938348532654, "6": 764.8607220035046, "7": 6185.517583973706, "8": 91.58273902721703, "9": 96.96333145257086, "10": 119.86984097165987, "11": 3581.477113009896, "12": 100.25986901018769}, "P99 TTFT (ms)": {"0": 96.35874245373998, "1": 119.31775196047965, "2": 320.1556353241901, "3": 5108.816966785234, "4": 272.46260001498723, "5": 360.7316683279349, "6": 1489.168742182664, "7": 12266.593238129279, "8": 3035.755477346246, "9": 187.09266467485577, "10": 1698.31733875093, "11": 3614.1338516841643, "12": 238.7010356097016}, "Mean ITL (ms)": {"0": 13.148679932792326, "1": 15.060293639288119, "2": 23.576922391558206, "3": 28.915838275540434, "4": 34.87059190775772, "5": 45.384935392648146, "6": 61.416337381312225, "7": 61.823793310224836, "8": 33.810426236413576, "9": 40.39587789803708, "10": 52.073068470635114, "11": 41.41707535523679, "12": 65.30155999316244}, "Median ITL (ms)": {"0": 12.754706491250545, "1": 13.971870997920632, "2": 21.21396898292005, "3": 24.72529048100114, "4": 31.85795748140663, "5": 38.472749991342425, "6": 47.986551508074626, "7": 47.73233801824972, "8": 29.4533499982208, "9": 35.02565098460764, "10": 39.54946296289563, "11": 40.50918703433126, "12": 53.78645652672276}, "P99 ITL (ms)": {"0": 27.035617469809953, "1": 43.021992800640874, "2": 52.905109739513136, "3": 52.723591158283014, "4": 96.65118003322276, "5": 143.3681922208052, "6": 185.70811799028888, "7": 129.39867164823227, "8": 65.20481075160204, "9": 121.15739312488584, "10": 353.185171848163, "11": 81.76825514063223, "12": 254.8112686467357}}}

You can also check the raw experiment data in the Artifact tab of the Buildkite page.

Performance Benchmark

yaml