Latency tests

Input length: 32 tokens.
Output length: 128 tokens.
Batch size: fixed (8).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: end-to-end latency (mean, median, p99).

Test name	GPU	Mean latency (ms)	Median latency (ms)	P99 latency (ms)
latency_llama8B_tp1	A100-SXM4-80GB	1581.23	1581.07	1582.17
latency_llama70B_tp4	A100-SXM4-80GB	4009.77	4008.23	4023.56
latency_mixtral8x7B_tp2	A100-SXM4-80GB	3601.12	3605.58	3638.14

Throughput tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm to achieve maximum throughput.
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
Evaluation metrics: throughput.

Test name	GPU	Tput (req/s)
throughput_llama8B_tp1	A100-SXM4-80GB	11.5862
throughput_llama70B_tp4	A100-SXM4-80GB	5.04876
throughput_mixtral8x7B_tp2	A100-SXM4-80GB	5.28197

Serving tests

Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
Output length: the corresponding output length of these 200 prompts.
Batch size: dynamically determined by vllm and the arrival pattern of the requests.
Average QPS (query per second): 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
We also added a speculative decoding test for llama-3 70B, under QPS 2
Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).

Test name	GPU	Tput (req/s)	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean ITL (ms)	Median ITL (ms)	P99 ITL (ms)
serving_llama8B_tp1_sharegpt_qps_1	A100-SXM4-80GB	0.996231	41.7151	36.12	85.2946	12.8135	12.4152	32.3985
serving_llama8B_tp1_sharegpt_qps_4	A100-SXM4-80GB	3.58132	46.54	38.5767	100.444	14.4047	13.2772	35.1649
serving_llama8B_tp1_sharegpt_qps_16	A100-SXM4-80GB	8.73152	75.2574	67.0459	166.507	21.4684	18.5655	44.6772
serving_llama8B_tp1_sharegpt_qps_inf	A100-SXM4-80GB	11.0453	2668.6	2618.81	4887.29	25.3366	21.3996	47.1266
serving_llama70B_tp4_sharegpt_qps_1	A100-SXM4-80GB	0.948572	117.243	95.6752	247.351	34.9282	31.6531	101.144
serving_llama70B_tp4_sharegpt_qps_4	A100-SXM4-80GB	2.8667	154.648	139.028	357.478	44.9985	37.4091	118.342
serving_llama70B_tp4_sharegpt_qps_16	A100-SXM4-80GB	4.66126	818.903	886.611	1635.54	60.2651	45.4716	134.859
serving_llama70B_tp4_sharegpt_qps_inf	A100-SXM4-80GB	4.70437	6645.03	6524.62	13075.4	60.6237	45.6135	131.268
serving_mixtral8x7B_tp2_sharegpt_qps_1	A100-SXM4-80GB	0.953785	321.505	62.9261	2994.86	33.0813	28.665	64.1739
serving_mixtral8x7B_tp2_sharegpt_qps_4	A100-SXM4-80GB	2.8415	72.6527	66.3744	143.941	39.0473	33.3613	109.977
serving_mixtral8x7B_tp2_sharegpt_qps_16	A100-SXM4-80GB	4.74567	101.049	98.6947	214.196	46.2396	36.2078	264.145
serving_mixtral8x7B_tp2_sharegpt_qps_inf	A100-SXM4-80GB	5.28146	4440.16	4098.73	5099.34	40.3893	38.0073	71.2875
serving_llama70B_tp4_sharegpt_specdecode_qps_2	A100-SXM4-80GB	1.85435	103.48	104.481	246.27	64.6142	52.833	248.151

json version of the benchmarking tables

This section contains the data of the markdown tables above in JSON format. You can load the benchmarking tables into pandas dataframes as follows:

import json
import pandas as pd

benchmarking_results_json = """The json string"""
benchmarking_results = json.loads(benchmarking_results_json)
latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])

The json string for all benchmarking tables:

{"latency": {"Test name": {"0": "latency_llama8B_tp1", "1": "latency_llama70B_tp4", "2": "latency_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB"}, "Mean latency (ms)": {"0": 1581.2345918190356, "1": 4009.771526201318, "2": 3601.1203796680397}, "Median latency (ms)": {"0": 1581.0680750291795, "1": 4008.2302839728072, "2": 3605.583952041343}, "P99 latency (ms)": {"0": 1582.1672522951849, "1": 4023.555895676837, "2": 3638.137021972798}}, "throughput": {"Test name": {"0": "throughput_llama8B_tp1", "1": "throughput_llama70B_tp4", "2": "throughput_mixtral8x7B_tp2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB"}, "Tput (req/s)": {"0": 11.586154383322661, "1": 5.0487558585540695, "2": 5.281974017289086}}, "serving": {"Test name": {"0": "serving_llama8B_tp1_sharegpt_qps_1", "1": "serving_llama8B_tp1_sharegpt_qps_4", "2": "serving_llama8B_tp1_sharegpt_qps_16", "3": "serving_llama8B_tp1_sharegpt_qps_inf", "4": "serving_llama70B_tp4_sharegpt_qps_1", "5": "serving_llama70B_tp4_sharegpt_qps_4", "6": "serving_llama70B_tp4_sharegpt_qps_16", "7": "serving_llama70B_tp4_sharegpt_qps_inf", "8": "serving_mixtral8x7B_tp2_sharegpt_qps_1", "9": "serving_mixtral8x7B_tp2_sharegpt_qps_4", "10": "serving_mixtral8x7B_tp2_sharegpt_qps_16", "11": "serving_mixtral8x7B_tp2_sharegpt_qps_inf", "12": "serving_llama70B_tp4_sharegpt_specdecode_qps_2"}, "GPU": {"0": "A100-SXM4-80GB", "1": "A100-SXM4-80GB", "2": "A100-SXM4-80GB", "3": "A100-SXM4-80GB", "4": "A100-SXM4-80GB", "5": "A100-SXM4-80GB", "6": "A100-SXM4-80GB", "7": "A100-SXM4-80GB", "8": "A100-SXM4-80GB", "9": "A100-SXM4-80GB", "10": "A100-SXM4-80GB", "11": "A100-SXM4-80GB", "12": "A100-SXM4-80GB"}, "Tput (req/s)": {"0": 0.9962311097317644, "1": 3.5813201358832303, "2": 8.731515894866588, "3": 11.045250675666502, "4": 0.948571868810484, "5": 2.866703518632254, "6": 4.661264798460147, "7": 4.704374644481578, "8": 0.9537846663194779, "9": 2.841499770944359, "10": 4.745666720422895, "11": 5.2814625791258045, "12": 1.8543482993797944}, "Mean TTFT (ms)": {"0": 41.71508113737218, "1": 46.53999873204157, "2": 75.25740994140506, "3": 2668.598015375901, "4": 117.24251082458068, "5": 154.64827167219482, "6": 818.9027324324707, "7": 6645.026940930984, "8": 321.50521200266667, "9": 72.65270059695467, "10": 101.04924640269019, "11": 4440.161882250686, "12": 103.47976516815834}, "Median TTFT (ms)": {"0": 36.11997200641781, "1": 38.57670654542744, "2": 67.04588001593947, "3": 2618.8126539927907, "4": 95.6752304919064, "5": 139.02789453277364, "6": 886.6105620400049, "7": 6524.616534530651, "8": 62.92606447823346, "9": 66.37439248152077, "10": 98.69470895500854, "11": 4098.728025506716, "12": 104.48082751827314}, "P99 TTFT (ms)": {"0": 85.29461560538036, "1": 100.44375726836728, "2": 166.50700119207602, "3": 4887.293133385247, "4": 247.35109049244772, "5": 357.4779474409295, "6": 1635.5378972087049, "7": 13075.43139310088, "8": 2994.855835365597, "9": 143.9412862842437, "10": 214.19564192416138, "11": 5099.336822686018, "12": 246.27022858010602}, "Mean ITL (ms)": {"0": 12.813451297957977, "1": 14.404681953778752, "2": 21.46842945099097, "3": 25.33657363538229, "4": 34.928174087474034, "5": 44.998509931003696, "6": 60.265088008154905, "7": 60.62374158779377, "8": 33.081305114028495, "9": 39.047330551924404, "10": 46.23961980547412, "11": 40.38933014460196, "12": 64.6142487762102}, "Median ITL (ms)": {"0": 12.415243545547128, "1": 13.277159014251083, "2": 18.5654564993456, "3": 21.399641002062708, "4": 31.65305551374331, "5": 37.40905347513035, "6": 45.471562014427036, "7": 45.61350948642939, "8": 28.664962970651686, "9": 33.36131398100406, "10": 36.20784101076424, "11": 38.0073010455817, "12": 52.833033027127385}, "P99 ITL (ms)": {"0": 32.39847831311637, "1": 35.16494511510252, "2": 44.67722342349589, "3": 47.12659605895169, "4": 101.14404625841416, "5": 118.34192156442441, "6": 134.85855089733377, "7": 131.26798910670914, "8": 64.17385465465485, "9": 109.9766490794718, "10": 264.1454911744222, "11": 71.2874795822425, "12": 248.15092891221866}}}

You can also check the raw experiment data in the Artifact tab of the Buildkite page.

Performance Benchmark

[Model] Add OLMoE (#7922)