Skip to content

Latest commit

 

History

History
92 lines (65 loc) · 9.98 KB

README.md

File metadata and controls

92 lines (65 loc) · 9.98 KB

Reproducible benchmark recipes for GPUs

License

Welcome to the reproducible benchmark recipes repository for GPUs! This repository contains recipes for reproducing training and serving benchmarks for large machine learning models using GPUs on Google Cloud.

Overview

  1. Identify your requirements: Determine the model, GPU type, workload, framework, and orchestrator you are interested in.
  2. Select a recipe: Based on your requirements use the Benchmark support matrix to find a recipe that meets your needs.
  3. Follow the recipe: each recipe will provide you with procedures to complete the following tasks:
    • Prepare your environment
    • Run the benchmark
    • Analyze the benchmarks results. This includes not just the results but detailed logs for further analysis

Benchmarks support matrix

Training benchmarks A3 Mega

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
GPT3-175B A3 Mega (NVIDIA H100) NeMo Pre-training GKE Link
Llama-3-70B A3 Mega (NVIDIA H100) NeMo Pre-training GKE Link
Llama-3.1-70B A3 Mega (NVIDIA H100) NeMo Pre-training GKE Link
Mixtral-8-7B A3 Mega (NVIDIA H100) NeMo Pre-training GKE Link

Training benchmarks A3 Ultra

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
Llama-3.1-70B A3 Ultra (NVIDIA H200) MaxText Pre-training GKE Link
Llama-3.1-70B A3 Ultra (NVIDIA H200) NeMo Pre-training GKE Link
Llama-3.1-405B A3 Ultra (NVIDIA H200) MaxText Pre-training GKE Link
Llama-3.1-405B A3 Ultra (NVIDIA H200) NeMo. Pre-training GKE Link
Mixtral-8-7B A3 Ultra (NVIDIA H200) NeMo Pre-training GKE Link

Training benchmarks A4

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
Llama-3.1-70B A4 (NVIDIA B200) MaxText Pre-training GKE Link
Llama-3.1-70B A4 (NVIDIA B200) NeMo Pre-training GKE Link
Llama-3.1-405B A4 (NVIDIA B200) MaxText Pre-training GKE Link
Llama-3.1-405B A4 (NVIDIA B200) NeMo Pre-training GKE Link
Mixtral-8-7B A4 (NVIDIA B200) NeMo Pre-training GKE Link

Inference benchmarks A3 Mega

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
Llama-4 A3 Mega (NVIDIA H100) SGLang Inference GKE Link
DeepSeek R1 671B A3 Mega (NVIDIA H100) SGLang Inference GKE Link
DeepSeek R1 671B A3 Mega (NVIDIA H100) vLLM Inference GKE Link

Inference benchmarks A3 Ultra

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
Llama-4 A3 Ultra (NVIDIA H200) vLLM Inference GKE Link
Llama-3.1-405B A3 Ultra (NVIDIA H200) TensorRT-LLM Inference GKE Link
DeepSeek R1 671B A3 Ultra (NVIDIA H200) SGLang Inference GKE Link
DeepSeek R1 671B A3 Ultra (NVIDIA H200) vLLM Inference GKE Link

Checkpointing benchmarks

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
Llama-3.1-70B A3 Mega (NVIDIA H100) NeMo Pre-training using Google Cloud Storage buckets for checkpoints GKE Link
Llama-3.1-70B A3 Mega (NVIDIA H100) NeMo Pre-training using Google Cloud Parallelstore for checkpoints GKE Link

Goodput benchmarks

Models GPU Machine Type Framework Workload Type Orchestrator Link to the recipe
Llama-3.1-70B A3 Mega (NVIDIA H100) NeMo Pre-training using the Google Cloud Resiliency library GKE Link

Repository structure

  • training/: Contains recipes to reproduce training benchmarks with GPUs.
  • inference/: Contains recipes to reproduce inference benchmarks with GPUs.
  • src/: Contains shared dependencies required to run benchmarks, such as Docker and Helm charts.
  • docs/: Contains supporting documentation for the recipes, such as explanation of benchmark methodologies or configurations.

Getting help

If you have any questions or if you found any problems with this repository, please report through GitHub issues.

Disclaimer

This is not an officially supported Google product. The code in this repository is for demonstrative purposes only.