Modern AI Chips & Data Center Compute Architectures Cheat Sheet: CPU, GPU, TPU, LPU, NPU, DPU & Global Silicon Selection

In the era of industrial generative large language models (LLMs) and multimodal AI agents, computing infrastructure choices dictate training convergence speeds, Time-to-First-Token (TTFT), cost per generated token, and cluster energy efficiency. Modern AI compute has shifted from single-GPU dominance to an orchestrated ecosystem of Heterogeneous Computing and Domain-Specific Architectures (DSA).

This cheat sheet strips out legacy models and marketing noise, anchoring directly on production silicon microarchitectures and data center topologies. It provides a compact overview of CPU, GPU, TPU, LPU, NPU, and DPU architectures, alongside major global AI accelerators (Huawei Ascend, Cambricon, Hygon, AWS Trainium, Meta MTIA, Microsoft Maia, NVIDIA Blackwell, etc.), including LLM VRAM sizing rules, cluster operational diagnostics, and MFU evaluation scripts.

🔗 Related Cheat Sheets: LLM & Agent Technical Concepts Cheat Sheet · Core AI Algorithms Cheat Sheet · Operations & High-Concurrency Systems Cheat Sheet

---

Modern AI Chip Microarchitecture & Data Center Pipeline Overview

---

1. Modern AI Data Center End-to-End Pipeline (PD Disaggregation)

The complete lifecycle of an LLM request in a modern cloud AI data center follows the pipeline: Data Offload ➔ Orchestration ➔ Prefill Compute ➔ Decode Generation ➔ Response Dispatch:

---

2. Six Core Computing Architectures Comparison Matrix

---

3. Global AI Compute Accelerator Landscapes

A. Domestic Independent Compute Accelerators

B. North American Cloud Hyperscaler Custom ASIC Matrix

---

4. LLM VRAM & Bandwidth Calculation Guide

1. Mathematical Formulas

A. Static Model Weights Footprint

$$\text{VRAM}_{\text{weights}} = \frac{P \times \text{Bits per Parameter}}{8 \times 1024^3} \text{ (GB)}$$

• FP16 / BF16 (16-bit): Each parameter requires 2 Bytes. Example: 70B model requires $70 \times 2 = 140 \text{ GB}$.
• INT8 (8-bit): Each parameter requires 1 Byte. Example: 70B model requires $70 \times 1 = 70 \text{ GB}$.
• FP4 / INT4 (4-bit): Each parameter requires 0.5 Bytes. Example: 70B model requires $70 \times 0.5 = 35 \text{ GB}$.

B. Dynamic KV-Cache VRAM Footprint (Per Concurrent Request, Per Token)

$$\text{Memory}{\text{KV-Cache}} = 2 \times L \times H{\text{kv}} \times D_{\text{head}} \times \text{Precision (Bytes)} \times S \text{ (Bytes)}$$

• $L$: Number of Transformer layers.
• $H_{\text{kv}}$: Number of Key-Value attention heads (Grouped Query Attention reduces this significantly compared to full MHA).
• $D_{\text{head}}$: Dimension per attention head (typically 128).
• $S$: Context sequence length (tokens).

2. Standard LLM VRAM Sizing Reference Table

💡 KV-Cache Sizing Rule: At concurrency $B = 64$ and context $32\text{k}$, a 70B model's KV-Cache alone consumes $64 \times 2.62\text{ GB} = 167.7\text{ GB}$! This exceeds the static weights footprint. Employing PagedAttention and FP8 KV-Cache quantization is essential.

---

5. AI Cluster Operations & Monitoring Command Matrix

1. NVIDIA GPU Diagnostic Commands

# 1. Output GPU status continuously in CSV format (Index, Name, Temp, Utilization, Memory, Power)
nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,utilization.memory,memory.used,memory.total,power.draw --format=csv -l 1

# 2. Check NVLink hardware link status and health
nvidia-smi nvlink -s

# 3. Check NVLink CRC error packet counts (rising counts indicate loose physical cables or slots)
nvidia-smi nvlink -e

# 4. View inter-GPU physical topology (identifies NVLink / PCIe / NUMA boundaries)
nvidia-smi topo -m

# 5. Lock GPU core and memory clocks (benchmarking and jitter elimination)
sudo nvidia-smi --lock-gpu-clocks=2100,2100

# 6. Run DCGM Level 3 deep hardware diagnostics (includes memory stress test)
dcgmi diag -r 3

2. Huawei Ascend NPU (CANN) Diagnostic Commands

# 1. Query all Ascend NPU chips, HBM usage, and driver versions
npu-smi info

# 2. Continuously monitor NPU compute utilization, temperature, and power draw (1s interval)
npu-smi info -w 1

# 3. Check HCCS high-speed interconnect topology between Ascend chips
npu-smi info -t hccs -i 0

# 4. View motherboard temperature sensors and power health
npu-smi info -t board -i 0

# 5. Query device error codes and hardware alarm logs
npu-smi info -t error-code -i 0

3. AMD Instinct GPU (ROCm) Commands

# 1. Monitor MI300X VRAM, utilization, temperature, and power
rocm-smi --showuse --showmeminfo vram --showtemp --showpower

# 2. Display intra-node XGMI interconnect bus topology
rocm-smi --showtopo

4. Model FLOPs Utilization (MFU) Production Estimation Script

def calculate_mfu(
    num_params_billion: float,      # Model parameters in billions (e.g., 70.0)
    throughput_tokens_per_sec: float,# Overall cluster generated tokens per second
    num_gpus: int,                  # Total number of participating GPUs (e.g., 8)
    peak_tflops_per_gpu: float,     # Theoretical peak TFLOPS per GPU (e.g., H100 FP16 peak = 989.0)
    is_training: bool = False       # True for training (6*P), False for inference (2*P)
) -> float:
    """Calculates Model FLOPs Utilization (MFU). Typical top-tier clusters achieve 40% - 55%."""
    # Forward pass: 2*P FLOPs/token. Backward pass: 4*P FLOPs/token. Training total: 6*P.
    factor = 6.0 if is_training else 2.0
    flops_per_token = factor * (num_params_billion * 1e9)
    
    # Achieved compute throughput in TFLOPS
    achieved_tflops = (throughput_tokens_per_sec * flops_per_token) / 1e12
    
    # Cluster theoretical peak compute in TFLOPS
    total_peak_tflops = num_gpus * peak_tflops_per_gpu
    
    mfu = (achieved_tflops / total_peak_tflops) * 100.0
    return round(mfu, 2)

# Example: 8x H100 running 70B model inference at 800 tokens/s
mfu = calculate_mfu(num_params_billion=70, throughput_tokens_per_sec=800, num_gpus=8, peak_tflops_per_gpu=989)
print(f"Observed Cluster MFU: {mfu}%")
# An MFU < 15% indicates severe memory bandwidth starvation or inter-node network bottlenecks.

---

6. Architectural Evolution Summary

1. Domain-Specific Architectures (DSA) End Monolithic GPU Monopoly: Compute is transitioning from homogeneous scaling to heterogeneous orchestration. CPUs handle dynamic control flow; DPUs offload networking and security; GPUs/TPUs process large-context prefill; LPUs specialize in low-latency autoregressive decode; and NPUs power ultra-efficient edge inference.
2. PD Disaggregation Is the Production Paradigm: Compute-intensive Prefill and memory-intensive Decode phases demand diametrically opposed hardware characteristics. Separating them across RDMA-linked nodes prevents resource contention and multiplies cluster throughput per watt.
3. Software-Hardware Co-Design Is the True Differentiator: Without a high-efficiency compiler (TensorRT, Triton, CANN, OpenXLA), specialized hardware yields little real-world benefit. Operator coverage, automated kernel fusion, and virtualized memory management remain the ultimate yardsticks of silicon maturity.