Tencent Hunyuan AI Infra New Open Source: HPC-Ops Inference Core Kernels Fully Upgraded

Tencent Hunyuan announced that the HPC-Ops inference kernel library has undergone a system-level upgrade, evolving from individual operators to a set of optimization capabilities covering the entire inference chain, including five key kernels. This upgrade effectively alleviates practical engineering bottlenecks such as Attention long-tail latency, memory copy overhead, and cross-card communication on mainstream inference platforms, with multiple performance indicators significantly exceeding existing open-source baselines.

HPC-Ops is an industrial-grade, high-performance set of underlying kernel libraries for large model inference, open-sourced and long-term maintained by Tencent Hunyuan's AI Infra team. The main highlights of this upgrade include:

Attention: Dynamic Load Scheduling Alleviates Inference Long-Tail Latency

To address the computational imbalance and inference long-tail latency caused by mixed long and short requests under real workloads, a runtime dynamic load scheduling solution is adopted. Real-world tests show up to 2.95x acceleration for long text, and end-to-end QPM improvement of up to 17%.

Router GEMM: Dual BF16 GEMM Combination Achieves FP32-Level Accuracy

A dual BF16 GEMM combination achieves FP32-level high-precision computation, balancing inference accuracy and GPU compute utilization. Accuracy significantly outperforms conventional BF16/TF32 schemes, with up to 3.22x speedup compared to CuBLAS FP32.

FusedMoE: MoE Full-Module Pipeline Reduces Memory and Kernel Launch Overhead

Builds an MoE full-module pipeline, integrating multi-stage processes and eliminating memory copy and kernel launch overhead. Compared to mainstream frameworks like vLLM and SGLang, performance improves by 1.2x-1.6x.

Fused AllReduce+Norm: Fusion of Cross-GPU Communication, Residual Addition, and Normalization

Deeply fuses cross-GPU communication, residual addition, and normalization computation. Compared to mainstream solutions like NCCL and FlashInfer, performance achieves 1.04x-1.68x speedup.

Sampler: Decoding Sampling Computation Fused into Fewer CUDA Kernels

Fuses the decoding-stage sampling computation (originally requiring over ten operators) into 2 CUDA kernels, significantly reducing scheduling, read/write, and synchronization redundancy overhead. Compared to vLLM, speedup is 4.0x-7.5x; compared to FlashInfer, speedup is 1.9x-4.7x, filling the gap at the end of inference.