HELM: A Scalable Multi-GPU System for Homomorphic LLM Inference

Recommended citation: HELM: A Scalable Multi-GPU System for Homomorphic LLM Inference. Lei Chen, Yiming Sun, Erci Xu, Yi Chen, Shengyu Fan, Xianglong Deng, Guang Fan, Liang Kong, Yilan Zhu, Geng Yang, Fu Xiao, Shoumeng Yan, Yisong Chang, Mingzhe Zhang. 33rd IEEE International Symposium on High-Performance Computer Architecture. HPCA 2027.

Abstract

Fully Homomorphic Encryption (FHE) enables LLM inference on encrypted inputs without exposing them to the server. However, deploying FHE LLMs on GPUs faces three bottlenecks: preprocessed plaintexts require tens to hundreds of TiB of storage and are costly to generate online; executing FHE operations individually incurs excessive memory traffic and results in low GPU utilization; and existing multi-GPU schemes suffer from load imbalance and excessive communication.

This paper presents Helm, a multi-GPU system for FHE LLM inference that addresses all three bottlenecks. First, Helm introduces a two-stage plaintext generation method that decouples the expensive encoding pipeline from lightweight layout transformations, enabling efficient on-the-fly data generation from a compact basis set. Second, Helm represents the FHE program as parameterized statements with explicit iteration domains, enabling statement-level operation optimizations, including operation fusion, lazy modular reduction, and operation batching to reduce instruction count and memory traffic and improve GPU utilization. Third, Helm introduces a heuristic multi-GPU partitioning method that uses a cost model to evaluate candidate plans and ILP to select a low-cost plan, accounting for both load balance and inter-GPU communication.

Our evaluation on eight NVIDIA H20 GPUs shows that, for 128-token inputs, Helm outperforms baselines by an average of 26.4× on Llama3-8B and 31.6× on Llama3-70B in terms of Time-To-First-Token (TTFT), while reducing the storage requirement by 96×. Under this setting, Helm achieves a TTFT of 48.8 seconds on Llama3-8B, bringing FHE-based LLM inference one step closer to practical deployment.