LUTHE: Breaking the Memory Wall of High-Precision Encrypted LLM Nonlinearities via GPU–CSD Co-Design

Recommended citation: LUTHE: Breaking the Memory Wall of High-Precision Encrypted LLM Nonlinearities via GPU–CSD Co-Design. Xiurui Pan, Guoci Chen, Guangyu Sun, Mingzhe Zhang, Jie Zhang. 59th IEEE/ACM International Symposium on Microarchitecture. 2026.

Abstract

Cloud-based Large Language Model (LLM) inference raises severe privacy concerns, motivating end-to-end encrypted inference with Fully Homomorphic Encryption (FHE). However, practical encrypted inference is bottlenecked by high-precision nonlinear operators (e.g., Softmax and GELU). TFHE’s Without-Padding Programmable Bootstrapping (WoP-PBS) enables exact high-precision nonlinear computation on encrypted data, but each invocation streams hundreds of megabytes to gigabytes of cryptographic keys. Once the key working set exceeds on-device memory capacity, execution hits a sharp capacity cliff: repeated streaming of KSK-family keys over PCIe dominates end-to-end execution time. To address this, we propose LUTHE, a heterogeneous architecture that co-designs a GPU with a Computational Storage Drive (CSD). Specifically, LUTHE partitions the WoP-PBS chain at primitive granularity: key-bound primitives that consume the dominant KSK-family keys execute on the CSD near flash-resident keys, while compute-bound primitives that consume the compact BSK remain on the GPU. LUTHE introduces a unified kernel that fuses all three WoP-PBS evaluation stages to eliminate cross-mode arbitration bubbles, and a FTL-bypass key-stream subsystem for jitter-free key delivery. Evaluations demonstrate that when key working sets exceed on-device memory, LUTHE reduces the latency of encrypted high-precision nonlinear computation by up to 16× over GPU-spilled baselines.