CertiK and Columbia University Unveil TensorLean, Identify 25 Bugs in AI GPU Kernels

Summary
-
CertiK and Columbia University introduced TensorLean, a mathematical verification framework designed to detect computational errors in production AI GPU kernels.
-
Researchers examined over 100 GPU kernels, uncovering 25 software bugs, including two in DeepSeek and 23 in FlashAttention implementations.
-
TensorLean enables developers to verify optimized GPU calculations, reducing risks of costly training repetitions and errors affecting AI model performance.
CertiK and Columbia University researchers have introduced TensorLean, a verification framework that identified 25 software bugs in artificial intelligence GPU kernels. The findings expose computational errors affecting software used in AI training and inference.
According to a November 20 announcement, TensorLean uses mathematical proofs to determine whether GPU kernels execute calculations correctly. Researchers describe the project as the first formal verification of production AI kernels. The framework targets computational errors that conventional software testing may overlook, particularly when developers optimize systems for faster processing.
TensorLean Verification Uncovers Errors in DeepSeek and FlashAttention
Researchers examined more than 100 production GPU kernels across several generations of graphics processing hardware. Their investigation uncovered two bugs in DeepSeek kernels and another 23 in FlashAttention implementations.
FlashAttention accelerates calculations within AI models, while DeepSeek uses optimized computational systems to support complex workloads. Errors in these operations can affect processing accuracy without necessarily interrupting software execution.
TensorLean examines compiled PTX instructions to determine whether calculations meet predefined mathematical requirements. The framework also uses the Lean proof assistant to generate independently verifiable evidence of computational correctness.
CertiK co-founder Ronghui Gu emphasized that identifying errors represents only one part of the verification process. Developers must also establish whether corrected programs satisfy their original requirements under specified conditions.
Framework Targets Costly Errors in Large-Scale AI Training
The research highlights financial risks associated with computational mistakes during expensive artificial intelligence training operations. Incorrect calculations can propagate through later steps, potentially forcing developers to repeat parts of their work.
For example, repeating 1% of a training operation costing $100 million would generate an additional $1 million expense. TensorLean examines optimization techniques that use reduced-precision arithmetic, overlap data transfers, and vary execution configurations. These techniques improve processing efficiency but can introduce errors that conventional testing struggles to detect.
Additionally, developers can repeat verification after modifying GPU kernels to determine whether performance improvements affect computational accuracy. The framework supports reusable mathematical specifications, simplifying verification across related operations.
Researchers aim to integrate these checks into development workflows, helping engineering teams assess computational accuracy alongside performance improvements. The framework provides machine-checkable proofs that developers can use when evaluating optimized GPU software for large-scale AI systems.
This story was originally published by 36Crypto. Read the original article.





