Pipeline for efficient quantization of large language models for resource-constrained deployment

Tchiwewe@csir.co.zaOnumanyi, Adeiza J2026-06-052026-06-052025-12979-8-3315-6518-32153-0033DOI: 10.1109/AFRICON66545.2025.11533795http://hdl.handle.net/10204/14817Recent years have seen breakthroughs in large language models (LLMs), such as the GPT and LLaMA family of models that have transformed natural language processing. Despite this, their considerable computational and memory requirements inhibit their deployment in edge and mobile environments. In this paper we introduce a modular quantization pipeline that reduces the memory footprint of LLMs while preserving core performance. We evaluate the basic and advanced quantization techniques, including Absolute Maximum Quantization, Zero-Point Quantization, GPTQ, and NF4, and make use of popular toolkits that include bitsandbytes and AutoGPTQ. Experimental results on representative tasks show that 4- and 8- bit quantized models can be run on commodity GPUs and CPUs with acceptable quality loss. Our experiments demonstrated up to 8x compression, making our pipeline suitable for LLM deployment in both edge and industrial scenarios.AbstractenArtificial intelligenceEdge AIInferenceLatencyLarge Language ModelsModel compressionPerplexityQuantizationResource efficiencyPipeline for efficient quantization of large language models for resource-constrained deploymentConference PresentationN/A