Skip to yearly menu bar Skip to main content


Poster
in
Workshop: Workshop on Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference

Post-LoRA Restoration: Utilizing Transferability of Low-Rank Adapter in Quantized Foundation Models

Yuto Kanda · Kenji Hatano

[ ] [ Project Page ]
Sun 27 Apr 1:30 a.m. PDT — 2:30 a.m. PDT

Abstract:

In this study, we consider the transferability of LoRA adapters in quantized foundation models.Specifically, we investigate whether LoRA adapters trained on a low-bit-width foundation model can still function effectively when merged into a higher-bit-width foundation model.By leveraging this transferability, it becomes possible to construct models with performance comparable to conventional LoRA using QLoRA adapters trained under resource-constrained conditions.Our method can be utilized to not only improve the performance of trained QLoRA models without additional training but also accelerate the construction of LoRA.

Chat is not available.


OSZAR »