Home | Publications | CDS+24

Variational Low-Rank Adaptation Using IVON

MCML Authors

Link to Profile Daniel Cremers

Daniel Cremers

Prof. Dr.

Director

Computer Vision & Artificial Intelligence

Abstract

We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models.

inproceedings CDS+24

FITML @NeurIPS 2024

Workshop Fine-Tuning in Modern Machine Learning: Principles and Scalability at the 38th Conference on Neural Information Processing Systems. Vancouver, Canada, Dec 10-15, 2024.

Authors

B. Cong • N. Daheim • Y. Shen • D. Cremers • R. Yokota • M. Khan • T. Möllenhoff

Links

URL GitHub

Research Area

 B1 | Computer Vision

BibTeXKey: CDS+24

#p-cremers
Back to Top