PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Significantly improved finetuned perf by simply changing the initialization of LoRAs AB matrix from Gaussian/zero to principal components. On GSM8K, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, outperforming LoRA’s 67.7% by 5.16%. Github: https://github.com/GraphPKU/PiSSA Paper: https://arxiv.org/abs/2404.02948 @opendatascience 12.6K viewsedited 08:06
