Text-speech collaboration LLM embedding low-rank adaptation, activation-aware weight quantization and knowledge distillation

Large Language Model, or LLM, are enabling more and more real world applications. To fuse a rich variety of information sources into LLM provides powerful solutions. This paper designs a text and speech fusion framework on LLM. It embeds LoRA fine-tuning, AWQ quantization, and knowledge distillation to improve the output models' performance. The proposed framework is evaluated on emotional applications and medical applications and compared with comprehensive large language models.