基于迁移学习与强化学习的自动音频标注系统

Automated audio captioning task aims at generating text description of a piece of audio. This system based on an encoder-decoder architecture, consisting of a Convolutional Neural Network(CNN) encoder and a Transformer decoder. In order to solve the problem of the mismatch between evaluation metrics and the loss function, reinforcement learning is investigated for generating more accurate caption. Furthermore, the system has a pre-trained encoder via transfer learning. Clotho dataset was chosen as our dataset. The results show that both techniques can further improve the performance of the captioning system.