Fedmed: A federated learning framework for language modeling

Xing Wu*, Zhaowang Liang, Jianjia Wang

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

31 Citations (Scopus)

Abstract

Federated learning (FL) is a privacy-preserving technique for training a vast amount of decentralized data and making inferences on mobile devices. As a typical language modeling problem, mobile keyboard prediction aims at suggesting a probable next word or phrase and facilitating the human-machine interaction in a virtual keyboard of the smartphone or laptop. Mobile keyboard prediction with FL hopes to satisfy the growing demand that high-level data privacy be preserved in artificial intelligence applications even with the distributed models training. However, there are two major problems in the federated optimization for the prediction: (1) aggregating model parameters on the server-side and (2) reducing communication costs caused by model weights collection. To address the above issues, traditional FL methods simply use averaging aggregation or ignore communication costs. We propose a novel Federated Mediation (FedMed) framework with the adaptive aggregation, mediation incentive scheme, and topK strategy to address the model aggregation and communication costs. The performance is evaluated in terms of perplexity and communication rounds. Experiments are conducted on three datasets (i.e., Penn Treebank, WikiText-2, and Yelp) and the results demonstrate that our FedMed framework achieves robust performance and outperforms baseline approaches.

Original languageEnglish
Article number4048
Pages (from-to)1-17
Number of pages17
JournalSensors (Switzerland)
Volume20
Issue number14
DOIs
Publication statusPublished - 2 Jul 2020
Externally publishedYes

Keywords

  • Communication efficiency
  • Federated learning
  • Language modeling
  • TopK ranking

Fingerprint

Dive into the research topics of 'Fedmed: A federated learning framework for language modeling'. Together they form a unique fingerprint.

Cite this