Big Data Ingestion and Lifelong Learning Architecture

Gautam Pal, Gangmin Li, Katie Atkinson

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

6 Citations (Scopus)


Lifelong Machine Learning (LML) mimics common human learning experiences. Humans undergo through long learning phase at start while studying followed by updating knowledge base incrementally from everyday instances. The objective is to retain past learnt knowledge and transfer learning to the next task iteratively. Training on the large data pool through a one-shot long running batch job limits the responsiveness and increases the infrastructure cost through large cluster requirements. The full dataset may not be available as well at the initiation of the training process. Through a review of previous work on lifelong machine leaning, we propose a Multi-agent Lambda Architecture (MALA) model to combine historical batch data with live streaming data to develop a lifelong learning system. MALA allows the streaming process to initialize itself with trained model from the batch data. Streaming process takes the batch data offset and incrementally updates the model iteratively with new waves of data. Reasons for our claim are presented through implementation of a recommender engine.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE International Conference on Big Data, Big Data 2018
EditorsNaoki Abe, Huan Liu, Calton Pu, Xiaohua Hu, Nesreen Ahmed, Mu Qiao, Yang Song, Donald Kossmann, Bing Liu, Kisung Lee, Jiliang Tang, Jingrui He, Jeffrey Saltz
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages4
ISBN (Electronic)9781538650356
Publication statusPublished - 2 Jul 2018
Event2018 IEEE International Conference on Big Data, Big Data 2018 - Seattle, United States
Duration: 10 Dec 201813 Dec 2018

Publication series

NameProceedings - 2018 IEEE International Conference on Big Data, Big Data 2018


Conference2018 IEEE International Conference on Big Data, Big Data 2018
Country/TerritoryUnited States


  • Incremental learning
  • Lifelong learning
  • Multi-agent System
  • Recommender systems


Dive into the research topics of 'Big Data Ingestion and Lifelong Learning Architecture'. Together they form a unique fingerprint.

Cite this