Big Data Ingestion and Lifelong Learning Architecture

Lifelong Machine Learning (LML) mimics common human learning experiences. Humans undergo through long learning phase at start while studying followed by updating knowledge base incrementally from everyday instances. The objective is to retain past learnt knowledge and transfer learning to the next task iteratively. Training on the large data pool through a one-shot long running batch job limits the responsiveness and increases the infrastructure cost through large cluster requirements. The full dataset may not be available as well at the initiation of the training process. Through a review of previous work on lifelong machine leaning, we propose a Multi-agent Lambda Architecture (MALA) model to combine historical batch data with live streaming data to develop a lifelong learning system. MALA allows the streaming process to initialize itself with trained model from the batch data. Streaming process takes the batch data offset and incrementally updates the model iteratively with new waves of data. Reasons for our claim are presented through implementation of a recommender engine.