Multimodal Machine Learning

1. Introduction1.1What is multimodal Machine learning? 1.1.1 Major characteristics of modality system1.2 General framework of Single modal system1.3 General framework for multimodal learning system1.4 Major challenges and motivation 1.5 Representation1.6 Translation and Alignment, and fusion 1.7 Co-learning-based system 1.8 Advantages of multimodal machine learning1.9 Prerequisites for promising applications1.10 Opportunities, challenges and recommendations1.11 Promising fields of application1.12 Community, communication, and data and tool sharing1.13 Technological challenges1.13.1 Intelligent Data Analysis for Applications: Biomedical, IoT-based Intelligent sensors, respective research areas1.14 Summary1.15 References

2 Co-Learning and Fusion based learning Techniques 2.1 Transfer knowledge between modalities2.2 Representations and predictive models of knowledge transfer learning techniques 2.3 Parallel data and Non-parallel data learning techniques 2.4 Transfer learning techniques 2.5 Non-parallel data- Zero-shot learning methods 2.6 Fusion of multiple modalities using Deep learning techniques2.7 Application and case studies: Surveillance Tracking Systems, Medical Systems and other emerging research fields2.8 Summary2.9 References

3. Multimodal representation and descriptive System 3.1 Introduction: Motivation and Challenges3.2 Joint representations 3.3 Visual semantic spaces, multimodal auto encoder3.4 Orthogonal joint representations3.4.1 Component analysis3.4.2 Parallel multimodal representations3.4.3 Similarity metrics, canonical correlation analysis3.5 Language models3.5.1 Unigrams, bigrams, skip-grams, skip-thought3.5.2 Unimodal sequence modelling3.5.2.1 Recurrent neural networks, LSTMs3.6 Optimization method 3.6.1 Backpropagation based learning and metaheuristic algorithms 3.7 Multimodal translation and mapping methods 3.7.1 Encoder-decoder models3.7.2 Machine translation, image captioning methods3.8 Generative vs retrieval approaches3.9 Joint representation methods 3.9.1 Joint Multimodal Representation based systems3.9.2 Performance comparisons and potential challenges3.10 Case study and evaluation: Intelligent data sensing, processing and representation and Applications3.11 Summary3.12 References

4.1 Joint Multimodal Representations4.2 Introduction: Fundamentals: Issues and challenges 4.3 Audio-visual speech recognition methodology4.3.1 Image captioning approaches 4.3.2 Multimodal Deep Boltzmann Machine4.3.3 Freeze branding based identification approach4.3.4 Audio-visual emotion recognition system 4.4 Issues and challenges of classical multimodal representation methods4.5 Generative method for Model evaluation and discussion: Case study 4.6 Summary4.7 References

5 Representation and Translation5.1 Introduction: Fundamental and Motivation 5.2 Coordinated representations5.3 Image Representation based Models and Coordinated Representation: Deep CCA5.4 Similarity matching and localization based techniques for detecting keypoint features5.5 Appearance based recognition of species using texture feature based descriptor techniques5.6 Efficient and robust frontalization of human and animal faces5.7 Applications and case study: Representation on example-based, and generative methods 5.8 Translation: example based method (dictionary based method): case study5.9 Summary 5.10 Reference

6 Alignment and Fusion Methods 6.1. Model-Based (Intermediate) Approaches6.2. Deep neural networks 6.3. Kernel-based methods: Kernel-based fusion, Multiple kernel learning6.4. Graphical models 6.5. Model Representation for behavioural activities using Fusion methods6.6. Benchmark tools and models6.7 Modality alignment: Latent alignment approaches6.8 Attention models, multi instance learning6.8.1 Explicit alignment and Dynamic time warping6.9 Multimodal fusion and co-learning techniques 6.10 Model free approaches: Early and late fusion, hybrid models6.10.1 Multimodal graphical models: Factorial HMM, 6.10.2 Multi-view Hidden state Conditional Random Field framework 6.10.3 Hidden Markov Models (HMMs) and Dynamic Bayesian Networks6.11 Case studies and applications: unimodal, multimodal and hybrid model based learning techniques6.12 Summary6.13 References

7. Multimodal Machine Learning Techniques and framework for Biometric-based System7.1 Introduction: Fundamental working of multi-modal Biometric System 7.2 Unimodal representations: Modality vs Multi-modality system 7.3 Classification of Framework and Multimodal system and framework 7.3.1 Enrolment and recognition phases of multi-modal biometric system7.3.2 Components of multimodal biometric system7.3.2.1 Data acquisition phase based Sensor 7.3.2.2 Data acquisition phase7.3.2.3 Data processing and Enhancement Phase7.3.2.4 Multimodal representation learning method 7.3.2.5 Multimodal tensor representation method 7.3.2.6 Feature extraction and data representation phase7.3.2.7 Database preparation and description module 7.3.2.8 Similarity matching techniques 7.3.2.9 Decision making phase 7.4 Learning strategy and tuning of multimodal system and framework 7.5 Fusion and co-learning algorithm of multimodal system 7.6 Joint distribution based feature learning, analysis and co-learning of multimodal systems 7.6 Multimodal Co-learning Techniques: Case study and Applications 7.7 Case study: Effects of learning methods based on various categories and parameters 7.8 Summary7.9 References

8. Emerging Trends and Future Challenges8.1 Multimodal Machine learning system: The Current State-of-the –Art8.2 Fusion based system for identification and retrieval for endanger species8.3 Multimodal fusion Focus on animal biometrics based recognition system8.4 Application in animal biometrics based fusion system 8.5 Application in Health and well-being Application in Industrial IOT and Health8.6 Future challenges: 8.7 Case studies: Emerging multimodal based learning techniques and applications:8.8 Deep learning based fusion techniques: Face recognition 8.9 Computational Intelligence for Multimedia Big Data on the Cloud with Engineering Applications8.10 Multimodal Joint Generative Discriminative methods and its applications8.11 Summary8.12 References