Immersive Video Technologies

1st Edition - September 29, 2022
Editors: Giuseppe Valenzise, Martin Alain, Emin Zerman, Cagri Ozcinar
Language: English
Paperback ISBN:
9 7 8 - 0 - 3 2 3 - 9 1 7 5 5 - 1
eBook ISBN:
9 7 8 - 0 - 3 2 3 - 9 8 6 2 3 - 6

Get a broad overview of the different modalities of immersive video technologies—from omnidirectional video to light fields and volumetric video—from a multimedia processing persp… Read more

Purchase options

Institutional subscription on ScienceDirect

Request a sales quote

Get a broad overview of the different modalities of immersive video technologies—from omnidirectional video to light fields and volumetric video—from a multimedia processing perspective.

From capture to representation, coding, and display, video technologies have been evolving significantly and in many different directions over the last few decades, with the ultimate goal of providing a truly immersive experience to users. After setting up a common background for these technologies, based on the plenoptic function theoretical concept, Immersive Video Technologies offers a comprehensive overview of the leading technologies enabling visual immersion, including omnidirectional (360 degrees) video, light fields, and volumetric video. Following the critical components of the typical content production and delivery pipeline, the book presents acquisition, representation, coding, rendering, and quality assessment approaches for each immersive video modality. The text also reviews current standardization efforts and explores new research directions.

With this book the reader will a) gain a broad understanding of immersive video technologies that use three different modalities: omnidirectional video, light fields, and volumetric video; b) learn about the most recent scientific results in the field, including the recent learning-based methodologies; and c) understand the challenges and perspectives for immersive video technologies.

Cover image
Title page
Table of Contents
Copyright
List of contributors
Preface
Part 1: Foundations
Chapter 1: Introduction to immersive video technologies
Abstract
1.1. Introduction
1.2. What is immersion?
1.3. Immersive video
1.4. Challenges in the processing and content delivery pipeline
1.5. Discussion and perspectives
References
Part 2: Omnidirectional video
Chapter 2: Acquisition, representation, and rendering of omnidirectional videos
Abstract
Acknowledgement
2.1. Introduction
2.2. Acquisition and projection model
2.3. Sphere representation
2.4. Rendering
2.5. Towards multi-view acquisition
References
Chapter 3: Streaming and user behavior in omnidirectional videos
Abstract
3.1. Introduction
3.2. Streaming pipeline: evolution towards ODV
3.3. System-centric streaming
3.4. The role of the user in ODV
3.5. User-centric ODV streaming
3.6. Conclusions and perspectives
References
Chapter 4: Subjective and objective quality assessment for omnidirectional video
Abstract
4.1. Introduction
4.2. Subjective quality assessment
4.3. Short-term video assessment
4.4. Long-term video assessment
4.5. Tile-based streaming
4.6. Objective quality assessment
4.7. VIVA-Q: omnidirectional video quality assessment based on planar Voronoi patches and visual attention
4.8. Conclusion
References
Chapter 5: Omnidirectional video saliency
Abstract
5.1. Introduction
5.2. Collecting user data
5.3. User behavior analysis
5.4. Modeling
5.5. Metrics
5.6. Conclusion and outlook
References
Part 3: Light fields
Introduction
Preliminaries on light fields
Chapter 6: Acquisition of light field images & videos
Abstract
6.1. Plenoptic cameras
6.2. Light field gantries
6.3. Camera arrays
References
Chapter 7: Light field representation
Abstract
7.1. Space domain representation
7.2. Frequency domain representation
7.3. Depth-based representation
7.4. Alternative representations
References
Chapter 8: Compression of light fields
Abstract
8.1. Introduction
8.2. Transform-based methods
8.3. Prediction-based methods
8.4. JPEG Pleno
8.5. Deep learning-based methods
8.6. Performance analysis
8.7. Conclusions and perspectives
References
Chapter 9: Light field processing for media applications
Abstract
Acknowledgements
9.1. Light field processing chain overview
9.2. Image acquisition and geometric camera calibration
9.3. Depth reconstruction
9.4. Interactive light field processing
9.5. Light field rendering
9.6. Real-time rendering in game engines
9.7. Neural rendering and 3D reconstruction
9.8. Conclusion and open challenges
References
Chapter 10: Quality evaluation of light fields
Abstract
10.1. Introduction
10.2. Characteristics of LF-related distortions
10.3. Subjective quality assessment
10.4. Objective quality assessment
10.5. Conclusion
References
Part 4: Volumetric video
Chapter 11: Volumetric video – acquisition, interaction, streaming and rendering
Abstract
Acknowledgements
11.1. Creation of volumetric video
11.2. Animating and interacting with volumetric video
11.3. Streaming volumetric video
11.4. Conclusions
References
Chapter 12: MPEG immersive video
Abstract
12.1. MIV description and profiles
12.2. TMIV description
12.3. Common test conditions
12.4. Evaluation of MIV with different configurations and 2D codecs
12.5. Conclusion
References
Chapter 13: Point cloud compression
Abstract
13.1. Introduction
13.2. Basic tools for point cloud compression
13.3. MPEG PCC standardization
13.4. Learning-based techniques
13.5. Conclusions and perspectives
References
Chapter 14: Coding of dynamic 3D meshes
Abstract
14.1. Introduction
14.2. Mesh fundamentals
14.3. Static meshes
14.4. Constant-connectivity mesh sequences
14.5. Variable-connectivity mesh sequences
14.6. Conclusion and future directions
References
Chapter 15: Volumetric video streaming
Abstract
15.1. Theoretical approaches to volumetric video streaming
15.2. Volumetric video streaming systems
15.3. Conclusion
References
Chapter 16: Processing of volumetric video
Abstract
16.1. Introduction
16.2. Restoration
16.3. Semantic segmentation
16.4. Object detection
16.5. Chapter at a glance
References
Chapter 17: Computational 3D displays
Abstract
Acknowledgements
17.1. Introduction
17.2. Perceptual considerations
17.3. Computational 3D displays
17.4. Summary and discussion
References
Chapter 18: Subjective and objective quality assessment for volumetric video
Abstract
18.1. Subjective quality assessment
18.2. Objective quality assessment
18.3. Conclusion
References
Part 5: Applications
Chapter 19: MR in video guided liver surgery
Abstract
19.1. Introduction
19.2. Medical images pre-processing
19.3. 3D liver modeling
19.4. Model-organ registration
19.5. Mixed reality guided surgery
19.6. Medical validation
19.7. Conclusion
References
Chapter 20: Immersive media productions involving light fields and virtual production LED walls
Abstract
20.1. Light fields in media production
20.2. Immersive LED wall productions
References
Chapter 21: Volumetric video as a novel medium for creative storytelling
Abstract
21.1. Volumetric video and creative practice
21.2. Case studies
21.3. Reflection on V-SENSE practices
21.4. Conclusion
References
Chapter 22: Social virtual reality (VR) applications and user experiences
Abstract
22.1. Introduction
22.2. The social VR clinic
22.3. CakeVR and the birthday celebration
22.4. MediaScape XR: the social VR museum experience
22.5. The social VR movie
22.6. Measuring user experiences in social VR
22.7. Discussion
22.8. Conclusion
References
Index

Giuseppe Valenzise

Giuseppe Valenzise is a CNRS researcher (chargé de recherches) at theUniversité Paris-Saclay, CNRS, CentraleSupélec, Laboratoire des Signaux et Systèmes (L2S, UMR 8506), in the Telecom and Networking hub. Giuseppe completed a master degree and a Ph.D. in Information Technology at the Politecnico di Milano, Italy, in 2007 and 2011, respectively. In 2012, he joined the French Centre National de la Recherche Scientifique (CNRS) as a permanent researcher, first at the Laboratoire Traitement et Communication de l’Information (LTCI) Telecom Paristech, and from 2016 at L2S. He got the French « Habilitation à diriger des recherches » (HDR) from Université Paris-Sud in 2019. His research interests span different fields of image and video processing, including traditional and learning-based image and video compression, light fields and point cloud coding, image/video quality assessment, high dynamic range imaging and applications of machine learning to image and video analysis. He is co-author of more than 70 research publications and of several award-winning papers. He is the recipient of the EURASIP Early Career Award 2018. Giuseppe serves/has served as Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology, IEEE Transactions on Image Processing, Elsevier Signal Processing: Image communication. He is an elected member of the MMSP and IVMSP technical committees of the IEEE Signal Processing Society for the term 2018-2023, as well as a member of the Technical Area Committee on Visual Information Processing of EURASIP.

Affiliations and expertise

Researcher, CentraleSupelec, Laboratoire des Signaux et Systemes, Universite Paris-Saclay, CNRS, France

Martin Alain

Dr. Martin Alain received the Master's degree in electrical engineering from the Bordeaux Graduate School of Engineering (ENSEIRB-MATMECA), Bordeaux, France in 2012 and the PhD degree in signal processing and telecommunications from University of Rennes 1, Rennes, France in 2016. As a PhD student working in Technicolor and INRIA in Rennes, France, he explored novel image and video compression algorithms. Since September 2016, he is a postdoctoral researcher in the V-SENSE project at the School of Computer Science and Statistics in Trinity College Dublin, Ireland. His research interests lie at the intersection of signal and image processing, computer vision, and computer graphics. His current topic involves light field imaging, with a focus on denoising, super-resolution, compression, scene reconstruction, and rendering. Martin is a reviewer for the Irish Machine Vision and Image Processing conference, IEEE International Conference on Image Processing, IEEE Transactions on Image Processing, IEEE International Conference on Multimedia & Expo, IEEE International Workshop on Multimedia Signal Processing, Elsevier Signal Processing: Image Communication, IEEE Transaction on Image Processing, IEEE Transactions on Circuits and Systems for Video Technology, IEEE Transactions on Circuits and Systems I, and IEEE Transactions on Multimedia. He is co-organizer of the special session on Recent Advances in Immersive Imaging Technology held at EUSIPCO 2018 in Rome, ICIP 2019 in Taipei, ICME 2020 in London, and MMSP 2020 in Tampere. He co-organized the tutorial “Immersive Imaging Technologies: from Capture to Display” at ICME 2020 and ACM Multimedia 2020.

Affiliations and expertise

Postdoctoral Researcher, V-SENSE Project, School of Computer Science and Statistics, Trinity College Dublin, Ireland

Emin Zerman

Emin Zerman is an Associate Senior Lecturer at Mid Sweden University. Previously, he worked as a postdoctoral research fellow at Trinity College Dublin, Ireland, working in V-SENSE project on immersive imaging technologies. He received his Ph.D. degree (2018) in Signals and Images from Télécom ParisTech, France, and his M.Sc. degree (2013) and B.Sc. degree (2011) in Electrical and Electronics Engineering from the Middle East Technical University, Turkey. He is a member of IEEE and IEEE Signal Processing Society. He has been acting as a reviewer for several conferences and peer-reviewed journals, including Signal Processing: Image Communications, IEEE TCSVT, IEEE TIP, IEEE TMM, ACM ToG, ACM JoCCH, ACM Multimedia, QoMEX, EUSIPCO, IEEE ICASSP, IEEE MMSP, IEEE ICME, IEEE ICIP, and NordiCHI. He organized several special sessions on immersive imaging technologies at ICME 2020, MMSP 2020, ICIP 2021, and MMSP 2021. He also organized tutorials at IEEE ICME 2020, ACM Multimedia 2020, and IEEE VCIP 2021. He is interested in human visual perception, user interaction, immersive multimedia, 3D technologies, multimedia quality assessment, and information visualization.

Affiliations and expertise

Associate Senior Lecturer, STC Research Center, Mid Sweden University, Sweden

Cagri Ozcinar

Dr. Cagri Ozcinar is a researcher at Samsung R&D Institute UK. Before joining Samsung, he was a research fellow within the V-SENSE project at Trinity College Dublin, Ireland. He was a post-doctoral fellow in the Multimedia group at Institut Mines-Telecom Telecom ParisTech, Paris, France. He received the M.Sc. (Hons.) and the Ph.D. degrees in electronic engineering from the University of Surrey. He is an associate editor of signal, image and video Processing (Springer). He has also been serving as a reviewer for many journals and conference proceedings, such as IEEE TIP, IEEE TCSVT, IEEE TMM, CVPR, IEEE ICASSP, IEEE ICME, IEEE ICIP, IEEE QoMEX, IEEE MMSP, EUSIPCO, BMVC, and WACV. He has been involved in organizing workshops, challenges, and special sessions in EUSIPCO, ICIP, ICME, and MMSP. He co-organized the tutorial “Immersive Imaging Technologies: from Capture to Display” at ICME 2020 and ACM Multimedia 2020. He has been involved in R&D projects that have resulted in commercialized on Samsung DTVs. His research interests include deep learning, computer vision, and immersive media.

Affiliations and expertise

Researcher, Samsung R&D Institute, UK

Life Sciences

Physical Sciences & Engineering

Social Sciences & Humanities

Health

Immersive Video Technologies

Purchase options

Institutional subscription on ScienceDirect

Giuseppe Valenzise

Martin Alain

Emin Zerman

Cagri Ozcinar