View synthesis from silhouette using deep convolutional neural network

Multiview video could be the basis to support various applications, such as Three-Dimensional video (3DV), Virtual Reality (VR), and Free Viewpoint Video (FVV). Multiview data is intrinsically redundant, in fact, the semantic contents of different views are almost similar, and this is especially true for small baseline views. Obviously, wide baseline views might significantly differ in their contents, where some objects might be completely absent in some views. However, the current approaches of representing multiview data, even if they exploit inter-view correlation, require large bandwidth for transmission. This bandwidth is almost linear with the number of transmitted views. Thus, in this paper we propose to address this problem by representing lateral views solely using their edges, while dropping their texture content. The texture content is synthesized, at the receiver side, by a convolutional neural network (CNN) exploiting the edges and the information in the central view. The edges of the lateral views represent the location of the “objects” in their corresponding views, whereas, we assume that their texture in other views does not change significantly, consequently there is no need to represent them in the lateral views. In this work, in addition to the proposed paradigm of representing multiview data, we also propose a training framework for the CNN network. Experimental results demonstrate the effectiveness of the proposed framework and demonstrate that the network is able to synthesis accurate and reliable lateral views starting from their edges.