A Simplified Framework for Zero-shot Cross-Modal Sketch Data Retrieval

Abstract

We deal with the problem of zero-shot cross-modal imageretrieval involving color and sketch images through a noveldeep representation learning technique. The problem of asketch to image retrieval and vice-versa is of practical im-portance, and a trained model in this respect is expectedto generalize beyond the training classes, e.g., the zero-shot learning scenario. Nonetheless, considering the dras-tic distributions-gap between both the modalities, a fea-ture alignment is necessary to learn a shared feature spacewhere retrieval can efficiently be carried out. Additionally,it should also be guaranteed that the shared space is se-mantically meaningful to aid in the zero-shot retrieval task.The very few existing techniques for zero-shot sketch-RGBimage retrieval extend the deep generative models for learn-ing the embedding space; however, training a typical GANlike model for multi-modal image data may be non-trivialat times. To this end, we propose a multi-stream encoder-decoder model that simultaneously ensures improved map-ping between the RGB and sketch image spaces and highdiscrimination in the shared semantics-driven encoded fea-ture space. Further, it is guaranteed that the class topologyof the original semantic space is preserved in the encodedfeature space, which subsequently reduces the model biastowards the training classes. Experimental results obtainedon the benchmark Sketchy and TU-Berlin datasets estab-lish the efficacy of our model as we outperform the existingstate-of-the-art techniques by a considerable margin.