Multitype Learning via Multimodal Data Embedding

Abstract

This paper creates a multimodal retrieval system for image and text data in a multi-type learning approach that enables text-to-image, image-to-text, text-to-text, and image-to-image retrievals. As a practical solution, a mobile application is developed in which the users can upload their images to search a description sentence for the images. The user system is created on the application, which is done with React Native, and crucial features like e-mail authentication and reset password options are added to the application. An essential database system is designed with PostgreSQL to store user information and search for the user. The multimodal embedding study is worked, and the model that recognizes multitype retrievals is formed. The image-to-text retrieval model, which is our application's idea, is applied to the mobile application. © 2021 IEEE.