Fine-grained image analysis of ultra-deep learning： projects, reviews and tutorials are all caught.
Selected from arXiv
Authors: Xiu-Shen Wei, Jianxin Wu, Quan Cui,
Machine heart compilation
Participation: Li Shimeng, zhangqian.
In this paper, researchers from Defiance Technology, Nanjing University and Waseda University have summarized the fine-grained image analysis based on deep learning, and discussed it from three aspects: fine-grained image recognition, retrieval and generation. In addition, they also discussed the future development direction of this field.
Computer vision (CV) is a process of understanding and analyzing images with machines, and it is an important branch of artificial intelligence. Fine-grained image analysis, FGIA) is a long-standing basic problem in various research fields of CV, and it is ubiquitous in various practical applications (such as bird species, car models, commodity identification, etc.). Small inter-class changes and large intra-class changes caused by fine-grained characteristics make this problem extremely challenging. Due to the vigorous development of deep learning, FGIA with deep learning has made remarkable progress in recent years.
This paper systematically summarizes FGIA technology based on deep learning. Specifically, this paper divides the research on FGIA technology into three categories: fine-grained image recognition, fine-grained image retrieval and fine-grained image generation. This paper also discusses other important issues of FGIA, such as publicly available benchmark data sets and their specific applications in related fields. At the end of this paper, several directions that need to be further discussed in the future and problems to be solved are emphasized.
Paper: Deep Learning for Fine-Grained Image Analysis: A Survey Paper Link: https://arxiv.org/pdf/1907.03069.pdf
Review the structure.
In another important AI conference held by Pacific Rim countries, the authors (Wei Xiushen and Wu Jianxin) organized a specific course for fine-grained image analysis. This tutorial provides some additional details about fine-grained image analysis, so I recommend this tutorial to readers who want to know more about it.
Course outline.
Tutorial address: http://www.weixiushen.com/tutorial/PRICAI18/FGIA.html
In addition, the author has also opened a home page of fine-grained image analysis, which contains representative papers, codes, data sets and so on.
Project catalogue.
Project address: http://www.weixiushen.com/project/awesome _ fgia/awesome _ fgia.html.
Background: Main problems and challenges in FGIA.
The difference between FGIA and general image analysis is that in general image analysis, the target objects belong to coarse-grained meta-categories (such as birds, oranges and dogs), so they look very different. However, in FGIA, objects are all subclasses of a meta-category, and their fine-grained characteristics make them look very similar. Let’s take image recognition as an example. As shown in Figure 1.
Figure 1: Fine-grained image analysis (right) and general image analysis (left)
In addition, the fine-grained characteristics will also lead to the problems of small inter-class variation caused by the high similarity of subcategories and large intra-class variation caused by different postures, sizes and angles, as shown in Figure 3.
Figure 3: Key Challenges of Fine-grained Image Analysis
Benchmark data set
Table 1: Summary of mainstream fine-grained image data sets
Table 1 lists the image data sets commonly used in fine-grained problems, and specially marks their meta-categories, the number of fine-grained images, the number of fine-grained categories and different kinds of supervision (i.e. bounding box, part annotation, hierarchical label, attribute label and text visual description, etc.), as shown in Figure 5.
Figure 5: Sample image with supervision information of CUB200-2011
Fine-grained image recognition
These fine-grained recognition methods can be summarized into three paradigms: (1) Fine-grained recognition with location classification subnet; (2) Fine-grained recognition with end-to-end feature coding; (3) Fine-grained recognition with external information.
Among them, the first paradigm and the second paradigm are limited only by the supervision related to fine-grained images (such as image labels, bounding boxes and some comments). In addition, due to the challenge of fine granularity, the automatic identification system can not achieve good performance. Therefore, researchers gradually try to integrate external but easily available information (such as web page data, text description, etc.) into the fine-grained recognition problem to further improve the accuracy, which corresponds to the third paradigm of fine-grained recognition. The commonly used evaluation index in fine-grained recognition is the average classification accuracy of all subordinate categories of data set.
4.1 Fine-grained identification with location classification subnetwork
In order to alleviate the problem of great changes within the class, fine-grained communities focus on capturing the discriminating semantic parts of fine-grained objects, and then establish intermediate representations related to these semantic parts for final classification. Specifically, in order to locate these key parts, the researchers designed a positioning sub-network. Then connect a classification subnet for identification. The framework formed by the cooperation of these two sub-networks is the first paradigm, that is, fine-grained recognition with positioning and classification sub-networks.
With positioning information (such as part bounding box or segmentation mask), a more discriminating intermediate (part) representation can be obtained. In addition, it further improves the learning ability of the classification sub-network, which can significantly enhance the accuracy of the final recognition.
The early work belonging to this paradigm relies on additional dense part annotations (also known as key point positioning) to locate semantic key parts (such as head and trunk) of the target. Some of them have learned part-based detectors [Zhang et al., 2014; Lin et al., 2015a], and some use segmentation method to locate parts. Then, these methods regard the features of multiple parts as the representation of the whole image, and feed them into the next classification sub-network for final recognition. Therefore, these methods are also called location-based recognition methods.
However, such intensive part annotation is labor-intensive, which limits the scalability and practicability of fine-grained applications in the real world. Recently, there is a trend that only image tags are needed [Jade Berg et al., 2015; Fu et al.，2017； Zheng et al.，2017； Sun et al., 2018] The technology that can accurately locate these parts has emerged. Their common idea is to find the corresponding parts first, and then compare their appearances. Specifically, we hope to capture the semantic parts (such as head and trunk) shared in fine-grained categories, and also hope to find the slight differences between these parts. Advanced technologies such as attention mechanism [Yang et al., 2018] and multi-stage strategy [He and Peng, 2017b] can carry out complex joint training for integrated positioning and classification sub-networks.
4.2 Fine-grained recognition with end-to-end feature coding
Different from the first paradigm, the second paradigm is end-to-end feature coding, which is realized by developing a powerful depth model for fine-grained recognition to directly learn more discriminating representations. The most representative of these methods is bilinear CNN (Lin et al., 2015b]), which uses the outer product of features from two deep CNN pools to represent images, thus encoding convolution-activated higher-order statistics to enhance intermediate learning ability. Because of its high model capacity, bilinear CNN has achieved excellent performance in fine-grained recognition. However, the dimension of bilinear feature is extremely high, so it cannot be applied in the real world, especially in large-scale applications.
Recently, there have been some attempts to solve this problem, such as [Gao et al., 2016; Kong and Fowlkes, 2017; Cui et al., 2017], [Pham and Pagh, 2013; Charikar et al., 2002] Try to aggregate low-dimensional embedding with tensor sketching, which can approximate bilinear features and maintain a considerable or higher accuracy. Other work, such as [Dubey et al., 2018], specially designed a specific loss function for fine-grained, which can drive the whole depth model to learn a discriminating fine-grained representation.
4.3 Fine-grained identification with external information
As mentioned above, in addition to the traditional recognition paradigm, another paradigm is to use external information (such as network data, multimodal data or human-computer interaction) to further help fine-grained recognition. See the paper for details.
Fine-grained image retrieval
Besides image recognition, fine-grained retrieval is another important aspect of FGIA, and it is also a hot research topic at present. In fine-grained retrieval, the commonly used evaluation index is mean average precision (mAP). In fine-grained image retrieval, given the database image of the same subclass (such as birds or cars) and the image to be queried, it can return the image belonging to the same category as the queried image without relying on any other supervision signals, as shown in Figure 7.
Figure 7: Fine-grained retrieval diagram.
General image retrieval is based on the similarity of image content (such as texture, color and shape) to retrieve very similar images, while fine-grained retrieval focuses on retrieving images belonging to the same category (such as creatures of the same species or a car type). At the same time, the difference of targets in fine-grained images is very small, but there are differences in posture, size and angle.
[Wei et al., 2017] Try to use deep learning for fine-grained image retrieval for the first time. This model uses the pre-trained CNN model to select meaningful depth descriptors by locating the main target in fine-grained images without supervision, and further reveals that the performance of retrieval tasks can be significantly improved only by removing the depth descriptors of background or noise. In order to break the limitation of unsupervised fine-grained retrieval by pre-training model, some experiments [Zheng et al., 2018; Zheng et al., 2019] tends to develop a brand-new loss function under the supervised index learning paradigm. At the same time, they also designed additional specific sub-modules for fine-grained targets, for example, [Zheng et al., 2018] proposed a weak supervised positioning module inspired by [Wei et al., 2017].
Fine-grained image generation
In addition to supervised learning tasks, image generation is also a representative topic in unsupervised learning. It uses a depth generation model like GAN[Goodfellow et al., 2014] to learn to synthesize realistic images that look real. With the quality of generated images getting higher and higher, a more challenging task-fine-grained image generation appears. As the name implies, fine-grained generation can synthesize images in fine-grained categories, such as the face of a specific person or objects in subordinate categories.
The first work in this field is CVAE-GAN proposed by [Bao et al., 2017], which combines the variational self-encoder with the generated countermeasure network in the conditional generation process to solve this problem. Specifically, CVAE-GAN models an image as a combination of labels and hidden attributes in a probabilistic model. By changing the fine-grained category of the feed generation model, it can generate images of a specific category. Recently, generating images from text descriptions [Xu et al., 2018b] has become popular because of its diversity and practicality, such as art generation and computer-aided design. After the attention-equipped spanning network is implemented, the model can synthesize the fine-grained details of the fine area according to the relevant details in the text description.
Specific applications related to fine-grained image analysis
In the real world, fine-grained image analysis technology based on deep learning has been applied in different fields, and has shown good performance, such as searching clothes or shoes in recommendation system [Song et al., 2017], identifying fashion images in e-commerce platform [Wei et al., 2016] and identifying products in intelligent retail platform [Wei et al., 2019a]. These applications are highly related to fine-grained retrieval and recognition of FGIA.
In addition, if we move down the granularity range, at the extreme, we can also regard face recognition as an example of fine-grained recognition, in which the granularity falls below the identity granularity level. In addition, the re-recognition of people or motor vehicles is also a fine-grained related task. The goal of this task is to determine whether two images belong to the same specific person or motor vehicle. Obviously, the granularity level of the re-identification task is also below the identity granularity.
In practical application, these tasks all follow the idea of FGIA to solve specific tasks in related fields. The idea of FGIA includes capturing the parts of the target that are highly distinguishable (faces, people and motor vehicles) [Suh et al., 2018], discovering coarse-grained structural information [Wei et al., 2018b] and developing attribute-based models [Liu et al., 2016].
Future direction
In this part, the researcher clearly points out the unsolved problems in FGIA related fields and some future research trends.
Automatic fine-grained model
The latest methods of AutoML and NAS have achieved comparable or even better results in various applications of computer vision. Therefore, it is hoped that AutoML or NAS technology can be used to develop automatic fine-grained models, and it is expected to find a better and more suitable depth model, which can also promote the progress of AutoML and NAS research in reverse.
Fine-grained few-shot learning
Our best deep learning fine-grained system needs hundreds of labeled samples. To make matters worse, the supervision of fine-grained images is not only time-consuming but also expensive, because fine-grained targets are accurately marked by experts in this field. Therefore, it is urgent to develop a fine-grained few-shot (FGFS) based on small samples in practical applications [Wei et al., 2019b]. The FGFS task requires the learning system to build a classifier for new fine-grained categories according to a small number of samples (only one or less than five) in the way of meta-learning. Robust FGFS method can greatly enhance the usability and scalability of fine-grained recognition.
Fine-grained hash
In practical applications such as fine-grained image retrieval, there will naturally be such a problem-in the case of very large reference data, the cost of finding accurate nearest neighbors is very high. Hash [Wang et al., 2018; Li et al., 2016] is one of the most popular and effective technologies in approximate nearest neighbor search, which has the potential to process a large number of fine-grained data. Therefore, fine-grained hashing is a direction worthy of further exploration in FGIA.
Fine-grained analysis in a more realistic environment
There are many novel topics in fine-grained image analysis-fine-grained analysis with domain adaptation, fine-grained analysis with knowledge transfer, fine-grained analysis with long tail distribution and running fine-grained analysis on embedded devices with limited resources. These more advanced and practical FGIA are worthy of a lot of research work.
Reporting/feedback