Learning Generative ConvNet with Continuous Latent Factors

• Model: a non-linear generalization of factor analysis where the mapping is parametrized by CNN
• Optimized image synthesis training on large-scale images by batch normalization
• Used new Back-Propagation inferenced by gradient descent / Langevin dynamics

Paper Abstract

This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the latent factors to the observed vector is parametrized by a convolutional neural network. The alternating back-propagation algorithm iterates between the following two steps: (1) Inferential back-propagation, which infers the latent factors by Langevin dynamics or gradient descent. (2) Learning back-propagation, which updates the parameters given the inferred latent factors by gradient descent.

• The project page : Link
• The paper online : Link *My name is listed in the Acknowledgement
• The poster : Link
• The presentation : Link

Generative Hierarchical Structure Learning of Sparse FRAME Models

• Model: Sparse FRAME, a multi-layer probability distribution model captured the part deformation
• Designed experiments for Sparse FRAME model on detection and clustering
• Compared Sparse FRAME model with DPM, And-or Graph on point, part, object level detection

Paper Abstract

This paper proposes a framework for generative learning of hierarchical structure of visual objects, based on training hierarchical random field models. The resulting model, which we call structured sparse FRAME model, is a straightforward variation on decomposing the original sparse FRAME model into multiple parts that are allowed to shift their locations, orientations and scales, so that the resulting model becomes a reconfigurable template.

• The project page : Link
• The paper online : Link
• The poster : Link

SAS: Self-Augmented Strategy for Language Model Pre-training

• A novel generating strategy for contextualized data argmentation
• Researched on unsupervised pretrain model for Transformers in NLP
• Outperform the SOTA result ELECTRA with 30% less computing cost

Paper Abstract

The core of a self-supervised learning method for pre-training language models includes the design of appropriate data augmentation and corresponding pre-training task(s). Most data augmentations in language model pre-training are context-independent. The seminal contextualized augmentation recently proposed by the ELECTRA requires a separate generator, which leads to extra computation cost as well as the challenge in adjusting the capability of its generator relative to that of the other model component(s). We propose a self-augmented strategy (SAS) that uses a single forward pass through the model to augment the input data for model training in the next epoch. Essentially our strategy eliminates a separate generator network and uses only one network to generate the data augmentation and undertake two pre-training tasks (the MLM task and the RTD task) jointly, which naturally avoids the challenge in adjusting the generator's capability as well as reduces the computation cost. Additionally, our SAS is a general strategy such that it can seamlessly incorporate many new techniques emerging recently or in the future, such as the disentangled attention mechanism recently proposed by the DeBERTa model. Our experiments show that our SAS is able to outperform the ELECTRA and other state-of-the-art models in the GLUE tasks with the same or less computation cost.

• The paper online : Arxiv

Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets

• Model : An energy-based model applied on 3D pointcloud generation
• Compared on PointFlow, PointGlow, Autoencoder, etc.

Paper Abstract

We propose a generative model of point clouds in the forms of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network. The energy function learns a coordinate encoding of each point and then aggregates all individual point features into an energy for the whole point cloud. We show that our model can be derived from the discriminative PointNet. The model is trained by MCMC-based maximum likelihood learning (as well as its variants), without the help of any assisting networks like those in GANs and VAEs. Our model does not rely on hand-crafting distance metric for point clouds in generation. It synthesizes point clouds that match to the observed examples. The learned point cloud representation can be useful for point cloud classification. Experiments demonstrate the advantages of the proposed model. Furthermore, we can learn a short-run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpretation.

• The paper online : Arxiv

Continuous Inverse Optimal Control by Energy-based Model

• Model: A sample-based inverse reinforcement learning model driven by Energy-based Model and Langevin Dynamic
• Energy function is formed by neural network enhanced human-crafted cost function
• Langevin Dynamic is used to generate trajectories by sampling in energy-based distribution

Paper Abstract

Autonomous driving is a challenging multiagent domain which requires optimizing complex, mixed cooperative-competitive interactions. Learning to predict contingent distributions over other vehicles' trajectories simplifies the problem, allowing approximate solutions by trajectory optimization with dynamic constraints. We take a model-based approach to prediction, in order to make use of structured prior knowledge of vehicle kinematics, and the assumption that other drivers plan trajectories to minimize an unknown cost function. We introduce a novel inverse optimal control (IOC) algorithm to learn other vehicles' cost functions in an energy-based generative model. Langevin Sampling, a Monte Carlo based sampling algorithm, is used to directly sample the control sequence. Our algorithm provides greater flexibility than standard IOC methods, and can learn higher-level, non-Markovian cost functions defined over entire trajectories. We extend weighted feature-based cost functions with neural networks to obtain NN-augmented cost functions, which combine the advantages of both model-based and model-free learning. Results show that model-based IOC can achieve state-of-the-art vehicle trajectory prediction accuracy, and naturally take scene information into account.

• The paper online : Link

Large-scale Image Retrieval Competition

• Model: A model with saliency detection, image classification and image retrieval
• Implemented Saliency Detection combining Dense and Sparse Reconstruction by Bayesian Integration
• Classified large-scale images by SVM and Convolution Neural Network

This is a competition hold by Alibaba. The goal is to output the picture with the most similarity by the given picture. The database is a million web pictures. There are three part for our model. They are saliency detection, CNN classification and text matching. I am in charge of saliency detection and classification. Our team ranked in the top 16 in the competition(Over 2000 teams).

Interactive Image Search for Clothing Recommendation

• Model: Hybrid Topics Model, An LDA based model integrates both visual and text information
• Used multi-trained Fast-RCNN to localize regions
• Extracted 3 types of visual descriptors: HOG, LBP, Color
• Implemented Hybrid Topics Model and introduced a demand-adaptive retrieval strategy

Paper Abstract

This paper proposes a novel approach to meet users' multi-dimensional requirements in clothing image retrieval.We propose the Hybrid Topic (HT) model to learn the intricate semantic representation of the descriptors above. The model provides an effective multi-dimensional representation of clothes and is able to perform automatic image annotation by probabilistic reasoning from image search. Furthermore, we develop a demand-adaptive retrieval strategy which refines users' specific requirements and removes users' unwanted features. Our experiments show that the HT method significantly outperforms the deep neural network methods.

• The system page : Magic Wardrobe
• The paper online : Link
• The poster : Link

Joint Face Detection and Alignment via Cascaded Compositional Learning

• Model: Sparse FRAME, a multi-layer probability distribution model captured the part deformation
• Jointed cascade face detection and alignment by advanced boosting algorithm
• Considered multi-domain to overcome unconstrained face data
• Trained multi domain on same random forest with both detection and alignment in parallel

This work is based on "Joint cascade face detection and alignment" and "Unconstrained Face Alignment via Cascaded Compositional Learning". We aim to provide domain partition on the Joint cascade face detection and alignment method.