We propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31 × 31, in contrast to commonly used 3 × 3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g ., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive ﬁelds and higher shape bias rather than texture bias. 2022: Xiaohan Ding, X. Zhang, Yi Zhou, Jungong Han, Guiguang Ding, Jian Sunhttps://arxiv.org/pdf/2203.06717v4.pdf

We propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31 × 31, in contrast to commonly used 3 × 3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g ., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive ﬁelds and higher shape bias rather than texture bias. 2022: Xiaohan Ding, X. Zhang, Yi Zhou, Jungong Han, Guiguang Ding, Jian Sun <a href="https://arxiv.org/pdf/2203.06717v4.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2203.06717v4.pdf</a>

Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local but large attention mechanism, showing appealing performance and efﬁciency. We propose Sparse Large Kernel Network ( SLaK ), a pure CNN architecture equipped with 51 × 51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classiﬁcation as well as typical downstream tasks.2022: S. Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Q. Xiao, Boqian Wu, Mykola Pechenizkiy, D. Mocanu, Zhangyang Wanghttps://arxiv.org/pdf/2207.03620v1.pdf

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local but large attention mechanism, showing appealing performance and efﬁciency. We propose Sparse Large Kernel Network ( SLaK ), a pure CNN architecture equipped with 51 × 51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classiﬁcation as well as typical downstream tasks. 2022: S. Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Q. Xiao, Boqian Wu, Mykola Pechenizkiy, D. Mocanu, Zhangyang Wang <a href="https://arxiv.org/pdf/2207.03620v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2207.03620v1.pdf</a>

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

We propose a new setup for evaluating existing knowledge intensive tasks in which we generalize the background corpus to a universal web snapshot. We investigate a slate of NLP tasks which rely on knowledge - either factual or common sense, and ask systems to use a subset of CCNet—the S PHERE corpus—as a knowledge source. In contrast to Wikipedia, otherwise a common background corpus in KI-NLP, S PHERE is orders of magnitude larger and better reﬂects the full diversity of knowledge on the web. 2021: Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Ouguz, Edouard Grave, Wen-tau Yih, Sebastian Riedelhttps://arxiv.org/pdf/2112.09924v2.pdf

We propose a new setup for evaluating existing knowledge intensive tasks in which we generalize the background corpus to a universal web snapshot. We investigate a slate of NLP tasks which rely on knowledge - either factual or common sense, and ask systems to use a subset of CCNet—the S PHERE corpus—as a knowledge source. In contrast to Wikipedia, otherwise a common background corpus in KI-NLP, S PHERE is orders of magnitude larger and better reﬂects the full diversity of knowledge on the web. 2021: Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Ouguz, Edouard Grave, Wen-tau Yih, Sebastian Riedel <a href="https://arxiv.org/pdf/2112.09924v2.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2112.09924v2.pdf</a>

The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus

Deploying convolutional neural networks (CNNs) on embedded devices is difficult due to the limited memory and computation resources. The redundancy in feature maps is an important characteristic of those successful CNNs, but has rarely been investigated in neural architecture design. This paper proposes a novel Ghost module to generate more feature maps from cheap operations. Based on a set of intrinsic feature maps, we apply a series of linear transformations with cheap cost to generate many ghost feature maps that could fully reveal information underlying intrinsic features. 2019: Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, Chang Xu https://arxiv.org/pdf/1911.11907v2.pdf

Deploying convolutional neural networks (CNNs) on embedded devices is difficult due to the limited memory and computation resources. The redundancy in feature maps is an important characteristic of those successful CNNs, but has rarely been investigated in neural architecture design. This paper proposes a novel Ghost module to generate more feature maps from cheap operations. Based on a set of intrinsic feature maps, we apply a series of linear transformations with cheap cost to generate many ghost feature maps that could fully reveal information underlying intrinsic features. 2019: Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, Chang Xu <a href="https://arxiv.org/pdf/1911.11907v2.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/1911.11907v2.pdf</a>

GhostNet: More Features From Cheap Operations

In this work, we propose to address this problem by performing object-centric alignment of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image-level supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. 2022: Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, F. Khanhttps://arxiv.org/pdf/2207.03482v1.pdf

In this work, we propose to address this problem by performing object-centric alignment of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image-level supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. 2022: Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, F. Khan <a href="https://arxiv.org/pdf/2207.03482v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2207.03482v1.pdf</a>

Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT).2021: Kai Han, An Xiao, E. Wu, Jianyuan Guo, Chunjing Xu, Yunhe WangRanked #6 on Fine-Grained Image Classification on Oxford-IIIT Petshttps://arxiv.org/pdf/2103.00112v3.pdf

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). 2021: Kai Han, An Xiao, E. Wu, Jianyuan Guo, Chunjing Xu, Yunhe Wang Ranked #6 on Fine-Grained Image Classification on Oxford-IIIT Pets <a href="https://arxiv.org/pdf/2103.00112v3.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2103.00112v3.pdf</a>

Transformer in Transformer

Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graph level feature for visual tasks. 2022: Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, E. Wuhttps://arxiv.org/pdf/2206.00272v1.pdf

Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graph level feature for visual tasks. 2022: Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, E. Wu <a href="https://arxiv.org/pdf/2206.00272v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2206.00272v1.pdf</a>

Vision GNN: An Image is Worth Graph of Nodes

Remotely sensed geospatial data are critical for applications including precision agriculture, urban planning, disaster monitoring and response, and climate change research, among others. Deep learning methods are particularly promising for modeling many remote sensing tasks given the success of deep neural networks in similar computer vision tasks and the sheer volume of remotely sensed imagery available. 2021: A. Stewart, Caleb Robinson, Isaac A. Corley, Anthony Ortiz, J. Ferres, Arindam Banerjeehttps://arxiv.org/pdf/2111.08872v3.pdf

Remotely sensed geospatial data are critical for applications including precision agriculture, urban planning, disaster monitoring and response, and climate change research, among others. Deep learning methods are particularly promising for modeling many remote sensing tasks given the success of deep neural networks in similar computer vision tasks and the sheer volume of remotely sensed imagery available. 2021: A. Stewart, Caleb Robinson, Isaac A. Corley, Anthony Ortiz, J. Ferres, Arindam Banerjee <a href="https://arxiv.org/pdf/2111.08872v3.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2111.08872v3.pdf</a>

TorchGeo: deep learning with geospatial data

We introduce OmniXAI 1 (short for Omni eXplainable AI), an open-source Python library of eXplainable AI (XAI), which oﬀers omni-way explainable AI capabilities and various interpretable machine learning techniques to address the pain points of understanding and interpreting the decisions made by machine learning (ML) in practice. OmniXAI aims to be a one-stop comprehensive library that makes explainable AI easy for data scientists, ML researchers and practitioners who need explanation for various types of data, models and explanation methods at diﬀerent stages of ML process.2022: Wenzhuo Yang, Hung Le, S. Savarese, S. Hoihttps://arxiv.org/pdf/2206.01612v3.pdf

We introduce OmniXAI 1 (short for Omni eXplainable AI), an open-source Python library of eXplainable AI (XAI), which oﬀers omni-way explainable AI capabilities and various interpretable machine learning techniques to address the pain points of understanding and interpreting the decisions made by machine learning (ML) in practice. OmniXAI aims to be a one-stop comprehensive library that makes explainable AI easy for data scientists, ML researchers and practitioners who need explanation for various types of data, models and explanation methods at diﬀerent stages of ML process. 2022: Wenzhuo Yang, Hung Le, S. Savarese, S. Hoi <a href="https://arxiv.org/pdf/2206.01612v3.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2206.01612v3.pdf</a>

OmniXAI: A Library for Explainable AI

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic.2018: Mikolaj Binkowski, Danica J. Sutherland, M. Arbel, A. GrettonGradient, MikuMikuDancehttps://arxiv.org/pdf/1801.01401v5.pdf

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. 2018: Mikolaj Binkowski, Danica J. Sutherland, M. Arbel, A. Gretton Gradient, MikuMikuDance <a href="https://arxiv.org/pdf/1801.01401v5.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/1801.01401v5.pdf</a>

Demystifying MMD GANs

Keeping you up to date with the latest trends and best performing architectures in this fast evolving field in computer science.Selecting papers by comparative results, citations and influence we educate you on the latest research.Consider supporting us on Patreon.com/PapersRead for feedback and ideas.

We introduce Codex, a GPT language model ﬁne-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Fur-thermore, we ﬁnd that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difﬁcult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difﬁculty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.2021: Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, F. Such, D. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, I. Babuschkin, S. Balaji, Shantanu Jain, A. Carr, J. Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, M. Knight, Miles Brundage, Mira Murati, Katie Mayer, P. Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zarembahttps://arxiv.org/pdf/2107.03374v2.pdf

We introduce Codex, a GPT language model ﬁne-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Fur-thermore, we ﬁnd that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difﬁcult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difﬁculty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics. 2021: Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, F. Such, D. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, I. Babuschkin, S. Balaji, Shantanu Jain, A. Carr, J. Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, M. Knight, Miles Brundage, Mira Murati, Katie Mayer, P. Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba <a href="https://arxiv.org/pdf/2107.03374v2.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2107.03374v2.pdf</a>

Evaluating Large Language Models Trained on Code

We present a self-supervised method to learn dynamic 3D deformations of garments worn by parametric human bodies. State-of-the-art data-driven approaches to model 3D garment deformations are trained using supervised strategies that require large datasets, usually obtained by expensive physics-based simulation methods or professional multi-camera capture setups. 2022: I. Santesteban, M. Otaduy, D. Casashttps://arxiv.org/pdf/2204.02219v1.pdf

We present a self-supervised method to learn dynamic 3D deformations of garments worn by parametric human bodies. State-of-the-art data-driven approaches to model 3D garment deformations are trained using supervised strategies that require large datasets, usually obtained by expensive physics-based simulation methods or professional multi-camera capture setups. 2022: I. Santesteban, M. Otaduy, D. Casas <a href="https://arxiv.org/pdf/2204.02219v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2204.02219v1.pdf</a>

SNUG: Self-Supervised Neural Dynamic Garments

In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance. We open-source the training and evaluation code, as well as the model weights. A 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission.2022: Sid Black, Stella Rose Biderman, Eric Hallahan, Quentin G. Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, M. Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, J. Tow, Ben Wang, Samuel WeinbachRanked #7 on Multi-task Language Understanding on MMLUhttps://arxiv.org/pdf/2204.06745v1.pdf

In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance. We open-source the training and evaluation code, as well as the model weights. A 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. 2022: Sid Black, Stella Rose Biderman, Eric Hallahan, Quentin G. Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, M. Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, J. Tow, Ben Wang, Samuel Weinbach Ranked #7 on Multi-task Language Understanding on MMLU <a href="https://arxiv.org/pdf/2204.06745v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2204.06745v1.pdf</a>

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

In this paper we introduce BEVFusion, an efﬁcient and generic multi-task multi-sensor fusion framework. It uniﬁes multimodal features in the shared bird’s-eye view (BEV) representation space, which nicely preserves both geometric and semantic information.2022: Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, D. Rus, Song HanRanked #1 on 3D Object Detection on nuSceneshttps://arxiv.org/pdf/2205.13542v1.pdf

In this paper we introduce BEVFusion, an efﬁcient and generic multi-task multi-sensor fusion framework. It uniﬁes multimodal features in the shared bird’s-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. 2022: Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, D. Rus, Song Han Ranked #1 on 3D Object Detection on nuScenes <a href="https://arxiv.org/pdf/2205.13542v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2205.13542v1.pdf</a>

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation

Remote

Annotated datasets have become one of the most crucial preconditions for the development and evaluation of machine learning-based methods designed for the automated interpretation of remote sensing data. In this paper, we review the historic development of such datasets, discuss their features based on a few selected examples, and address open issues for future developments.2021: M. Schmitt, S. A. Ahmadi, R. Hänschhttps://arxiv.org/pdf/2105.11726v2.pdf

Annotated datasets have become one of the most crucial preconditions for the development and evaluation of machine learning-based methods designed for the automated interpretation of remote sensing data. In this paper, we review the historic development of such datasets, discuss their features based on a few selected examples, and address open issues for future developments. 2021: M. Schmitt, S. A. Ahmadi, R. Hänsch <a href="https://arxiv.org/pdf/2105.11726v2.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2105.11726v2.pdf</a>

There is No Data Like More Data - Current Status of Machine Learning Datasets in Remote Sensing

We suggest “Hopular”, a novel Deep Learning architecture for medium and small sized datasets, where each layer is equipped with continuous modern Hopfield networks. The modern Hopfield networks use stored data to identify feature-feature, feature-target, and sample-sample dependencies. Hopular’s novelty is that every layer can directly access the original input as well as the whole training set via stored data in the Hopfield networks. Hopular outperforms XGBoost, CatBoost, LightGBM and a state-of-the art Deep Learning method designed for tabular data. Thus, Hopular is a strong alternative to these methods on tabular data.2022: Bernhard Schafl, Lukas Gruber, Angela Bitto-Nemling, S. HochreiterRanked #1 on General Classification on Shrutimehttps://arxiv.org/pdf/2206.00664v1.pdf

We suggest “Hopular”, a novel Deep Learning architecture for medium and small sized datasets, where each layer is equipped with continuous modern Hopfield networks. The modern Hopfield networks use stored data to identify feature-feature, feature-target, and sample-sample dependencies. Hopular’s novelty is that every layer can directly access the original input as well as the whole training set via stored data in the Hopfield networks. Hopular outperforms XGBoost, CatBoost, LightGBM and a state-of-the art Deep Learning method designed for tabular data. Thus, Hopular is a strong alternative to these methods on tabular data. 2022: Bernhard Schafl, Lukas Gruber, Angela Bitto-Nemling, S. Hochreiter Ranked #1 on General Classification on Shrutime <a href="https://arxiv.org/pdf/2206.00664v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2206.00664v1.pdf</a>

Hopular: Modern Hopfield Networks for Tabular Data

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality.2022: Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, Fang Wenhttps://arxiv.org/pdf/2205.12952v1.pdf

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. 2022: Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, Fang Wen <a href="https://arxiv.org/pdf/2205.12952v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2205.12952v1.pdf</a>

Pretraining is All You Need for Image-to-Image Translation

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-ﬁdelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample ﬁdelity and image-text alignment much more than increasing the size of the image diffusion model.2022: Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. S. Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, D. Fleet, Mohammad NorouziRanked #1 on Text-to-Image Generation on COCO (using extra training data)https://arxiv.org/pdf/2205.11487v1.pdf

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-ﬁdelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample ﬁdelity and image-text alignment much more than increasing the size of the image diffusion model. 2022: Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. S. Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, D. Fleet, Mohammad Norouzi Ranked #1 on Text-to-Image Generation on COCO (using extra training data) <a href="https://arxiv.org/pdf/2205.11487v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2205.11487v1.pdf</a>

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

We propose a recipe on how to build a general, powerful, scalable (GPS) graph Transformer with linear complexity and state-of-the-art results on a diverse set of benchmarks. Graph Transformers (GTs) have gained popularity in the field of graph representation learning with a variety of recent publications but they lack a common foundation about what constitutes a good positional or structural encoding, and what differentiates them. 2022: Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, A. Luu, Guy Wolf, D.Beaini https://arxiv.org/pdf/2205.12454v1.pdf

We propose a recipe on how to build a general, powerful, scalable (GPS) graph Transformer with linear complexity and state-of-the-art results on a diverse set of benchmarks. Graph Transformers (GTs) have gained popularity in the field of graph representation learning with a variety of recent publications but they lack a common foundation about what constitutes a good positional or structural encoding, and what differentiates them. 2022: Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, A. Luu, Guy Wolf, D.Beaini <a href="https://arxiv.org/pdf/2205.12454v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2205.12454v1.pdf</a>

Recipe for a General, Powerful, Scalable Graph Transformer

Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. 2021: Minguk Kang, Woohyeon Shim, Minsu Cho, Jaesik ParkRanked #1 on Conditional Image Generation on CIFAR-10https://arxiv.org/pdf/2111.01118v1.pdf

Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. 2021: Minguk Kang, Woohyeon Shim, Minsu Cho, Jaesik Park Ranked #1 on Conditional Image Generation on CIFAR-10 <a href="https://arxiv.org/pdf/2111.01118v1.pdf" rel="nofollow" target="_blank">https://arxiv.org/pdf/2111.01118v1.pdf</a>

Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training

Keeping you up to date with the latest trends and best performing architectures in this fast evolving field in computer science. Selecting papers by comparative results, citations and influence we educate you on the latest research. Consider supporting us on <a href="http://Patreon.com/PapersRead" rel="nofollow" target="_blank">Patreon.com/PapersRead</a> for feedback and ideas.

Papers Read on AI

The best podcast episodes of the most interesting stories and things. From skyping someone while they're in space, to tracking down and meeting a telephone scammer in person, to the story of a Galapagos. These episodes will keep you and all your friends entertained and captivated for hours. Learn something new in every episode! Add episodes you're interested into your queue below!

Best Episodes: Interesting Stories & Things

The best episodes for anyone looking to learn more about topics such as happiness, fulfillment, depression and psychology/mental health in general. With interviews with industry experts, personal stories of individuals, this topic has lessons applicable to everyone. Add episodes you're interested into your queue below!