AI Post Transformers

This 2019 paper, "Language Models are Unsupervised Multitask Learners," introduces GPT-2, a large language model designed for zero-shot learning, meaning it can perform tasks without explicit, task-specific training. The research highlights the model's ability to learn various natural language processing (NLP) tasks, such as question answering, summarization, and translation, by bei...

This 2023 paper Gaussian Error Linear Units (GELUs), a novel activation function for neural networks that outperforms traditional activations like Rectified Linear Units (ReLUs) and Exponential Linear Units (ELUs) across various tasks. GELUs operate by weighting inputs by their value using the standard Gaussian cumulative distribution function, providing a probabilistic interpretation unlike the s...

This 2014 journal article introduces "Dropout", a novel technique designed to combat overfitting in deep neural networks, which are powerful but prone to memorizing training data. The core concept involves randomly deactivating a subset of neurons and their connections during the training phase, which prevents hidden units from overly relying on each other. This process effective...

This paper details "Constitutional AI," a novel method for training AI assistants to be harmless without extensive human-labeled data for harmful outputs. This approach involves a supervised learning phase, where an AI critiques and revises its own responses based on a set of pre-defined principles or a "constitution." Following this, a reinforcement learning (RL) phase uses AI...

The Chinchilla research by DeepMind investigates the optimal model size and training tokens for large language models, aiming to maximize performance within a fixed computational budget. They challenge prior beliefs by demonstrating that model size and training data should scale proportionally, not primarily focusing on larger models with constant data. Their new model, Chinchilla, with 70 billion...

This paper provides an overview of Chamfer Matching, a classical image registration method primarily used for segmented features. They explain its theoretical underpinnings, including its reliance on distance transforms, cost functions, and optimization algorithms. The texts highlight its applications, particularly in medical imaging for radiotherapy, where it aids in treatment verification, plann...

Review of the 2019 pun titled paper "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" by the folks at Facebook. This research introduces BART, a novel denoising autoencoder designed for pre-training sequence-to-sequence models, which proves effective for various natural language processing tasks, including generation, tran...

Review of the 2017 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", leveraging the transformer architecture, by Google. This paper introduces BERT (Bidirectional Encoder Representations from Transformers), a novel language representation model designed for pre-training deep bidirectional representations from unlabeled text. Unlike prior models tha...

This academic paper introduces Batch Normalization (BN), a novel technique designed to accelerate the training of Deep Neural Networks (DNNs) by addressing the issue of internal covariate shift. Internal covariate shift refers to the phenomenon where the distribution of inputs to each layer changes during training, slowing down the learning process and making it difficult to train models with cert...

Review of the seminal 2017 paper Attention is all you need. These paper introducea the Transformer architecture, a dominant model in natural language processing that relies entirely on multi-head attention instead of recurrent or convolutional networks. The first paper, "Attention Is All You Need," introduces the Transformer, showcasing its superior performance and training efficiency in...