AI Post Transformers

This paper outlines the advancements in Optical Character Recognition (OCR), particularly focusing on handwritten character and word recognition using Neural Networks. The authors, affiliated with AT&T Labs-Research, detail various machine learning techniques, including Gradient-Based Learning and Convolutional Neural Networks (CNNs) like LeNet-5, highlighting their effectiveness in handling h...

The first source introduces Layer Normalization (LN), a technique designed to accelerate and stabilize the training of deep neural networks, particularly recurrent neural networks, by normalizing summed inputs across a layer for each training case. It contrasts LN with Batch Normalization, highlighting LN's independence from mini-batch size and its consistent computation during both training...

This 2022 paper explores the significant negative impact of repeated data on the performance of large language models, even when such repetitions constitute a small fraction of the total training data. The authors observe a "double descent" phenomenon, where model performance initially improves, then degrades at a specific repetition frequency, and finally improves again with excessive r...

This 2023 paper, GPT-4 Technical Report from OpenAI introduces GPT-4, a multimodal AI model capable of processing both image and text inputs to produce text outputs, demonstrating human-level performance on various professional and academic benchmarks, such as the bar exam. The report highlights the predictable scaling of the model's performance and its improved factual accuracy and adherence...

This paper introduces Grouped-Query Attention (GQA), a novel approach designed to enhance the inference efficiency of large language models. It addresses the limitations of Multi-Query Attention (MQA), which, while fast, can compromise model quality, and Multi-Head Attention (MHA), which offers high quality but is slower due to memory bandwidth overhead. The authors propose uptraining existing MHA...

This 2020 paper outlines the development and evaluation of GPT-3, a large language model, exploring its performance across various natural language processing tasks under zero-shot, one-shot, and few-shot learning conditions, which involve providing minimal to no task-specific examples during inference. It details the model's architecture, training methodology, including its use of a massive...

This 2019 paper, "Language Models are Unsupervised Multitask Learners," introduces GPT-2, a large language model designed for zero-shot learning, meaning it can perform tasks without explicit, task-specific training. The research highlights the model's ability to learn various natural language processing (NLP) tasks, such as question answering, summarization, and translation, by bei...

This 2023 paper Gaussian Error Linear Units (GELUs), a novel activation function for neural networks that outperforms traditional activations like Rectified Linear Units (ReLUs) and Exponential Linear Units (ELUs) across various tasks. GELUs operate by weighting inputs by their value using the standard Gaussian cumulative distribution function, providing a probabilistic interpretation unlike the s...

This 2014 journal article introduces "Dropout", a novel technique designed to combat overfitting in deep neural networks, which are powerful but prone to memorizing training data. The core concept involves randomly deactivating a subset of neurons and their connections during the training phase, which prevents hidden units from overly relying on each other. This process effective...

This paper details "Constitutional AI," a novel method for training AI assistants to be harmless without extensive human-labeled data for harmful outputs. This approach involves a supervised learning phase, where an AI critiques and revises its own responses based on a set of pre-defined principles or a "constitution." Following this, a reinforcement learning (RL) phase uses AI...

The Chinchilla research by DeepMind investigates the optimal model size and training tokens for large language models, aiming to maximize performance within a fixed computational budget. They challenge prior beliefs by demonstrating that model size and training data should scale proportionally, not primarily focusing on larger models with constant data. Their new model, Chinchilla, with 70 billion...

This paper provides an overview of Chamfer Matching, a classical image registration method primarily used for segmented features. They explain its theoretical underpinnings, including its reliance on distance transforms, cost functions, and optimization algorithms. The texts highlight its applications, particularly in medical imaging for radiotherapy, where it aids in treatment verification, plann...

Review of the 2019 pun titled paper "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" by the folks at Facebook. This research introduces BART, a novel denoising autoencoder designed for pre-training sequence-to-sequence models, which proves effective for various natural language processing tasks, including generation, tran...

Review of the 2017 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", leveraging the transformer architecture, by Google. This paper introduces BERT (Bidirectional Encoder Representations from Transformers), a novel language representation model designed for pre-training deep bidirectional representations from unlabeled text. Unlike prior models tha...

This academic paper introduces Batch Normalization (BN), a novel technique designed to accelerate the training of Deep Neural Networks (DNNs) by addressing the issue of internal covariate shift. Internal covariate shift refers to the phenomenon where the distribution of inputs to each layer changes during training, slowing down the learning process and making it difficult to train models with cert...

Review of the seminal 2017 paper Attention is all you need. These paper introducea the Transformer architecture, a dominant model in natural language processing that relies entirely on multi-head attention instead of recurrent or convolutional networks. The first paper, "Attention Is All You Need," introduces the Transformer, showcasing its superior performance and training efficiency in...