Artificial Discourse

This research paper investigates the mathematical reasoning abilities of large language models (LLMs) and finds that their performance on mathematical problems is not as robust as initially thought. The authors introduce a new benchmark,  GSM-Symbolic , which generates diverse versions of math problems to assess LLMs' reasoning skills more thoroughly. Their findings indicate that LLMs struggle to...

The research paper introduces the  Differential Transformer , a new architecture for large language models that aims to improve the performance of these models by reducing the amount of attention they pay to irrelevant information. This architecture accomplishes this through a  differential attention mechanism  that calculates attention scores as the difference between two separate attention maps....

This technical report from Microsoft introduces  phi-3 , a new series of language models (LLMs) designed for various tasks. The core of the report focuses on  phi-3-mini , a small but highly capable LLM that rivals models like Mixtral and GPT-3.5 in performance despite being compact enough to run locally on a smartphone. This achievement is attributed to the use of  optimized training data  that p...

"You Only Look Once: Unified, Real-Time Object Detection", introduces YOLO, a novel approach to object detection. This method frames object detection as a regression problem, allowing a single neural network to predict bounding boxes and associated class probabilities directly from full images in a single evaluation. YOLO is incredibly fast, processing images at 45 frames per second, making it sui...

This research paper introduces Whisper, a speech recognition system trained on a massive, weakly supervised dataset of 680,000 hours of audio. The paper argues that scaling weakly supervised training has been underappreciated in speech recognition and that Whisper's robust, zero-shot performance demonstrates its ability to generalize well across different domains, languages, and tasks, even surpas...

WaveNet, a deep neural network designed to generate raw audio waveforms. The paper highlights WaveNet's ability to produce audio signals with unprecedented naturalness, surpassing the performance of existing text-to-speech systems. Key to WaveNet's success is the use of dilated causal convolutions, which enable the model to capture long-range temporal dependencies in audio data. The authors demons...

The paper introduces LLaMA, a series of open-source foundation language models ranging in size from 7B to 65B parameters, trained on trillions of tokens from publicly available datasets. LLaMA-13B surpasses GPT-3 on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. The authors demonstrate that training state-of-the-art models with publicly available...

This study describes the AlphaZero algorithm, a general-purpose reinforcement learning algorithm that can achieve superhuman performance in challenging domains. It surpasses the capabilities of traditional game-playing programs in chess, shogi, and Go, demonstrating its ability to learn from scratch and master complex games without relying on handcrafted domain knowledge. AlphaZero utilizes deep n...

This research paper introduces a new language representation model called BERT (Bidirectional Encoder Representations from Transformers). BERT's key innovation is its ability to learn deep bidirectional representations from unlabeled text, enabling it to outperform existing language models on a wide range of natural language processing tasks, including question answering, language inference, and s...

The authors demonstrate that  deep residual learning  overcomes the problem of  vanishing/exploding gradients  that often hinders the training of very deep networks by explicitly letting stacked layers fit a residual mapping. This technique enables the training of extremely deep networks, leading to significant accuracy gains in various tasks, including ImageNet classification, object detection on...

"Playing Atari with Deep Reinforcement Learning," focuses on training deep convolutional neural networks to play Atari 2600 games using reinforcement learning. The authors use a novel approach called Deep Q-learning, which combines Q-learning with experience replay, a technique that allows the agent to learn from past experiences and improve its performance. This paper explores the ability of deep...

The research paper outlines a new approach to training generative models called  Generative Adversarial Nets (GANs) . GANs utilize a  minimax two-player game  where a  generative model (G)  learns to produce realistic data samples, while a  discriminative model (D)  learns to distinguish between real and generated samples. Through this competitive process,  G improves its ability to create convinc...

This paper describes the application of a back-propagation network to handwritten digit recognition. The authors demonstrate how a network architecture, constrained by geometric knowledge, can achieve high accuracy in classifying digits without extensive preprocessing. The network, trained on a real-world dataset of handwritten zip codes, achieves a 1% error rate with a 9% rejection rate, showing...

The research paper "ImageNet Classification with Deep Convolutional Neural Networks" details the development and training of a large-scale convolutional neural network (CNN) for image classification. The authors, Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, present a groundbreaking architecture that achieved state-of-the-art results on the ImageNet dataset, a challenging benchmark in c...

This research paper revisits the traditional Recurrent Neural Networks (RNNs) – specifically, LSTMs and GRUs – and shows how to adapt them for modern parallel training. The authors demonstrate that by removing certain dependencies within the RNN structure, these models can be trained using the Parallel Scan algorithm, making them significantly faster than their traditional counterparts. The paper...

Attention is all you need: The Transformer is a new network architecture based solely on attention mechanisms that excel in sequence transduction tasks like language modelling and machine translation. Unlike traditional recurrent models, the Transformer allows for parallelization during training, leading to faster training times, especially with longer sequences. Notably, the Transformer utilizes...