AI: Origins

This 2022 academic paper explores Fill-in-the-Middle (FIM) capabilities in causal decoder-based language models, demonstrating that these models can learn to infill text effectively by simply rearranging parts of the training data. The authors propose a method where a middle section of text is moved to the end of a document during training, showing this data augmentation does not negatively impact...

This 2016 academic paper addresses the challenge of translating rare and unknown words in Neural Machine Translation (NMT), a common issue as NMT models typically operate with a fixed vocabulary while translation itself is an open-vocabulary problem. The authors propose a novel approach where rare and unknown words are encoded as sequences of subword units, eliminating the need for a back-off dict...

This 2013 paper introduces advancements to the continuous Skip-gram model, a method for learning high-quality distributed vector representations of words. The authors present extensions like subsampling frequent words and negative sampling to enhance vector quality and training speed. A significant contribution is the method for identifying and representing idiomatic phrases as single tokens, impr...

This 2013 academic paper introduces two new model architectures, Continuous Bag-of-Words (CBOW) and Skip-gram, designed for efficiently computing continuous vector representations of words from vast datasets. The authors compare the quality and computational cost of these new models against existing neural network language models, demonstrating significant improvements in accuracy at a lower compu...

This paper published in 2003 introduces a neural probabilistic language model designed to address the curse of dimensionality inherent in modeling word sequences. The authors propose learning a distributed representation for words, which enables the model to generalize from seen sentences to an exponential number of semantically similar, unseen sentences. This approach simultaneously learns word f...

The paper published in 1989, "Training Stochastic Model Recognition Algorithms as Networks can lead to Maximum Mutual Information Estimation of Parameters"  by John S. Bridle, proposes a novel approach to  pattern recognition , specifically improving  Hidden Markov Models (HMMs)  used in  speech recognition . It focuses on  discrimination-based training methods  within  neural networks (NNs) . The...

The paper on backpropagation was published in 1986. The paper presents a  collaborative research effort  focusing on  back-propagation  as a method for  learning representations  within  neural networks . One document, "Learning representations by back-propagating errors (1).pdf," introduces the  theoretical framework  and  mathematical underpinnings  of this learning algorithm, explaining how con...

This paper published in 1986 introduces the concept of Parallel Distributed Processing (PDP) models, offering a new perspective on how human cognition works, contrasting it with traditional sequential processing. It explores how the brain handles complex tasks like perception, motor control, language understanding, and memory retrieval by simultaneously considering multiple, often ambiguous, piece...

The 1958 paper on Perceptrons, by Marvin L. Minsky and Seymour A. Papert, offers an expanded edition exploring artificial intelligence, particularly pattern recognition, and learning through linear parallel predicates and geometrical theory of linear inequalities. It discusses the historical development of neural networks and connectionism from the 1940s through the 1980s, providing mathematical s...

Perhaps the first related papers influencing the rise of the design of neural networks, published in 1943! The paper, "A Logical Calculus of the Ideas Immanent in Nervous Activity," is a  foundational paper  in fields such as  cognitive science  and  artificial intelligence . This work models  neural networks  as discrete-time systems where neurons have "all-or-none" states (firing or not firing)...