Large Language Model (LLM) Talk

LLaMA-2 is a collection of large language models (LLMs), with pretrained and fine-tuned versions ranging from 7 billion to 70 billion parameters. The fine-tuned models, called Llama 2-Chat, are designed for dialogue and outperform open-source models on various benchmarks. The models were trained on 2 trillion tokens of publicly available data, and were optimized for both helpfulness and safety usi...

LLaMA-1 is a collection of large language models ranging from 7B to 65B parameters, trained on publicly available datasets. LLaMA models achieve competitive performance compared to other LLMs like GPT-3, Chinchilla, and PaLM, with the 13B model outperforming GPT-3 on most benchmarks, despite being much smaller, and the 65B model being competitive with the best large language models. The document a...

The surveys of large language models (LLMs), covering their development, training, and applications. Key areas include data collection and preprocessing, which is crucial for model quality, and methods for adapting LLMs using instruction tuning or reinforcement learning with human feedback. The survey also discusses prompt engineering, which is important for task performance and involves designing...

Mixture of Experts (MoE) models use multiple sub-models, or experts, to handle different parts of the input space, orchestrated by a router or gating mechanism. MoEs are trained by dividing data, specializing experts, and using a router to direct inputs. Not all parameters are activated for each input, using sparse activation, and techniques such as load balancing and expert capacity are used to i...

Multi-task learning (MTL) is a machine learning approach where a model learns multiple tasks simultaneously, leveraging the shared information between related tasks to improve generalization. MTL can be motivated by human learning and is considered a form of inductive transfer. Two common methods for MTL in deep learning are hard and soft parameter sharing. Hard parameter sharing involves sharing...

Gradient descent is a widely used optimization algorithm in machine learning and deep learning that iteratively adjusts model parameters to minimize a cost function. It operates by moving parameters in the opposite direction of the gradient. There are three main variants: batch gradient descent, which uses the whole training set; stochastic gradient descent (SGD), which uses individual training ex...

Generative Pre-trained Transformers (GPTs) are a family of large language models that use a transformer deep learning architecture. They are pre-trained on vast amounts of text data and then fine-tuned for specific tasks. GPT models can generate human-like text, translate languages, summarize content, analyze data, and write code. These models utilize self-attention mechanisms to process input and...

Linear Transformers address the computational limitations of standard Transformer models, which have a quadratic complexity, O(n^2), with respect to input sequence length. Linear Transformers aim for linear complexity, O(n), making them suitable for longer sequences. They achieve this through methods such as low-rank approximations, local attention, or kernelized attention. Examples include Linfor...

BERT (Bidirectional Encoder Representations from Transformers) is a groundbreaking NLP model from Google that learns deep, bidirectional text representations using a transformer architecture. This allows for a richer contextual understanding than previous models that only processed text unidirectionally. BERT is pre-trained using a masked language model and a next sentence prediction task on large...

Sora is an AI model from OpenAI that creates videos from text using a diffusion process, starting with noise and refining it. It employs a transformer architecture and handles videos as spacetime patches. Sora can extend existing footage, animate images, and blend videos. It has shown an ability to simulate elements of the real world, but has some shortcomings in depicting accurate physics and cau...

The sources explore word embeddings, representing words as numerical vectors to capture meaning. The Skip-gram model is a key method for learning these high-quality, distributed vector representations from large text datasets. This model predicts surrounding words in a sentence, resulting in word vectors that encode linguistic patterns. To enhance the Skip-gram model, the sources introduce techniq...

Diffusion models are generative models that learn to create data by reversing a process that gradually adds noise to a training sample. Stable Diffusion uses a U-Net architecture to map images to images, incorporating text prompts with CLIP embeddings and cross-attention, operating in a compressed latent space for efficiency. These models can be adapted for video generation by adding temporal laye...

The sources describe RETRO (Retrieval-Enhanced Transformer), a language model that enhances its performance by retrieving information from a large database. RETRO uses a key-value store where keys are BERT embeddings of text chunks and values are the text chunks themselves. When processing input, it retrieves similar text chunks from the database to augment the input, allowing it to perform compar...

GPT-2 language model is a large, transformer-based model using a decoder-only architecture. It predicts the next word in a sequence, much like an advanced keyboard app. GPT-2 is auto-regressive, adding each predicted token to the input for the next step. It uses masked self-attention, focusing on previous tokens, unlike BERT's self-attention. Input tokens are processed through multiple decoder blo...

GPT3 is a large language model that generates text based on its training on a massive dataset of 300 billion tokens. It outputs text one token at a time, influenced by input text. The model encodes what it learns in 175 billion parameters and has a context window of 2048 tokens. The core calculations happen within 96 transformer decoder layers, each with 1.8 billion parameters. Words are converted...

The Transformer model is a neural network architecture that uses self-attention to understand relationships between elements in sequential data like words in a sentence. Unlike recurrent neural networks (RNNs) that process data sequentially, the Transformer can process all words in parallel. It has an encoder to read the input and a decoder to generate the output. Positional encoding accounts for...

Prompt engineering is the iterative process of creating text inputs to guide AI models toward desired outputs. It involves using techniques such as clear instructions, delimiters, and specified output formats. Effective prompts may include examples, reference texts, and persona instructions. Advanced techniques like Chain-of-Thought (CoT) prompting for step-by-step reasoning, and the use of extern...

LLM-based autonomous agents are a developing area of AI focused on creating systems that can perceive, reason, and act autonomously using large language models (LLMs). These agents use planning, memory (sensory, short-term, and long-term), and tools to accomplish tasks. They are applied in fields like social science, natural science, and engineering. Evaluation includes human assessments and objec...