Artificial Discourse

This research paper investigates the impact of different language models (LLMs) used as "teachers" to generate synthetic responses for instruction tuning. The authors demonstrate a surprising phenomenon they call the "Larger Models' Paradox," where larger and supposedly "stronger" teacher models do not always lead to improved instruction-following abilities in smaller base models. They propose a n...

This research paper investigates the potential for large language models (LLMs) to self-improve in long-context reasoning, which involves processing and understanding complex information spread across long stretches of text. The authors propose a novel approach called SEALONG that leverages the LLMs' ability to generate multiple outputs for a given question and then scores these outputs using a me...

LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models, introduces a new method for generating 3D models using large language models (LLMs). The authors address the challenge of tokenizing 3D mesh data for LLMs by representing the mesh data as plain text using the OBJ file format, a standard text-based format for 3D models. This approach allows for direct integration with LLMs without modify...

The researchers introduce LLaVA-o1, a vision language model designed to perform structured reasoning by breaking down problem-solving into four distinct stages: summary, caption, reasoning, and conclusion. They compiled a new dataset, LLaVA-o1-100k, and proposed a stage-level beam search method to improve model performance during inference. Experimental results demonstrate that LLaVA-o1 outperform...

The BlueLM-V-3B, a multimodal large language model (MLLM) designed specifically for mobile devices. The researchers address the challenges of deploying large models on mobile phones, such as limited memory and processing power, by implementing a novel algorithm and system co-design approach. This includes a dynamic resolution scheme that optimizes image processing and a token downsampler that redu...

CORAL, a novel benchmark dataset for evaluating Retrieval-Augmented Generation (RAG) systems in a multi-turn conversational setting. The authors highlight the limitations of existing datasets in assessing conversational RAG and detail CORAL's unique features, including open-domain coverage, knowledge intensity, free-form responses, topic shifts, and citation labeling. They explain how CORAL is der...

This research paper surveys small language models (SLMs) and explores their applications, design, training, and model compression techniques. The authors explain that while large language models (LLMs) have proven effective, their resource demands have led to the development of SLMs, which are more efficient and can be deployed on a wider range of devices. The paper examines various techniques to...

This research explores whether transformers, a type of neural network architecture, can learn to reason implicitly over knowledge. The authors find that transformers can learn to reason implicitly, but only through a phenomenon called  grokking , where training extends far beyond overfitting. The study investigates two reasoning types:  composition  and  comparison . They find that while the trans...

This research paper details the development of Llama 3, a large language model with improved capabilities in language understanding, code generation, mathematical reasoning, and multimodality. The paper emphasizes the importance of high-quality data, scaling up compute power, and using simple, efficient methods to achieve optimal results. It also explores the integration of vision and speech capab...

Unlike traditional  Multi-Layer Perceptrons (MLPs) , which have fixed activation functions on nodes, KANs have learnable activation functions on edges. This seemingly simple change allows KANs to outperform MLPs in terms of accuracy and interpretability, particularly for small-scale artificial intelligence and scientific tasks. The text explores the mathematical foundations of KANs, highlighting t...

The document describes the development of  MMIE , a large-scale benchmark designed to evaluate the performance of  Large Vision-Language Models (LVLMs)  in interleaved multimodal comprehension and generation tasks. MMIE comprises a dataset of 20,000 meticulously curated multimodal queries across various domains, including mathematics, coding, and literature, which are designed to challenge LVLMs t...

This research paper proposes a novel method called Thought Preference Optimization (TPO) to train large language models (LLMs) to "think" before responding to user instructions. TPO utilizes a preference-based training framework where LLMs generate internal thoughts alongside their responses, and these thoughts are then optimized based on the quality of the resulting responses. The authors argue t...

The paper, "VIT-LENS: Towards Omni-modal Representations," introduces a novel approach to enable Artificial Intelligence (AI) agents to perceive information from various modalities beyond just vision and language. It proposes a method that leverages a pre-trained visual transformer (ViT) to efficiently encode information from diverse modalities, such as 3D point clouds, depth, audio, tactile, and...

This research paper proposes a new method for efficiently training linear transformers, which are a type of neural network that uses linear attention to process sequences of data. Unlike traditional transformers, which have quadratic complexity in sequence length, linear transformers can process long sequences in linear time, making them more efficient for certain tasks. However, existing linear t...

This research explores how the architecture of pre-trained language models influences their base capabilities, specifically focusing on the FFN-Wider Transformer architecture. The study identifies a key factor in model performance: the contribution ratio of the Multi-Head Attention (MHA) layer, which acts as a combination function that reflects the model's ability to combine linguistic features. T...

This research paper proposes a new method called  Reinforcement Learning from Execution Feedback (RLEF)  to improve the ability of large language models (LLMs) to generate code that successfully completes tasks. The authors demonstrate the effectiveness of RLEF by training LLMs on a challenging competitive programming benchmark called  CodeContests . RLEF trains the models to iteratively generate...

This research paper investigates how variations in the phrasing of prompts impact the performance of large language models (LLMs) across 120 tasks and five models. The study systematically analyzes six families of paraphrase types, including morphology, syntax, lexicon, lexico-syntax, discourse, and others, to determine their influence on model outputs. The findings demonstrate a potential for sig...

LLaMA-Omni , designed to improve the seamless interaction between speech and large language models (LLMs). This model integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder, allowing it to generate text and speech responses directly from speech instructions with minimal latency. To enhance the model's performance, the authors create a speech instruction da...

"Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models," details the development of a new family of multimodal language models (VLMs) called Molmo. Molmo is notable for its open-weight and open-data approach, meaning the model's weights, training data, and code are publicly available. This contrasts with the current trend of proprietary VLMs which keep their models clo...

This technical paper proposes a novel technique called  Low-Rank Adaptation (LoRA)  for adapting large language models (LLMs) to specific downstream tasks. LoRA addresses the challenge of fine-tuning LLMs, which requires updating all model parameters, by injecting low-rank decomposition matrices into each layer of the Transformer architecture. This significantly reduces the number of trainable par...

MCTrack  is a new method for tracking multiple 3D objects, particularly designed for autonomous driving. The authors claim that MCTrack outperforms existing methods across popular datasets like KITTI, nuScenes, and Waymo. The paper also standardizes the format of perception results across various datasets, making it easier for researchers to focus on algorithm development. Additionally, the paper...

This paper proposes  Synatra , a system for generating large amounts of training data for digital agents. The goal is to overcome the problem of  expensive human annotation  by using  indirect knowledge  like online tutorials and random web pages as input. Synatra leverages  LLMs  to transform this indirect knowledge into  direct demonstrations  in the form of action sequences, which are then used...

This research paper investigates the phenomenon of "lazy layers" in large language models (LLMs). Lazy layers occur when deeper layers in LLMs lose the ability to learn meaningful information, leading to a decline in model performance. The authors introduce a new training technique called Inheritune, which addresses this issue by inheriting the initial layers of a larger, pre-trained model and gra...

This research paper proposes a polynomial-time approximation scheme (PTAS) for finding perfect equilibria in dynamic games. This is a significant contribution to game theory because it has long been an open question whether such an algorithm exists. The authors introduce a new geometric object called the "equilibrium bundle," which allows them to formalize perfect equilibria as zero points of its...

MLE-bench  is a benchmark that evaluates the performance of AI agents on machine learning engineering tasks. The benchmark is comprised of 75 real-world Kaggle competitions, each with a dataset, description, and grading code. The authors evaluated various language models and agent frameworks on MLE-bench, finding that the best-performing agent achieved at least the level of a Kaggle bronze medal i...