Arxiv Papers

Orca 2, a small language model, is trained using various reasoning techniques and is able to outperform larger models on complex tasks. The model's performance is similar to models 5-10x larger, demonstrating the potential of smaller models in advanced reasoning. https://arxiv.org/abs//2311.11045 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple P...

FastBERT is a BERT variant that uses only 0.3% of its neurons during inference, achieving similar performance to other BERT models. It selectively engages 12 out of 4095 neurons per layer using fast feedforward networks, resulting in significant speedup. https://arxiv.org/abs//2311.10770 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: http...

FastBERT is a BERT variant that uses only 0.3% of its neurons during inference, achieving similar performance to other BERT models. It selectively engages 12 out of 4095 neurons per layer using fast feedforward networks, resulting in significant speedup. https://arxiv.org/abs//2311.10770 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: http...

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection This paper proposes a unified vision-language model that combines images and videos into a single feature space, resulting in improved performance on various visual-language tasks compared to models that treat images and videos separately. https://arxiv.org/abs//2311.10122 YouTube: https://www.youtube.com/@ArxivPaper...

This paper proposes a unified vision-language model that combines images and videos into a single feature space, resulting in improved performance on various visual-language tasks compared to models that treat images and videos separately. https://arxiv.org/abs//2311.10122 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.ap...

The paper analyzes the effectiveness of using shallow feed-forward networks to mimic the attention mechanism in the Transformer model. Results show that these "attentionless Transformers" can rival the performance of the original architecture, highlighting the potential to streamline complex architectures for sequence-to-sequence tasks. https://arxiv.org/abs//2311.10642 YouTube: https://...

This paper presents a fully automated solution for generating consistent characters from text prompts. The proposed method outperforms baseline methods in terms of prompt alignment and identity consistency, as demonstrated through quantitative analysis and a user study. Several practical applications are showcased. https://arxiv.org/abs//2311.10093 YouTube: https://www.youtube.com/@ArxivPapers Tik...

This paper presents a fully automated solution for generating consistent characters from text prompts. The proposed method outperforms baseline methods in terms of prompt alignment and identity consistency, as demonstrated through quantitative analysis and a user study. Several practical applications are showcased. https://arxiv.org/abs//2311.10093 YouTube: https://www.youtube.com/@ArxivPapers Tik...

The paper proposes Striped Attention, an extension to the Ring Attention algorithm, to address workload imbalance in causal transformer models. It achieves significant throughput improvements on GPUs and TPUs at long sequence lengths. https://arxiv.org/abs//2311.09431 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.c...

The paper proposes Striped Attention, an extension to the Ring Attention algorithm, to address workload imbalance in causal transformer models. It achieves significant throughput improvements on GPUs and TPUs at long sequence lengths. https://arxiv.org/abs//2311.09431 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.c...

SiRA is a sparse mixture of low rank adaption approach that leverages sparse computation to improve the performance of large language models on downstream tasks. It outperforms other approaches in single task and multitask settings. https://arxiv.org/abs//2311.09179 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com...

SiRA is a sparse mixture of low rank adaption approach that leverages sparse computation to improve the performance of large language models on downstream tasks. It outperforms other approaches in single task and multitask settings. https://arxiv.org/abs//2311.09179 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com...

FastCoT is a model-agnostic framework that uses parallel decoding and auto-regressive decoding simultaneously to improve inference time without sacrificing performance in language models. https://arxiv.org/abs//2311.08263 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify...

FastCoT is a model-agnostic framework that uses parallel decoding and auto-regressive decoding simultaneously to improve inference time without sacrificing performance in language models. https://arxiv.org/abs//2311.08263 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify...

The paper introduces the problem of adversarial arithmetic, where language models are tested on arithmetic questions with adversarial prompts. The authors propose an algorithm for finding successful attacks and show that models can be partially hardened against these attacks, but not fully robust. https://arxiv.org/abs//2311.07587 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.t...

The paper introduces the problem of adversarial arithmetic, where language models are tested on arithmetic questions with adversarial prompts. The authors propose an algorithm for finding successful attacks and show that models can be partially hardened against these attacks, but not fully robust. https://arxiv.org/abs//2311.07587 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.t...

The paper explores the ability of large language models (LLMs) to judge the quality of their own generations. It introduces a reasoning with refinement strategy called ART, which improves performance on multistep reasoning tasks by asking necessary questions and ranking refinements. The approach achieves a 5-point gain over self-refinement baselines while using a smaller model as the decision make...

The paper explores the ability of large language models (LLMs) to judge the quality of their own generations. It introduces a reasoning with refinement strategy called ART, which improves performance on multistep reasoning tasks by asking necessary questions and ranking refinements. The approach achieves a 5-point gain over self-refinement baselines while using a smaller model as the decision make...

LUMOS is a framework for training language agents using a unified data format and modular architecture. It achieves comparable or superior performance to state-of-the-art agents and exhibits advantages in complex question answering, web tasks, math problems, and generalization to unseen tasks. https://arxiv.org/abs//2311.05657 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tikto...

LUMOS is a framework for training language agents using a unified data format and modular architecture. It achieves comparable or superior performance to state-of-the-art agents and exhibits advantages in complex question answering, web tasks, math problems, and generalization to unseen tasks. https://arxiv.org/abs//2311.05657 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tikto...

This study investigates the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, revealing distinct patterns in encoders and decoders. Initial training expands dimensionality, while later training refines into more compact representations. https://arxiv.org/abs//2311.05928 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers...

This study investigates the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, revealing distinct patterns in encoders and decoders. Initial training expands dimensionality, while later training refines into more compact representations. https://arxiv.org/abs//2311.05928 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers...

Prompt Cache is a method for accelerating inference in large language models by reusing attention states of frequently occurring text segments. It significantly reduces latency without compromising output accuracy or requiring modifications to the model parameters. https://arxiv.org/abs//2311.04934 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Pod...

Prompt Cache is a method for accelerating inference in large language models by reusing attention states of frequently occurring text segments. It significantly reduces latency without compromising output accuracy or requiring modifications to the model parameters. https://arxiv.org/abs//2311.04934 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Pod...

The paper explores the use of speculative sampling to reduce latency in text generation, comparing it to autoregressive sampling. The authors also discuss the use of model-based optimizations and provide a Jupyter notebook and sample executions. https://arxiv.org/abs//2311.04951 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podca...