Arxiv Papers

Online speculative decoding is introduced as a technique to improve the efficacy of speculative decoding in large language models. By continually updating draft models using excess computational power, the draft models can more accurately predict the target model's outputs, resulting in reduced latency. https://arxiv.org/abs//2310.07177 YouTube: https://www.youtube.com/@ArxivPapers TikTok: htt...

MatFormer is a nested Transformer architecture that allows for the extraction of smaller models from a larger trained model, enabling fine-grained control over tradeoffs in deployment constraints while maintaining accuracy. It is effective across different model classes, modalities, and scales. https://arxiv.org/abs//2310.07707 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tikt...

MatFormer is a nested Transformer architecture that allows for the extraction of smaller models from a larger trained model, enabling fine-grained control over tradeoffs in deployment constraints while maintaining accuracy. It is effective across different model classes, modalities, and scales. https://arxiv.org/abs//2310.07707 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tikt...

Step-Back Prompting is a technique that improves the performance of language models by enabling them to derive high-level concepts and principles from specific details, leading to better reasoning abilities. Experimental results show substantial performance gains on various reasoning-intensive tasks. https://arxiv.org/abs//2310.06117 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://ww...

Step-Back Prompting is a technique that improves the performance of language models by enabling them to derive high-level concepts and principles from specific details, leading to better reasoning abilities. Experimental results show substantial performance gains on various reasoning-intensive tasks. https://arxiv.org/abs//2310.06117 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://ww...

Mistral 7B is a language model with 7 billion parameters that outperforms other models in various benchmarks. It uses grouped-query attention and sliding window attention for faster and more efficient inference. There is also a fine-tuned version for following instructions. The models are available under the Apache 2.0 license. https://arxiv.org/abs//2310.06825 YouTube: https://www.youtube.com/@Ar...

The paper introduces SYNTRA, a method to reduce hallucination in large language models (LLMs) on abstractive summarization tasks. By optimizing the LLM's system message via prefix-tuning on a synthetic task, hallucination is reduced on real-world downstream tasks. https://arxiv.org/abs//2310.06827 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple...

The paper introduces SYNTRA, a method to reduce hallucination in large language models (LLMs) on abstractive summarization tasks. By optimizing the LLM's system message via prefix-tuning on a synthetic task, hallucination is reduced on real-world downstream tasks. https://arxiv.org/abs//2310.06827 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple...

This paper presents a systematic approach for fusing transformer-based networks using Optimal Transport to align architectural components. The proposed method outperforms vanilla fusion and individual parent models, showcasing the potential of fusing multiple Transformers. https://arxiv.org/abs//2310.05719 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers A...

This paper presents a systematic approach for fusing transformer-based networks using Optimal Transport to align architectural components. The proposed method outperforms vanilla fusion and individual parent models, showcasing the potential of fusing multiple Transformers. https://arxiv.org/abs//2310.05719 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers A...

The paper introduces the concept of linear mapping number (LMN) as a measure of network complexity and argues its superiority over the L2 norm in characterizing neural network compression and generalization. https://arxiv.org/abs//2310.05918 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/...

The paper introduces the concept of linear mapping number (LMN) as a measure of network complexity and argues its superiority over the L2 norm in characterizing neural network compression and generalization. https://arxiv.org/abs//2310.05918 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/...

The paper introduces ITIT, a training paradigm for vision-language models that allows training on unpaired image and text data. ITIT achieves similar performance to models trained on paired data with significantly fewer data samples. https://arxiv.org/abs//2310.03734 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.co...

The paper introduces ITIT, a training paradigm for vision-language models that allows training on unpaired image and text data. ITIT achieves similar performance to models trained on paired data with significantly fewer data samples. https://arxiv.org/abs//2310.03734 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.co...

Large language models (LLMs) like Llama-2 learn coherent models of data generating processes, as evidenced by their linear representations of space and time across multiple scales, robustness to prompting variations, and identification of "space neurons" and "time neurons". https://arxiv.org/abs//2310.02207 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tikto...

Large language models (LLMs) like Llama-2 learn coherent models of data generating processes, as evidenced by their linear representations of space and time across multiple scales, robustness to prompting variations, and identification of "space neurons" and "time neurons". https://arxiv.org/abs//2310.02207 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tikto...

This paper compares retrieval-augmentation and long context window methods for improving the performance of large language models (LLMs) on downstream tasks. The study finds that retrieval-augmentation with a 4K context window can achieve comparable performance to a finetuned LLM with a 16K context window, while requiring less computation. Retrieval also significantly improves LLM performance rega...

This paper compares retrieval-augmentation and long context window methods for improving the performance of large language models (LLMs) on downstream tasks. The study finds that retrieval-augmentation with a 4K context window can achieve comparable performance to a finetuned LLM with a 16K context window, while requiring less computation. Retrieval also significantly improves LLM performance rega...

This paper examines the role and effectiveness of self-correction in large language models (LLMs) and finds that LLMs struggle to self-correct without external feedback, sometimes even degrading performance. Suggestions for future research and applications are provided. https://arxiv.org/abs//2310.01798 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Appl...

This paper examines the role and effectiveness of self-correction in large language models (LLMs) and finds that LLMs struggle to self-correct without external feedback, sometimes even degrading performance. Suggestions for future research and applications are provided. https://arxiv.org/abs//2310.01798 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Appl...

The paper proposes a new approach called "pause-training" for language models, where the model delays outputting the next token until it has processed additional hidden vectors. Experimental results show improved performance on various tasks, suggesting the potential of this new paradigm. https://arxiv.org/abs//2310.02226 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www....

The paper proposes a new approach called "pause-training" for language models, where the model delays outputting the next token until it has processed additional hidden vectors. Experimental results show improved performance on various tasks, suggesting the potential of this new paradigm. https://arxiv.org/abs//2310.02226 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www....

The paper proposes a framework called PIT that uses human preference data to implicitly train large language models (LLMs) to improve their response quality. PIT outperforms prompting-based methods in experiments on real-world and synthetic datasets. https://arxiv.org/abs//2310.00898 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://...

The paper proposes a framework called PIT that uses human preference data to implicitly train large language models (LLMs) to improve their response quality. PIT outperforms prompting-based methods in experiments on real-world and synthetic datasets. https://arxiv.org/abs//2310.00898 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://...

The paper proposes using an abstract graphics language as an intermediate representation for generating vector graphics from text. They introduce a large-scale dataset and a new model that outperforms existing models in terms of similarity to human-created figures. The framework and datasets are publicly available. https://arxiv.org/abs//2310.00367 YouTube: https://www.youtube.com/@ArxivPapers Tik...