Mechanical Dreams

Optimal Linear Decay Learning Rate Schedules and Further Refinements 05.01.2025 18:28 ↓

In this episode: • The Death of Cosine?: Introduction to the episode and the paper. Professor Norris expresses his skepticism about changing established habits like Cosine Annealing, while Linda teases a shake-up in the status quo. • Theory vs. Reality: A discussion on the massive gap between theoretical learning rates (like 1/t) and what practitioners actually use. Linda explains why the theory h...