PyTorch Developer Podcast

Structured kernels are a new way to write kernels in PyTorch. Why did they take so long? What finally convinced us that we should do them? Why did it end up taking me the better part of a year to only be half done with them? Further reading: Structured kernels RFC https://github.com/pytorch/rfcs/blob/rfc-0005/RFC-0005-structured-kernel-definitions.md Taxonomy of PyTorch operators by shape behavior...

Functionalization is the process by which we remove mutation from autograd graphs in PyTorch, leaving us with a purely functional graph that we can execute in the normal way. Why do we need to do functionalization? What makes it not so easy to do? How do we do it? And how does it compare to mutation removal that you might see in a compiler? Further reading: Section 3.1 of this paper on PyTorch AD...

Ever wanted to learn about CUDA but not sure where to start? In this sixteen minute episode I try to jam in as much CUDA knowledge as could be reasonably expected in a podcast. You won't know how to write a kernel after this episode, but you'll know about what a GPU is, what the general CUDA programming model is, why asynchronous execution makes everything complicated, and some general principles...

What's inference mode? Why doesn't my code run fast if I use no_grad or make sure requires_grad=False? How come inference mode is safe but AutoNonVariableTypeMode is not? Further reading: Inference mode RFC https://github.com/.../rfc0011/RFC-0011-InferenceMode.md Inference mode docs for C++ frontend users https://github.com/.../cpp/source/notes/inference_mode.rst Tracking issue for Python frontend...

What is vectorization? How do you use it in PyTorch? What are some of the traps and pitfalls of writing vectorized code in PyTorch? Further reading: native/cpu README https://github.com/.../aten/src/ATen/native/cpu/README.md Vec256 classes https://github.com/.../tree/master/aten/src/ATen/cpu/vec256 AVX512 support tracking issue https://github.com/pytorch/pytorch/issues/56187

Why is PyTorch split into so many libraries? What's the point of these splits? What do Windows, mobile and CUDA have to do with the library splits? Further reading: c10 folder architecture description https://github.com/.../wiki/Software-Architecture-for-c10 Implementation of the TORCH_API visibility macros https://github.com/.../blob/master/c10/macros/Export.h An example of virtual call based hoo...

Why is the dispatcher the way it is today? How did evolve over time, and what constraints got added so that it is the kind of complicated piece it is today? Further reading: How the dispatcher actually works http://blog.ezyang.com/.../lets-talk-about-the-pytorch.../ Zachary DeVito's original version of ATen, before it got merged back into PyTorch mainline https://github.com/zdevito/ATen The multip...

In this episode, we will discuss how to bind a C++ object in Python? We'll try to answer the following questions: How does pybind11 do it? What's different about how we implement it for Tensor? What are some downsides of the approach? Note from the future: I recorded and then decided I didn't like my follow up episode about how to preserve PyObjects even when they go dead in Python. Maybe some day...