What’s a learning rate? Why might you want to schedule it? How does the LR scheduler API in PyTorch work? What the heck is up with the formula implementation? Why is everything terrible?

What’s a learning rate? Why might you want to schedule it? How does the LR scheduler API in PyTorch work? What the heck is up with the formula implementation? Why is everything terrible?

Learning rate schedulers

What are they good for? (Caches. Private fields.) C++ side support, how it’s implemented / release resources. Python side support, how it’s implemented. Weak ref tensor hazard due to resurrection. Downsides of weak references in C++. Scott Wolchok’s release resources optimization.Other episodes to listen to first: https://pytorch-dev-podcast.simplecast.com/episodes/reference-counting https://pytorch-dev-podcast.simplecast.com/episodes/pyobject-preservation

What are they good for? (Caches. Private fields.) C++ side support, how it’s implemented / release resources. Python side support, how it’s implemented. Weak ref tensor hazard due to resurrection. Downsides of weak references in C++. Scott Wolchok’s release resources optimization.Other episodes to listen to first: <a href="https://pytorch-dev-podcast.simplecast.com/episodes/reference-counting" rel="nofollow" target="_blank">https://pytorch-dev-podcast.simplecast.com/episodes/reference-counting</a> <a href="https://pytorch-dev-podcast.simplecast.com/episodes/pyobject-preservation" rel="nofollow" target="_blank">https://pytorch-dev-podcast.simplecast.com/episodes/pyobject-preservation</a>

Weak references

Mike Ruberry has an RFC about stride-agnostic operator semantics (https://github.com/pytorch/pytorch/issues/78050), so let's talk about strides. What are they? How are they used to implement views and memory format? How do you handle them properly when writing kernels? In what sense are strides overspecified, and therefore, not worth slavishly reimplementing in a system like PrimTorch? What does Edward think we should do about them?My blog post that covers strides along with other topics can be found at http://blog.ezyang.com/2019/05/pytorch-internals/

Mike Ruberry has an RFC about stride-agnostic operator semantics (<a href="https://github.com/pytorch/pytorch/issues/78050" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/78050</a>), so let's talk about strides. What are they? How are they used to implement views and memory format? How do you handle them properly when writing kernels? In what sense are strides overspecified, and therefore, not worth slavishly reimplementing in a system like PrimTorch? What does Edward think we should do about them?My blog post that covers strides along with other topics can be found at <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="nofollow" target="_blank">http://blog.ezyang.com/2019/05/pytorch-internals/</a>

Strides

AOTAutograd is a cool new feature in functorch for capturing both forward and backward traces of PyTorch operators, letting you run them through a compiler and then drop the compiled kernels back into a normal PyTorch eager program. Today, Horace joins me to tell me how it works, what it is good to use for, and what our future plans for it are.

AOTAutograd is a cool new feature in functorch for capturing both forward and backward traces of PyTorch operators, letting you run them through a compiler and then drop the compiled kernels back into a normal PyTorch eager program. Today, Horace joins me to tell me how it works, what it is good to use for, and what our future plans for it are.

AOTAutograd

Sherlock recently joined the PyTorch team, having previously worked on ONNX Runtime at Microsoft, and Sherlock’s going to ask me some questions about the dispatcher, and I’m going to answer them. We talked about the history of the dispatcher, how to override dispatching order, multiple dispatch, how to organize various dispatch keys and torch function mode. The companion video is at https://youtu.be/_qB2Ho1O3u4

Sherlock recently joined the PyTorch team, having previously worked on ONNX Runtime at Microsoft, and Sherlock’s going to ask me some questions about the dispatcher, and I’m going to answer them. We talked about the history of the dispatcher, how to override dispatching order, multiple dispatch, how to organize various dispatch keys and torch function mode. The companion video is at <a href="https://youtu.be/_qB2Ho1O3u4" rel="nofollow" target="_blank">https://youtu.be/_qB2Ho1O3u4</a>

Dispatcher questions with Sherlock

PyTorch recently moved all of its CI from CircleCI to GitHub Actions. There were a lot of improvements in the process, making my old podcast about CI obsolete! Today, Eli Uriegas joins me to talk about why we moved to GitHub Actions, how the new CI system is put together, and what some cool features about our new CI.

PyTorch recently moved all of its CI from CircleCI to GitHub Actions. There were a lot of improvements in the process, making my old podcast about CI obsolete! Today, Eli Uriegas joins me to talk about why we moved to GitHub Actions, how the new CI system is put together, and what some cool features about our new CI.

New CI

C++ has exceptions, Python has exceptions. But they’re not the same thing! How do exceptions work in CPython, how do we translate exceptions from C++ to Python (hint: it’s different for direct bindings versus pybind11), and what do warnings (which we also translate from C++ to Python) have in common with this infrastructure?

C++ has exceptions, Python has exceptions. But they’re not the same thing! How do exceptions work in CPython, how do we translate exceptions from C++ to Python (hint: it’s different for direct bindings versus pybind11), and what do warnings (which we also translate from C++ to Python) have in common with this infrastructure?

Python exceptions

PyTorch’s torch API is the Python API everyone knows and loves, but there’s also another API, the ATen API, which most of PyTorch’s internal subsystems are built on. How to tell them apart? What implications do these have on our graph mode IR design? Also, a plug for PrimTorch, a new set of operators, not designed for eager mode, that is supposed to be even lower level than ATen.

PyTorch’s torch API is the Python API everyone knows and loves, but there’s also another API, the ATen API, which most of PyTorch’s internal subsystems are built on. How to tell them apart? What implications do these have on our graph mode IR design? Also, a plug for PrimTorch, a new set of operators, not designed for eager mode, that is supposed to be even lower level than ATen.

Torch vs ATen APIs

PyTorch is in the business of shipping numerical software that can run fast on your CUDA-enabled NVIDIA GPU, but it turns out there is a lot of heterogeneity in NVIDIA’s physical GPU offering and when it comes to what is fast and what is slow, what specific GPU you have on hand matters quite a bit. Yet there are literally hundreds of distinct NVIDIA GPU models on the market, how do you make sense of the madness? Today, Natalia Gimelshein joins me to talk about everything that’s going on in the NVIDIA GPU market, and what, as a framework developer, you have to care about to make sense of it all.Further reading.NVIDIA microarchitectures on Wikipedia https://en.wikipedia.org/wiki/Category:Nvidia_microarchitecturesA slightly old post about matching SM to architecture https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

PyTorch is in the business of shipping numerical software that can run fast on your CUDA-enabled NVIDIA GPU, but it turns out there is a lot of heterogeneity in NVIDIA’s physical GPU offering and when it comes to what is fast and what is slow, what specific GPU you have on hand matters quite a bit. Yet there are literally hundreds of distinct NVIDIA GPU models on the market, how do you make sense of the madness? Today, Natalia Gimelshein joins me to talk about everything that’s going on in the NVIDIA GPU market, and what, as a framework developer, you have to care about to make sense of it all.Further reading.<ul><li>NVIDIA microarchitectures on Wikipedia <a href="https://en.wikipedia.org/wiki/Category:Nvidia_microarchitectures" rel="nofollow" target="_blank">https://en.wikipedia.org/wiki/Category:Nvidia_microarchitectures</a></li><li>A slightly old post about matching SM to architecture <a href="https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/" rel="nofollow" target="_blank">https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/</a></li></ul>

All about NVIDIA GPUs

A lot of recent work going in PyTorch is all about adding new and interesting Tensor subclasses, and this all leads up to the question of, what exactly is OK to make a tensor subclass? One answer to this question comes from an old principle from Barbara Liskov called the Liskov substitution principle, which informally can be stated as S is a subtype of T if anywhere you have T, it can be replaced with S without altering "desirable" properties of this program. In this podcast I'll talk about LSP and how it relates to the design of Tensor subclasses and a hypothetical "abstract Tensor specification" which really doesn't exist but which sort of implicitly exists in the corpus of existing PyTorch programs.Further reading:This is a cool interview with Barbara Liskov that I quote in the podcast https://www.youtube.com/watch?v=-Z-17h3jG0AMax Balandat talking about linear operators in PyTorch https://github.com/pytorch/pytorch/issues/28341At the end I talk a little bit about multiple dispatch; an earlier discussion about this topic is in this podcast https://pytorch-dev-podcast.simplecast.com/episodes/multiple-dispatch-in-torch-function

A lot of recent work going in PyTorch is all about adding new and interesting Tensor subclasses, and this all leads up to the question of, what exactly is OK to make a tensor subclass? One answer to this question comes from an old principle from Barbara Liskov called the Liskov substitution principle, which informally can be stated as S is a subtype of T if anywhere you have T, it can be replaced with S without altering "desirable" properties of this program. In this podcast I'll talk about LSP and how it relates to the design of Tensor subclasses and a hypothetical "abstract Tensor specification" which really doesn't exist but which sort of implicitly exists in the corpus of existing PyTorch programs.Further reading:<ul><li>This is a cool interview with Barbara Liskov that I quote in the podcast <a href="https://www.youtube.com/watch?v=-Z-17h3jG0A" rel="nofollow" target="_blank">https://www.youtube.com/watch?v=-Z-17h3jG0A</a></li><li>Max Balandat talking about linear operators in PyTorch <a href="https://github.com/pytorch/pytorch/issues/28341" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/28341</a></li><li>At the end I talk a little bit about multiple dispatch; an earlier discussion about this topic is in this podcast <a href="https://pytorch-dev-podcast.simplecast.com/episodes/multiple-dispatch-in-torch-function" rel="nofollow" target="_blank">https://pytorch-dev-podcast.simplecast.com/episodes/multiple-dispatch-in-torch-function</a></li></ul>

Tensor subclasses and Liskov substitution principle

The PyTorch Developer Podcast is a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch.

In this episode I talk about reduced precision floating point formats float16 (aka half precision) and bfloat16. I'll discuss what floating point numbers are, how these two formats vary, and some of the practical considerations that arise when you are working with numeric code in PyTorch that also needs to work in reduced precision. Did you know that we do all CUDA computations in float32, even if the source tensors are stored as float16? Now you know!Further reading.The Wikipedia article on IEEE floating point is pretty great https://en.wikipedia.org/wiki/IEEE_754How bfloat16 works out when doing training https://arxiv.org/abs/1905.12322Definition of acc_type in PyTorch https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/AccumulateType.h

In this episode I talk about reduced precision floating point formats float16 (aka half precision) and bfloat16. I'll discuss what floating point numbers are, how these two formats vary, and some of the practical considerations that arise when you are working with numeric code in PyTorch that also needs to work in reduced precision. Did you know that we do all CUDA computations in float32, even if the source tensors are stored as float16? Now you know!Further reading.<ul><li>The Wikipedia article on IEEE floating point is pretty great <a href="https://en.wikipedia.org/wiki/IEEE_754" rel="nofollow" target="_blank">https://en.wikipedia.org/wiki/IEEE_754</a></li><li>How bfloat16 works out when doing training <a href="https://arxiv.org/abs/1905.12322" rel="nofollow" target="_blank">https://arxiv.org/abs/1905.12322</a></li><li>Definition of acc_type in PyTorch <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/AccumulateType.h" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/AccumulateType.h</a></li></ul>

Half precision

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers &gt; 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.Further reading.A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440Cinder Python https://github.com/facebookincubator/cinder

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers &gt; 0 causes memory leak (<a href="https://github.com/pytorch/pytorch/issues/13246" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/13246</a>). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.Further reading.<ul><li>A nice summary of the full issue <a href="https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662</a></li><li>DataLoader architecture RFC <a href="https://github.com/pytorch/pytorch/issues/49440" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/49440</a></li><li>Cinder Python <a href="https://github.com/facebookincubator/cinder" rel="nofollow" target="_blank">https://github.com/facebookincubator/cinder</a></li></ul>

DataLoader with multiple workers leaks memory

PyTorch operates on its input data in a batched manner, typically processing multiple batches of an input at once (rather than once at a time, as would be the case in typical programming). In this podcast, we talk a little about the implications of batching operations in this way, and then also about how PyTorch's API is structured for batching (hint: poorly) and how Numpy introduced a concept of ufunc/gufuncs to standardize over broadcasting and batching behavior. There is some overlap between this podcast and previous podcasts about TensorIterator and vmap; you may also be interested in those episodes.Further reading.ufuncs and gufuncs https://numpy.org/doc/stable/reference/ufuncs.html and https://numpy.org/doc/stable/reference/c-api/generalized-ufuncs.htmlA brief taxonomy of PyTorch operators by shape behavior http://blog.ezyang.com/2020/05/a-brief-taxonomy-of-pytorch-operators-by-shape-behavior/Related episodes on TensorIterator and vmap https://pytorch-dev-podcast.simplecast.com/episodes/tensoriterator and https://pytorch-dev-podcast.simplecast.com/episodes/vmap

PyTorch operates on its input data in a batched manner, typically processing multiple batches of an input at once (rather than once at a time, as would be the case in typical programming). In this podcast, we talk a little about the implications of batching operations in this way, and then also about how PyTorch's API is structured for batching (hint: poorly) and how Numpy introduced a concept of ufunc/gufuncs to standardize over broadcasting and batching behavior. There is some overlap between this podcast and previous podcasts about TensorIterator and vmap; you may also be interested in those episodes.Further reading.<ul><li>ufuncs and gufuncs <a href="https://numpy.org/doc/stable/reference/ufuncs.html" rel="nofollow" target="_blank">https://numpy.org/doc/stable/reference/ufuncs.html</a> and <a href="https://numpy.org/doc/stable/reference/c-api/generalized-ufuncs.html" rel="nofollow" target="_blank">https://numpy.org/doc/stable/reference/c-api/generalized-ufuncs.html</a></li><li>A brief taxonomy of PyTorch operators by shape behavior <a href="http://blog.ezyang.com/2020/05/a-brief-taxonomy-of-pytorch-operators-by-shape-behavior/" rel="nofollow" target="_blank">http://blog.ezyang.com/2020/05/a-brief-taxonomy-of-pytorch-operators-by-shape-behavior/</a></li><li>Related episodes on TensorIterator and vmap <a href="https://pytorch-dev-podcast.simplecast.com/episodes/tensoriterator" rel="nofollow" target="_blank">https://pytorch-dev-podcast.simplecast.com/episodes/tensoriterator</a> and <a href="https://pytorch-dev-podcast.simplecast.com/episodes/vmap" rel="nofollow" target="_blank">https://pytorch-dev-podcast.simplecast.com/episodes/vmap</a></li></ul>

Batching

Python is a single dispatch OO language, but there are some operations such as binary magic methods which implement a simple form of multiple dispatch. torch_function__ (through its Numpy predecessor __array_function) generalizes this mechanism so that invocations of torch.add with different subclasses work properly. This podcast describes how this mechanism works and how it can be used (in an unconventional way) to build composable subclasses ala JAX in functorch. Further reading: This podcast in written form https://dev-discuss.pytorch.org/t/functorch-levels-as-dynamically-allocated-classes/294Multiple dispatch resolution rules in the RFC https://github.com/pytorch/rfcs/blob/master/RFC-0001-torch-function-for-methods.md#process-followed-during-a-functionmethod-call

Python is a single dispatch OO language, but there are some operations such as binary magic methods which implement a simple form of multiple dispatch. torch_function__ (through its Numpy predecessor __array_function) generalizes this mechanism so that invocations of torch.add with different subclasses work properly. This podcast describes how this mechanism works and how it can be used (in an unconventional way) to build composable subclasses ala JAX in functorch. Further reading: <ul><li>This podcast in written form <a href="https://dev-discuss.pytorch.org/t/functorch-levels-as-dynamically-allocated-classes/294" rel="nofollow" target="_blank">https://dev-discuss.pytorch.org/t/functorch-levels-as-dynamically-allocated-classes/294</a></li><li>Multiple dispatch resolution rules in the RFC <a href="https://github.com/pytorch/rfcs/blob/master/RFC-0001-torch-function-for-methods.md#process-followed-during-a-functionmethod-call" rel="nofollow" target="_blank">https://github.com/pytorch/rfcs/blob/master/RFC-0001-torch-function-for-methods.md#process-followed-during-a-functionmethod-call</a></li></ul>

Multiple dispatch in __torch_function__

Writing multithreading code has always been a pain, and in PyTorch there are buckets and buckets of multithreading related issues you have to be aware about and deal with when writing code that makes use of it. We'll cover how you interface with multithreading in PyTorch, what goes into implementing those interfaces (thread pools!) and also some miscellaneous stuff like TLS, forks and data structure thread safety that is also relevant.Further reading:TorchScript CPU inference threading documentation https://github.com/pytorch/pytorch/blob/master/docs/source/notes/cpu_threading_torchscript_inference.rstc10 thread pool https://github.com/pytorch/pytorch/blob/master/c10/core/thread_pool.h and autograd thread pool https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cppTracking issue for TLS propagation across threads https://github.com/pytorch/pytorch/issues/28520

Writing multithreading code has always been a pain, and in PyTorch there are buckets and buckets of multithreading related issues you have to be aware about and deal with when writing code that makes use of it. We'll cover how you interface with multithreading in PyTorch, what goes into implementing those interfaces (thread pools!) and also some miscellaneous stuff like TLS, forks and data structure thread safety that is also relevant.Further reading:<ul><li>TorchScript CPU inference threading documentation <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/notes/cpu_threading_torchscript_inference.rst" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/blob/master/docs/source/notes/cpu_threading_torchscript_inference.rst</a></li><li>c10 thread pool <a href="https://github.com/pytorch/pytorch/blob/master/c10/core/thread_pool.h" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/blob/master/c10/core/thread_pool.h</a> and autograd thread pool <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cpp" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cpp</a></li><li>Tracking issue for TLS propagation across threads <a href="https://github.com/pytorch/pytorch/issues/28520" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/28520</a></li></ul>

Multithreading

CUDA is asynchronous, CPU is synchronous. Making them play well together can be one of the more thorny and easy to get wrong aspects of the PyTorch API. I talk about why non_blocking is difficult to use correctly, a hypothetical "asynchronous CPU" device which would help smooth over some of the API problems and also why it used to be difficult to implement async CPU (but it's not hard anymore!) At the end, I also briefly talk about how async/sync impedance can also show up in unusual places, namely the CUDA caching allocator.Further reading.CUDA semantics which discuss non_blocking somewhat https://pytorch.org/docs/stable/notes/cuda.htmlIssue requesting async cpu https://github.com/pytorch/pytorch/issues/44343

CUDA is asynchronous, CPU is synchronous. Making them play well together can be one of the more thorny and easy to get wrong aspects of the PyTorch API. I talk about why non_blocking is difficult to use correctly, a hypothetical "asynchronous CPU" device which would help smooth over some of the API problems and also why it used to be difficult to implement async CPU (but it's not hard anymore!) At the end, I also briefly talk about how async/sync impedance can also show up in unusual places, namely the CUDA caching allocator.Further reading.<ul><li>CUDA semantics which discuss non_blocking somewhat <a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="nofollow" target="_blank">https://pytorch.org/docs/stable/notes/cuda.html</a></li><li>Issue requesting async cpu <a href="https://github.com/pytorch/pytorch/issues/44343" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/44343</a></li></ul>

Asynchronous versus synchronous execution

We talk about gradcheck, the property based testing mechanism that we use to verify the correctness of analytic gradient formulas in PyTorch. I'll talk a bit about testing in general, property based testing and why gradcheck is a particularly useful property based test. There will be some calculus, although I've tried to keep the math mostly to intuitions and pointers on what to read up on elsewhere.Further reading.Gradcheck mechanics, a detailed mathematical explanation of how it works https://pytorch.org/docs/stable/notes/gradcheck.html In particular, it also explains how gradcheck extends to complex numbersJAX has a pretty good explanation about vjp and jvp at https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.htmlFast gradcheck tracking issue https://github.com/pytorch/pytorch/issues/53876

We talk about gradcheck, the property based testing mechanism that we use to verify the correctness of analytic gradient formulas in PyTorch. I'll talk a bit about testing in general, property based testing and why gradcheck is a particularly useful property based test. There will be some calculus, although I've tried to keep the math mostly to intuitions and pointers on what to read up on elsewhere.Further reading.<ul><li>Gradcheck mechanics, a detailed mathematical explanation of how it works <a href="https://pytorch.org/docs/stable/notes/gradcheck.html" rel="nofollow" target="_blank">https://pytorch.org/docs/stable/notes/gradcheck.html</a> In particular, it also explains how gradcheck extends to complex numbers</li><li>JAX has a pretty good explanation about vjp and jvp at <a href="https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html" rel="nofollow" target="_blank">https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html</a></li><li>Fast gradcheck tracking issue <a href="https://github.com/pytorch/pytorch/issues/53876" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/53876</a></li></ul>

gradcheck

torch.use_deterministic_algorithms lets you force PyTorch to use deterministic algorithms. It's very useful for debugging!There are some errors in the recording: the feature is called torch.use_deterministic_algorithms, and there is not actually a capability to warn (this was in an old version of the PR but taken out), we just error if you hit nondeterministic code.Docs: https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms

torch.use_deterministic_algorithms lets you force PyTorch to use deterministic algorithms. It's very useful for debugging!There are some errors in the recording: the feature is called torch.use_deterministic_algorithms, and there is not actually a capability to warn (this was in an old version of the PR but taken out), we just error if you hit nondeterministic code.Docs: <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms" rel="nofollow" target="_blank">https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms</a>

torch.use_deterministic_algorithms

Reference counting is a common memory management technique in C++ but PyTorch does its reference counting in a slightly idiosyncratic way using intrusive_ptr. We'll talk about why intrusive_ptr exists, the reason why refcount bumps are slow in C++ (but not in Python), what's up with const Tensor&amp; everywhere, why the const is a lie and how TensorRef lets you create a const Tensor&amp; from a TensorImpl* without needing to bump your reference count.Further reading.Why you shouldn't feel bad about passing tensor by reference https://dev-discuss.pytorch.org/t/we-shouldnt-feel-bad-about-passing-tensor-by-reference/85Const correctness in PyTorch https://github.com/zdevito/ATen/issues/27TensorRef RFC https://github.com/pytorch/rfcs/pull/16

Reference counting is a common memory management technique in C++ but PyTorch does its reference counting in a slightly idiosyncratic way using intrusive_ptr. We'll talk about why intrusive_ptr exists, the reason why refcount bumps are slow in C++ (but not in Python), what's up with const Tensor&amp; everywhere, why the const is a lie and how TensorRef lets you create a const Tensor&amp; from a TensorImpl* without needing to bump your reference count.Further reading.<ul><li>Why you shouldn't feel bad about passing tensor by reference <a href="https://dev-discuss.pytorch.org/t/we-shouldnt-feel-bad-about-passing-tensor-by-reference/85" rel="nofollow" target="_blank">https://dev-discuss.pytorch.org/t/we-shouldnt-feel-bad-about-passing-tensor-by-reference/85</a></li><li>Const correctness in PyTorch <a href="https://github.com/zdevito/ATen/issues/27" rel="nofollow" target="_blank">https://github.com/zdevito/ATen/issues/27</a></li><li>TensorRef RFC <a href="https://github.com/pytorch/rfcs/pull/16" rel="nofollow" target="_blank">https://github.com/pytorch/rfcs/pull/16</a></li></ul>

Reference counting

Memory layout specifies how the logical multi-dimensional tensor maps its elements onto physical linear memory. Some layouts admit more efficient implementations, e.g., NCHW versus NHWC. Memory layout makes use of striding to allow users to conveniently represent their tensors with different physical layouts without having to explicitly tell every operator what to do.Further reading.Tutorial https://pytorch.org/tutorials/intermediate/memory_format_tutorial.htmlMemory format RFC https://github.com/pytorch/pytorch/issues/19092Layout permutation proposal (not implemented) https://github.com/pytorch/pytorch/issues/32078

Memory layout specifies how the logical multi-dimensional tensor maps its elements onto physical linear memory. Some layouts admit more efficient implementations, e.g., NCHW versus NHWC. Memory layout makes use of striding to allow users to conveniently represent their tensors with different physical layouts without having to explicitly tell every operator what to do.Further reading.<ul><li>Tutorial <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="nofollow" target="_blank">https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html</a></li><li>Memory format RFC <a href="https://github.com/pytorch/pytorch/issues/19092" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/19092</a></li><li>Layout permutation proposal (not implemented) <a href="https://github.com/pytorch/pytorch/issues/32078" rel="nofollow" target="_blank">https://github.com/pytorch/pytorch/issues/32078</a></li></ul>

Memory layout

PyTorch Developer Podcast

The best podcast episodes on Startups. For anyone working in a startup, or anyone interested in the incredible stories behind how people built companies like Instagram, and Facebook. Offering both practical advice and insights, as well as the stories for how many of the biggest startups got to where they are today. Add episodes you're interested into your queue below!

OwlTail

Best Startups Podcasts

The best philosophy podcast episodes. Asking questions about how to live a better life, what it means to be and do good, to the idea of God. This topic includes episodes that bring about important questions that everyone should be asking themselves. Add episodes you're interested into your queue below!