HN Reader

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra.

In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class was winding up, he joked and said "just kidding, don't do that... here's how we can do this with a Taylor expansion", and proceeded to derive the same identity in what felt like 30 seconds.

Also, don't forget the Jacobian and gradient aren't the same thing!

> The class involved numerous example numerical computations using the Julia language, which you can install on your own computer following these instructions. The material for this class is also located on GitHub at https://github.com/mitmath/matrixcalc

Those looking for a shorter primer could consult https://arxiv.org/abs/1802.01528

3b1b classic on this topic with beautiful visualizations: https://youtu.be/O85OWBJ2ayo

Great course. I highly recommended anyone interested in this topic to check it out on the MIT website, taught by the same authors. They are great lecturers.

The Matrix Cookbook [1] can be handy when learning this topic.

https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

I have skimmed it, and it looks very good. It is actually not solely about matrix calculus, but shows a practical approach to differentiation in different vector spaces with many examples and intuitions.

What does calculus mean ?

wait what - another math textbook recommendation by academicians. ML and MLL are arts of tinkering not academic subjects.

Though Steven Johnson is the real deal and writes lots of code, Edelman is a shyster/imposter who used to ride the coattails of G. Strang and now shills for Julia where he makes most of his money. You don't need, and won't understand ML/LLM by reading textbooks.

1. If you want to have a little fun with ML/LLM, fire up Google Collab and run one of tutorials on the web - Karpathy, Hugging Face or PyTorch examples.

2. If you don't want to do, but just read for fun, Howard & Parr's essay as recommended by someone else here is much shorter and more succinct. https://explained.ai/matrix-calculus/ this link renders better

3. If you insist on academic textbooks, Boyd & Vandenberghe skips calculus and has more applications (engineering). Unfortunately, code examples are in Julia! https://web.stanford.edu/~boyd/vmls/vmls.pdf https://web.stanford.edu/~boyd/vmls/. link to python version

4. If you Want to become a tensor & differential programming ninja, learn Jax, XLA https://docs.jax.dev/en/latest/quickstart.html https://colab.research.google.com/github/exoplanet-dev/jaxop...