I went down this rabbit hole not so long ago too.
There was a discussion open on containerd's GitHub on removing the dependency on the pause image but it has been closed as won't fix:
https://github.com/containerd/containerd/issues/10505
Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration:
https://github.com/kubernetes/kubeadm/issues/2020