HN Reader

thank God, Pytorch gained so much momentum before this came out, Now we have a true platform independent semi standard For parallel computations. We are not stuck with NVIDIA specifics.

It's great that parts of pie torch which concern the NVIDIA backend can now be implemented in Python directly, The important part that it doesn't really matter or shouldn't matter for end users / Developers

that being said, maybe this new platform will extend the whole concept of on GPU computation via Python to even more domains like maybe games.

Imagine running rust the Game performantly mainly on the GPU via Python

Very curious how this compares to JAX [1].

JAX lets you write Python code that executes on Nvidia, but also GPUs of other brands (support varies). It similarly has drop-in replacements for NumPy functions.

This only supports Nvidia. But can it do things JAX can't? It is easier to use? Is it less fixed-size-array-oriented? Is it worth locking yourself into one brand of GPU?

[1] https://github.com/jax-ml/jax

This is huge. Anyone who was considering AMD + ROCm as an alternative to NVIDIA in the AI space isn’t anymore.

I’m one of those people who can’t (won’t) learn C++ to the extent required to effectively write code for GPU execution…. But to have a direct pipeline to the GPU via Python. Wow.

The efficiency implications are huge, not just for Python libraries like PyTorch, but also anything we write that runs on an NVIDIA GPU.

I love seeing anything that improves efficiency because we are constantly hearing about how many nuclear power plants OpenAI and Google are going to need to power all their GPUs.

Rust support next? RN I am manually [de]serializing my data structures as byte arrays to/from the kernels. It would be nice to have truly shared data structures like CUDA gives you in C++!

Python is really shaping up to be the lingua franca of programming languages. Its adoption is soaring in this FOSS renaissance and I think it's the closest thing to a golden hammer that we've ever had.

The PEP model is a good vehicle for self-improvement and standardization. Packaging and deployment will soon be solved problems thanks to projects such as uv and BeeWare, and I'm confident that we're going to see continued performance improvements year over year.

Is NVIDIA's JIT-based approach here similar JAX's, except targeting CUDA directly rather than XLA? Would like to know how these different JIT compilers relate to one another.

Technically speaking, all of this exists (including the existing library integration and whatnot) through CuPy and Numba already, but the fact that it’s getting official support is cool.

Doesn't this mean AMD could make the same python API that targets their hardware, and now Nvidia GPUs aren't as sticky?

Will be interesting to compare the API to what Modular has [1] with Mojo.

1. https://docs.modular.com/mojo/stdlib/gpu/

Here is the repo: https://github.com/NVIDIA/cuda-python

Kind of late to the game, but can anyone recommend a good primer on GPU programming?

Hmm wonder what vulnerabilities will now be available with this addition.

Source?

See also: https://tinygrad.org/

Reverse-engineered python-only GPU API, works with not only CUDA but Also AMD's ROCm

Other runtimes: https://docs.tinygrad.org/runtime/#runtimes

>In 2024, Python became the most popular programming language in the world — overtaking JavaScript — according to GitHub’s 2024 open source survey.

I wonder why Python take over the world? Of course, it's easy to learn, it might be easy to read and understand. But it also has a few downsides: low performance, single threaded, lack of static typing.

python is the winner, turning pseudo code into interesting stuff

it's only the beginning, there is no need to create new programming languages anymore

Could pandas benefit from this integration?

We should find a new word since GPU is is from back when it was used for graphics.

CuTile, in many ways, feels like a successor to OpenAI's Triton... And not only are we getting tile/block-level primitives and TileIR, but also a proper SIMT programming model in CuPy, which I don't think enough people noticed even at this year's GTC. Very cool stuff!

That said, there were almost no announcements or talks related to CPUs, despite the Grace CPUs being announced quite some time ago. It doesn't feel like we're going to see generalizable abstractions that work seamlessly across Nvidia CPUs and GPUs anytime soon. For someone working on parallel algorithms daily, this is an issue: debugging with NSight and CUDA-GDB still isn't the same as raw GDB, and it's much easier to design algorithms on CPUs first and then port them to GPUs.

Of all the teams in the compiler space, Modular seems to be among the few that aren't entirely consumed by the LLM craze, actively building abstractions and languages spanning multiple platforms. Given the landscape, that's increasingly valuable. I'd love to see more people experimenting with Mojo — perhaps it can finally bridge the CPU-GPU gap that many of us face daily!

I'm no GPU programmer, but seems easy to use even for someone like me. I pulled together a quick demo of using the GPU vs the CPU, based on what I could find (https://gist.github.com/victorb/452a55dbcf59b3cbf84efd8c3097...) which gave these results (after downloading 2.6GB of dependencies of course):

    Creating 100 random matrices of size 5000x5000 on CPU...
    Adding matrices using CPU...
    CPU matrix addition completed in 0.6541 seconds
    CPU result matrix shape: (5000, 5000)
    
    Creating 100 random matrices of size 5000x5000 on GPU...
    Adding matrices using GPU...
    GPU matrix addition completed in 0.1480 seconds
    GPU result matrix shape: (5000, 5000)

Definitely worth digging into more, as the API is really simple to use, at least for basic things like these. CUDA programming seems like a big chore without something higher level like this.

This will probably lead to what, I think, python has led to in general: A lot more things tried quicker and targeted things that stay in a faster language. All in all this is a great move. I am looking forward to playing with it for sure.

What makes Python such a target for these kind of things?

I've noticed alot of projects add Python support like this. Does the Python codebase allow for it to compile down to different targets easier than others?

The GTC 2025 announcement session that's mentioned in this article has video here: https://www.nvidia.com/en-us/on-demand/session/gtc25-s72383/

It's a holistic approach to all levels of the stack, from high-level frameworks to low-level bindings, some of which is highlighting existing libraries, and some of which are completely newly announced.

One of the big things seems to be a brand new Tile IR, at the level of PTX and supported with a driver level JIT compiler, and designed for Python-first semantics via a new cuTile library.

https://x.com/JokerEph/status/1902758983116657112 (without login: https://xcancel.com/JokerEph/status/1902758983116657112 )

Example of proposed syntax: https://pbs.twimg.com/media/GmWqYiXa8AAdrl3?format=jpg&name=...

Really exciting stuff, though with the new IR it further widens the gap that projects like https://github.com/vosen/ZLUDA and AMD's own tooling are trying to bridge. But vendor lock-in isn't something we can complain about when it arises from the vendor continuing to push the boundaries of developer experience.

Slightly related, I had a go at doing llama 3 inference in luajit using cuda as one compute backend for just doing matrix multiplication

https://github.com/CapsAdmin/luajit-llama3/blob/main/compute...

While obviously not complete, it was less than I thought was needed.

It was a bit annoying trying to figure out which version of the function (_v2 suffix) I have to use for which driver I was running.

Also sometimes a bit annoying is the stateful nature of the api. Very similar to opengl. Hard to debug at times as to why something refuse to compile.

Pytorch???

  CUDA was born from C and C++

It would be nice if they actually implemented a C variant of CUDA instead of extending C++ and calling it CUDA C.