HN Reader

Pure vision will never be enough because it does not contain information about the physical feedback like pressure and touch, or the strength required to perform a task.

For example, so that you don't crush a human when doing massage (but still need to press hard), or apply the right amount of force (and finesse?) to skin a fish fillet without cutting the skin itself.

Practically in the near term, it's hard to sample from failure examples with videos on Youtube, such as when food spills out of the pot accidentally. Studying simple tasks through the happy path makes it hard to get the robot to figure out how to do something until it succeeds, which can appear even in relatively simple jobs like shuffling garbage.

With that said, I suppose a robot can be made to practice in real life after learning something from vision.

Extremely oversold article.

> the core insight: predict in representation space, not pixels

We've been doing this since 2014? Not only that, others have been doing it at a similar scale. e.g. Nvidia's world foundation models (although those are generative).

> zero-shot generalization (aka the money shot)

This is easily beaten by flow-matching imitation learning models like what Pi has.

> accidentally solved robotics

They're doing 65% success on very simple tasks.

The research is good. This article however misses a lot of other work in the literature. I would recommend you don't read it as an authoritative source.

This article contains so many falsehoods and history rewrites that it's pretty painful to read.

This was a bit hard to read. It would be good to have a narrative structure and more clear explanation of concepts.

IMO, VideoMimic is a better proof-of-concept

https://www.videomimic.net/

https://www.videomimic.net/page1.html

My mom said I was throwing away my life watching YouTube all day and clearly I just haven’t been watching YouTube enough. 1 million YouTube videos here I come!

Solving robotics is some claim.

>camera pose sensitivity

>the model is basically a diva about camera positioning. move the camera 10 degrees and suddenly it thinks left is right and up is down.

Reminds that several years ago Tesla had to finally start to explicitly extract 3D model from the net. Similarly i expect here that it would get pipelined - one model extracts/builds 3D, and the other is actually the "robot" working in that 3D. Each one can be alone trained much better and efficiently, with much better transfer and generalization, than the large monolithic model working from the 2D video. In pipeline approach, it is very easy to generate synthetic input 3D data better covering interesting scenario space for the "robot" model.

And, for example, you can't just, without significant training, feed the large monolithic model a lidar point space instead of videos. Whereis in a pipelined approach, you just switch the 3D generating pipeline input model.

I do not know and do not care much about robotics per se, but I wish LLM's were better with spatial reasoning. If the new insight helps with that - great!

I dabbled a bit in geolocation with LLM's recently. It is still surprising to me how good they are with finding the general area a picture was taken. Give it a photo of a random street corner on this earth and it is likely will not only tell you the correct city or town but most often even the correct quarter.

On the other hand, if you ask it for a birds eye view of a green, a brown and a white house on the north side of a one-way street (running west to east) east of an intersection running north to south, it may or may not get it right. If you want it to add an arrow going in the direction of the one-way street, it certainly has no clue at all and the result is 50/50.

Solved??? Where?

Someone watched 'Devs' ?

if you havent - highly recommended.

Friendly unit conversion man at your service: 114 years.

> right now, you need to show the robot pictures of what you want it to do. want it to "clean the kitchen"? better have a photo of a clean kitchen handy.

What about using Flux Kontext (or Controlnets) to turn the messy kitchen into a clean kitchen?

So video gen models basically can be extrapolated to control robotics ? How long until Veo3 robots take over?

Put tiny cams on robot arms and let it control them. They can be flimsy for safety. If it is sure something is happening say nothing, if it is 70-99% sure have it guess what is going on, if <70% have it ask what is going on.

i thought all the cool data driven robotics stuff was like reinforcement learning from sensors that track moving effectors in the real world with online retraining that mimics the sensorimotor experimentation that is observed during the developmental phases of real neurobiological systems?

so you just kinda let it run for a while and it bumps and squirms around until it stands up or whatever.

seems also the future for real ai?

I don't know. I'm not the expert, but if you've ever tried to a backflip or anything where your toes are above your head, then you'll know that spatial awareness goes well beyond vision. Or if you throw a frisbee for the dog to catch, they don't actually look at it while running; they look, predict position, then move in. Veni, vidi, vici. So any model that "learns physics" just through vision seems flawed from the start. What's your thought there?

I wonder how much language does this model understand. If we pan across text will it fill in sensible next word? How good will it be?

I just wrote a reply to a comment talking about the AI tells this writing has, but it got flagged so my comment disappeared when I hit post. I'll rephrase out of spite:

My first thought upon reading this was that an LLM had been instructed to add a pithy meme joke to each paragraph. They don't make sense in context, and while some terminally online people do speak in memes, those people aren't quoting doge in 2025.

There's also a sense of incoherence in the whole piece. For instance, this section:

"- after: 22 million videos + 1 million images (now we're talking)

they basically hoovered up everything: something-something v2, kinetics, howto100m, and a billion youtube videos"

Was it a billion vids or 22m? It turns out the latter sentence is just rephrasing the list of sources in a cool casual way, and the last one is called YT-Temporal-1B. That's a billion frames of video, not a billion videos.

"why didn't we think of this sooner?", asks the article. Not sure who the "we" is supposed to be, but the robotics community has definitely thought of this before. https://robo-affordances.github.io/ from 2023 is one pretty relevant example that comes to mind, but I have recollections of similar ideas going back to at least 2016 or so (many of which are cited in the V-JEPA2 paper). If you think data-driven approaches are a good idea for manipulation, then the idea of trying to use Youtube as a source of data (an extremely popular data source in computer vision for the past decade) isn't exactly a huge leap. Of course, the "how" is the hard part, for all sorts of reasons. And the "how" is what makes this paper (and prior research in the area) interesting.

I was unable to make through the article (now we're talking).

It seems that in order for robotics and automation to work properly, AI models including LLMs, YOLO, RL and others need helps from their distant cousins namely logic, optimization and constraint programming that can be attributed as intelligent automation or namely IA [1],[2],[3],[4].

[1] Logic, Optimization, and Constraint Programming: A Fruitful Collaboration - John Hooker - CMU (2023) [video]:

https://www.youtube.com/live/TknN8fCQvRk

[2] "We Really Don't Know How to Compute!" - Gerald Sussman - MIT (2011) [video]:

https://youtube.com/watch?v=HB5TrK7A4pI

[3] Google OR-Tools:

https://developers.google.com/optimization

[4] MiniZinc:

https://www.minizinc.org/

This is interesting for generalized problems ("make me a sandwich") but not useful for most real world functions ("perform x within y space at z cost/speed"). I think the number of people on the humanoid bandwagon trying to implement generalized applications is staggering right now. The physics tells you they will never be as fast as purpose-built devices, nor as small, nor as cheap. That's not to say there's zero value there, but really we're - uh - grasping at straws...

No you didn't, and I don't even need to click on the link to know it

I didn't understand a single word about this post and what was supposed to be solved and had to stop reading.

Was this actually written by a human being? If so, the author(s) suffer from severe language communication problems. Doesn't seem to be grounded at least with reality and my personal experience with robotics. But here's my real world take:

Robotics is going to be partially solved when ROS/ROS2 becomes effectively exterminated and completely replaced by a sane robotics framework.

I seriously urge the authors to use ROS/ROS2. Show us, implementing your solution with ROS, pushing it to a repository and allow others to verify what you solved, maybe?. Suffer a bit with the framework and then write a real post about real robotics hands-on, and not just wander on fancy uncomprehensible stuff that probably no-one will ever do.

Then we can maybe start talking about robotics.

Does YouTube allow massive scraping like this in their ToS?

Dr Fei-Fei Li talks about this as the LWM (Large World Model) during this interview: https://youtu.be/fQGu016AlVo and with https://www.worldlabs.ai/