HN Reader

ETH Zurich and EPFL to release a LLM developed on public infrastructure

361

I hope they do well. AFAIK they’re training or finetuning an older LLaMA model, so performance might lag behind SOTA. But what really matters is that ETH and EPFL get hands-on experience training at scale. From what I’ve heard, the new AI cluster still has teething problems. A lot of people underestimate how tough it is to train models at this scale, especially on your own infra.

Disclaimer: I’m Swiss and studied at ETH. We’ve got the brainpower, but not much large-scale training experience yet. And IMHO, a lot of the “magic” in LLMs is infrastructure-driven.

11 hours agoby isusmelj

"respecting web crawling opt-outs during data acquisition produces virtually no performance degradation"

Great to read that!

12 hours agoby k__

Is this setting the bar for dataset transparency? It seems like a significant step forward. Assuming it works out, that is.

They missed an opportunity though. They should have called their machine the AIps (AI Petaflops Supercomputer).

11 hours agoby bee_rider

The open training data is a huge differentiator. Is this the first truly open dataset of this scale? Prior efforts like The Pile were valuable, but had limitations. Curious to see how reproducible the training is.

11 hours agoby WeirderScience

Yeah, that's what "democratizing AI" means.

9 hours agoby amelius

The press release talks a lot about how it was done, but very little about how capabilities compare to other open models.

11 hours agoby oytis

The article says

“ Open LLMs are increasingly viewed as credible alternatives to commercial systems, most of which are developed behind closed doors in the United States or China”

It is obvious that the companies producing big LLMs today have the incentive to try to enshitify them. Trying to get subscriptions at the same time as trying to do product placement ads etc. Worse, some already have political biases they promote.

It would be wonderful if a partnership between academia and government in Europe can do a public good search and AI that endeavours to serve the user over the company.

11 hours agoby wood_spirit

Pretty proud to see this at the top of HN as a Swiss (and I know many are lurking here!). These two universities produce world-class founders, researchers, and engineers. Yet, we always stay in the shadow of the US. With our top-tier public infrastructure, education, and political stability (+ neutrality), we have a unqiue opportunity to build something exceptional in the open LLM space.

11 hours agoby hubraumhugo

Looking forward to proof test it.

12 hours agoby Bengalilol

gross use of public infrastructure

7 hours agoby nektro

Use case for science and code LLMs: Superhydrodynamic gravity (SQR / SQG, )

LLMs do seem to favor general relativity but probably would've favored classical mechanics at the time given the training corpora.

Not-yet unified: Quantum gravity, QFT, "A unified model must: " https://news.ycombinator.com/item?id=44289148

Will be interested to see how this model responds to currently unresolvable issues in physics. Is it an open or a closed world mentality and/or a conditioned disclaimer which encourages progress?

What are the current benchmarks?

From https://news.ycombinator.com/item?id=42899805 re: "Large Language Models for Mathematicians" (2023) :

> Benchmarks for math and physics LLMs: FrontierMath, TheoremQA, Multi SWE-bench: https://news.ycombinator.com/item?id=42097683

Multi-SWE-bench: A Multi-Lingual and Multi-Modal GitHub Issue Resolving Benchmark: https://multi-swe-bench.github.io/

Add'l LLM benchmarks and awesome lists: https://news.ycombinator.com/item?id=44485226

Microsoft has a new datacenter that you don't have to keep adding water to; which spares the aquifers.

How to use this LLM to solve energy and sustainability problems all LLMs exacerbate? Solutions for the Global Goals, hopefully

5 hours agoby westurner

Why would you announce this without a release? Be honest.

11 hours agoby greenavocado