[asboans]

Firstly, automobiles are really impressive.

Second, with that out the way, these cars are not playing the same game as horses… first, and quite obviously they have massive amounts of horsepower, which is kind of like giving a team of horses… many more horses. But also cars have an absolutely massive fuel capacity. Petrol is such an efficient store of chemical energy compared to hay and cars can store gallons of it.

I think if you give my horse the ability of 300 horses and fed it pure gasoline, I would be kind of embarrassed if it wasn’t able to win a horse race.

[furyofantares]

Yeah man, and it would be wild to publish an article titled "Ford Mustang and Honda Civic win gold in the 100 meter dash at the Olympics" if what happened was the companies drove their cars 100 meters and tweeted that they did it faster than the Olympians had run.

Actually that's too generous, because the humans are given a time limit in ICPC, and there's no clear mapping to say how the LLM's compute should be limited to make a comparison.

It IS an interesting result to see how models can do on these tests - and it's also a garbage headline.

[krisoft]

> what happened was the companies drove their cars 100 meters and tweeted that they did it faster than the Olympians had run

That would be indeed an interesting race around the time cars were invented. Today that would be silly, since everyone knows what cars are capable of, but back then one can imagine a lot more skepticism.

Just as there is a ton of skepticism today of what LLMs can achieve. A competition like this clearly demonstrates where the tech is, and what is possible.

> there's no clear mapping to say how the LLM's compute should be limited to make a comparison

There is a very clear mapping of course. You give the same wall clock time to the computer you gave to the humans.

Because what it is showing is that the computer can do the same thing a human can under the same conditions. With your analogy here they are showing that there is such a thing as a car and it can travel 100 meters.

Once it is a foregone conclusion that an LLM can solve the ICPC problems and that question has been sufficiently driven home to everyone who cares we can ask further ones. Like “how much faster can it solve the problems compared to the best humans” or “how much energy it consumes while solving them”? It sounds like you went beyond the first question and already asking these follow up questions.

[in-silico]

Cars going faster than humans or horses isn't very interesting these days, but it was 100+ years ago when cars were first coming on the scene.

We are at that point now with AI, so a more fitting headline analogy would be "In a world first, automobile finishes with gold-winning time in horse race".

Headlines like those were a sign that cars would eventually replace horses in most use-cases, so the fact that we could be in the the same place now with AI and humans is a big deal.

[hnfong]

All the while with skeptics snarkily commenting "Cars can move fast, but they can't really run like a human!"

[LPisGood]

I think your analogy is interesting but it falls apart because “moving fast” is not something we consider uniquely human, but “solving hard abstract problems” is

[apstls]

This metaphor drops some pretty key definitional context. If the common belief prior to this race was that cars could not beat horses, maybe someday but not today, then the article is completely reasonable, even warranted.

[Swizec]

> Firstly, automobiles are really impressive. Second, with that out the way, these cars are not playing the same game as horses

Yes. That’s why cars don’t compete in equestrian events and horses don’t go to F1 races.

This non-controversial surely? You want different events for humans, humans + computers, and just computers.

Notice that self driving cars have separate race events from both horses and human-driven cars.

[in-silico]

The point is that up until now, humans were the best at these competitions, just like horses were the best at racing up until cars came around.

The other commenter is pointing out how ridiculous it would be for someone to downplay the performance of cars because they did it differently from horses. It doesn't matter if they did it using different methods, that fact that the final outcome was better had world-changing ramifications.

The same applies here. Downplaying AI because it has different strengths or plays by different rules is foolish, because that doesn't matter in the real world. People will choose the option that that leads to the better/faster/cheaper outcome, and that option is quickly becoming AI instead of humans - just like cars quickly became the preferred option over horses. And that is crazy to think about.

[gxs]

Yeah I think the only thing OP was passing judgement on is on the competition aspect of it, not the actual achievement of any human or non human participant

That’s how I read it at least - exactly how you put it

[throw310822]

I think you missed that the whole point of this race was:

"did we build a vehicle faster than a horse, yes/no?"

Which matters a lot when horses are the fastest land vehicle available. (We're so used to thinking of horses as a quaint and slow mean of transport that maybe we don't realize that for millennia they've been the fastest possible way to get from one place to another.)

[lbrandy]

I was struck how the argument is also isomorphic to how we talked about computers and chess. We're at the stage where we are arguing the computer isn't _really_ understanding chess, though. It's just doing huge amounts of dumb computation with huge amounts of opening book and end tables and no real understanding, strategy or sense of whats going on.

Even though all the criticism were, in a sense, valid, in the end none of it amounted to a serious challenge to getting good at the task at hand.

[LaffertyDev]

I don’t think you’ll find many race tracks that permit horses and cars to compete together.

(I did enjoy the sarcasm, though!)

[j_timberlake]

This response is good but the more general problem is that people are in "It doesn't look like anything to me" mode like Westworld robots seeing advanced technology. If there's a way to snap people out of that, I've never seen it.

[GoatInGrey]

Snark aside, I would expect a car partaking in a horse race to beat all of the horses. Not because it's a better horse, but because it's something else altogether.

Ergo, it's impressive with nuance. As the other commenter said.

[rich_sasha]

There's a difference. How much money went into training the computer here Vs the human? If you want to prove that a computer can, at extreme cost and effort, beat a human - sure, it's possible.

But you can also conclude that putting a lot of money and effort pays off. It's more like comparing a horse to a Ferrari that had millions of development costs, has a team of engineers maintaining it, isn't reusable, and just about beats Chestnut. It's a long way until the utility of both is matched.

[melenaboija]

Comparing power with reasoning does not make any sense at all.

Humans have surpassed their own strength since the invention of the lever thousands of years ago. Since then, it has been a matter of finding power sources millions of times greater such as nuclear energy

[LunaSea]

Power is one thing, efficiency is another.

Humans are more efficient watt for watt than any AI ever invented.

Now if you were to limit AIs to 400 watts we could probably thinks it's fair.

[matheusd]

> Humans are more efficient watt for watt than any AI ever invented.

Indeed they are. For now. The long term trend is not in our favor.

[Gud]

Your analogy is flawed.

Are the humans allowed to bring their laptops and use the internet? Or a downloaded copy?

[bgwalter]

The massive amounts of compute power is not the major issue. The major issue is unlimited amount of reference material.

If a human can look up similar previous problems just as the "AI" can, it is a huge advantage.

Syzygy tables in chess engines are a similar issue. They allow perfect play, and there is no reason why a computer gets them and a human does not (if you compare humans against chess engines). Humans have always worked with reference material for serious work.

[chpatrick]

Humans are allowed to look up and learn from as many previous problems as they want before the competition. The AI is also trained on many previous problems before the competition. What's the difference?