WingNews logo WingNews
top | item 47133261

(no title)

danpalmer | 5 days ago

LLMs using code to answer questions is nothing new, it's why the "how many Rs in strawberry" question doesn't trip them up anymore, because they can write a few lines of Python to answer it, run that, and return the answer.

Mathematica / Wolfram Language as the basis for this isn't bad (it's arguably late), because it's a highly integrated system with, in theory, a lot of consistency. It should work well.

That said, has it been designed for sandboxing? A core requirement of this "CAG" is sandboxing requirements. Python isn't great for that, but it's possible due to the significant effort put in by many over years. Does Wolfram Language have that same level? As it's proprietary, it's at a disadvantage, as any sandboxing technology would have to be developed by Wolfram Research, not the community.

discuss

order

adius|5 days ago

I also think that sandboxing is crucial. That’s why I’m working on a Wolfram Language interpreter that can be run fully sandboxed via WebAssembly: https://github.com/ad-si/Woxi

danpalmer|5 days ago

Awesome. I'm pretty unfamiliar with the Wolfram Language, but my understanding that the power of it came from the fact it was very batteries-included in terms of standard library and even data connections (like historical weather or stock market data).

What exactly does Woxi implement? Is it an open source implementation of the core language? Do you have to bring your own standard library or can you use the proprietary one? How do data connections fit into the sandboxing?

I realise I may be uninformed enough here that some of these might not make sense though, interested to learn.

Someone|5 days ago

> it's why the "how many Rs in strawberry" question doesn't trip them up anymore, because they can write a few lines of Python to answer it, run that, and return the answer.

That still requires the LLM to ‘decide’ that consulting Python to answer that question is a good idea, and for it to generate the correct code to answer it.

Questions similar to ”how many Rs in strawberry" nowadays likely are in their training set, so they are unlikely to make mistakes there, but it may be still be problematic for other questions.

simianwords|5 days ago

>LLMs using code to answer questions is nothing new, it's why the "how many Rs in strawberry" question doesn't trip them up anymore, because they can write a few lines of Python to answer it, run that, and return the answer.

False. It has nothing to do with tool use but just reasoning.

FrustratedMonky|5 days ago

What is reasoning?

I also can not multiply large numbers without a paper and pencil, and following an algorithm learned in school.

That is the same as an LLM running some python, is the same as me following instructions to perform multiplication.