Depends on the definition of "effective", of course. They're certainly more prolific, but "provably satisfies a specification" (from the first line of the Wikipedia page) is kind of the opposite of what they do.
As someone who worked in the field, I think there were 3 big directions in the last ~20 years that remain relevant:
* Experiments in interface design, the equivalent of language design. These balance being able to guide the system and being able to maintain code written this way. Cleverness here ended up leading to a lot of gains in the more practical systems, and IMO, is one of the more important problems in scaling LLM tools to be a bigger part of how teams build software.
* Optimized search procedures. Ex: Concolic execution and counter-example guided execution, that are the outer loop and tricks over whatever per-step model (SMT, LLM) is used. Think algorithms behind optimal agents that go beyond "we run lint, feed it into the LLM, and loop until the output is good."
* How to discharge useful fragments to SMT solvers and other technologies. Not used much publicly nor discussed, but we already find value here, such as generating database queries.
The last 10 years has seen a move to combining neural methods with symbolic, so I view them as complementary, and so should most folks doing agent-based LLM work (read: most). It was frustrating to me that the ML methods (largely bayesian) at the time I was attempting failed so our group found more mileage in sticking to the symbolic side. Likewise, even LLMs aren't the be-all for AI methods -- there's a lot of cool recommender work we do that is outside of the LLM.
Nowadays, we're building louie.ai to finally do both sides of the equation. Our focus is less on SMT and more on search + LLM, but we've already found cases where combining LLMs with classic PL approaches helped some of our query synthesis problems. If folks are interested in opportunities here, the graphistry team has open roles, immediately on a security principle researcher / senior SA, and we expect a senior backend/infra SE soon as well.
Smaug123|1 year ago
lmeyerov|1 year ago
* Experiments in interface design, the equivalent of language design. These balance being able to guide the system and being able to maintain code written this way. Cleverness here ended up leading to a lot of gains in the more practical systems, and IMO, is one of the more important problems in scaling LLM tools to be a bigger part of how teams build software.
* Optimized search procedures. Ex: Concolic execution and counter-example guided execution, that are the outer loop and tricks over whatever per-step model (SMT, LLM) is used. Think algorithms behind optimal agents that go beyond "we run lint, feed it into the LLM, and loop until the output is good."
* How to discharge useful fragments to SMT solvers and other technologies. Not used much publicly nor discussed, but we already find value here, such as generating database queries.
The last 10 years has seen a move to combining neural methods with symbolic, so I view them as complementary, and so should most folks doing agent-based LLM work (read: most). It was frustrating to me that the ML methods (largely bayesian) at the time I was attempting failed so our group found more mileage in sticking to the symbolic side. Likewise, even LLMs aren't the be-all for AI methods -- there's a lot of cool recommender work we do that is outside of the LLM.
Nowadays, we're building louie.ai to finally do both sides of the equation. Our focus is less on SMT and more on search + LLM, but we've already found cases where combining LLMs with classic PL approaches helped some of our query synthesis problems. If folks are interested in opportunities here, the graphistry team has open roles, immediately on a security principle researcher / senior SA, and we expect a senior backend/infra SE soon as well.