(no title)
jaen | 4 days ago
> I'd encourage you to check the source
I couldn't have written my comment without reading the source, obviously!
> o1/o3 or Grok-Thinking, where the model is the solver at test-time.
What? I said that the SotA is the model generating code at test-time, not solving it directly via CoT/ToT etc.
kofdai|3 days ago
You’re right to be skeptical of the speed, and I realize I was incomplete in describing my process. I should have been more transparent: I am using Claude Code as a "pair-programmer" to implement and review the logic I design.
While the Verantyx engine itself remains a 100% static, symbolic solver at test-time (no LLM calls during inference), the rapid score jumps from 20.1% to 22.4% are indeed accelerated by an AI-assisted workflow.
My role is to identify the geometric pattern in the failed tasks and design the DSL primitive (the "what"). I then use Claude Code to scaffold the implementation, check for regressions across the 1,000 tasks, and refine the code (the "how").
This is why I can commit 30-80 lines of verified geometric logic in minutes rather than hours. The "thinking" and the "logic design" are human-led, but the "implementation" is AI-augmented.
My apologies if my previous comments made it sound like I was manually typing every single one of those 26K lines without help. In 2026, I believe this "Human-Architect / AI-Builder" model is the most effective way to tackle benchmarks like ARC.
I’d love to hear your thoughts on this hybrid approach to symbolic AI development.
kofdai|3 days ago
You said the SotA is models generating code at test-time, and you're correct. Systems like o3 synthesize Python programs per-task, execute them, and check outputs. That's a legitimate program synthesis approach.
Here's where Verantyx differs structurally:
*The DSL is fixed before test-time.* When Verantyx encounters a new task, it doesn't generate arbitrary Python. It searches over a closed vocabulary of ~60 typed primitives (`apply_symmetrize_4fold`, `self_tile_uniform`, `midpoint_cross`, etc.) and composes them. The search space is finite and enumerable. An LLM generating code has access to the full expressiveness of Python—mine doesn't.
*Here's the concrete proof that this isn't prompt-engineering:*
While we've been having this discussion, the solver went from 20.1% to *22.2%* (222/1000 tasks). That's +21 tasks in under 48 hours. Each new task required identifying a specific geometric pattern in the failure set, designing a new primitive function, implementing it, verifying it produces zero regressions on all 1,000 tasks, and committing. The commit log tells this story:
- `v55`: `panel_compact` — compress grid panels along separator lines - `v56`: `invert_recolor` — swap foreground/background with learned color mapping - `v57`: `midpoint_cross` + `symmetrize_4fold` + `1x1_feature_rule` (+5 tasks) - `v58`: `binary_shape` lookup + `odd_one_out` extraction (+2) - `v59`: `self_tile_uniform` + `self_tile_min_color` + `color_count_upscale` (+4)
Each of these is a 30-80 line Python function with explicit geometric semantics. You can read any one of them in `arc/cross_universe_3d.py` and immediately understand what spatial transformation it encodes. An LLM prompt-tuning loop cannot produce this kind of monotonic, regression-free score progression on a combinatorial benchmark—you'd see random fluctuations and regressions, not a clean staircase.
*The uncomfortable reality for "just use an LLM" approaches:*
My remaining ~778 unsolved tasks each require a new primitive that encodes a geometric insight no existing primitive covers. Each one I add solves 1-3 tasks. This is the grind of actual program synthesis research—expanding a formal language one operator at a time. It's closer to compiler design than machine learning.
I'd genuinely welcome a technical critique of the architecture. The code is right there: [cross_universe_3d.py](https://github.com/Ag3497120/verantyx-v6/blob/main/arc/cross...) — 1,200 lines, zero imports from any ML library.