wsxiaoys's comments

Hi HN,

I wrote a 4-part series on how we built the AI edit model behind Pochi’s coding agent.

It covers everything from real-time context management and request lifecycles to dynamically rendering code edits using only VS Code’s public APIs.

I’ve written this as openly and concretely as possible, with implementation details and trade-offs.

Full series:

1. The Edit Model Behind Tab Completion: https://docs.getpochi.com/developer-updates/how-we-created-n...

2. Real-Time Context Management in Your Code Editor: https://docs.getpochi.com/developer-updates/context-manageme...

3. Request Management Under Continuous Typing: https://docs.getpochi.com/developer-updates/request-manageme...

4. Dynamic Rendering Strategies for AI Code Edits: https://docs.getpochi.com/developer-updates/dynamic-renderin...

OP here - I've talked in detail about how we rendered NES suggestions using only VS Code public APIs.

Most tools fork the editor or build a custom IDE so they can skip the hard interaction problems.

Our NES is a VS Code–native feature. That meant living inside strict performance budgets and interaction patterns that were never designed for LLMs proposing multi-line, structural edits in real time.

In this case, surfacing enough context for an AI suggestion to be actionable, without stealing attention, is much harder.

That pushed us toward a dynamic rendering strategy instead of a single AI suggestion UI. Each path gets deliberately scoped to the situations where it performs best, aligning it with the least disruptive representation for a given edit.

If AI is going to live inside real editors, I think this is the layer that actually matters.

Happy to hear your thoughts!

OP here - happy to answer any questions.

This was one of the more unexpectedly tricky layers of building real-time LLM suggestions, and I’d love to hear how others have approached timing, cancellation, and speculative prediction in their editors or agents.

OP here - this is Part 2 of a series documenting how we built NES (Next Edit Suggestions), our real-time edit model inside the Pochi editor extension.

The real challenge (and what ultimately determines whether NES feels “intent-aware”) was how we manage context in real time while the developer is editing live. For anyone building real-time AI inside editors, IDEs, or interactive tools.

I hope you find this interesting. Happy to answer any questions!

I’ve been experimenting with next-edit prediction for a while and wrote up how we trained the edit model that powers our Tab completion feature. This post is part of a broader series where we share how we built this feature from the low-level modeling right up to the editor extension.

The cool part is we fine-tuned Gemini Flash Lite with LoRA instead of an OSS model, helping us avoid all the infra overhead and giving us faster responses with lower compute cost.

I've spent the last few months working on a custom RL model for coding tasks. The biggest headache has been the lack of good tooling for tuning the autorater's prompt. (That's the judge that gives the training feedback.) The process is like any other quality-focused task—running batch rating jobs and doing SxS evaluations—but the tooling really falls short. I think I'll have to build my own tools once I wrap up the current project

Appreciated! Fixed

Yes - however, the FIM model requires careful configuration to properly set the prompt template.

> So using 2 NVLinked GPU's with inference is not supported?

To make better use of multiple GPUs, we suggest employing a dedicated backend for serving the model. Please refer to https://tabby.tabbyml.com/docs/references/models-http-api/vl... for an example

Yes - AMD GPU is supported through vulkan backend:

https://github.com/TabbyML/tabby/releases/tag/v0.23.0

https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/

Tabby comes with builtin RAG support so you can add this api framework to it.

Example: https://demo.tabbyml.com/search/how-to-configure-sso-in-tabb...

Settings page: https://demo.tabbyml.com/settings/providers/doc

Not yet, consider subscribe https://github.com/TabbyML/tabby/issues/322 for future updates!

Check https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ to see a local setup with 3090.

Tabby is engineered for team usage, intended to be deployed on a shared server. However, with robust local computing resources, you can also run Tabby on your individual machine. Check https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ to see a local setup with 3090.

Never imagined our project would make it to the HN front page on Sunday!

Tabby has undergone significant development since its launch two years ago [0]. It is now a comprehensive AI developer platform featuring code completion and a codebase chat, with a team [1] / enterprise focus (SSO, Access Control, User Authentication).

Tabby's adopters [2][3] have discovered that Tabby is the only platform providing a fully self-service onboarding experience as an on-prem offering. It also delivers performance that rivals other options in the market. If you're curious, I encourage you to give it a try!

[0]: https://www.tabbyml.com

[1]: https://demo.tabbyml.com/search/how-to-add-an-embedding-api-...

[2]: https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ

[3]: https://www.linkedin.com/posts/kelvinmu_last-week-i-introduc...

Fun fact: We've implemented binary embedding search [1] without the need for a specialized vector database. Instead, dimensional tokens like 'embedding_0_0', 'embedding_1_0' are created and being built into the tantivy index [2].

We're satisfied with the quality and performance this approach yields, while still keep Tabby embed everything into a single binary.

[1] My binary vector search is better than your FP32 vectors: https://blog.pgvecto.rs/my-binary-vector-search-is-better-th...

[2] Tantivy: https://github.com/quickwit-oss/tantivy

TabbyML | https://tabbyml.com | Software Engineer (Rust) / Product Engineer – Full-Time | Remote

Tabby strives to become the AI Intelligence Stack for the entire development lifecycle. We are a fully distributed, all-remote team.

Our tech stack includes:

  * Frontend: TypeScript, React, Next.js
  * Backend: Rust, GraphQL
  * IDE/Extension: TypeScript, Node.js
  * Tools: GitHub, Slack, Linear, Lark

Please apply here: https://tabbyml.vercel.app/

Checkout tabby: https://github.com/TabbyML/tabby

Blog post on repository context: https://tabby.tabbyml.com/blog/2023/10/16/repository-context...

(Disclaimer: I started this project)

Can the diagram being part of the code it self (e.g special comment string on top of certain piece, or in a special metalfiel)

It'll be really nice if the code diagram itself could be version tracked together with code.

Nice implementation! It should serve as a great reference for a minimal Tabby's backend API. Thank you for sharing it!

Yeah - ultimately, it won't be as performant or feature-rich compared to https://github.com/TabbyML/tabby, but it's still perfect for educational purposes!