WingNews logo WingNews GitHub [2]

WeMoveOn's comments

WeMoveOn | 2 years ago | on: Grok

but is switch c even usable? iirc the training set was nowhere near enough for a model of that size to be coherent in a conversation

WeMoveOn | 2 years ago | on: DeepSeek Coder: Let the Code Write Itself

> much of the work is repetitive, but it comes with its edge cases

for the repetitive stuff, just use copilot embedded in whatever editor you use.

the edge cases are tricky, to actually avoid these the model would need an understanding of both the use case (which is easy to describe to the model) and the code base itself (which is difficult, since description/docstring is not enough to capture the complex behaviors that can arise from interactions between parts of your codebase).

idk how you would train/finetune a model to somehow have this understanding of your code base, I doubt just doing next token prediction would help, you'd likely have to create chat data discussing the intricacies of your code base and do DPO/RLFH to bake it into your model.

look into techniques like qlora that'll reduce the needed memory during tuning. look into platforms like vast ai to rent GPUs for cheap.

RAG/Agents could be useful but probably not. could store info about functions in your codebase such as the signature, the function it calls, its docstring, and known edge cases associated with it. if you don't have docstrings using a LLM to generate them is feasible.

WeMoveOn | 2 years ago | on: Word2Vec received 'strong reject' four times at ICLR2013

Did you now?? I'll have you know that I wrote the full word2vec paper on a roll of shabby two-ply tissue paper during my time in a Taco Bell stall. Sadly, it was then used to mop up my dietary regrets and was subsequently lost to foul wretches of the sewage system. Left with nothing but the memories of my groundbreaking thoughts and the lingering aroma of liquid feces, I texted Mikolov an idea I had about using neural nets to map sequences of text tokens from one language to another only for him to reply "lol thx" and ghost me. I was quite negatively surprised when he decided to take this to the public courts of Facebook and failed to mention the "brainest boi alive™" who gave him this idea in the first place.

WeMoveOn | 2 years ago | on: 3Blue1Brown Calculus Blog Series

Seriously, his explanations on topics go well beyond the lectures some of my professors provide and could probably benefit a lot of students if given as a resource... If only academia wasn't so distrustful of those outside their circles...

WeMoveOn | 2 years ago | on: AI: Nvidia Is Taking All the Money

Eh its more so that the entire ecosystem is built on CUDA. I don't think most developers would care or even notice if you swapped out CUDA for something that'll work seamlessly with the existing Python libraries. Sadly such an alternative doesn't exist and probably won't exist.
page 1