SteelPh0enix's comments

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

oh, hey, thanks!

i wonder what was there, i like spicy comments...

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

I use llama.cpp mostly for working with code that i can't share with any online provider. Simple NDA stuff. Some refactors are easier to do via LLM than manually. It's a decent debugging duck too.

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

Read the README of the model. You'll probably find some benchmark metrics there that can tell you more-or-less how "good" the model is, but keep in mind that it's not that hard to artifically boost those scores, so don't reject every model that isn't at the top of the benchmark.

I've listed some good starting models at the end of the post. Usually, most LLM models like Qwen or Llama are general-purpose, some are fine-tuned for specific stuff, like CodeQwen for programming.

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

>Yeah the mingw method on windows is a ludicrous thing to even think about

Building most stuff on Windows is ludicrous, that's not something uncommon. I've chosen MSYS there as it's the easiest and least paninful way of installing deps for Vulkan build.

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

If you're satisfied with ollama, I don't think there's a point to go "lower" and use llama.cpp directly.

Unless you want to tinker.

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

Yeah, I should have mentioned that in the post because I know that not everyone likes to tinker as much as I do. I'll add a note about this site in a revision soon.

However, I still find it useful to know how to do that manually.

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

I have spent unreasonable amounts of time building llama.cpp for my hardware setup (AMD GPU) on both Windows and Linux. That was one of the main reasons of writing that blog post for me. Lmao.

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

Hey, author of the blog post here. Check out avante.nvim if you're already a vim/nvim user, I'm using it as assistant plugin with llama-server and it works great.

Small models, like Llama 3.2, Qwen and SmolLM are really good right now, compared to few years ago

SteelPh0enix | 1 year ago | on: Llama.cpp guide – Running LLMs locally on any hardware, from scratch

Wow, i did not know about ccmake. I'll check it out and edit the post if it's really that easy to use, thanks.

SteelPh0enix | 2 years ago | on: Aerugo – RTOS for aerospace uses written in Rust

Hey, one of the devs of Aerugo here! Thanks for feedback!

You are absolutely right - at current state, this is not a safety-critical OS, however the project doesn't claim that explicitly - it's "safety-critical applications oriented". It's a small detail, but you're right to point it out.

The lack of mentioned documents is due to the fact that this RTOS was not qualified for any criticality. And this is due to our resource constraints for this project - ESA provided us with a year of time and funds for ~2 full-time developers. It would be physically impossible to create this project from scratch (as we did) and qualify it, even for crit C, in that timeline.

We would love to do a follow-up activity on Aerugo, and one of our ideas was the qualification (maybe not for Crit A, but B would be nice for an RTOS). However, that's a thing for the future, and we don't know what exactly will happen next with Aerugo yet - we're working on it.

I'd also like to point out that we have designed this system with safety in mind - we've been regularly analyzing potentially problematic design choices and code with that in mind (especially unsafe code and functions). There is a ground for criticality qualification, it just needs a lot of work to make it a fact.

PS: We do intend to release our "Lessions Learned" report in near future to the public!

SteelPh0enix | 2 years ago | on: Aerugo – RTOS for aerospace uses written in Rust

While our RTOS in fact doesn't care about it, because it's up to the user to implement the abstraction for the platform, the bigger issue is Rust support for more exotic architectures used in space - for example, SPARC (LEON3), which we'd like to consider for a Rust project, but it's compiler support seems relatively bad.

SteelPh0enix | 2 years ago | on: Aerugo – RTOS for aerospace uses written in Rust

Hello, one of the devs of Aerugo here.

The rationale is rather simple - SAMV71 is used in critical applications (it has some functional safety certificates), and I have past experience of writing critical software for it at N7.