Not jus being impressed that every paper coming out is SOTA, but also leads the way in being Open-Source in the pure definition of OSS, even with permissible licensing.
Let's not confuse the company with the country by over-fitting a narrative. Popular media is reenforcing hatred or anything that sponsors them, especially to weaker groups. Less repercussions and more clicks/money to be made I guess.
While Politicians may hate each other, Scientists love to work with other aspiring Scientists who have similar ambitions and the only competition is in achieving measurable success and the reward it means to the greater public.
Without any bias, but it's genuinely admirable when companies release their sources to enable faster scientific progress cycles. It's ironic that this company is dedicated to finance, yet shares their progress, while non-profits and companies dedicated purely to AI are locking all knowledge about their findings from access.
Are there other companies like DeepSeek that you know of that commonly release great papers? I am following Mistral already, but I'd love to enrich my sources of publications that I consume. Highly appreciated!
I love open source and the general vibe of good vibes you're bringing, but...this isn't SOTA, or close, even on the papers own terms. (i.e. excluding models released the last 6 months, including their own, which is a strange, yet understandable, choice given the results they report)
Quickest way to show this:
- Table 2, top of page 7
- Gemma 2 27B, 0 interventions, has 94.1/56.6/60.2
- Gemma 2 27B, with all their interventions, has 86/64/69.
- Gemma 2 27B, with all their interventions, sampled 32 times, is at 90.4/67.2/70.3.
- Gemma 2 27B came out in...June 2024. :/
Quick heuristics employed here:
- What models did they compare against? (this isn't strictly an issue, the big screaming tell is "What models did they compare against compared to their last N papers?"
- How quickly does the paper have to move towards N samples, and how big does N get before they're happy enough to conclude? (32). How much does that improve performance on their chosen metric? (1.8%)
Happy to see deekseek using the correct (and much more idiomatic) term "inference-time scaling", instead of the grotesque construction of "test-time compute" that openAI came up with.
Any idea why I lost interest in deep seek? I used it and grok3 a whole bunch when they first came out but now I’ve fallen back to Claude for everything.
For coding, I‘m finding Claude‘s responses most to the point and on-task. While many other models try to extrapolate or lecture or patronize. DeepSeek is pretty good though. Maybe it’s the high latency (probably due to prompt processing)?
Deepseek is super bad at personal advice. I can tell it was trained on an oddly stodgy data set. It gives advice that would suit a hyper conservative world view. Like IBM 1950s middle management training course level advice.
Gemma is by far the best at giving advice and planning ones days and life priorities. Not sure how to benchmark that.
> The idea of role-playing as different characters is novel.
It is not. I remember Karpathy being really excited about the "1 million gpt personas" dataset and highlighted it as a way to avoid reward hacking in RLAIF. That was 3-6 months ago I believe.
Of course paper / code / weights beats idea, and it's exciting to see how far this can go.
ALLTaken|11 months ago
Let's not confuse the company with the country by over-fitting a narrative. Popular media is reenforcing hatred or anything that sponsors them, especially to weaker groups. Less repercussions and more clicks/money to be made I guess.
While Politicians may hate each other, Scientists love to work with other aspiring Scientists who have similar ambitions and the only competition is in achieving measurable success and the reward it means to the greater public.
Without any bias, but it's genuinely admirable when companies release their sources to enable faster scientific progress cycles. It's ironic that this company is dedicated to finance, yet shares their progress, while non-profits and companies dedicated purely to AI are locking all knowledge about their findings from access.
Are there other companies like DeepSeek that you know of that commonly release great papers? I am following Mistral already, but I'd love to enrich my sources of publications that I consume. Highly appreciated!
wood_spirit|11 months ago
Is DeepSeeks openness in part to reduce the big American tech companies?
Febra33|11 months ago
What's wrong with China? They're wonderful in the OSS ecosystem.
refulgentis|11 months ago
Quickest way to show this:
- Table 2, top of page 7
- Gemma 2 27B, 0 interventions, has 94.1/56.6/60.2
- Gemma 2 27B, with all their interventions, has 86/64/69.
- Gemma 2 27B, with all their interventions, sampled 32 times, is at 90.4/67.2/70.3.
- Gemma 2 27B came out in...June 2024. :/
Quick heuristics employed here:
- What models did they compare against? (this isn't strictly an issue, the big screaming tell is "What models did they compare against compared to their last N papers?"
- How quickly does the paper have to move towards N samples, and how big does N get before they're happy enough to conclude? (32). How much does that improve performance on their chosen metric? (1.8%)
resters|11 months ago
Paste in a snippet from a book and ask the model to continue the story in the style of the snippet. It's surprising how bad most of the models are.
Grok-3 comes in a close second, likely because it is actually DeepSeek R1 with a few mods behind the scenes.
vessenes|11 months ago
gmerc|11 months ago
DS3: 5M training run Grok3: 400M training run
for 2% difference in the benchmarks.
mentalgear|11 months ago
bilsbie|11 months ago
manmal|11 months ago
narrator|11 months ago
Gemma is by far the best at giving advice and planning ones days and life priorities. Not sure how to benchmark that.
cma|10 months ago
UltraSane|11 months ago
ftbsqcfjm|11 months ago
[deleted]
NitpickLawyer|11 months ago
It is not. I remember Karpathy being really excited about the "1 million gpt personas" dataset and highlighted it as a way to avoid reward hacking in RLAIF. That was 3-6 months ago I believe.
Of course paper / code / weights beats idea, and it's exciting to see how far this can go.