Where did the decades of classical NLP go? No gold-standard resources like WordNet? No statistical methods?
There's nothing wrong with this, the solution is a good pragmatic choice. It's just interesting how our collective consciousness of expansive scientific fields can be so thoroughly purged when a new paradigm arises.
LLMs have completely overshadowed ML NLP methods from 10 years ago, and they themselves replaced decades statistical NLP work, which also replaced another few decades of symbolic grammar-based NLP work.
Progress is good, but it's important not to forget all those hard-earned lessons, it can sometimes be a real superpower to be able to leverage that old toolbox in modern contexts. In many ways, we had much more advanced methods in the 60s for solving this problem than what Harper is doing here by naively reinventing the wheel.
I'll admit it's something of a bold label, but there is truth in it.
Before our rule engine has a chance to touch the document, we run several pre-processing steps that imbue semantic meaning to the words it reads.
> LLMs have completely overshadowed ML NLP methods from 10 years ago, and they themselves replaced decades statistical NLP work, which also replaced another few decades of symbolic grammar-based NLP work.
This is a drastic oversimplification. I'll admit that transformer-based approaches are indeed quite prevalent, but I do not believe that "LLMs" in the conventional sense are "replacing" a significant fraction of NLP research.
I appreciate your skepticism and attention to detail.
I would much rather check my writing against grammatical rules that are hard coded in an open source program—meaning that I can change them—than ones that I imagine would be subject to prompt fiddling or worse; implicitly hard coded in a tangle of training data that the LLM would draw from.
There are two versions of the LanguageTool: open source and cloud-based. Open source checks the individual words in the dictionary just like the system's spell checker. Maybe there is something more to it, but in my tests, it did not fix even obvious errors. It's not an alternative to Grammarly or this tool.
IMO not using LLMs is a big plus in my book. Grammarly has been going downhill since they've been larding it with "AI features," it has become remarkably inconsistent. It will tell me to remove a comma one hour, and then tell me to add it back the next.
Being dyslexic, I was an avid Grammarly user. Once it started adding "AI features" the deterioration was noticeable, I cancelled my subscription and stopped using it a year ago.
I also only ever used the web app, so copy+pasting as installing the app is for all intentness and purposes is installing a key logger.
Grammar works on rules, not sure why that needs an LLM, Grammarly certainly worked better for me when it was more dumb, using rules.
Grammarly sometimes gets stuck in a loop, where it suggests changing from A to B. It then immediately suggests changing from B to A again, continuing to suggest the opposite change every time I accept the suggestion.
It's not a problem; I make the determination which option I like better, but it is funny.
General purpose LLMs seem to get very confused about punctuation, in my experience. It's one of their big areas of obvious failing. I'm surprised Grammarly would allow this to happen.
'imo' and 'in my book' are redundant in the same sentence. Are there rules-based techniques to catch things like that? Btw I loved the use of 'larding' outside the context of food.
I've relied on Grammarly to spellcheck all my writing for a few years (dyslexia
prevents me from seeing the errors even when reading it 10 times). However, I find its increasing focus on LLMs and its insistence on rewriting sentences in more verbose ways bothers me a lot. (It removes personality and makes human-written text read like AI text.)
So I've tried out alternatives, and Harper is the closest I've found at the moment... but i still feel like grammarly does a better job at the basic word suggestion.
Really, all I wish for is a spellcheck that can use the context of the sentence to suggest words. Most ordinary dictionary spellchecks can pick the wrong word because it's syntactically closer. They may replace "though" with "thought" because I wrote "thougt" when the sentence clearly indicates "though" is correct; and I see no difference visually between any of the three words.
What's wild is that OpenAI's earlier models were trained to guess the next word in a sentence. I wonder if GPT-2 would get "though" correct more often than the latest AI-assisted writing tools like Grammerly.
There are some areas where it seems like LLMs (or even SLMs) should be way more capable. For example, when I touch a word on my Kindle, I'd think Amazon would know how to pick the most relevant definition. Yet it just grabs the most common definition. For example, consider the proper definition of "toilet" in this passage: "He passed ten hours out of the twenty-four in Saville Row, either in sleeping or making his toilet."
Why wouldn't you want an LLM for a language learning tool? Language is one of things I would trust an LLM completely on. Have you ever seen ChatGPT make an English mistake?
Grammarly is all in on AI and recently started recommended splitting "wasn't" and added the contraction to the word it modified. Example: "truly wasn't" becomes "was trulyn't"
Yeah, I agree. An open-source LLM-based grammar checker with a user interface similar to Grammarly is probably what I'm looking for. It doesn't need to be perfect (none of the options are); it just needs to help me become a better writer by pointing out issues in my text. I can ignore the false positives, and as long as it helps improve my text, I don't mind if it doesn't catch every single issue.
Using an LLM would also help make it multilingual. Both Grammarly and Harper only support English and will likely never support more than a few dozen very popular languages. LLMs could help cover a much wider range of languages.
I tried to use one LLM based tool to rewrite sentence in more official corporate form, and it rewrote something like "we are having issues with xyz" into "please provide more information and I'll do my best to help".
LLMs are trained so hard to be helpful that it's really hard to contain them into other tasks
uh. yes? it's far from uncommon, and sometimes it's ludicrously wrong. Grammarly has been getting quite a lot of meme-content lately showing stuff like that.
it is of course mostly very good at it, but it's very far from "trustworthy", and it tends to mirror mistakes you make.
Seeing Harper as an implementation of natural language's LSP brings me great joy, as it proves an idea I've had for a long time—natural language and programming languages are interconnected. Many concepts and techniques from programming languages can also be applied to natural language, making our lives more convenient. The development of LLMs and vibe coding has further blurred the boundary between natural language and programming languages, offering similar insights.
Surprised coming into this that I don't see anyone mentioning vale[0]. I've been using it for ~4 years now and love it.
I use grammarly briefly when it came out and liked the idea. Admittedly it has more polish than vale for people writing in google docs, &c. Still, I stick with Vale. Is there any case for moving to Harper?
Would be nice if they had a website where you could demo/test it before downloading extensions and stuff. Their firefox extension opens to this page https://writewithharper.com/install-browser-extension but when you paste in anything more than a few paragraphs the highlighting is all messed up.
Slightly controversial compared to other comments here but I haven't used Grammerly at all since LLMs came out. Even a 4B local LLM is good enough to rephrase all forms of text and fix most grammer mistakes.
I think a lot of value comes by integrating with a language server and/or browser extensions.
Do you have a setup where this is possible or do you copy paste between text fields? (Genuine question. I’d love to use a local LLM integrating with an LSP).
I'm a long-time Grammarly user. I just tried Harper, and it simply performs very poorly. It is a good initiative, but I don't feel the current state of this software to be worthwhile.
I wish it had keyboard shortcuts. As a Vim user, in Chrome it's tedious to click on every suggestion given by the app. Also, maybe add a "delay" so it doesn't think the currently-being-typed word is a mistake (let me finish typing first!).
Otherwise, it's great work. There should be an option to import/export the correction rules though.
this solution is just fundamentally insufficient. in the age of LLMs it's pretty insane to imagine programmers manually hard-coding an arbitrary subset of grammatical corrections (sure: it's faster, it's local first, but it's not enough). on top of that, English (like any other natural language) is such a complicated beast that you will never write a classic deterministic parser that's sophisticated enough to allow you to reliably implement even the most basic of grammatical corrections (check the other comments for examples). it's just not gonna happen.
i guess it's a nice and lightweight enhancement on top of the good old spellchecker, though
[+] [-] oersted|8 months ago|reply
I'm just a bit skeptical about this quote:
> Harper takes advantage of decades of natural language research to analyze exactly how your words come together.
But it's just a rather small collection of hard-coded rules:
https://docs.rs/harper-core/latest/harper_core/linting/trait...
Where did the decades of classical NLP go? No gold-standard resources like WordNet? No statistical methods?
There's nothing wrong with this, the solution is a good pragmatic choice. It's just interesting how our collective consciousness of expansive scientific fields can be so thoroughly purged when a new paradigm arises.
LLMs have completely overshadowed ML NLP methods from 10 years ago, and they themselves replaced decades statistical NLP work, which also replaced another few decades of symbolic grammar-based NLP work.
Progress is good, but it's important not to forget all those hard-earned lessons, it can sometimes be a real superpower to be able to leverage that old toolbox in modern contexts. In many ways, we had much more advanced methods in the 60s for solving this problem than what Harper is doing here by naively reinventing the wheel.
[+] [-] chilipepperhott|8 months ago|reply
Before our rule engine has a chance to touch the document, we run several pre-processing steps that imbue semantic meaning to the words it reads.
> LLMs have completely overshadowed ML NLP methods from 10 years ago, and they themselves replaced decades statistical NLP work, which also replaced another few decades of symbolic grammar-based NLP work.
This is a drastic oversimplification. I'll admit that transformer-based approaches are indeed quite prevalent, but I do not believe that "LLMs" in the conventional sense are "replacing" a significant fraction of NLP research.
I appreciate your skepticism and attention to detail.
[+] [-] tough|8 months ago|reply
[+] [-] tolerance|8 months ago|reply
The Neovim configuration for the LSP looks neat: https://writewithharper.com/docs/integrations/neovim
The whole thing seems cool. Automattic should mention this on their homepage. Tools like this are the future of something.
[+] [-] triknomeister|8 months ago|reply
[+] [-] Polarity|8 months ago|reply
[+] [-] shortformblog|8 months ago|reply
https://github.com/languagetool-org/languagetool
I generally run it in a Docker container on my local machine:
https://hub.docker.com/r/erikvl87/languagetool
I haven't messed with Harper closely but I am aware of its existence. It's nice to have options, though.
It would sure be nice if the Harper website made clear that one of the two competitors it compares itself to can also be run locally.
[+] [-] akazantsev|8 months ago|reply
[+] [-] unfitted2545|8 months ago|reply
[+] [-] pram|8 months ago|reply
[+] [-] tiew9Vii|8 months ago|reply
I also only ever used the web app, so copy+pasting as installing the app is for all intentness and purposes is installing a key logger.
Grammar works on rules, not sure why that needs an LLM, Grammarly certainly worked better for me when it was more dumb, using rules.
[+] [-] InsideOutSanta|8 months ago|reply
It's not a problem; I make the determination which option I like better, but it is funny.
[+] [-] boplicity|8 months ago|reply
[+] [-] raincole|8 months ago|reply
Not that I think LLM is always better, but it would be interesting to compare these two approaches.
[+] [-] chneu|8 months ago|reply
[+] [-] harvey9|8 months ago|reply
[+] [-] Alex-Programs|8 months ago|reply
[+] [-] raverbashing|8 months ago|reply
So just like English teachers I see
[+] [-] aDyslecticCrow|8 months ago|reply
I've relied on Grammarly to spellcheck all my writing for a few years (dyslexia prevents me from seeing the errors even when reading it 10 times). However, I find its increasing focus on LLMs and its insistence on rewriting sentences in more verbose ways bothers me a lot. (It removes personality and makes human-written text read like AI text.)
So I've tried out alternatives, and Harper is the closest I've found at the moment... but i still feel like grammarly does a better job at the basic word suggestion.
Really, all I wish for is a spellcheck that can use the context of the sentence to suggest words. Most ordinary dictionary spellchecks can pick the wrong word because it's syntactically closer. They may replace "though" with "thought" because I wrote "thougt" when the sentence clearly indicates "though" is correct; and I see no difference visually between any of the three words.
[+] [-] Breza|8 months ago|reply
There are some areas where it seems like LLMs (or even SLMs) should be way more capable. For example, when I touch a word on my Kindle, I'd think Amazon would know how to pick the most relevant definition. Yet it just grabs the most common definition. For example, consider the proper definition of "toilet" in this passage: "He passed ten hours out of the twenty-four in Saville Row, either in sleeping or making his toilet."
[+] [-] aagha|8 months ago|reply
[+] [-] demarq|8 months ago|reply
No errors detected. So this needs a lot of rule contributions to get to Grammarly level.
[+] [-] alpb|8 months ago|reply
[+] [-] Tsarp|8 months ago|reply
[+] [-] wellthisisgreat|8 months ago|reply
[+] [-] marginalia_nu|8 months ago|reply
> In large, this is _how_ anything crawler-adjacent tends to be
It suggests
> In large, this is how _to_ anything crawler-adjacent tends to be
[+] [-] healsdata|8 months ago|reply
[+] [-] josephcsible|8 months ago|reply
[+] [-] jantissler|8 months ago|reply
[+] [-] icapybara|8 months ago|reply
[+] [-] healsdata|8 months ago|reply
https://imgur.com/a/RQZ2wXA
[+] [-] InsideOutSanta|8 months ago|reply
Using an LLM would also help make it multilingual. Both Grammarly and Harper only support English and will likely never support more than a few dozen very popular languages. LLMs could help cover a much wider range of languages.
[+] [-] Szpadel|8 months ago|reply
LLMs are trained so hard to be helpful that it's really hard to contain them into other tasks
[+] [-] Groxx|8 months ago|reply
it is of course mostly very good at it, but it's very far from "trustworthy", and it tends to mirror mistakes you make.
[+] [-] dartharva|8 months ago|reply
[+] [-] VTimofeenko|8 months ago|reply
https://writewithharper.com/docs/integrations/language-serve...
[+] [-] novoreorx|8 months ago|reply
[+] [-] sdtransier|8 months ago|reply
https://automattic.com/2024/11/21/automattic-welcomes-harper...
[+] [-] loughnane|8 months ago|reply
I use grammarly briefly when it came out and liked the idea. Admittedly it has more polish than vale for people writing in google docs, &c. Still, I stick with Vale. Is there any case for moving to Harper?
[0] https://vale.sh/
[+] [-] raybb|8 months ago|reply
[+] [-] ErrorNoBrain|8 months ago|reply
i honestly don't trust grammarly ... i mean, its essentially a keylogger.
i did try it a bit once, and i never seem to have it work that well for me. But i am multilingual so maybe thats part of my hurdle
[+] [-] IceWreck|8 months ago|reply
[+] [-] gglanzani|8 months ago|reply
Do you have a setup where this is possible or do you copy paste between text fields? (Genuine question. I’d love to use a local LLM integrating with an LSP).
[+] [-] ibobev|8 months ago|reply
[+] [-] thr0waway001|8 months ago|reply
Passes.
For reference: https://youtu.be/w-R_Rak8Tys?si=h3zFCq2kyzYNRXBI
[+] [-] iwinux|8 months ago|reply
[+] [-] behnamoh|8 months ago|reply
Otherwise, it's great work. There should be an option to import/export the correction rules though.
[+] [-] cAtte_|8 months ago|reply
i guess it's a nice and lightweight enhancement on top of the good old spellchecker, though