[LeonidBugaev]

One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability

[haddr]

Some years ago I compared those boilerplate removal tools and I remember that jusText was giving me the best results out of the box (tried readability and few other libraries too). I wonder what is the state of the art today?

[jot]

This is worth having a look at: https://mixmark-io.github.io/turndown/

With some configuration you can get most of the way there.

[asadalt]

oh AI is optional here. I do use readability to clean the html before converting to .md.

[jot]

Last time I tried readability it worked well with articles but struggled with other kinds of pages. Took away far more content than I wanted it to.

[IanCal]

How do you achieve the same things without AI here using that tool?

[chrisweekly]

"How do you do it without AI" is a question I (sadly) expect to see more often.

[fbdab103]

I was honestly expecting it to be mostly black magic, but it looks like the meat of the project is a bunch of (surely hard won) regexes. Nifty.

[nyokodo]

> I was … expecting it to be mostly black magic, but … the meat of the project is a bunch of … regexes

Wait, regexes are the epitome of black magic. What do you consider as black magic?

[foundzen]

how is it compared to mozilla/readability?

[asadm]

it uses readibility but does some additional stuff like relink images to local paths etc., which I needed