[pradn]

Just complaining that the world is bad is a good way to waste your energy and end up being a cynic.

So why is not all information organized in structured, open formats? Because there's not enough of an incentive to label/structure your documents/data that way. That's if you even want to open your data to the public - paywalls fund business models.

There have been some smaller successes with semantic web, however. While a recipe site might not want to make it easy for everyone to scrape their recipes, people do want Twitter to generate a useful link preview from their sites' metadata. They do that with special tags Twitter recognizes, and other sites can use as well.

The good news is that LLMs can generate structured data from unstructured documents. It's not perfect, but has two advantages: it's cheaper than humans doing it manually, and you don't have to ask the author to do anything. The structuring can happen on the read side, not the write side - that's powerful. This means we could generate large corpuses of open data from previously-inaccessible opaque documents.

This massive conversion of unstructured to structured data has already been happening in private, with efforts like Google's internal Knowledge Graph. That project has probably seen billions in cumultative investment over the years.

What we need is open data orgs like Wikipedia pick up this mantle. They already have Wikidata, whose facts you can query with a graph querying language. The flag example in the article could be decomposed into motifs by an LLM and added to the flag's entry. And then you could use SPARQL to do the structured query. (And that structured query can be generated from LLMs, too!)

LLMs and structured data are friends.