How do you get access get all the local government meetings? Do you have a crawler that looks up every city in the country then visits each website and pull down the info? A public listing site?
Hi! I've written crawlers for about a dozen municipal hosting platforms, and you can learn the bare-bones of it from our "How" page: https://civic.band/how.html
The short answer is: there's no common API for any of these sites, and even the ones that do have an API are sometimes misconfigured. It's why I wrote all the scrapers by hand.
There isn’t any sort of standard for recording public meetings. I’ve seen everything mic less live streams with obstructed cameras to well curated flawless back and forth with great audio and transcripts. Meeting to meeting it can vary.
whstl|2 months ago
Apparently there is scraping of public data + keyword matching + moderators filtering the matches.
An example that he shows a bit earlier in the video comes from this page, which has an RSS feed: https://www.cityofsanbenito.com/AgendaCenter/City-Commission...
The video says it's open source but I can't find the source.
stronglikedan|2 months ago
phildini|2 months ago
I also gave a talk on this concept that walks through the whole process: https://www.youtube.com/watch?v=MtWzNnZvQ6w
The short answer is: there's no common API for any of these sites, and even the ones that do have an API are sometimes misconfigured. It's why I wrote all the scrapers by hand.
nyjah|2 months ago
c0brac0bra|2 months ago