I'm using this for the AI annotations at https://pagewatch.ai/
Its strange how much different it makes the whole page feel from regular boxes, much more of a tangible/informal feeling. Felt perfect to go with the type of feedback LLM's give.
I tried signing up and scanning apkmirror.com but it immediately came up with a 403. Perhaps your user agent is blocked, you should probably use a custom user agent.
It is cloudflare that blocked us, we scrape using a regular chrome instance (with the latest user-agent), but cloudflare are sometimes very aggressive in their blocking.
I'm busy adding a proxy option to handle these cases automatically, and some way to whitelist our scraper.
archon810|1 year ago
lpellis|1 year ago