ChangeDetection, monitor any website change

[shortrounddev2]

A few years ago when I was a Junior engineer, I worked for a company which would provide us with free tickets to any sports team in our city. I'm not a sports fan, but my stepdad loves hockey. When the tickets went up, they got reserved pretty fast.

I noticed that the site we used to reserve the tickets had a predictable slug like: `{company}-hockey-tickets-2016-season`, and `{company}-hockey-tickets-2015-season`. The sites also didn't change between each refresh.

So, when it was nearly time for the 2017 season to start, I wrote a script which hashed a GET request of the site at `{company}-hockey-tickets-2017-season`. If the hash changed, it would send me a text message using twilio, and I would know to immediately get on a machine and reserve the tickets we wanted. After a couple false positives, the page eventually went up before it was announced and I reserved the tickets I wanted.

Unfortunately, the office manager who did these things told me I couldn't jump the gun and un-reserved my tickets. After the official announcement went up, the spots I had originally reserved quickly went to someone else

[HardlyCurious]

Yeah, that office manager probably knew who would be getting the tickets in advance.

[solardev]

It seems like bad company policy to let a bunch of techies fight over a limited resource via web requests. It's like network inspector gladiators. Maybe that was the point? Lol

[mhb]

Why is the hash of the request better than comparing strings?

[hdivider]

Good stuff. Niche idea for ya: consider government contractors. Gov sites tend to be atrocious. Manual change tracking can mean you find a $1M RFP weeks earlier, which increases chance of winning. (Consider mda.mil. There's no email signup. What they consider a 'newsletter' is in fact a list of PDFs hosted on their site.)

And that's federal gov contracting. State/local is even more of a Wild West. So much so that many, maany companies just never have the bandwidth to even find RFPs and navigate those ancient sites.

[MoOmer]

While that may be true if contractors/competitors for contracts were generally unaware of the contract until posting, that’s usually not the reality. A lot of government contracting advice from experienced contractors is that if you’re discovering it via a public source (i.e. the website) you’re too late.

[solardev]

Funny how much better government might get if we just let their employees post on Craigslist

[serhack_]

I'm OP. I'm not the developer of the project, but I think it deserved a post on HN because it's pretty an useful tool to track website changes. Setup can be done in a few minutes and the lead developer is always looking for solid feedbacks.

[TekMol]

It cannot be a Show HN then:

https://news.ycombinator.com/showhn.html

Please change the title.

[kierenj]

First time I've noticed a GH repo having so much SEO in the About section! Not sure how I feel about it..

[mnstngr]

It's actually refreshing to see end-user documentation and answering the basic questions of “what can this do” and “why should I care” in the readme. Most projects jump straight into how to build it locally, with not even a screenshot of what they’ve created.

[Lalabadie]

Right? First time I run into a README file on Github and I think "Oh, this text is for search engines, not readers".

[realPubkey]

They should add an url into the description text. This will render as backlink any there are many pages out there which replicate github-topic pages which will then give your site many links for free.

[serhack_]

I actually found it via the about section so I guess it helped haha

[Aulig]

I've been self-hosting this for a year or so and it's pretty neat.

It happens quite often that I need to wait for something which doesn't have a builtin alert.

Currently I'm for example using it to get an alert when a flutter package has been updated on pub.dev

[_flux]

I've used https://urlwatch.readthedocs.io but this definitely seems much easier to use—though possibly not quite as powerful regarding page filtering. But at least ChangeDetection supports jq which is already quite a nice feature in that department.

[mchenier]

https://visualping.io does a nice similar job and is free for moderate personal use.

[davidinosauro]

On the theme of alternatives, I've been using https://www.followthatpage.com/ on and off for years. In particular for my partner's job hunt, watching some local trade association billboards or single school's `/jobs.html` pages.

Also has a reasonable free plan for personal use (20 pages daily + 20 pages weekly + 1 page hourly).

Definitely much simpler in terms of diffing (only text), but it has this 2000s vibe.

[cloudyporpoise]

Curious how this handles sites behind CloudFlare using bot detection.

[_boffin_]

It’s not too difficult to get past cloudflare using puppeteer. Being normal is the key.

[acheong08]

I see selenium and playwright in the dependencies. I would’ve just gone with tls-client

[Mo3]

Kudos, this is a very feature rich solution. I like it. 8.99 seems a little bit much, but I guess it's probably the sweet spot on the curve.

[serhack_]

You can install it for free on your machine. I think it's a decent price, considering that they offer you to check a website even with a ratio of seconds.

[johnnyworker]

For Windows, there is the OG https://www.aignes.com/

[ephbit]

Here's an addon for Firefox that also checks websites in regular intervals to detect changes:

https://github.com/WaldiPL/webpageScanner

Not affiliated with it and I don't know if it still works, haven't used it lately. When I used it, it worked to my satisfaction.

[abbaselmas]

my favorite one is visualping.io

[wkat4242]

Interesting! I've been making some scripts privately for specific tasks, like an Nvidia video card I wanted to buy. However many sites are really hostile to scraping these days. I'll give it a try,.

[weinzierl]

camelcamelcamel is a really old service, but it's still around and still works well for product price changes.

[champagnepapi]

How does this handle A/B tests that sites are running?

[masukomi]

wouldn't that depend entirely on which group its request got put in, which would be something it would have no control over?

[sccxy]

I have used it for some time now. It has been very useful for me.

Some use cases for me:

* price changes

* calendar changes for sport events

* document changes for local gov

* new firmware releases

* terms of conditions changes

[accidbuddy]

Anyone has been utilized this tool to monitoring Amazon Prices (with login)?

[stewx]

Use CamelCamelCamel