//spoof google bot datacenter ip (in x-forwarded-for header)
//disable cookies
All this, and more, can be done in haproxy. Smaller, faster than golang. Nice to know that Google cache works for www.ft.com. Another method that worked, e.g., for www.wsj.com, for years was using AMP URL. But WSJ has recently ditched AMP. As a poor substitute I have been using WSJ URLs from podcasts. The advantage of using haproxy versus golang is there's no need to edit and recompile the program when one wants to add a new method. It's possible to set it up so that one only has to add a line to the config file, or add a hostname to a map file. Even more, haproxy lets one reload the configuration without restarting the server.
[+] [-] 1vuio0pswjnm7|2 years ago|reply
// pull page from google cache
//spoof twitter referer
//spoof google bot ua
//spoof google bot datacenter ip (in x-forwarded-for header)
//disable cookies
All this, and more, can be done in haproxy. Smaller, faster than golang. Nice to know that Google cache works for www.ft.com. Another method that worked, e.g., for www.wsj.com, for years was using AMP URL. But WSJ has recently ditched AMP. As a poor substitute I have been using WSJ URLs from podcasts. The advantage of using haproxy versus golang is there's no need to edit and recompile the program when one wants to add a new method. It's possible to set it up so that one only has to add a line to the config file, or add a hostname to a map file. Even more, haproxy lets one reload the configuration without restarting the server.
[+] [-] simonblack|2 years ago|reply
[+] [-] KomoD|2 years ago|reply