I find the use of rake to be kind of unorthodox, and yet I don't know what else you'd use in the Go world, other than maybe just Makefiles. Any particular reason to choose Rake? It's probably not easy to get it running on Windows based on my experience playing with Rails on Windows.
Other than that it looks quite useful, and it's definitely something to keep in the tool belt. Bonus points for the subtle Undertale references too :)
When I've been working in Go projects which required external helpers of some sort, I've always just written them as separate Go binaries because they're relatively simple and quick to get up and running.
I've started doing this for my Rust projects too because Rust is becoming a really nice language for writing these sorts of things, and having a separate binary as a Rust file in src/bin is convenient.
Often you want to use some logic from the main application in your tasks and using the same language streamlines that.
I'm also curious about this. It seems that many go developers out there are using Makefiles. Makefiles are a good solution for golang projects in some cases, but I've seen a lot of people really abusing Makefiles and trying to use them for more generic task running.
In a past life, we used invoke [1] for task running. It was incredible but has the same problem as rake: it introduces another language (Python) and more dependencies.
There's a fairly new task runner being developed in go called mage [2], but it didn't seem worth the jump yet to me as it's still pretty immature (I haven't played with it in a few months, though). Did you consider trying that out?
I use matr[1], which is lets you define your build functions/tasks as ordinary go functions. There's a complementary library for issuing shell commands.
I like keeping just one language and tooling, but the biggest benefit to me is I'm always forgetting the arcane bits of shell syntax, and using Go saves me from having to search it all the time. Despite being more verbose than bash, I find it saves me time overall.
A comprehensive prompt is an extremely useful and an underappreciated productivity booster.
But please, when making screencast or screenshots, use a simple prompt.
Because the information included in the prompt is often not only irrelevant, it can be actively distracting and makes reading through less pleasant by increasing the required eye-movement and cognitive effort to filter out the useless content.
I wrote a script bash-for-recording for myself a few years back, for invocation by termrec, which set the window size to 80×24, set HOME to a new empty directory, set USER and NAME to dummy values (creating an actual new throwaway user account would be better here, but I was lazy), set TERM to xterm-256color (I think it was), cleared the environment (env -i) and possibly one or two other things, set a deliberately very simple and obvious prompt (which sets the title as well), cleared the screen, then finally started a nice clean bash profile.
I should pull out my old laptop or my backups and resuscitate the script.
Although the concurrency level (512 connections) is a bit too aggressive for most servers. You'll get throttled, blocked or your backend will crash (which isn't too bad in any case, except that might not be what you were after with a link checker).
Actually, I totally agree with you. I decided the number based on the default maximum number of open files on Linux because I was not sure about common limits of concurrent connections between clients and HTTP servers. Or, it should probably regulate numbers of requests per second sent to the same hosts. If someone suggests other options, I would adopt it.
I have found linkchecker to be unreasonably slow by default, very fragile if you try to make it any faster (e.g. occasional socket errors with almost any concurrency, regardless of the purported ulimits, in a way that suggested some kind of socket leaking to me at the time), and possessing fairly bad reporting. Also, on Windows, being built with ancient Python meant it didn’t support SNI, so I had to delve into it to figure out a way of turning TLS verification off to make it work pretty much at all, which also hints at its generally poor configurability and surprisingly poor documentation (given the fact that there is actually a moderate amount of it).
I still use linkchecker (because I’ve never completed my Rust-based link checker that I started several years ago), and have extended it at work to support client-side certificates which we use on CI, but I’m generally fairly unimpressed with linkchecker.
Thank you for your feedback! Can you open an issue on the GitHub repository? I would appreciate it if you add some concrete use cases as then it'll be clear what kinds of options need to be implemented.
I don't know Go, but looking at the code it doesn't look like it handles sites that are rendered on the client, if so it has limited utility in today's web.
Of course it does not, for that you need a headless browser ( = Chrome Headless).
Neither Python/Go/Java/Rust/C/C++/D/Ruby/C#/F#/... nor any other language (not even Node afaik, but Puppeteer handles that) can handle Javascript rendered sites.
Are there any good checkers for URLs in text files?
I wrote https://github.com/jwilk/urlycue , but I'm not quite happy about it, I don't have energy to improve it either; so I'm looking for alternatives.
I guess one of the bigger challenges when it comes to unstructured data is identifying URLs. Is there a canonical way of identifying a URL embedded in text? Is it an impossible problem?
pylinkvalidator seems to be a pretty good python alternative for this project. It's also very fast, scans 700+ URLs in under a minute
https://github.com/bartdag/pylinkvalidator
Awesome tool. I have one small request, is it possible to put an architecture diagram connecting different moving parts like parsing, fetcher, daemon etc. IMHO this might be useful for people who are trying to go through the source code to understand how the tool functions. Thanks anyways :)
[+] [-] jchw|8 years ago|reply
Other than that it looks quite useful, and it's definitely something to keep in the tool belt. Bonus points for the subtle Undertale references too :)
[+] [-] beefsack|8 years ago|reply
I've started doing this for my Rust projects too because Rust is becoming a really nice language for writing these sorts of things, and having a separate binary as a Rust file in src/bin is convenient.
Often you want to use some logic from the main application in your tasks and using the same language streamlines that.
[+] [-] mjk7841|8 years ago|reply
In a past life, we used invoke [1] for task running. It was incredible but has the same problem as rake: it introduces another language (Python) and more dependencies.
There's a fairly new task runner being developed in go called mage [2], but it didn't seem worth the jump yet to me as it's still pretty immature (I haven't played with it in a few months, though). Did you consider trying that out?
[1] https://github.com/pyinvoke/invoke [2] https://github.com/magefile/mage
[+] [-] eloff|8 years ago|reply
I like keeping just one language and tooling, but the biggest benefit to me is I'm always forgetting the arcane bits of shell syntax, and using Go saves me from having to search it all the time. Despite being more verbose than bash, I find it saves me time overall.
[1] https://github.com/matr-builder/matr
[+] [-] barsonme|8 years ago|reply
[+] [-] reificator|8 years ago|reply
[+] [-] jatoben|8 years ago|reply
[+] [-] komuW|8 years ago|reply
https://github.com/magefile/mage
[+] [-] omeid2|8 years ago|reply
But please, when making screencast or screenshots, use a simple prompt.
Because the information included in the prompt is often not only irrelevant, it can be actively distracting and makes reading through less pleasant by increasing the required eye-movement and cognitive effort to filter out the useless content.
Small little things makes a difference.
[+] [-] chrismorgan|8 years ago|reply
I should pull out my old laptop or my backups and resuscitate the script.
The example at http://tty-player.chrismorgan.info/ was generated using that script.
[+] [-] jakear|8 years ago|reply
[+] [-] gtirloni|8 years ago|reply
Although the concurrency level (512 connections) is a bit too aggressive for most servers. You'll get throttled, blocked or your backend will crash (which isn't too bad in any case, except that might not be what you were after with a link checker).
[+] [-] raviqqe42|8 years ago|reply
[+] [-] zdw|8 years ago|reply
How does it compare to the older python "linkchecker", which was resurrected here (and is quite fast): https://github.com/linkchecker/linkchecker
[+] [-] chrismorgan|8 years ago|reply
I still use linkchecker (because I’ve never completed my Rust-based link checker that I started several years ago), and have extended it at work to support client-side certificates which we use on CI, but I’m generally fairly unimpressed with linkchecker.
[+] [-] fwip|8 years ago|reply
[+] [-] raviqqe42|8 years ago|reply
[+] [-] bryanrasmussen|8 years ago|reply
[+] [-] OoooooooO|8 years ago|reply
Neither Python/Go/Java/Rust/C/C++/D/Ruby/C#/F#/... nor any other language (not even Node afaik, but Puppeteer handles that) can handle Javascript rendered sites.
[+] [-] jpsim|8 years ago|reply
Separately, it’d be awesome if this also checked that anchor links resolve to id values to validate linking to specific elements on a page.
[0]: https://github.com/realm/jazzy
[+] [-] jamietanna|8 years ago|reply
[+] [-] jwilk|8 years ago|reply
I wrote https://github.com/jwilk/urlycue , but I'm not quite happy about it, I don't have energy to improve it either; so I'm looking for alternatives.
[+] [-] oneeyedpigeon|8 years ago|reply
[+] [-] andyonthewings|8 years ago|reply
I run it in a CI job which builds a static site. It works pretty well for me.
[+] [-] gruzh|8 years ago|reply
[+] [-] thamizhan2611|8 years ago|reply
[+] [-] peterwwillis|8 years ago|reply
[+] [-] pknopf|8 years ago|reply
[+] [-] ryanlol|8 years ago|reply
Although, based on a quick look at the code this thing isn't going to go particularly fast.