[sdeziel]

No, you just have to pin the current and future/backup intermediate CAs (X3 and X4) and be done with it.

[SadWebDeveloper]

Let's consider the scenario were a "hacker" can get another cert from one or all intermediate CAs from Let's encrypt or even worst a rogue government with corrupted ties inside the Let's Encrypt team, both scenarios not so far fetched since anyone could change the DNS server for a couple of minutes and ask Let's encrypt to issue a new one so using the intermediate CA's is pointless making it irrelevant to use HPKP this days.