Quench of LHC inner triplet magnet causes a small leak with major consequences

[ThePhysicist]

Helium leaks are a nightmare. During my PhD I worked with a self-built dilution cryostat that would often have leaks in the custom-built Indium seals. To find the leaks you'd have to pump the cryostat to high vacuum, hook up a portable mass spectrometer tuned to Helium to the pump circuit and then spray different parts of the cryostat with Helium from a regular gas canister. The He would then get sucked in through the leak and show up on the mass spectrometer, which was coupled to a loudspeaker so it would cause a sound whose pitch increased with the measured He density. Once you found the leak you had to vent the entire system, remove the faulty seal and replace it with a new one. All seals were handmade, i.e. you took a small filament of Indium, placed it between the flange and the housing (after carefully cleaning everything with Acetone) and carefully screwed it shut, turning each screw only a tiny bit at each turn and going around all the screws until you could see the Indium squeeze out of the edges.

Even worse, some leaks would only show up when the system got cooled down to liquid Helium temperature. When that happened you were out of luck as you can't cool the system down to 4K before spraying it with Helium, so you had to just guess where the leak might be and replace all seals in that area until you found the right one. Going even deeper in temperature would eventually turn the He4 suprafluid, which means that it loses all internal friction. In that state it would squeeze through even the tiniest molecular cracks, so again if that happened you just had to redo all the seals and hope they would hold.

[javajosh]

I think I'll bookmark this comment and come back to it the next time I feel annoyed by tracking down a software bug.

[jerf]

AlphaPheonix on YouTube has an excellent and informative video of a very similar process of testing seals, though the vacuum is pulled for a different reason: https://www.youtube.com/watch?v=VD69crOFx10

[londons_explore]

One would imagine there must be some kind of paint you can paint onto the outside of a vacuum flask, and have it be sucked into any gaps and solidify there.

It seems they already make a product for that - vacuum grease. It can get sucked into even a fairly large crack, and plug it up, as long as the depth of the crack is much longer than the width, then atmospheric pressure won't provide enough force to send it deeper into the crack.

And the grease itself is designed not to evaporate into the vacuum.

Whats wrong with using that?

[YakBizzarro]

Oh yes, way too familiar situation... dry cryostats were really a game changer. Apart of the incredible advantage of not having to refill them frequently and on Sunday, they are much more reliable and without any cold seal

[simonjgreen]

That's sounds horrific. It makes me wonder what proportion of time you'd spend massaging and debugging your lab equipment vs. actually using it?

[throwawaymobule]

Helium leaks are especially annoying, because they'll brick most modern electronics for a day or so. iPhones especially.

If they have a PCB oscillator, the gas will seep its way in there and cause a clock failure.

[willis936]

A good friend of mine made a leak checker from scrap parts while I was working at a university lab. It was just for a UHV system (no cryo). I wish I still worked with him.

The whizz sound leak checkers make is good fun.

[latchkey]

Could you hear the indium scream as it was being squeezed out?

https://youtu.be/mjiP5Q6g_aM?t=153

[NotYourLawyer]

Ha, I also remember using a helium canister to look for leaks in a vacuum system in the lab. At least ours wasn’t at cryo temperatures though. Sounds like a real pain.

[phkahler]

Maybe we should "just" build this in space. ;-)

[mensetmanusman]

Was it ever an option to cycle the system with temporal atomic layer deposition chemistry to slowly build up aluminum oxide barriers?

[throwaway290]

Based on you description... Couldn't you just go so cold He goes superfluid so you find all the leaks right up?

[COGlory]

This sounds miserable!

[user3939382]

This is a more extreme description of why I hate plumbing.

[SiempreViernes]

Man, so relieved it's just cooling one section that is meant by "major consequences", title made me think of the explosion during commissioning. Looks like it's only sector S78 that needs attention: https://op-webtools.web.cern.ch/vistar/vistars.php?usr=LHC2

This report is impressively detailed for being put out just two days of the incident, but I guess LHC having world class technical diagnostics teams is not too surprising.

[bayindirh]

When I was visiting CERN, the whole accelerator was in maintenance mode. We have seen a detector (IIRC it was ATLAS) which was disassembled in layers, so we were able to see almost all of it.

Our guide explained that whole schedule for maintenance and operation is set in stone and whole schedule is planned almost to hour resolution for the next six months or so. Also it's worth noting that, there is periodic maintenance that needs to be done, and that's also planned in advance.

Because of this stringent requirements, "several weeks" of shift is indeed a major problem, and by several I guess they are looking to ~8 weeks, if not more.

[vasco]

The "major consequences" seem to be for the schedule:

> This incident will probably have a great impact on the LHC schedule, with machine operation unlikely to resume for at least several weeks.

I also read it instantly in case they had a black hole on their hands, even though I'd expect us all to be dead almost instantly if any real "major consequence" event truly happened in the accelerators.

[tux3]

Vistar is so cool :)

Also, here's a public grafana I just found with temperature charts in S78 and a bunch of other cryo data: https://dash.web.cern.ch/d/iDRuWWHGz/sector-78-trend

Not that I can do anything very interesting with that particular bit of information, but I'm always happy to see open data. We could really use more of that in public science, and CERN is really doing a great job

[fancyfredbot]

It is amazing that the beam was "dumped" just milliseconds before the magnet quenched! If that hadn't happened then the beam could have crashed into the magnet and caused a lot more damage.

Dumping the beam means diverting it out of the main LHC ring and crashing it into a specially designed buffer (I think it is a lump of steel or something). So cool to see this all happening automatically.

[tuardoui]

If like me, you want to see what it looks like : https://home.cern/news/news/accelerators/autopsy-lhc-beam-du...

[zelos]

It hadn't occurred to me that there would be significant energy contained in a beam of sub-atomic particles. Something like 100kg TNT equivalent (~500MJ)? Wow.

[s0rce]

According to the link below its an 8m long steel tube filled with various types of graphite.

[aredox]

I was there when there was a (much bigger) quench during commissioning of the LHC [0][1]. It was also at the focusing quadrupole modules - maybe the same ones?

I could make a joke about how it's Fermilab's vengeance for not having their own accelerator, but that would be dishonest: these focusing modules are the most complex ones of the whole ring.

[0]http://stephatcern.blogspot.com/2008/12/photos-of-lhc-damage... [1]https://home.cern/news/press-release/cern/cern-releases-anal...

[notanote]

The 2008 quench happened at the other side of the ring. (Sector 3-4 then, sector 7-8 now.) The physics run was going very smoothly up to now [0]. Last year part of the machine (RF) had to be brought to room temperature as well, after a cooling tower fault, and there was no beam for four weeks. I get the impression that this will take longer, but I hope not by much.

[0]https://atlas.web.cern.ch/Atlas/GROUPS/DATAPREPARATION/Publi...

[fifteen1506]

For major click-bait for Half-Life 1 players, I propose the title to be prefixed with "UNFORESEEN CONSEQUENCES: ".

[ComodoHacker]

This report really has disturbingly strong Half-Life 1 vibes for me.

[misterflibble]

Gordon! Get away from the beam!!!!

[Gordonjcp]

I never thought I'd see a resonance cascade...

[dr_faustus]

Just reading that gives you an inkling of an idea what kind of amazing feat of engineering the LHC is. I really hope governments keep funding basic research like that because it teaches us not only about nature but also how to built those incredible machines.

[lhoff]

Partially related recommendation for the german-speaking crowd here:

The Raumzeit podcast released the second episode of a longer series about the cern. In the frist one he discussed the history and success of the cern with the leader of the experimental physics department and in the second one he talks to the guy who is in charge of operations of the proton synchrotron about the actual accelerators.

https://raumzeit-podcast.de/2023/07/05/rz111-cern-geschichte...

https://raumzeit-podcast.de/2023/07/19/rz112-cern-die-beschl...

[formerly_proven]

This is a very nicely written, clear article. This is what good science communication looks like.

[ggm]

The inner tube is hard vacuum? Does freeze over risk rupture and also incur re evacuation and time to get those last pesky digits of nothing?

[CrampusDestrus]

you mean "consequenches"

[Animats]

Not too bad. The 2008 magnet failure took two years to repair, with over 50 magnets affected.

[ur-whale]

"Major consequences", and here I was already think a beam of protons flying at near the speed of light was let loose towards the city of Geneva.

But no, just a section of the accelerator that needs fixing.

[noobermin]

Bad time to be an LHC grad student.

[codeulike]

Quench

Nice verb

[unixhero]

Unforeseen consequences

Don't forget your crowbar

[trenchgun]

Tiny black holes that are going to swallow the earth in the coming hours?

[enslavedrobot]

Quenching a magnet is the result of poor planning and poor maintenance. The only time I ever hear about magnets quenching is at CERN. I remember they quenched ~40 at once when they first turned it on, a multi-million dollar error. What the hell are they up to over there!?

[sakex]

Do we know if this could have an impact on the local population? Also is there any research on the effect of the LHC on people living in Geneva and around?