I hate big power. When I was an IBM engineer, power into the computer room was a big deal. Big diesel generators feeding into the main switchboard along with redundant mains connections.
One incident I remember is when an electrician was working on one of the power feeds while it was live. Foolish, but the kind of thing people did when getting approval to shut off power was a bureaucratic nightmare.
The sparky was tightening down a bolt on one of the phases with a big crescent spanner when he inadvertantly made contact with another phase.
There was a flash that lit up the room and a loud bang. His crescent spanner literally melted in the middle. Incredibly he was uninjured, although he did take the whole room down, which was an extremely serious matter.
Modern computers that plug into the wall socket are much more pleasant to work on.
Nice, my favorite big power story was a technician who was bringing up the next set of "rows" in a very large web scale data center.
One component of that task was running thousands of feet of CAT-6e cable. Of course you want to be sure your spool of CAT-6e cable has no wire breaks in it before you start using it[1] to wire up racks so there was a step that involved putting a connector on the stub of cable that was inside the reel, and a connector on the end of the cable that came off the reel, and then hooking up an instrument between them to measure the resistance/propagation/etc.
It is convenient to work at waist height for this, so our hapless technician had the cable spool put on a convenient big square brick like metal structure outside the building. He added the connectors, hooked up the tester, and the tester exploded. (well not completely but it did experience some rapid unplanned disassembly due to a large number of capacitors suddenly not existing.
The "brick like structure" that the technician was using as a table top happened to be one of the transformers that was converting high voltage down to 440V 3-phase for the data center. When he effectively shorted the cable loop with his tester he air coupled into the field around the transformer and put > 400V into the tester. Whoops!
[1] tracking down a bad network cable is not a lot of fun.
Electricians work on live stuff all the time. They just rarely do so in residential settings which is why most people don't realize that. He was just sloppy. There procedures you're supposed to be following (like using insulated tools) to prevent stuff like that.
Good thing tools are made of such resistive metals. I can only imagine how that would have gone if the metal with the best tensile strength for a wrench was copper.
If anyone is in Victoria, Australia and wants to play with an IBM z9 mainframe, i picked one up for about $500 and although I've managed to power and boot it, i haven't managed to load zOS on it properly. Got some time to spend on it and would love it if anyone else is interested in experimenting/getting it up and running.
I also bought the relevant storage array (but no disks). Unfortunately that takes up a heap of space and I'm looking to offload it, it's free if anyone wants it, otherwise ill probably have to scrap it :(
I usually lurk here but feel free to reach out via twitter (same username)
I’ve seen those “support element” laptops hanging off of racks before, but I always assumed that in this case (IBM mainframe, IBM support element, vertical integration) they were not laptops as such, but rather thin clients in the most literal sense: just IO interfaces with a ribbon cable feeding back to the mainframe, where the framebuffer of the mainframe’s dom0/maintenance VM was pushed over VGA and keyboard+mouse+peripherals were returned over USB. Essentially as if the support element were a Thunderbolt Display with USB HID devices plugged into it. Like a modern VT100!
I’m honestly surprised to learn that I’m wrong, as having the laptop be a real laptop would introduce all sorts of headaches compared to having the laptop be a dumb terminal—it’d have active components like disks and fans that could to burn out (and not be hot-swappable like the server’s own components are); and more importantly, it’d have to talk to the server using some form of network interface (unless it’s just a glorified glass terminal for a USB-serial adapter.) In which case, bad behaviour on the server—exactly the case where you need a local admin console—could block or steal resources from your connection attempt. You may as well just be SSHing into the dom0 from across the room over the SBC VLAN at that point.
The support laptop on an IBM mainframe is analogous to the setup on a Cray machine. The System Management Workstation or 'SMW' (previously a Supermicro tower workstation with a bunch of ethernet), now a Dell Poweredge rackmount is its own system which controls the rest of the machine--booting, shutdown, and all of the other myriad maintenance tasks on a cluster like that. It's also responsible for building boot images and interrogating the low level controllers on each rack and on system blades themselves.
During bringup of a machine like that, the SMW host is installed first from scratch with a Linux distribution, hardware inventory is enumerated, boot images are built for individual node roles in the system and then the system performs its boot procedure.
The new iSeries we bought (Power S914) uses a Windows computer as its terminal. I would prefer a dedicated terminal with all the keys, but that how it goes now. One thing about it, the tech that is doing the installation is remoted into that PC doing the steps to copy our old machine to the new one.
On a side note, the new machine has LTO-7 and our old machine (9406-800) predates LTO, so it makes the upgrade a bit longer.
Support element has to be more or less full-blown computer as the mainframe itself does not have any real firmware to boot^H^H^H^HIPL itself. Before support elements the IPL process involved initalizing various things by toggling switches on the console.
Also why the mainframe would even have a framebuffer?
Providing a console interface (ie. 3270 terminal emulator) to whatever OS runs on the mainframe is only small part of what SE does.
I like seeing posts like this showing that there is still interest in operating with mainframes and bare-metal servers these days.
My biggest fear is that you are in the minority of developers that can single-handedly operate one in the age of buzzword-ridden lingo such as AWS, GCP, Heroku, etc which forces a dogma to run rampant in our industry to host all of your startup/company only on other peoples's servers rather than to setup up your own in-house bare-metal servers instead.
So I am very impressed to see this, as an added bonus it is a mainframe. Please post more of this.
It's not about clouds. The problem is with mainframe manufacturers. Their prices are ridiculous, so no sane man with limited budgets would buy it. I can build incredible powerful server from a desktop or workstation components for a few thousand dollars. Probably can reduce it to hundreds with using old parts. I can increase price to x2 and buy some HP or Dell blade and that would be real server hardware. But nobody is going to increase price to x100 and buy mainframe. That's just too much.
> My biggest fear is that you are in the minority of developers that can single-handedly operate one
You're right. About 6 months ago the University of Canberra, Australia started it's 'Bachelor of Information Technology in Mainframe Computing' degree program because of the shortage.
This is a fascinating bit of nerdery. Would someone be willing to explain what mainframes are used for in a modern context, and how they compare to a really high-capacity x86 machine? I know that one difference is that you can hot-swap pretty much any component on a mainframe, but would be very curious to learn more about these beasts.
A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus.
Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.
I have interviewed at one of the biggest retailers in Italy (Esselunga, a retailer in groceries and pretty much all the related things).
While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one.
What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way.
They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption.
So basically what I got from that is that the benefits are:
That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.
I looked at the author's previous post where he details the process of buying the thing. A fair amount of it goes over my head, I must admit. But that final price tag.... You've gotta really love mainframes. I can't imagine the utility bills for it. The power usage was reduced to 1.7kW. I guess some people buy a sports car and some buy a mainframe. Personally, I'd never be able to drop that kind of money for a hobby. I think emulating a mainframe on a fancy PC would scratch my itch. But to each their own.
If you asked me how much I thought 700kg of computer should cost, I probably would have guessed more than $12k. Is this thing modern? Specs don't seem too impressive but I don't know anything about these things.
Interesting it runs straight off normal 3phase mains - the only mainframe I worked (a bit ) on was a small Unisys one and it had its own 400v supply as part of the set up.
The Bulk Power Regulator mentioned in the blog post is essentially an 400V DC supply that powers everything else in the rack (including the blowers with threephase motors and their own DC/AC invertor).
So it is similar except the power supply is built on modern power electronics and is quite small and internal in the rack.
I was a former mechanical maintenance tech for an auto manufacturer, so this story feels familiar to connecting a CNC lathe, or a multi-axis mill, but reading this guys ham-fisted accounts made me pretty livid..
>The datacenter staff, not needing a forklift in their day-to-day, had managed to solicit the services of a forklift, the associated operator, and some very handy folks to help navigate the mainframe from the storage space to its final location.
Since this thing can approach the cost of a rolls-royce,and weighs half as much as a prius, movers and millwrights should have been called in. watching two chubby guys in tennis shoes and no gloves move a 700 kilo server in reverse would have given me a heart attack.
> The various guides did hint towards how the phases are connected to various Bulk Power Regulators (BPR) but nothing was very definite.
that should have been your clue to hire an electrician. there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR.
>I tried to do my best to construct a power cable
Make sure you tell your building supervisor and the fire brigade you "tried your best." there is no "tried your best" section in the electrical code. You either construct a cable to spec, or not.
>Now, I assumed that given there is no neutral it has to be a delta system
Since youre handling voltages that can kill you, perhaps a meter?
>The uncertainty was which phase-pair powered the lowest BPR? I guessed A-B
OR your BPR explodes, your HV breaker trips, and you wipe out power for half the building. Christ on his throne.
> there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR
No, in IBM mainframes, the rack component that takes the mains AC and starts doing something with it is, indeed, called a "bulk power regulator". For example, IBM part number 6186-7040, "Bulk Power Regulator".
> that should have been your clue to hire an electrician
Note in the article he did say "I confirmed the wiring scheme with some friends I have that do the power installations at Dreamhack, and even had them do the wiring as they have all the qualifications needed to do so safely and legally."
Why? It shouldn't be. They use 3-phase in Europe just like in the US, for industrial locations. The differences in residential or lower-voltage standards really have nothing to do with industrial power availability.
[+] [-] beachy|6 years ago|reply
One incident I remember is when an electrician was working on one of the power feeds while it was live. Foolish, but the kind of thing people did when getting approval to shut off power was a bureaucratic nightmare.
The sparky was tightening down a bolt on one of the phases with a big crescent spanner when he inadvertantly made contact with another phase.
There was a flash that lit up the room and a loud bang. His crescent spanner literally melted in the middle. Incredibly he was uninjured, although he did take the whole room down, which was an extremely serious matter.
Modern computers that plug into the wall socket are much more pleasant to work on.
[+] [-] ChuckMcM|6 years ago|reply
One component of that task was running thousands of feet of CAT-6e cable. Of course you want to be sure your spool of CAT-6e cable has no wire breaks in it before you start using it[1] to wire up racks so there was a step that involved putting a connector on the stub of cable that was inside the reel, and a connector on the end of the cable that came off the reel, and then hooking up an instrument between them to measure the resistance/propagation/etc.
It is convenient to work at waist height for this, so our hapless technician had the cable spool put on a convenient big square brick like metal structure outside the building. He added the connectors, hooked up the tester, and the tester exploded. (well not completely but it did experience some rapid unplanned disassembly due to a large number of capacitors suddenly not existing.
The "brick like structure" that the technician was using as a table top happened to be one of the transformers that was converting high voltage down to 440V 3-phase for the data center. When he effectively shorted the cable loop with his tester he air coupled into the field around the transformer and put > 400V into the tester. Whoops!
[1] tracking down a bad network cable is not a lot of fun.
[+] [-] dsfyu404ed|6 years ago|reply
Also relevant: https://i.imgur.com/HjqfZZh.jpg
[+] [-] derefr|6 years ago|reply
[+] [-] C1sc0cat|6 years ago|reply
Real Datacentres have diverse routing for every thing
[+] [-] cr0sh|6 years ago|reply
Usually people who make that mistake don't get away any where close to that cleanly. One of the more horrific ways to accidentally die...
[+] [-] infosecfriends|6 years ago|reply
I usually lurk here but feel free to reach out via twitter (same username)
[+] [-] derefr|6 years ago|reply
I’m honestly surprised to learn that I’m wrong, as having the laptop be a real laptop would introduce all sorts of headaches compared to having the laptop be a dumb terminal—it’d have active components like disks and fans that could to burn out (and not be hot-swappable like the server’s own components are); and more importantly, it’d have to talk to the server using some form of network interface (unless it’s just a glorified glass terminal for a USB-serial adapter.) In which case, bad behaviour on the server—exactly the case where you need a local admin console—could block or steal resources from your connection attempt. You may as well just be SSHing into the dom0 from across the room over the SBC VLAN at that point.
[+] [-] theideaofcoffee|6 years ago|reply
During bringup of a machine like that, the SMW host is installed first from scratch with a Linux distribution, hardware inventory is enumerated, boot images are built for individual node roles in the system and then the system performs its boot procedure.
[+] [-] protomyth|6 years ago|reply
On a side note, the new machine has LTO-7 and our old machine (9406-800) predates LTO, so it makes the upgrade a bit longer.
[+] [-] dfox|6 years ago|reply
Also why the mainframe would even have a framebuffer?
Providing a console interface (ie. 3270 terminal emulator) to whatever OS runs on the mainframe is only small part of what SE does.
[+] [-] iofiiiiiiiii|6 years ago|reply
[+] [-] rvz|6 years ago|reply
My biggest fear is that you are in the minority of developers that can single-handedly operate one in the age of buzzword-ridden lingo such as AWS, GCP, Heroku, etc which forces a dogma to run rampant in our industry to host all of your startup/company only on other peoples's servers rather than to setup up your own in-house bare-metal servers instead.
So I am very impressed to see this, as an added bonus it is a mainframe. Please post more of this.
[+] [-] vbezhenar|6 years ago|reply
[+] [-] eecc|6 years ago|reply
If you look hard enough at Kubernetes - perhaps with a Kafka queue in the mix - it will turn into a big Websphere cluster and wave back.
[+] [-] PinkMilkshake|6 years ago|reply
You're right. About 6 months ago the University of Canberra, Australia started it's 'Bachelor of Information Technology in Mainframe Computing' degree program because of the shortage.
[+] [-] WrtCdEvrydy|6 years ago|reply
I'm already hearing the term internally and within job postings -> 'repatriation'... bringing it back from AWS.
[+] [-] pjmlp|6 years ago|reply
So we did some small stuff on AWS and Azure and that was all about it, in max 5 years time the fad will be on the down path.
[+] [-] idlewords|6 years ago|reply
[+] [-] derefr|6 years ago|reply
Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.
[+] [-] znpy|6 years ago|reply
While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one.
What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way.
They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption.
So basically what I got from that is that the benefits are:
- write once, run forever code
- 100% uptime (not "five nines", literally 100% uptime)
- a reference environment that you can rely on (no runtime change, everything that works today will keep workin in 30 years)
- value added consulting from IBM
- a complete set of tools (runtime, database, primitives, libraries etc)
[+] [-] lboc|6 years ago|reply
https://www.ibm.com/support/knowledgecenter/zosbasics/com.ib...
http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf
Plus one that I seem to be posting on a monthly basis these days:
https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf
Here's one I just found, and will be reading tonight I think:
http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf
Enjoy!
[+] [-] ofrzeta|6 years ago|reply
[+] [-] vbezhenar|6 years ago|reply
[+] [-] nickpsecurity|6 years ago|reply
https://en.wikipedia.org/wiki/Channel_I/O
That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.
[+] [-] Lowkeyloki|6 years ago|reply
[+] [-] titanomachy|6 years ago|reply
[+] [-] C1sc0cat|6 years ago|reply
[+] [-] Merrill|6 years ago|reply
[+] [-] blattimwind|6 years ago|reply
[+] [-] dfox|6 years ago|reply
So it is similar except the power supply is built on modern power electronics and is quite small and internal in the rack.
[+] [-] nimbius|6 years ago|reply
>The datacenter staff, not needing a forklift in their day-to-day, had managed to solicit the services of a forklift, the associated operator, and some very handy folks to help navigate the mainframe from the storage space to its final location.
Since this thing can approach the cost of a rolls-royce,and weighs half as much as a prius, movers and millwrights should have been called in. watching two chubby guys in tennis shoes and no gloves move a 700 kilo server in reverse would have given me a heart attack.
> The various guides did hint towards how the phases are connected to various Bulk Power Regulators (BPR) but nothing was very definite.
that should have been your clue to hire an electrician. there is no such thing as a bulk power regulator. this is a BUCK POWER REGULATOR.
>I tried to do my best to construct a power cable
Make sure you tell your building supervisor and the fire brigade you "tried your best." there is no "tried your best" section in the electrical code. You either construct a cable to spec, or not.
>Now, I assumed that given there is no neutral it has to be a delta system
Since youre handling voltages that can kill you, perhaps a meter?
>The uncertainty was which phase-pair powered the lowest BPR? I guessed A-B
OR your BPR explodes, your HV breaker trips, and you wipe out power for half the building. Christ on his throne.
[+] [-] racingmars|6 years ago|reply
No, in IBM mainframes, the rack component that takes the mains AC and starts doing something with it is, indeed, called a "bulk power regulator". For example, IBM part number 6186-7040, "Bulk Power Regulator".
> that should have been your clue to hire an electrician
Note in the article he did say "I confirmed the wiring scheme with some friends I have that do the power installations at Dreamhack, and even had them do the wiring as they have all the qualifications needed to do so safely and legally."
[+] [-] 6thaccount2|6 years ago|reply
There has been a lot of mainframe related posts on HN lately.
[+] [-] tyingq|6 years ago|reply
[+] [-] dsfyu404ed|6 years ago|reply
*which slightly more complicated in Europe.
[+] [-] magduf|6 years ago|reply