TLDR: Looks like a failure in their CA-7 batch processing systems, which compounded so they weren't able to back out. Lots of the experts in RBS's implementation of CA-7 have been made redundant, and been replaced with offshore-expertise
I'm not going to question the proficiency of the offshore talent, however, I'm sure most on HN would agree that there's great value with legacy systems in not getting rid of the people who have been working on them for the last 20 years, and know all the idiosyncrasies and foibles.
Lots of the UK banks still use highly complex decades old main-frame based systems which interact with hundreds of other systems. It feels to me like the systems have got so complex, that its hard to see a way of replacing or rewriting and simplifying them.
"It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"
> there's great value with legacy systems in not getting rid of the people who have been working on them for the last 20 years, and know all the idiosyncrasies and foibles.
As someone who unfortunately has been in touch with CA-7 and who comes from a strong Unix/Linux background I can tell you I wouldn't want to touch that thing with a 30ft pole and on top of that these batches are usually extremely complicated in terms of dependencies and a ton of history and lessons-learned go into building those chains and the whole system is so complex yet so critical, we have 1 or 2 guys doing graveyard shifts for nothing but watching CA7 do its thing because if anything fails it could also shut down vital parts of the bank. (e.g. because literally some plain text files didn't get copied from A to B, in 2012...)
So not only is there possible great value in NOT getting rid of people who basically grew up with the system and know it inside out but when you do get rid of them it should be more than crystal clear to even the most disconnected manager that this is some really serious stuff you are outsourcing and just blindly trusting the friendly sales guy's promises is not going to cut it... it is almost like outsourcing a custom in-house software to an off-shore "expert" and hoping they can just instantly pick things up where your people left it off.
Now it is time to really take a very close look at those SLAs that the project team just wanted approved and signed asap....
I have been affected by this over the past week or so. I've been on holiday, and came back to a house with no fresh food, and no working bank account to pay for any, so I've been eating processed shite out of the freezer like a student for four days, and it's making me feel a bit poorly, not to mention the freezer is starting to look a little bare. I haven't had breakfast for days, and I have about two meals left if I stretch it. Surely this isn't acceptable in 2012?
Perhaps a little less seriously than the effect on my health from a lack of fresh food, I was paid my monthly salary yesterday, and that hasn't appeared in my account. I'm assuming that it'll appear in the next few days, but if it doesn't, I can't purchase the car I'd arranged to buy on Friday. It's all a massive balls-up and affecting my life in so many random ways.
Can any British readers recommend any alternative banks? I'm based in the centre of Sheffield, so I have the choice of pretty much all of them.
The Co-Op? (Although rumour has it that they're going to completely replace their computer systems either this year or early next year so that might be a case of jumping from one basket case to another!)
RBS appears to have fouled up the scheduling of their overnight batch run in a way that meant large tranches of transactions simply weren't going through the system. Fixing the problem has taken days (which may be down to poorly managed outsourcing) and it's then taking even more time to catch up with the backlog of unregistered transactions, because they can only fit so many extra into the overnight run.
If you've got a pay slip or something, then get yourself to an RBS branch and they'll give you cash to tide you over in the meantime. Don't starve yourself!
I moved from Natwest to First Direct a couple of months ago, and have had a great experience so far.
Their online banking isn't quite so nice as Natwest's, but that's more than made up for by the amazing phone service. I've had to call a couple of times (once to replace a lost card), and almost instantly been put through to someone who can solve my problem.
I keep the equivalent of about 500 USD in cash at a "hidden" spot in my apartment, in case something bad temporarily would happen to my bank account. Not enough for it to hurt if it was stolen, but enough to last for a while (I am a student). Perhaps that's something you could consider once this mess is sorted.
I've been with the Nationwide for years now, and never had a problem with them. Being a building society they're a bit less profit driven, though I don't know if anything can protect against offshore cost-cutting exercises in this day and age :/
I'm 22, so you might say I haven't experienced the full spectrum of banking products, but for what I have used I really like HSBC. Always quick to answer the phone, and maybe not so specific to their service levels but they were one of the few banks to not take bailout cash if I remember rightly.
They have their First Direct offering too, which gives you a higher interest rate for less face-time. You still get to use parts of the HSBC branches too.
I've used a few and ultimately ended up at HSBC for my business and personal account. Haven't had any problems so far (touch wood) and been pretty impressed so far with the service.
If I remember correctly, they were one of the few banks to come out the back of the banking crisis largely unscathed. Convinced a few friends to move too and have had good feedback so far (but ymmv as usual).
This thread is relevant to my interests: I'm totally going to switch when I can actually get at my cash. If the problem has been because of cuts to & outsourcing of IT staff, we can only assume there'll be more of them.
So, here's a question: does anyone know what the frontline IT setups of other UK banks are like? There'd be no point in jumping to an outfit which has similarly weak infrastructure.
Can any British readers recommend any alternative banks?
Consider building societies.
I've been very happy with the Nationwide for twenty odd years. Online banking is pretty good. Also, because of their organisational structure, building societies seem to have weathered the various banking stupidities of the last few years considerably better then some.
I'd strongly recommend Nationwide. Their staff are always friendly and helpful, it's a building society and their online banking site shits all over Natwest's offering (I had a NW account up until a year or so ago).
Alistair Darling's autobiography describes some pretty scary moments - apparently when the Chairman of RBS phoned him (in a bit of a panic) they only had enough cash to last, at most, 2 hours.
[NB Slightly amusing to be posting this to HN when I am maybe 200m from the old RBS HQ]
Does anyone else think that 110,000 transactions shouldn't take a whole weekend? Of course, we don't know exactly what a transaction involves but I would have expected this to take only a few hours at most. But, of course, I'm assuming modern hardware. It seems as if they just don't have the spare capacity to catch-up after an outage like this.
I wonder if this will make the conservative bank IT departments even more conservative or will they go in the opposite direction and realise the benefits of modern technology (performance, scalability and a pool of knowledgeable talent)?
The only explanation for 110,000 transactions taking a whole weekend is if bank employees are handling them manually. That amount of work should take 60 seconds.
Attention entrepreneurs: there is a real opportunity, at least in the UK, for a bank built the right way from the ground up. I'm talking about contemporary systems, real product ownership in the technology group where IT isn't just an outsourced cost, automated testing and continuous integration in the systems. Most of the efforts at 'new banks' in the UK have been built on the foundations of ancient, creaking mainframe systems. There is a lot of important logic in those legacy systems for sure, but I think progress could be made in building fresh solutions by starting at the relatively simple end of things e.g. customer setup, identity verification and basic savings accounts. Could built out from there to offer more complex products. The upsides of a fresh approach: lower maintenance costs, higher reliability, more operations can be carried out in realtime etc.
First problems with them at University, they gave away free 5 year railcards to all students who opened a student bank. No one I knew got one, you only got it if you went through a laborious complaint process. Once I realised I did not have my card after several months I complained, and they refused to issue me one because I waited too long even though it was their fault for not dispatching it in the first place like they promised.
Second problems was with their disgusting ethics when I fell upon hard times.
Third time was when I went travelling and they lied to me straight up about the fees I would be charged using my card abroad, which caused problems. (They did settle with me afterwards and give me some money back, but it was another laborious mission).
Throughout, general appalling service speckled with useless calls from customer service research departments asking me how long I had to queue for in my recent visit and other pointless questions. Flacid attempts at 'caring for customers'.
I'm at a stage now where I can move from them, and will do soon. I'm always happy to spend time telling people about my experiences with them and my recommendation to completely avoid them if possible.
I know banks are not exactly beacons of ethics and good business in society. My expectations for banks is low because of this yet Natwest consistently have managed to go to new depths.
Bankers do sometimes forget that these days, their entire business is an elaborate front end for a general ledger program.
I predict that banks which see IT as a key source of competitive advantage will prosper in the coming decades. Banks which see it as a cost centre will eventually be slugged with debacles like these.
Of course, such self-serving aggrandisement as this is not without its own problems. Banking software is enormously complex and is meant to be utterly reliable. Two things that rarely mix well.
Add in the storied difficulties of running enormous software projects and you begin to understand why most banks just don't want to hear about how the transactions occur, just that they occur.
[+] [-] tomgallard|13 years ago|reply
TLDR: Looks like a failure in their CA-7 batch processing systems, which compounded so they weren't able to back out. Lots of the experts in RBS's implementation of CA-7 have been made redundant, and been replaced with offshore-expertise
I'm not going to question the proficiency of the offshore talent, however, I'm sure most on HN would agree that there's great value with legacy systems in not getting rid of the people who have been working on them for the last 20 years, and know all the idiosyncrasies and foibles.
Lots of the UK banks still use highly complex decades old main-frame based systems which interact with hundreds of other systems. It feels to me like the systems have got so complex, that its hard to see a way of replacing or rewriting and simplifying them.
[+] [-] timrobinson|13 years ago|reply
http://www.guardian.co.uk/technology/2012/jun/25/how-natwest...
"It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"
[+] [-] kahawe|13 years ago|reply
As someone who unfortunately has been in touch with CA-7 and who comes from a strong Unix/Linux background I can tell you I wouldn't want to touch that thing with a 30ft pole and on top of that these batches are usually extremely complicated in terms of dependencies and a ton of history and lessons-learned go into building those chains and the whole system is so complex yet so critical, we have 1 or 2 guys doing graveyard shifts for nothing but watching CA7 do its thing because if anything fails it could also shut down vital parts of the bank. (e.g. because literally some plain text files didn't get copied from A to B, in 2012...)
So not only is there possible great value in NOT getting rid of people who basically grew up with the system and know it inside out but when you do get rid of them it should be more than crystal clear to even the most disconnected manager that this is some really serious stuff you are outsourcing and just blindly trusting the friendly sales guy's promises is not going to cut it... it is almost like outsourcing a custom in-house software to an off-shore "expert" and hoping they can just instantly pick things up where your people left it off.
Now it is time to really take a very close look at those SLAs that the project team just wanted approved and signed asap....
[+] [-] andyking|13 years ago|reply
Perhaps a little less seriously than the effect on my health from a lack of fresh food, I was paid my monthly salary yesterday, and that hasn't appeared in my account. I'm assuming that it'll appear in the next few days, but if it doesn't, I can't purchase the car I'd arranged to buy on Friday. It's all a massive balls-up and affecting my life in so many random ways.
Can any British readers recommend any alternative banks? I'm based in the centre of Sheffield, so I have the choice of pretty much all of them.
[+] [-] pja|13 years ago|reply
RBS appears to have fouled up the scheduling of their overnight batch run in a way that meant large tranches of transactions simply weren't going through the system. Fixing the problem has taken days (which may be down to poorly managed outsourcing) and it's then taking even more time to catch up with the backlog of unregistered transactions, because they can only fit so many extra into the overnight run.
If you've got a pay slip or something, then get yourself to an RBS branch and they'll give you cash to tide you over in the meantime. Don't starve yourself!
[+] [-] rlpb|13 years ago|reply
If you depend so much on one single provider then you have a problem, not the system.
Get some redundancy. Get at least one other bank account. Get a credit card, or two. Keep some emergency cash lying around.
[+] [-] JonWood|13 years ago|reply
Their online banking isn't quite so nice as Natwest's, but that's more than made up for by the amazing phone service. I've had to call a couple of times (once to replace a lost card), and almost instantly been put through to someone who can solve my problem.
[+] [-] Zirro|13 years ago|reply
[+] [-] davedx|13 years ago|reply
[+] [-] stevejalim|13 years ago|reply
[+] [-] geoffw8|13 years ago|reply
They have their First Direct offering too, which gives you a higher interest rate for less face-time. You still get to use parts of the HSBC branches too.
Hope all ends well for you sir!
[+] [-] bitdiffusion|13 years ago|reply
If I remember correctly, they were one of the few banks to come out the back of the banking crisis largely unscathed. Convinced a few friends to move too and have had good feedback so far (but ymmv as usual).
[+] [-] prehensile|13 years ago|reply
So, here's a question: does anyone know what the frontline IT setups of other UK banks are like? There'd be no point in jumping to an outfit which has similarly weak infrastructure.
[+] [-] maxer|13 years ago|reply
You just need to warn them when you travel or they will block your card.
[+] [-] adrianhoward|13 years ago|reply
Consider building societies.
I've been very happy with the Nationwide for twenty odd years. Online banking is pretty good. Also, because of their organisational structure, building societies seem to have weathered the various banking stupidities of the last few years considerably better then some.
[+] [-] Paul_S|13 years ago|reply
[+] [-] corford|13 years ago|reply
[+] [-] petercooper|13 years ago|reply
Fair enough, but this isn't endemic to RBS. It's like looking for a new neighborhood to move to if your cat gets run over.
[+] [-] megablast|13 years ago|reply
[+] [-] UnoriginalGuy|13 years ago|reply
[+] [-] arethuza|13 years ago|reply
http://www.youtube.com/watch?v=k1cRXipswfA
Alistair Darling's autobiography describes some pretty scary moments - apparently when the Chairman of RBS phoned him (in a bit of a panic) they only had enough cash to last, at most, 2 hours.
[NB Slightly amusing to be posting this to HN when I am maybe 200m from the old RBS HQ]
[+] [-] hopeless|13 years ago|reply
"The bank processed more than 110,000 transactions with a value of €500 million over the weekend" - http://www.irishtimes.com/newspaper/breaking/2012/0625/break...
Does anyone else think that 110,000 transactions shouldn't take a whole weekend? Of course, we don't know exactly what a transaction involves but I would have expected this to take only a few hours at most. But, of course, I'm assuming modern hardware. It seems as if they just don't have the spare capacity to catch-up after an outage like this.
I wonder if this will make the conservative bank IT departments even more conservative or will they go in the opposite direction and realise the benefits of modern technology (performance, scalability and a pool of knowledgeable talent)?
[+] [-] hopeless|13 years ago|reply
That's a two week outage. Is this sort of thing common in other countries?
[+] [-] qntm|13 years ago|reply
[+] [-] dasil003|13 years ago|reply
[+] [-] highlander|13 years ago|reply
[+] [-] TomGullen|13 years ago|reply
First problems with them at University, they gave away free 5 year railcards to all students who opened a student bank. No one I knew got one, you only got it if you went through a laborious complaint process. Once I realised I did not have my card after several months I complained, and they refused to issue me one because I waited too long even though it was their fault for not dispatching it in the first place like they promised.
Second problems was with their disgusting ethics when I fell upon hard times.
Third time was when I went travelling and they lied to me straight up about the fees I would be charged using my card abroad, which caused problems. (They did settle with me afterwards and give me some money back, but it was another laborious mission).
Throughout, general appalling service speckled with useless calls from customer service research departments asking me how long I had to queue for in my recent visit and other pointless questions. Flacid attempts at 'caring for customers'.
I'm at a stage now where I can move from them, and will do soon. I'm always happy to spend time telling people about my experiences with them and my recommendation to completely avoid them if possible.
I know banks are not exactly beacons of ethics and good business in society. My expectations for banks is low because of this yet Natwest consistently have managed to go to new depths.
[+] [-] jacques_chester|13 years ago|reply
I predict that banks which see IT as a key source of competitive advantage will prosper in the coming decades. Banks which see it as a cost centre will eventually be slugged with debacles like these.
Of course, such self-serving aggrandisement as this is not without its own problems. Banking software is enormously complex and is meant to be utterly reliable. Two things that rarely mix well.
Add in the storied difficulties of running enormous software projects and you begin to understand why most banks just don't want to hear about how the transactions occur, just that they occur.
[+] [-] yread|13 years ago|reply
The Register uses cookies. Some may have been set already. Read about managing our cookies.
Please click the button to accept our cookies. If you continue to use the site, we'll assume you're happy to accept the cookies anyway.
Is this really the way to handle the new cookie regulations?
[+] [-] maxer|13 years ago|reply
http://www.bbc.co.uk/news/uk-northern-ireland-18590028