top | item 26180785

Judge: Citibank isn't entitled to $500M it sent to various creditors last August

1232 points| danbr | 5 years ago |arstechnica.com

601 comments

order
[+] airstrike|5 years ago|reply
> Raj thought that checking the "principal" checkbox and entering the number of a Citibank wash account would ensure that the principal payment would stay at Citibank. He was wrong. To prevent payment of the principal, Raj actually needed to set the "front" and "fund" fields to the wash account as well as "principal." Raj didn't do that.

I don't even have anything to add. The paragraph speaks for itself... You can't make this up

[+] jacurtis|5 years ago|reply
It is also worth noting that Raj wasn't alone in the confusion on how to perform this task. Before he could complete the transfer his boss in India also had to look at what Raj did. Then their boss in Deleware also approved the transaction.

ALL THREE PEOPLE INVOLVED IN THIS TASK WERE CONFUSED by the interface and all three thought that this was the correct way to do this.

This wasn't the case of one person checked the wrong box. Everyone who looked at this was confused. This problem was not individual, but systemic.

[+] gzer0|5 years ago|reply
If you read literally the next few lines:

> Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta. All three believed that setting the "principal" field to an internal wash account number would prevent payment of the principal. As he approved the transaction, Fratta wrote: "looks good, please proceed. Principal is going to wash."

[+] redshirtrob|5 years ago|reply
These issues are everywhere if you bother to notice. My in-laws had a coffee maker with a built-in grinder. The grind function was default enabled, so if you went to brew some coffee from pre-ground beans and just turned it on, the grinder would kick on with an awful sound.

The solution they came up with was a "Grinder Off" button that lit up to indicate the grinder was turned off. It was...confusing to say the least. And as an infrequent user of this coffee maker it was all too easy to screw up. To add insult to injury, that off light turned off (i.e. enabled grinding) every time you brewed a new pot. You had to be sure to press it every time you brewed from pre-ground beans, which was always for us.

[+] cosmodisk|5 years ago|reply
You look at the screenshot,then read the summary on what happened and then wonder how on earth the bank still has any money left on their accounts with systems like this.
[+] JJMcJ|5 years ago|reply
So there wasn't a final screen that said something like

You are about to send $83,932,198.83 to (various account information fields)

Unlike, say, a site that sells hand knit mittens or maybe organic kibble.

[+] w23j|5 years ago|reply
Well one could add, that other possible options are "COLLAT", "COMPINTSF", "DEFAUL" and "DFLFTC". Hard to believe that anyone could possibly be confused by that or miss the clearly necessary "FRONT" and "FUND".
[+] 1024core|5 years ago|reply
Let's not blame the lowly sub here.

The next paragraph continues:

Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta. All three believed that setting the "principal" field to an internal wash account number would prevent payment of the principal. As he approved the transaction, Fratta wrote: "looks good, please proceed. Principal is going to wash."

[+] dheera|5 years ago|reply
I've used Citizens Bank before for business reasons and their UI is probably one of the worst I have ever seen. I don't understand why consumers get such good stuff and enterprises get such crap.
[+] dopidop|5 years ago|reply
I work on a large financial transaction system. A dumb backbone for bank the same size as citi.

This UI is pretty good ! Haha. No seriously today we had a one hour conversation on what happen when a user cancel a trade twice. I don’t think we’re gonna find a elegant solution within the contraints of the clusterfuck of sub-system we maintaining. Kinda disheartening.

[+] woobar|5 years ago|reply
Not defending this horrible UI, but this is a case of a non-standard operation and one should be extra careful when trying to override default behavior. Matt Levine has more details.[1]

If I am an approver of a $900M transfer using an edge case I will follow instructions:

"Citibank’s internal Fund Sighting Manual provides instructions for suppressing Flexcube’s default. When entering a payment, the employee is presented with a menu with several “boxes” that can be “checked” along with an associated field in which an account number can be input. The Fund Sighting Manual explains that, in order to suppress payment of a principal amount, “ALL of the below field[s] must be set to the wash account: FRONT[;] FUND[; and] PRINCIPAL” — meaning that the employee had to check all three of those boxes and input the wash account number into the relevant fields." [1]

[1] https://www.bloomberg.com/opinion/articles/2021-02-17/citi-c...

[+] DrBazza|5 years ago|reply
Having previously worked at "Megabank", this does not surprise me at all.

Conway's Law applies here. It's a hugely dysfunctional bank, saddled with bureaucracy, fragile software (company change control ran on a IBM mainframe as late as 2015, requiring IE6), teams that are 'silo-ed', and talking with other teams are often the equivalent of a divorce lawyer's communication between spouses.

Sandy Weill's crowning glory of mashing together Salomon Brothers, Smith Barney, Primerica, Travellers, Citicorp, and Diner's Club.

[+] bluishgreen|5 years ago|reply
Amusing and mildly racist use of the name Raj in the comment, not from the original article as far as I can see. Please note that John of Delaware, his boss confirmed that principal goes to wash and said to “please proceed”
[+] SpaceManNabs|5 years ago|reply
And Raj will still probably get fired for jointly discovering this vulnerability.
[+] joejerryronnie|5 years ago|reply
I can understand if you’re venmoing your buddy $20, but if your are transferring millions of dollars, and faced with that horrendous interface, you might think to consult the standard operating procedures.
[+] fortran77|5 years ago|reply
I also wonder what the action of setting an account number in the "PRINCIPAL" field does if the system basically ignores it unless other fields are set. Why is this configuration even possible?
[+] koheripbal|5 years ago|reply
To me this speaks more to the need for major money transfers to have more programmatic checks in place as well aa 2nd person signoffs.
[+] Shivetya|5 years ago|reply
as many pointed out, it still isn't uncommon to repurpose or "Extend" and existing UI to do stuff it was never meant to do and then never go back and clean up the efforts when such use is common place.

resulting in issues like Citi got. Seriously who thought that UI, requirements, text, and all, was a long term solution for handling any amount of money?

[+] Igelau|5 years ago|reply
Ars Technica must have redacted the name. It just says "subcontractor" now.
[+] kables|5 years ago|reply

[deleted]

[+] CivBase|5 years ago|reply
> The actual work of entering this transaction into Flexcube fell to a subcontractor in India named Arokia Raj.

> Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta.

The names of these individuals seems like an unnecessary detail, especially since the article names the software interface as the culprit. I can't help but think about the recent NYT/SSC incident.

[+] onli|5 years ago|reply
This really is an awesome example of a bad UI. These are exactly the kind of UIs you see in the enterprise all the time, with the article containing the totally crazy description of what was expected plus the screenshots to show that there was absolutely no way to get this, if not being the programmer who wrote it (and even then, no guarantee at all to get it right). To have a somewhat exact approach of detecting and fixing issues like this, that is what usability is all about.

So it's really the design of the software and not just the style of it that was wrong and had to be improved. A usability professional would have caught that immediately - and would have been way cheaper.

[+] xyzelement|5 years ago|reply
To be sure, this is an ugly UI but this is not necessarily a pure UI issue at root.

Internal applications have to be more complex that external applications - that's why you sometimes have to call a company to do something the consumer frontend doesn't support. The employees are expected to operate a more powerful/flexible system than the customer, and I think there's always a risk inherent there.

In this case, it's likely that it's not just a dumb UI thing that the employee "needed to set the "front" and "fund" fields to the wash account as well as "principal." I suspect thank "front" and "back" are actually real business concepts in how the bank models transfers which the employee/reviewer did not understand well. Instead they expressed a different model of a transfer (an external one) which is a totally legitimate use case, just not the intended one.

That's kinda one level of empathy, I suspect this really happened because these users think about these operations as "I check this box, then I check that box" rather than "I get how transfers work and I am going to express my intention using that understanding." So it's probably much more of a training issue, because to give these people a very narrow and polished consumer-style frontend would take away the flexibility they likely need to actually execute their roles.

[+] dekerta|5 years ago|reply
Not surprising that Oracle's promotional video for Flexcube is all corporate-speak and buzz-words, and doesn't actually show the product or even explain what it does. https://videohub.oracle.com/media/t/1_mxpp4dyv

All enterprise software is terrible like this. It's because the people ordering the software are disconnected from the people who will have to use it

[+] sebmellen|5 years ago|reply
> "Oracle's new machine learning adapter unlocks intelligence engrained in individual operational patterns."

This made me laugh so hard, I can just imagine some bank executive hearing this and nodding, thinking they're going to change the world somehow.

[+] zucker42|5 years ago|reply
I couldn't stop laughing while watching that video. It sounds like a parody. Does that type of video seriously convince CTOs to buy FlexCube?
[+] alecco|5 years ago|reply
This is golden. It's spoken like a voice synth. "cut operating costs, deliver high volumes, and transfer these benefits to its customers", then "meanwhile, at the back office, ... Oracle's new Machine Learning adapter unlocks intelligence ingrained in indivudual operational patterns".
[+] peterkelly|5 years ago|reply
I just watched that video and now I know even less about the product than I did before (which was nothing).
[+] bob1029|5 years ago|reply
I had to share this with my team today. We produce an app for banking that is used at the front line (i.e. in the branches). We have explicitly acknowledged the fact that the bulk of our users are going to be new to the business & are going to be paid peanuts while simultaneously being expected to produce consistent business outcomes. As a result, we have tailored our interfaces for self-discoverability and intuitiveness. Making a UI intuitive is a book all by itself, but suffice to say that we spend a lot of time agonizing over how UI elements are laid out so that users are less likely to make errors in judgement.

Our application is not immune to exposing exceptions to the business. But, we have added measures throughout to minimize this as much as possible. In areas where there is a lot of jargon which might confuse new employees, we put help buttons which allow a user to quickly review important terms & other documentation in-line with the actual functionality. This makes our application almost a training course in and of itself.

Additionally, in cases where we identify that poor choices could have broader impacts to other parts of the business (i.e. lighting 500mm on fire), we add explicit validations with hard cutoff limits to prevent insane things from happening. In this specific case, we would probably walk the user through a decision tree to force them down a valid path and ensure the funds were being tagged to the correct account(s). We also have approval loops in our application for more sensitive operations, but even these have integral validations & other measures to ensure that "experienced" employees don't screw up either.

[+] CobrastanJorji|5 years ago|reply
This is a great lesson. However, I'm afraid for you're designing your product to be good for the people who are being paid peanuts and are new to the business, but those are not the people who are buying your software. They're just the users of the software. Oracle put basically no thought into their UI but put a lot of money into promotional videos and sales, and they sold their software to giant international banks because the people buying the software experience it only through videos and salespeople.

I'm terribly afraid that you're going to be punished for focusing on the right things.

[+] lefstathiou|5 years ago|reply
I would like to point out that Flexcube is built by Oracle. Citi's mistake here was in tying themselves to what is likely the "safest", "market leading" solution.

If you all could see how frustrating Ariba (by SAP) is, you'd have a great laugh.

[+] gzer0|5 years ago|reply
> Ordinarily, the law would be on Citibank's side here. Under New York law, someone who sends out an erroneous wire transfer—for example, sending a payment to the wrong account—is entitled to get the money back.

> But the law makes an exception when a debtor accidentally wires money to a creditor. In that case, if the creditor doesn't have prior knowledge the payment was a mistake, it's free to treat it as a repayment of the loan.

There's a major UI issue here, no doubt about it.

But, it's important to also point out how bizzare this exception is, how did this even get included as the one and only exception?

[+] gregoriol|5 years ago|reply
Everyone here seems to focus on the bad UI and blames those who made the software because this particular operation was complicated to perform and failed. But all this is missing the points: the UI reflect the business, the UI has likely been designed to perform many very complex banking operations and works well for that, we just don't know how powerful it is from the screenshot and are not experts in the field. It's as if we were shown a shell and people were like "wow look at that horrible UI, no wonder they deleted the production database".

Sometimes a basic UI is the most efficient at some tasks, but one has to be trained to use it properly and processes have to be in place to prevent horrible issues. This is the problem here and this is where they failed.

[+] peterkelly|5 years ago|reply
This problem seems like it could have been fixed by a better UI though, specifically one which included a confirmation step that showed the implications of the decision i.e. the amounts that would be transferred and who they would be transferred to.

If there was a screen that said "the transaction you have selected will result in a total of $900 million being sent to the following parties, broken down as follows: ...; Please confirm you wish to proceed" - then the outcome would have been very different.

[+] perlgeek|5 years ago|reply
OK, so they payed $500M about two years early, right?

That's very bad for your cash flow, and you lose 2 years worth of interest from that, but in the long run, citibank isn't $500M poorer, right?

I'm trying to understand what the real financial impact is. Interest rates aren't very high right now, so can maybe a few percent of the $500M per year?

Regarding the UI issue: Not only is it bad that the user interface is weird, but that approval seems to use the same bad UI as entering the data. The approval stage really needs a line like "this will result in a $ X being sent out on $Date to $Recipient".

[+] ineedasername|5 years ago|reply
But the law makes an exception when a debtor accidentally wires money to a creditor

Okay, but Citibank wasn't the debtor here, they were simply the middleman. And:

"To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake... would have been borderline irrational"

Sure, if you hear hoofbeats, the likely guess is horses, not zebras. But if someone calls you up and says "Hey, I got some nice pics of zebras running by your house" then you know the probable guess was wrong, and should act accordingly.

Absolutely the mistake was Citibank's fault. Maybe you think that means, inherently, that they should lose the case. But if you look at the actual laws on the books, I really don't understand how the interpretation of the laws comes down against Citi here. Though a half $billion mistake on UI design might still be a very beneficial lesson for the entire UX design community to remember.

[+] ab_testing|5 years ago|reply
I have a contrarian view that this is not a UI design issue - instead a user training issue. Oracle and bad UI design often comes up on HN and every time the debate is that Oracle software is very bad and ripe for disruption. However, most of what Oracle has built or bought is complex because the purposes it serves is complex. You cannot expect a user to just sit in front of this screen (or any other Oracle screen for that matter) and figure it out. There is a lot of tribal knowledge involved and most of the times it is a user training issue.

If the user and his manager was unsure, they should have tested it on a dev system or with a smaller amount. Blaming bad software is easy. But having worked with complex ERP systems for more than a decade, I can easily say tat most of these systems are designed keeping user feedback in mind. In fact a lot of modules that Oracle sells were systems that their users built as add-ons to Oracle and were finally embraced by Oracle.

Everybody blames bad ERP systems from Oracle (PeopleSoft, JD Edwards, Oracle EBS) and SAP. Yet noboody is building competing systems at a mass scale. Because they are complex and needs lots of integrations.

That is why, you see lots of small business ERP systems but very few complex full scale ERP systems -the kind Oracle dabbles in.

[+] austincheney|5 years ago|reply
Imagine working on the complexities of financial software for a major financial institution. I don’t have to imagine this as I work for a much larger financial institution with many more financial products.

Now take that complexity that is in your imagination now and couple it with internal software. Think about the last megacorp you worked at and how awesome the internal tools are.

Now imagine writing the applications and processes that govern and that internal software. The internal end product doesn’t get the product attention it deserves, because well... it’s just internal and not seen by customers. The stuff you write to support that end product gets even less product attention. It’s an act of God to get a few more lines in on top of the millions of redundant lines already present. Now couple that with the inane process of financial management internal to the financial industry.

Shitty enough yet knowing the product management requirements to make this good user facing software? Try imagining then testing it with automation.

The complexities are vast enough that converging, in my area, to git for version control from various other tools was a multi year effort to avoid completely halting the business requirements.

[+] MattGaiser|5 years ago|reply
Used to work in banking innovation and have plenty of friends that work/have worked at banks. Bad UIs cause errors and inability to help customers all the time.
[+] tppiotrowski|5 years ago|reply
A lot of people are pointing to the UI here but I think there’s a more fundamental problem of using a system in a way it wasn’t designed to in the first place.

The need for transferring principal to your own “wash” account in order to make an interest payment to other parties seems unconventional.

It’s violating the problem domain. Similar to using special int values like -1 or -2 in an orderId DB field to indicate that order was cancelled or in progress instead of creating an explicit orderStatus field.

[+] hnedeotes|5 years ago|reply
So they put 3 guys who seem to never had done such a thing in charge of a $1B transaction, that requires special parameters, and they didn't even checked if their assumptions were right?

And here I am, tearing out my hair, sleepless nights, wondering if using iodash instead of pure JS incurs in needless overhead for people I don't know from anywhere visiting my website.

(probably someone got a nice check out of it though)

[+] redkoala|5 years ago|reply
Oracle Flexcube is a packaged banking product. While Citibank did make the choice to purchase the product, the UI design flaw is really inherent in the product itself.

https://www.oracle.com/industries/financial-services/banking...

[+] bigpeopleareold|5 years ago|reply
If the numbers don't convince, maybe the cozy picture of the man who enjoys doing large corporate transactions will seal the deal!
[+] freebee16|5 years ago|reply
on the link they say "A rich and intuitive user experience that helps bankers enrich customer guidance, advice and product recommendations to customers".
[+] 35fbe7d3d5b9|5 years ago|reply
Is it? Or is it like most Oracle enterprise products (E-Business Suite, PeopleSoft, yadda yadda) – and the actual UX is bespoke constructed by an army of contractors across the world?
[+] maxwell|5 years ago|reply
Wasn't actually about the UI, here's the buried lede:

> Ordinarily, paying back a loan early wouldn't be a big deal, since the parties could simply negotiate a new loan on similar terms. But in this case, some of the lenders were not on good terms with Revlon and Citibank.

[+] boatsie|5 years ago|reply
Obviously Citibank was at fault here for not understanding how the software was supposed to be used, but they did not write the software Flexcube, Oracle did. And if you look at some of the manuals for the software, it is all very poorly designed:

https://docs.oracle.com/cd/E53393_01/homepage.htm

However, Citibank's real mistake was trying to use the software for something it probably was not really designed for. It seems they created a "hack" to make this type of interest-only and rollover payment possible so that they didn't have to bother trying to figure out the correct interest payments for the loan holders.

[+] isolli|5 years ago|reply
Isn't this a core design failure, rather than merely a UI failure?

<< On Flexcube, the easiest (or perhaps only) way to execute the transaction was to enter it in the system as if paying off the loan in its entirety, but to direct the principal portion of the payment to a "wash account" (an internal Citibank account) to help ensure that money does not leave the bank. >>