I am not Indian but I work for a large Tata like IT firm. This hit way too close to home. There a lot of cultural issues here that comes down to management being rewarded if things are done cheaply and discouraging any agency or self-realization by the developers. If I saw this in the US, I’d walk out. They literally don’t have that option as there’s a 90 day salary clawback if they do. Some general thoughts:
- Most management has a non-tech background. So they get what they want to hear and don’t want to hear what’s wrong.
- Thinking this coming from the same team or from the same company is wrong. They silo developers like crazy. There likely was an API developer, an Office 365 developer, frontend developer (so specific it is down to the framework or stack!) and the developers themselves will not touch anything they aren’t “certified “ in.
- I have been in meetings on $100 million projects where they will seriously argue over the cost of sendgrid. Eventually this will come down to no one having “sendgrid experience”
And some developer saying they can do it in Office365.
- Security team will get the first cut in budget since it should “already be secure.”
- You are likely talking over the head of the nephew hired to do security for this. Will the government or anyone sue them? No, so why is this guy bugging us.
Developers aren’t encouraged to develop but get tickets out and not question them. The manga is “It wasn’t in the requirements” all the way down the chain.
I work with smart developers out of India but it is not a culture of innovation. This kind of work is treated like a call center. Don’t go off script, stick your little problem domain, if we aren’t failing we are winning.
> I have been in meetings on $100 million projects where they will seriously argue over the cost of sendgrid. Eventually this will come down to no one having “sendgrid experience” And some developer saying they can do it in Office365.
Reading this in your comment was physically painful.
There was a car dealer (Honda affiliate) I had the unfortunate "pleasure" of dealing with back in the mid-late 2000s that stored finance applications by numeric incrementing ids. I never did report it, but I was able to pull up a bunch of sensitive info (SSN, DOB, names, addresses) on folks living in NJ. (I didn't report it because bug bounties weren't really a thing back then and the CFAA was).
I managed to get my application removed, but the vulnerability existed for several years until they updated to a new system. The new system also appeared to have some vulnerabilities, but I never invested time to figure it out. I just did not do business with that dealer ever again, and I'm super wary about car dealerships and finance applications these days...I usually get my financing from elsewhere even if it means a bit higher of a payment...thankfully my vehicle is paid off.
There is a huge missing niche for trusted intermediaries of identity information. We’ve been working on this at https://cerebrum.com in a different niche (background checks), but this comment just triggered a slew of ideas…
The security blunders are obviously horrible, but MAYBE explained by inexperienced developers tasked with something way beyond their understanding.
But how on earth did anyone approve storing confidential customer documents in an email account? This seems to indicate there's nobody in charge that understands anything about how to run this business. And if it's a subsidiary or outsourcing partner, it also shows that nobody has ever audited this business.
This is criminally negligent behavior from the company owners, and whoever is contracting them to do this work.
> But how on earth did anyone approve storing confidential customer documents in an email account?
Given the competence shown here, I doubt anyone approved anything. Most likely saving sent mail was a feature of whatever mail server they're using and it was a byproduct of the dumb decision to use an actual account for a "noreply" address.
I saw a fairly large estate agency system that bcc’d every outgoing email from their system to a shared account everybody then synced to Outlook. It was part audit log, part debugging tool, part database backup.
They changed when they realised employees were taking all their customers’ details to new jobs.
Yikes! This an unusual exploit since it both has an absolutely massive impact (literally access to everything on SharePoint and Outlook??), with a relatively straightforward vector (just looking at client side JavaScript).
One nit: I'd rather see people redact sensitive data with solid blocks instead of blurs in screenshots. Can't be too careful!
Let’s also appreciate that a monitoring email endpoint that was designed more or less as a communication worker/agent/runner has been abandoned and was basically matastasizing. That tells me that they aren’t monitoring email utilization or any other compensatory mechanism for identifying anamolous behavior - eg “hey why is email alias costing us [multiple of others]/month in storage”
“The noreply account could be the most important account in an organization because it could potentially have a record of everything they have ever sent to customers”
> Everything after October 18 is a back-and-forth between CERT-In and me trying to determine if there would be a bug bounty reward. TTIBI never responded to the question, so I decided to close the case on December 22 and CERT-In sent me a nice appreciation letter.
If a "leading Insurance Broker across India" can't afford to hire competent developers the least they can do is throw a couple bucks at someone who took the time to identify the multiple severe problems that jeopardized their customers and who notified them responsibly.
The fact that they didn't and still haven't reset the password of the compromised email account blows my mind. Why would I ever trust a company that acts like this to do anything right? It seems like Toyota Tsusho Insurance Broker India should be avoided like the plague.
I've seen similar levels of incompetence first hand. This isn't someone actively ignoring important security warnings. This is someone not understanding what you are talking about. This is someone who, at a fundamental level, has no grasp of the landscape they are operating in or the challenges they are up against. This is someone who wants you to go away because the jargon you're talking doesn't make any sense to them or their team.
"Please stop sending me these confusing emails. I have important work to do."
The only way to fix this is a "changing of the guard" at the organizational level. The IT boss, and everything he has ever touched, has to go.
Part of the problem is effectively using this inbox as a "free" SMTP account so they don't have to pay for outbound emails. There would not be as much sensitive information in this account's sent/inbox if they using something like SES, which is incredibly cheap ($0.10/1000 emails).
Indian developers don't care about security unless you explain how to build the software step by step. Many firms got burnt because of this. We stopped working with individuals and teams in 2018. They don't care about security.
I was trying to say this in polite terms, but failed to find the words. I am absolutely not knocking India at all. My all time best manager came from India and he taught me A LOT about software development in the earlier 2000s. He was incredibly smart, and there were 2 other folks at that company from India that also were very smart.
However, I've also had to deal with the reverse. Folks from India and elsewhere that just blindly churn out code according to literal instructions and don't give any thought as to how that code might not be safe/efficient/whatever.
That being said, I blame the company, not the people. You could easily end up in a similar mess here in the states if you don't take some time to vet.
(note: watching someone code on an interview aka pair coding isn't vetting, even take home assignments don't. If you do either of these, you aren't vetting, you are subconsciously looking at speed/accuracy/ability to think quickly, which may also mean you are discriminating based on age/disability -- i.e. people that code slower or think slower tend to be older or have a disability -- which is a violation of federal law here in the states.)
A simple APP_DEBUG=false in their production .env would’ve prevented leaking the Laravel stack trace and logs (Issue 3). I imagine they were debugging in production and forgot to switch it back
> Logging into the Microsoft account was surprisingly easy. There was no two-factor authentication set up or any other login verification prompts. If there was, it probably would not have been possible for me to successfully login.
This shouldn't have been possible in the first place for a few months now as Microsoft did a massive push to disable anything but OAuth logins for O365.
Wild-assed guess before I read this: in their greed for personal information, they took what should be a purely client-side scripted job into something that connects to the back end.
Edit: Yup! Instead of just doing calculations, it involved some e-mail workflow.
> The password could be used to log into the “noreplyeicher@ttibi.co.in” Microsoft email account.
I'm surprised this is literally true as described.
The actual browser itself makes the actual SMTP connection to the Microsoft e-mail host! The authentication name used is the above e-mail address. I typed out the Base64 to check:
There is a second IT silliness here, a minor one compared to the password gaffe. "noreply" addresses should not be real mail accounts or working aliases!
The noreply address is just a fake you put into the SMTP envelope and From: which will bounce due to not resolving if someone replies to it.
> The actual browser itself makes the actual SMTP connection to the Microsoft e-mail host!
This is not generally possible, browsers cannot make arbitrary socket connections in the way that would be required to reliably communicate with an SMTP server. The article makes clear that the frontend is calling a poorly-coded email-sending API implemented as an HTTP endpoint.
This is a boggling level of disdain for customer security - even putting aside the insanely low levels of data security, it's mind boggling that the website remained up for months after the disclosure, and that even after being taken down the vulnerability remained open.
This is a boggling level of disdain for customer security
To be fair, this usually doesn't start as a boggling level of disdain. It usually starts out as 100% ignorance. It's how the people and the group respond to the negative feedback from experts and from reality, which brings in the disdain, even spiraling to boggling levels.
There are two deep lessons herein, rooted in game theory.
Given that the password hasn't changed, I'd assume that there are exactly 0 sysadmins or software engineers working at this insurance company. A web app was poorly hacked together a few years ago, and just ticks-over in the background. Nobody in the org knows about the exploit (and it's possible they don't have the capacity to understand the exploit).
It's a bit hard to imagine this specific problem existing outside of the Microsoft ecosystem. I can very well imagine that there are loads of corporate resources provided through a valid O365 account that are useful for targeted hacks -- heck, the metadata in the corporate directory alone is going to be useful to a ne'er-do-well.
I really can't believe they haven't changed the password. I wonder what part of their workflow that breaks?
So crazy that things like this still happen in production. I mean, maybe I have survivorship bias (we never hear about the companies that don't have security flaws, or the hundreds of APIs that are completely secure), but it should be super easy to make a site that is secure. Even I know how to do it. It shouldn't be that hard to find people who know how to make secure sites.
You are either young or don't know any better. All major companies have bug bounties program and consistently, every few weeks, payout CRITICAL level bounties, as in attacker managed to get full server/access to any account etc. Security breaches are just a matter of time. Who is to blame is debatable, since being a criminal and breaking and stealing (into digital or physical business) is against the law.
I am so with you. I should be the lowest common denominator when it comes to security- I am what in my head qualifies as a novice. But I notice basic flaws at almost every company I work for. Absolutely baffled how this keeps happening.
I've not very knowledgeable on the process of building a backend API but could someone explain how sending the email's password back in an error log could ever been a good idea?
Obviously, the answer is never (unless it's for _very_ specific testing in a dev only environment).
In this case, it's not that they were sending the password directly for any reason, but instead returning the raw SMTP log from sending the email; which as a byproduct had the password in it due to needing to authenticate with the SMTP server.
Passwords in error logs are only _ever_ good if doing very, very, low level debugging of why logins aren't working right. Even then it's usually enough to just log which auth backends are touched and their result state. However it MIGHT happen if an encoding issue is suspected. Ideally never on a production system.
>October 18, 2023: I noticed the vulnerability is now fixed – the email sending API now requires authentication. I ask CERT-In if TTIBI can offer a bug bounty reward.
>TTIBI never responded to the question, so I decided to close the case on December 22 and CERT-In sent me a nice appreciation letter.
The letter:
"Dear Eaton Zveare,
This email is written in appreciation and recognition of your efforts for bringing our attention to the "Cryptographic Failures" in one of the Indian websites on 08.08.2023.
The role of responsible security researchers is pivotal for creating a secure cyber ecosystem and CERT-In strongly believes in working actively with a researcher like you for the discovery of cyber security vulnerabilities and their subsequent remediation in a responsible manner.
We look forward to your valuable contribution in future as well.
randomgiy3142|2 years ago
- Most management has a non-tech background. So they get what they want to hear and don’t want to hear what’s wrong.
- Thinking this coming from the same team or from the same company is wrong. They silo developers like crazy. There likely was an API developer, an Office 365 developer, frontend developer (so specific it is down to the framework or stack!) and the developers themselves will not touch anything they aren’t “certified “ in.
- I have been in meetings on $100 million projects where they will seriously argue over the cost of sendgrid. Eventually this will come down to no one having “sendgrid experience” And some developer saying they can do it in Office365.
- Security team will get the first cut in budget since it should “already be secure.”
- You are likely talking over the head of the nephew hired to do security for this. Will the government or anyone sue them? No, so why is this guy bugging us.
Developers aren’t encouraged to develop but get tickets out and not question them. The manga is “It wasn’t in the requirements” all the way down the chain.
I work with smart developers out of India but it is not a culture of innovation. This kind of work is treated like a call center. Don’t go off script, stick your little problem domain, if we aren’t failing we are winning.
chias|2 years ago
Reading this in your comment was physically painful.
ClumsyPilot|2 years ago
This is what you get, ladies and gentlemen, without unions and labour right.
Coming soon, to us too.
rootsudo|2 years ago
To a T.
eek2121|2 years ago
I managed to get my application removed, but the vulnerability existed for several years until they updated to a new system. The new system also appeared to have some vulnerabilities, but I never invested time to figure it out. I just did not do business with that dealer ever again, and I'm super wary about car dealerships and finance applications these days...I usually get my financing from elsewhere even if it means a bit higher of a payment...thankfully my vehicle is paid off.
cjs_ac|2 years ago
https://eaton-works.com/2023/06/06/honda-ecommerce-hack/
airtonix|2 years ago
[deleted]
sebmellen|2 years ago
gorbachev|2 years ago
But how on earth did anyone approve storing confidential customer documents in an email account? This seems to indicate there's nobody in charge that understands anything about how to run this business. And if it's a subsidiary or outsourcing partner, it also shows that nobody has ever audited this business.
This is criminally negligent behavior from the company owners, and whoever is contracting them to do this work.
blcknight|2 years ago
Given the competence shown here, I doubt anyone approved anything. Most likely saving sent mail was a feature of whatever mail server they're using and it was a byproduct of the dumb decision to use an actual account for a "noreply" address.
blowski|2 years ago
They changed when they realised employees were taking all their customers’ details to new jobs.
qup|2 years ago
Hopefully they at least took the Base64 password out of the error log. I'm sure they did. Right? !?
zero5two|2 years ago
roland35|2 years ago
One nit: I'd rather see people redact sensitive data with solid blocks instead of blurs in screenshots. Can't be too careful!
rudasn|2 years ago
lxe|2 years ago
There should be a legal framework that holds companies liable for certain level of security mishandling when it comes to private customer data.
ahoka|2 years ago
kumarski|2 years ago
Persistent power being one of these.
Can't wait til there's enough electricity in India to where hacks become a primary concern.
They're laying down 100k kilometesr of fiber optic per a month and 350 5g cell sites per day.
unknown|2 years ago
[deleted]
AndrewKemendo|2 years ago
“The noreply account could be the most important account in an organization because it could potentially have a record of everything they have ever sent to customers”
autoexec|2 years ago
If a "leading Insurance Broker across India" can't afford to hire competent developers the least they can do is throw a couple bucks at someone who took the time to identify the multiple severe problems that jeopardized their customers and who notified them responsibly.
The fact that they didn't and still haven't reset the password of the compromised email account blows my mind. Why would I ever trust a company that acts like this to do anything right? It seems like Toyota Tsusho Insurance Broker India should be avoided like the plague.
zelon88|2 years ago
"Please stop sending me these confusing emails. I have important work to do."
The only way to fix this is a "changing of the guard" at the organizational level. The IT boss, and everything he has ever touched, has to go.
AndrewKemendo|2 years ago
which likely led to this issue in the first place
semiquaver|2 years ago
judge2020|2 years ago
ballenf|2 years ago
- store all sent emails
- create an interface for business/non-devs to view and search past messages
If they were using all the functionality of this "free" SMTP approach, that's quite a bit of development + maintenance cost.
system2|2 years ago
eek2121|2 years ago
However, I've also had to deal with the reverse. Folks from India and elsewhere that just blindly churn out code according to literal instructions and don't give any thought as to how that code might not be safe/efficient/whatever.
That being said, I blame the company, not the people. You could easily end up in a similar mess here in the states if you don't take some time to vet.
(note: watching someone code on an interview aka pair coding isn't vetting, even take home assignments don't. If you do either of these, you aren't vetting, you are subconsciously looking at speed/accuracy/ability to think quickly, which may also mean you are discriminating based on age/disability -- i.e. people that code slower or think slower tend to be older or have a disability -- which is a violation of federal law here in the states.)
subtra3t|2 years ago
prestonholt|2 years ago
Gelob|2 years ago
mschuster91|2 years ago
This shouldn't have been possible in the first place for a few months now as Microsoft did a massive push to disable anything but OAuth logins for O365.
kazinator|2 years ago
Edit: Yup! Instead of just doing calculations, it involved some e-mail workflow.
> The password could be used to log into the “noreplyeicher@ttibi.co.in” Microsoft email account.
I'm surprised this is literally true as described.
The actual browser itself makes the actual SMTP connection to the Microsoft e-mail host! The authentication name used is the above e-mail address. I typed out the Base64 to check:
There is a second IT silliness here, a minor one compared to the password gaffe. "noreply" addresses should not be real mail accounts or working aliases!The noreply address is just a fake you put into the SMTP envelope and From: which will bounce due to not resolving if someone replies to it.
semiquaver|2 years ago
danielklnstein|2 years ago
Great post!
stcredzero|2 years ago
To be fair, this usually doesn't start as a boggling level of disdain. It usually starts out as 100% ignorance. It's how the people and the group respond to the negative feedback from experts and from reality, which brings in the disdain, even spiraling to boggling levels.
There are two deep lessons herein, rooted in game theory.
EDIT: In this case, op did everything right!
OtherShrezzing|2 years ago
Dalewyn|2 years ago
azalemeth|2 years ago
I really can't believe they haven't changed the password. I wonder what part of their workflow that breaks?
SAI_Peregrinus|2 years ago
Probably their single sign-on. They probably only have the one company password, shared. That's the single sign-on!
orenlindsey|2 years ago
spaceheater|2 years ago
pavel_lishin|2 years ago
A "site" that's a static webpage? Sure.
A full application that just happens to use HTTP as one of its interfaces? More difficult than you'd think.
incangold|2 years ago
plussed_reader|2 years ago
dontCallUs_7381|2 years ago
This takes "don't call us, we'll call you" to the next level: running a vast chunk of your business from the noreply@ account!
pylua|2 years ago
Maybe applications that take personal and sensitive data should require a clearance. Companies that perform pen tests should also require clearances.
m-GDEV|2 years ago
elliottcarlson|2 years ago
In this case, it's not that they were sending the password directly for any reason, but instead returning the raw SMTP log from sending the email; which as a byproduct had the password in it due to needing to authenticate with the SMTP server.
mjevans|2 years ago
Tryanasorus|2 years ago
josu|2 years ago
>TTIBI never responded to the question, so I decided to close the case on December 22 and CERT-In sent me a nice appreciation letter.
The letter:
"Dear Eaton Zveare,
This email is written in appreciation and recognition of your efforts for bringing our attention to the "Cryptographic Failures" in one of the Indian websites on 08.08.2023. The role of responsible security researchers is pivotal for creating a secure cyber ecosystem and CERT-In strongly believes in working actively with a researcher like you for the discovery of cyber security vulnerabilities and their subsequent remediation in a responsible manner.
We look forward to your valuable contribution in future as well.
Thanks & Regards"
https://eaton-works.com/cdn-cgi/imagedelivery/VwwCqBIYNXeyNQ...
unknown|2 years ago
[deleted]
carlockkasey9|2 years ago
[deleted]
isabella2189|2 years ago
[deleted]
ajross|2 years ago
[deleted]
polishdude20|2 years ago
Why this one?
joosters|2 years ago
pphysch|2 years ago
hk__2|2 years ago
duckhelmet|2 years ago
[deleted]
Dalewyn|2 years ago
bloqs|2 years ago
JonChesterfield|2 years ago
Total incomprehension of the driving force behind society for decades is not an indicator of intelligence.