Took me a while searching around in that article to figure out what EDI is:
> Since each business has multiple trading partners, and each of its trading partner operates on different business systems, “point to point” integration of these transactions (that is, mapping Walmart’s internal database format directly to the QuickBooks JSON API) would require the recipient to have detailed knowledge of many different transaction formats – for example, a company selling to three customers running on NetSuite, SAP, and QuickBooks, respectively, would need to understand NetSuite XML, SAP IDoc, and QuickBooks JSON. Maintaining familiarity with so many systems isn’t practical; to avoid this explosion of complexity, businesses instead use commonly-accepted intermediary formats, which are broadly known as Electronic Data Interchange – EDI.
I cut my teeth on this kind of thing 25 years ago...it's the stuff that makes the world go around, but gets shuffled off to the back corners of the tech world. Way more interesting and nuanced than it gets credit. So many problems that were solved decades ago that have been forgotten as the world perpetually reinvents the wheel.
When I worked for a huge electronics distributor in the EU I got assigned the EDI hot potato by accident and kinda grew to like it. I got to talk to other companies's IT people that handled it on the their side instead of business facing sales or management people. Started with loving the XML formats and hating EDIFACT but that got reversed real fast. One of the projects I'm still most proud of from that time is consolidating all of our EDI systems into one, coherent (and beautifly designed I might add since no one interfered because no one cared ;) whole that used EDIFACT as an internal format to represent documents. That enabled developing integrations quickly by just writing a translator/parser between our format and whatever God awful format the other party used.
One thing I do hate with a passion still is the AS2 transmission later that some vendors used. My god, what a shitshow that is. No stable clients, discrepancies between implementations and all for what? Essentially transferring files. Ugh.
I develop an internal system for a company which sells diapers to big chain supermarkets in europe. It is connected to them through EDI. These are my 2 reasons why EDI is hard and fucked up in my opinion:
1) syntax - we are using XML which has weird element names and values and it's mixed camel case with other styles. It's not nice or possible to read without a manual/reference
2) brokers - you connect to others throught "EDI brokers" (dunno the right name in english). Basically you don't connect with other directly but it always goes through one or more brokers which are part of some EDI union or whatnot. So for example order from chain to a producer goes like this Supermarket -> Broker -> Producer. What this means is that different brokers pass you the orders in different way. So i guess it can vary a lot. Our broker is maybe the largest in the country and yet the only way how to programatically get messages from him is invoking a Java program, which downloads the message from his servers and that is possible to do exactly once. Then it's up to you to distribute it in your company as you wish. God forbid if you corrupt the message or something. This is very bad because you have to do a lot more work than just do a few REST requests to deal with anything. It prolongs basic integration by a lot.
Twenty years ago I was working with EDI without XML. The format was a series of lines, each with a starting tag followed by a delimited list of values.
Each transaction type had a grammar, so each starting tag could mean different things depending on its position within the grammatical structure of the transaction. We were new to EDI and had to roll out a bunch of partners fast, so we bought $60K software to help us. Turned out, that software had no understanding of the grammar; you had to use a cumbersome GUI to essentially write code from scratch to pull out what you wanted, with the only control structure available being GOTOs. Just pulling out a dozen important fields looked like a significant engineering effort.
Then our partners said they required us to display to the user everything they sent. I ended up ditching the $60K software and writing my own tool with a proper parser, so I could input the grammar of a new transaction type in an hour and sort everything out right. After that it was easy.
We didn't have to deal with brokers at least, we connected to everyone directly. I think the company became one of the brokers after I left.
This both brings back my past and a future project I have waiting for me.
The discipline of dealing with other people's systems is a great learning point.
The last person I integrated with was totally bemused by my huge list of questions. They always seem to assume that the way their system exports orders is the norm!
A:"When do you export orders, as soon as they are placed or on some schedule?"
B:"As soon as they are placed".
Some time later in the conversation...
B:"You needn't run it that early, we don't export orders until after 7.30am".
You really can't trust anything in EDI until you have seen the orders!
I just remembered a proprietry ERP system I worked with that accepted EDI orders placed in a certain directory on their Solaris server. Every 30 seconds it would empty that folder and begin to process the orders.
The company must have implemented a design internally to spawn a new process to deal with each new order. The first I noticed was that the log file it spat out was not in order, it had messages about different parts of different orders all intermingled... then 1 day I put 700 orders in and it launched 700 processes and the server ran out of swap space and crashed.... Their support team felt I was unreasonable! I had to rate limit afterwards with my system drip feeding the orders! It is funny to think that it is still in use I understand, and busier than ever!
The biggest problem with EDI is semantics. When humans talk to each other they have ways of disambiguating the words and jargon used. Mistakes still happen.
Semantics are embedded at different layers of different systems and machines don't have the common sense to notice different semantic schemas.
From the medical side; we had a pathology system, an ADT (admission, discharge, transfer) system, a clinical system, and a billing system. The semantics of what constitutes a demographic update of a patient record vs. an update of an encounter (visit) is bad enough when the same data may be denormalized in messages in both streams. Even with an established interchange format (HL7) each system encodes patients, their visits, their billing codes, and their results in slightly different ways. Sure; the pathology system and clinical system both understand HL7 and ICD9, but do they agree on what it means when a modifier gets added to one of the codes during an update? Or does an interface engine MITMing the connection have to generate a delete/insert in one direction to synchronize the semantic meaning in the downstream systems? Or what exactly does "gross description" mean to a pathologist vs. a clinician (not to mention a biller or admission clerk or me), not to mention the systems that are supposed to store this information in a meaningful way (it turns out the best quantifiable semantic representation of the latter is plain ASCII, or maybe a rich text formatted document).
I spent a decade working with ANSI X12. Writing software to generate it, streaming parsers for mutli-gigabyte batches, tools to display it, consume it, validate it, etc. Really interesting world to deal with. Every trading partner had their own interpretation of certain things. In my experience (healthcare) things slowly improved over time. Supposedly, FHIR will will take over in that space, but I have my doubts.
FHIR will take over, but I wouldn't call that a good thing. the development of FHIR is extremely haphazard, most of it will turn into a generic KV store and miss a lot of the great systems thinking in EDI. It's as if they lowered the barrier to entry at the expense of robust document exchange.
Hopefully Stedi can make adopting EDI easier and increase its popularily over the next 50 years and there isn't such a demand to create alternative systems.
FHIR can now be used instead of X12 for prior authorizations but in the US it can't be used for claims yet. Claims will continue to rely on X12 for many years.
I did medical billing EDI adjacent stuff with a VMS system (that ran decades past point of obsolescence).
One thing - in the past the EDI integrations just had to be simple, the feature set was so small - there wasn't room to go totally crazy.
Once they mandated full ehealth records and exploded the complexity of the medical IT side, clinician productivity (face to face with patient) went way down. Integration pain went way up. The old tab separated value row turned into GIGANTIC XML monstrosities for the smallest things - now thousands of these blobs in a document. If you put even 10,000 XML blobs in an exchange vs tab separated values - very quickly appreciated the old way of doing things. One annoying thing, even the person on other end you are dealing with lost all ability to troubleshoot issues.
For some reason in some areas - no EDI in the traditional sense is being used (ie, govt contracts with idiots who build latest buzzword filled solutions - total annoyance to interact with).
Yeah, worked in medical billing for a while. It's insane. One particular example I remember was that there was one code for "blood pressure taken while patient lying down" and a completely different code for "blood pressure taken while patient sitting up" (rather than, say, one code for "blood pressure" and another for "patient position").
I'm going to have to implement an EDI medical billing system soon (moving from XLS/FTP based billing, say what you will about it but relatively easy to understand and automate) and after looking over the spec, I am not looking forward to it at all. Ended up purchasing a product called Cozyroc which integrates into SSIS(SQL Server Integration Services) which can do SQL to EDI translation with a map provided, should make things easier but still not looking forward to it.
I don't think EDI itself is too complex or anything--after working with several REST/SOAP servers, EDI is just another standard you have to account for. Parsing 850s or generating 810s is just the name of the game.
What makes it super complicated and frankly pisses me off are the "exclusive" deals retailers have. Want to work with Target? Too bad, you have to work with CommerceHub. Macy's? Nope, gotta use SPS Commerce. This is why I like Shopify or even Amazon Vendor Central, which uses EDI. Wayfair was fortunately open to the idea of anyone linking to their EDI servers so long as there's a proper onboarding process.
My company, which is an ERP for wholesalers, doesn't have a way to directly work with the retailer. It always has to go through one of these middleware companies.
EDI is not hard!!!! Period! Like some file formats, it’s unnecessarily complicated. The biggest problem in this industry is that there’s a monopoly and many companies are being FORCED to use these arcane and costly service providers.
I build and integrate EDI solutions for small/mid and retail juggernauts.
We're a small business looking into EDI with some of our suppliers, but are finding the pricing quite painful for our level of business when talking to some service providers. Which service providers do you see as arcane and costly?
It's not hard, it's complex. I spent a few years going through 834 and later at a different company, 811. For the first I embedded a javascript interpreter to do translations. For the second I put most of the logic in the code.
I work in container shipping in Europe. Everything is coordinated by e-mailing EDI files (baplie, coprar, iftdgn etc.) Back and forth. Tens of millions containers per year.
Even though it should be a standard, you can't just e-mail a file to someone and except it to work. There's always integration work needed on both sides. Even then, errors frequently occur where you have to manually adjust something. I work on the operational side, but have become quite the edifact expert just to be able to do my work.
Overall solid introduction to the space. EDI was one of the things that boggled my mind early in my career.
> What we’re seeing here is that while a standard can be opinionated about the structure of a document and the naming of fields, it cannot be opinionated about the contents of a business transaction – the contents of a business transaction are dictated by the idiosyncrasies of the business itself.
Maybe it's the engineer in me speaking, but this seems like a pretty unfortunate stance to take. Sort of similar to saying, "we have the concept of HTTP status codes, but we cannot possibly opine on the codes themselves."
A bit hyperbolic for sure, but not having opinions on critical multi-party information has led to things like the reluctance of financial services to support non-ASCII values for customer names.
I haven’t done any work on EDI for many years so it might be a bit out of date, but in case it’s of any use to anyone I wrote an EDI parsing library and processor for many of the EDIFACT message specs:
I've had to do some EDI in the Automotive world and Financial world (ACH and Wires)
Maybe I've been lucky in my travels, but none of them were terribly difficult, so long as you had a spec doc in front of you. (As a C# Dev, I'd usually use some form of FileHelpers alongside of MultiRecordEngine, if you're having to deal with things like 'correction' records, treat the resulting records as an eventstream and use those to build the final dataset.)
What gets -nasty- is when companies violate a spec in a subtle way. In one case, the 'response' file for ACH transfers from one bank 'reused' the leftmost digit in a certain field for their own code. This however made it extraordinarily difficult to reconcile to the original check due to the relative shortness of this identifier.
Yeah a lot of people think those industries are "ripe for disruption" and then they realize no you can't represent it in JSON and you can't manage it on a phone-sized UI.
X12 is also used for freight rail messaging in the US. Anytime one railroad interchanges a train with another (in addition to many other types of events), messages are sent. I deal with it on most days.
i love how most of the comments on here are talking about EDI back when they first started and here I am consolidating 6 different systems into 1 EDI VAN and its considered new capabilities -_-
Ah, this triggered an old memory! I used to do integration implementations in Microsoft BizTalk Server, which was all XML based: all of the guts were designed to handle XML documents, so it had out-of-the-box components to translate non-XML ("flat file") messages to/from XML as they were received or sent. These components were universal and designed to operate off of annotations baked into the schemas that represented the XML versions of the messages - things like field sizes, expected locations of line breaks, records that could repeat, etc. The BizTalk developer tools included a schema builder tool ("BizTalk Editor") that helped you autogenerate, construct and mark up schemas with the declarative flat-file translation logic.
I only had to interact with EDI once or twice, but I remember when I discovered that BizTalk shipped with a huge set of premade schemas for most of the X12 and EDIFACT message types, plus extra pluggable components that were "enhanced" versions of the regular translator components that handled the more complex aspects that most "plain" flat files didn't have, mostly stuff around batching/splitting/validating I think. Many of the schemas were gigantic and unbelievably complicated.
[+] [-] eatonphil|4 years ago|reply
> Since each business has multiple trading partners, and each of its trading partner operates on different business systems, “point to point” integration of these transactions (that is, mapping Walmart’s internal database format directly to the QuickBooks JSON API) would require the recipient to have detailed knowledge of many different transaction formats – for example, a company selling to three customers running on NetSuite, SAP, and QuickBooks, respectively, would need to understand NetSuite XML, SAP IDoc, and QuickBooks JSON. Maintaining familiarity with so many systems isn’t practical; to avoid this explosion of complexity, businesses instead use commonly-accepted intermediary formats, which are broadly known as Electronic Data Interchange – EDI.
[+] [-] azalemeth|4 years ago|reply
[+] [-] poulsbohemian|4 years ago|reply
[+] [-] mszcz|4 years ago|reply
One thing I do hate with a passion still is the AS2 transmission later that some vendors used. My god, what a shitshow that is. No stable clients, discrepancies between implementations and all for what? Essentially transferring files. Ugh.
[+] [-] erik_seaberg|4 years ago|reply
[+] [-] jcims|4 years ago|reply
[+] [-] masa331|4 years ago|reply
1) syntax - we are using XML which has weird element names and values and it's mixed camel case with other styles. It's not nice or possible to read without a manual/reference
2) brokers - you connect to others throught "EDI brokers" (dunno the right name in english). Basically you don't connect with other directly but it always goes through one or more brokers which are part of some EDI union or whatnot. So for example order from chain to a producer goes like this Supermarket -> Broker -> Producer. What this means is that different brokers pass you the orders in different way. So i guess it can vary a lot. Our broker is maybe the largest in the country and yet the only way how to programatically get messages from him is invoking a Java program, which downloads the message from his servers and that is possible to do exactly once. Then it's up to you to distribute it in your company as you wish. God forbid if you corrupt the message or something. This is very bad because you have to do a lot more work than just do a few REST requests to deal with anything. It prolongs basic integration by a lot.
[+] [-] DennisP|4 years ago|reply
Each transaction type had a grammar, so each starting tag could mean different things depending on its position within the grammatical structure of the transaction. We were new to EDI and had to roll out a bunch of partners fast, so we bought $60K software to help us. Turned out, that software had no understanding of the grammar; you had to use a cumbersome GUI to essentially write code from scratch to pull out what you wanted, with the only control structure available being GOTOs. Just pulling out a dozen important fields looked like a significant engineering effort.
Then our partners said they required us to display to the user everything they sent. I ended up ditching the $60K software and writing my own tool with a proper parser, so I could input the grammar of a new transaction type in an hour and sort everything out right. After that it was easy.
We didn't have to deal with brokers at least, we connected to everyone directly. I think the company became one of the brokers after I left.
[+] [-] jimnotgym|4 years ago|reply
The discipline of dealing with other people's systems is a great learning point.
The last person I integrated with was totally bemused by my huge list of questions. They always seem to assume that the way their system exports orders is the norm!
A:"When do you export orders, as soon as they are placed or on some schedule?"
B:"As soon as they are placed".
Some time later in the conversation...
B:"You needn't run it that early, we don't export orders until after 7.30am".
You really can't trust anything in EDI until you have seen the orders!
[+] [-] jimnotgym|4 years ago|reply
The company must have implemented a design internally to spawn a new process to deal with each new order. The first I noticed was that the log file it spat out was not in order, it had messages about different parts of different orders all intermingled... then 1 day I put 700 orders in and it launched 700 processes and the server ran out of swap space and crashed.... Their support team felt I was unreasonable! I had to rate limit afterwards with my system drip feeding the orders! It is funny to think that it is still in use I understand, and busier than ever!
[+] [-] benlivengood|4 years ago|reply
Semantics are embedded at different layers of different systems and machines don't have the common sense to notice different semantic schemas.
From the medical side; we had a pathology system, an ADT (admission, discharge, transfer) system, a clinical system, and a billing system. The semantics of what constitutes a demographic update of a patient record vs. an update of an encounter (visit) is bad enough when the same data may be denormalized in messages in both streams. Even with an established interchange format (HL7) each system encodes patients, their visits, their billing codes, and their results in slightly different ways. Sure; the pathology system and clinical system both understand HL7 and ICD9, but do they agree on what it means when a modifier gets added to one of the codes during an update? Or does an interface engine MITMing the connection have to generate a delete/insert in one direction to synchronize the semantic meaning in the downstream systems? Or what exactly does "gross description" mean to a pathologist vs. a clinician (not to mention a biller or admission clerk or me), not to mention the systems that are supposed to store this information in a meaningful way (it turns out the best quantifiable semantic representation of the latter is plain ASCII, or maybe a rich text formatted document).
[+] [-] time0ut|4 years ago|reply
[+] [-] arpinum|4 years ago|reply
Hopefully Stedi can make adopting EDI easier and increase its popularily over the next 50 years and there isn't such a demand to create alternative systems.
[+] [-] nradov|4 years ago|reply
[+] [-] allynjalford|4 years ago|reply
[+] [-] onphonenow|4 years ago|reply
One thing - in the past the EDI integrations just had to be simple, the feature set was so small - there wasn't room to go totally crazy.
Once they mandated full ehealth records and exploded the complexity of the medical IT side, clinician productivity (face to face with patient) went way down. Integration pain went way up. The old tab separated value row turned into GIGANTIC XML monstrosities for the smallest things - now thousands of these blobs in a document. If you put even 10,000 XML blobs in an exchange vs tab separated values - very quickly appreciated the old way of doing things. One annoying thing, even the person on other end you are dealing with lost all ability to troubleshoot issues.
For some reason in some areas - no EDI in the traditional sense is being used (ie, govt contracts with idiots who build latest buzzword filled solutions - total annoyance to interact with).
[+] [-] commandlinefan|4 years ago|reply
[+] [-] joenathanone|4 years ago|reply
[+] [-] acwan93|4 years ago|reply
What makes it super complicated and frankly pisses me off are the "exclusive" deals retailers have. Want to work with Target? Too bad, you have to work with CommerceHub. Macy's? Nope, gotta use SPS Commerce. This is why I like Shopify or even Amazon Vendor Central, which uses EDI. Wayfair was fortunately open to the idea of anyone linking to their EDI servers so long as there's a proper onboarding process.
My company, which is an ERP for wholesalers, doesn't have a way to directly work with the retailer. It always has to go through one of these middleware companies.
[+] [-] iask|4 years ago|reply
I build and integrate EDI solutions for small/mid and retail juggernauts.
[+] [-] neolithic|4 years ago|reply
[+] [-] djweis|4 years ago|reply
[+] [-] tester756|4 years ago|reply
[+] [-] LeChuck|4 years ago|reply
Even though it should be a standard, you can't just e-mail a file to someone and except it to work. There's always integration work needed on both sides. Even then, errors frequently occur where you have to manually adjust something. I work on the operational side, but have become quite the edifact expert just to be able to do my work.
[+] [-] devadvance|4 years ago|reply
> What we’re seeing here is that while a standard can be opinionated about the structure of a document and the naming of fields, it cannot be opinionated about the contents of a business transaction – the contents of a business transaction are dictated by the idiosyncrasies of the business itself.
Maybe it's the engineer in me speaking, but this seems like a pretty unfortunate stance to take. Sort of similar to saying, "we have the concept of HTTP status codes, but we cannot possibly opine on the codes themselves."
A bit hyperbolic for sure, but not having opinions on critical multi-party information has led to things like the reluctance of financial services to support non-ASCII values for customer names.
[+] [-] AtlasBarfed|4 years ago|reply
[+] [-] nirav72|4 years ago|reply
[+] [-] BiggsHoson|4 years ago|reply
[+] [-] blipvert|4 years ago|reply
https://github.com/davidcoles/medici
[+] [-] to11mtm|4 years ago|reply
Maybe I've been lucky in my travels, but none of them were terribly difficult, so long as you had a spec doc in front of you. (As a C# Dev, I'd usually use some form of FileHelpers alongside of MultiRecordEngine, if you're having to deal with things like 'correction' records, treat the resulting records as an eventstream and use those to build the final dataset.)
What gets -nasty- is when companies violate a spec in a subtle way. In one case, the 'response' file for ACH transfers from one bank 'reused' the leftmost digit in a certain field for their own code. This however made it extraordinarily difficult to reconcile to the original check due to the relative shortness of this identifier.
[+] [-] tester756|4 years ago|reply
complexity of those industries is giant
[+] [-] throwawayboise|4 years ago|reply
[+] [-] jvolkman|4 years ago|reply
[+] [-] tamiral|4 years ago|reply
[+] [-] allynjalford|4 years ago|reply
[+] [-] nlawalker|4 years ago|reply
I only had to interact with EDI once or twice, but I remember when I discovered that BizTalk shipped with a huge set of premade schemas for most of the X12 and EDIFACT message types, plus extra pluggable components that were "enhanced" versions of the regular translator components that handled the more complex aspects that most "plain" flat files didn't have, mostly stuff around batching/splitting/validating I think. Many of the schemas were gigantic and unbelievably complicated.