Clearly it's the fault of those pesky engineers again, how else could you explain the same type of cheating across multiple divisions of the company. They only other explanation would involve Management, and obviously that's not it.
Though the fact that none of the other manufacturers spoke up about the violation of emissions standards when when they must have done their own tests to see how VW managed to get emissions so low, seems to point to the fact they they were all doing it to some degree. Perhaps not as brazenly as VW, but still no one wants to rock the boat they are in.
While I certainly don't buy the "pesky engineers" argument, it's actually somewhat plausible Porsche didn't know about the cheating (beyond what everyone knew, which was that the stated mileage and emissions were virtually impossible to achieve by any known mechanism). Since they dropped a VW manufactured engine (3.0 TDI) into a shared VW chassis (Touraeg) and built the Cayenne around that, there's a good chance the engine and engine management were treated as a black box and simply integrated at face value.
I wouldn't be surprised either way, but why would the Porche engineers, getting the engine control software from the common engine control group, spot check the emissions on the test track rather than in the globally approved testing setup?
If you presume they are doing their job and not cheating, why would you cross check that your emissions were valid in both the dyno test and on the track? It requires that you assume your fellow workers are cheating, which is hard to do at most places.
My impression is that the process was structured in a way to give higher levels deniability. I highly doubt direct orders were given from the top to cheat; instead it was made clear that the performance numbers would come out right AND the emissions standards would be met, or heads would roll. And lo and behold! The numbers and sales targets were met. And management can say "well, we don't know how they did it. We never asked!"
if those pesky engineers are looking for a job after getting fired from VW, I heard there is a young intern position available with the Donald Trump campaign.
Background: I used to be an automotive software engineer. I speak for
myself only here.
Whenever the topic of automotive software comes up on HN there are
comments alongs the lines of "global variables bad", but not much
construtive feedback.
I want to explain some of the tradeoffs that lead to the architecture
used in automotive software to get a better discussion going with HN
readers about that architecture.
tl;dr Given the hardware restrictions, real-time requirements and
measurement capabilities required in automotive software, shared global
variables without locks is a fast and safe way to share state between
different software components as long as each variable is only written
in one place in the program.
The microprocessor has to run for 10+ years in a wide range of
temperatures and be dirt cheap, so you end up with specs like 180 MHz, 4
MB of flash and 128 KB of RAM.
The program must run deterministicly with respect to memory. There is no
malloc/new in the code. All variables are statically allocated.
Because the physical world doesn't pause, no code is allowed to block
while waiting for resources, especially synchronization primitives like
mutexes.
The software architecture is in 2 main parts: basic software
containing the real-time OS and hardware drivers, and the
application layer which has the domain-specific code for controlling the
engine, brakes, etc.
The basic software is implemented using usual C programming techniques.
It has an API provided by function calls and structs to hide
implementation details of each microcontroller.
The application software is where the programming model is different.
To understand why, you need to know where automotive software comes
from and what it is trying to acheive.
Originally all controllers were mechanical: a valve opens proportionally
to the vacuum in a part of the system. Then some controllers were
implemented in analog electronics: take multiple voltages, feed them
through an op-amp and use the output to control a valve.
So automotive software reproduces this: get some inputs, compute the
same physical equations at a regular rate and generate outputs.
This is dataflow programming. Blocks of code have inputs and outputs.
They are executed at a fixed rate that depends on the physical phenomena
(air flow changes fast, temperature changes slowly). Different blocks
are conneceted together in a hierachical way to form subsystems.
Encapsulation is acheived by viewing these blocks as black boxes: you
don't need to care how the block works if you are only interested in
knowing which inputs it uses and outputs it produces.
Here's an example component to control a gizmo.
It might be implemented in a visual environment like Simulink by
MathWorks, or it implemented by hand from a spec.
It takes some inputs (EngineSpeed, CoolantTemp, Gizmo_PositionDesired,
Gizmo_Position), has some intermediate values (Gizmo_Error), and
outputs (Gizmo_DutyCycle). Those are implemented as global variables. It
also uses some constants (Gizmo_Gain_MAP, Gizmo_Offset_VAL). It has 2
processes, running every 100ms and 10ms. All this information would be
specified in an XML file.
The header GizmoController_data.h is auto-generated at compile time by a
tool from the XML file mentioned above. It will contain global variable
definitions for the inputs, intermediates and outputs with the
appropriate volatile, const, static and extern storage classes/type
qualifiers. This ensures that the compiler will enforce that inputs
can't be written to, intermediate values are private to the component
and outputs can be read by other modules.
Note that no explicit synchronization is needed to access inter-process
variables like Gizmo_Gain or inter-component variables like
Gizmo_Position. It's shared memory between 2 processes scheduled in OS
tasks that can potentially interrupt each other, but since the write is
atomic and happens only in one place, there is no data race. This is
huge! Concurrent programming, without locks, with the best efficiency
possible, using a simple technique anybody can understand: only one
place in the program is allowed to write to any global memory location.
Calibration is another aspect of automotive software. In most software
the constants either never change or can be set in some kind of
configuration file. For an automotive controller, the value of constants
(gains, offsets, limits, etc) depend on the vehicle so they must
be configurable at run time during development. This is implemented in
the C code by putting all constants in a memory area that is ROM in
production units, but RAM in development units. The compiler enforces
that application software cannot change constants, but the basic
software includes code so that constants can be changed from the outside
in development. This process is called calibration and is done by
calibration engineers who are usually not the ones who wrote the
software. Note that calibration can drastically affect the behavior of
the software. What would happen if Gizmo_Gain_MAP is set to all zeros?
Measurement of variables is essential to understanding what's going on
inside the embedded controller. Having all that state available in
global variables makes it possible for the calibration tool request the
value of any variable in the software at a fixed rate and display it in a
virtual oscilloscope.
The measurement and calibration tool needs to know how to access the
variables and constants. It uses a file that maps from names to
addresses for a particular version of software. That file can easily be
generated a compile time since all allocations are static.
Going back to the architecture of the application software, let's look
at where our gizmo controller fits. It is not the only component needed
to make the gizmo work. You also need components to calculate the gizmo
position from some external signal (let's say an analog voltage), to
route the output signal to the powerstage driver on the PCB, to
determine which position the gizmo should currently occupy. These would
form the gizmo subsystem package.
When the supplier releases gizmo 2.0 (TM) they upgrade the input signal
to be a PWM input instead of an analog input. Modularity in the software
allows the software team to simply replace the gizmo position component
with one that reads a PWM instead of an analog voltage and keep the rest
of the gizmo subsystem the same. In the future, projects that use gizmo
1.0 use one version of the gizmo subsystem and projects that use 2.0 use
another.
This is true at any level in the hierarchy: as long as the inputs and
outputs are the same, a component or subystem package can be replaced by
another.
Version control in automotive software reflects this. Instead of having
one tree of versions and releases like a typical software project, each
component, subsystem package and software project has its own tree of
versions. Each software project will reference the subsystem packages
required for their engine type, vehicle platform, sensors and actuators,
etc. This is how code reuse is acheived.
Testing is a mix of simulation (the sensor/actuator is simulated in
Simulink and connected to Simulink block diagram of the software
component), hardware-in-the-loop (a computer simulates the vehicle, but
the real electronic control unit is used) and vehicle testing.
Thanks for reading. I hope this improves your understanding of how
automotive software is structured.
I'm hoping the discussion will bring examples from other fields like
robotics, drones and aeronautics that have similar real-time
requirements on how they architect their software.
So you can use global variables and the like, but is there any progress in going higher level than C in industry? In which case you use global variables and whatnot, but you have static assertions about things like "this is only written to in one spot"
For example, I know there's an embedded systems company that writes everything in Haskell, and running their Haskell program actually generates "safe" C code that they compile down to their chips.
My impression from the oft-cited Toyota report was not only that there was a lot of global variable stuff, but that these "write-once" principles weren't super respected.
Testing also included very expensive (SW license cost, now HW), cycle accurate simulators at Delphi. ASIC specs were implemented into C and hooked into the simulator. Prototype firmware was compiled and run on the CPU simulator and "board" (netlist of simulated ASICs). This helped discover bugs/spec deficiencies before ASIC tape-out (save $$$). Of course, it was all wrapped in a Tk "dashboard/test unit" GUI for the other teams to consume. Tk was actually quite a pleasure to use, especially when interfacing with C!
EDIT: It basically provided at least two implementations for an ASIC by independent teams. The ASIC guys would implement the real deal in an HDL and test it. However, full system tests where the HDL simulation is hooked up with CPU simulation were too slow. By implementing the same spec in C, you gain simulation performance and get a second implementation and set of eyes that can help find bugs.
If some of these timers are implemented as separate OS tasks that can interrupt each others, what's to stop one from taking a bit too much time and throwing off the timing of another?
I'm not an embedded developer, so I don't have much value to add re: architecture; however, this caught my eye:
> The microprocessor has to run for 10+ years in a wide range of temperatures and be dirt cheap, so you end up with specs like 180 MHz, 4 MB of flash and 128 KB of RAM.
If I'm paying tens of thousands of dollars for a car, how come they're using the cheapest possible components? If tinkerers can ship the Raspberry Pi for $30 per board, inc. a 900Mhz quad-core chip and 1GB RAM, you'd think GM could get components at least that modern for an insignificant cost relative to the car they are controlling.
When coming to work from university (1989-90), I made a somewhat similar though more primitive design for the embedded software of a redundancy controller of a satellite modem. The various tasks (control of RF and baseband electronics, display and buttons UI, network management interface, etc) would run according to a scheduler, each in turn. No dynamic memory allocation, internal communication via global variables that are written only in one place. There was no mechanism to actually enforce this, it was just the principle in code, and the enforcement was possible because it was a one-man software project so I had no one else to blame.
This was written in PL/M (which looks like Pascal but is more similar to a primitive variant of C) on a 8031 microcontroller, from scratch (no 3rd party SW components at all - that was a long ago).
Later on, I used the same platform for a design we did on a telephone/data cable modem we designed for domestic use (We saw a need for internet enabled cable modem in 1992 and started work on a thing that provided POTS and 128 kbit/s over SLIP). That was also running a 8031, but as we wanted to make it dirt cheap, the board did not have dual-port RAM. The hardware wizards made it run with a hack on the regular RAM access so that the SW could only access RAM if it read any memory location twice: first read would just place the address bits on the bus, and doing another read would actually fetch the content. This enabled us to save several dollars per device.
(This was possible because the 8031 has an architecture which is not strictly von Neumann design: the code and address spaces are different memory spaces and you use different CPU instructions to access the different memories, so your program memory on EPROM works normally but your data memory in external RAM does not. There's also a small internal RAM in yet another overlapping address space which was not impacted by address bus weirdness.)
This is just to show what kind of strange limitations you sometimes have on the hardware in embedded devices. Unlimited virtual memory, large non-volatile storage instead of a puny i2C bus EEPROM... you just dreamed of those. Modern controllers with 4G flash are really huge compared to that, but they are still far below what people have on desktop computers, or even an Android phone.
The limitations in embedded devices indeed come from cost per device as well as heat, vibration, radiation, humidity and similar environmental factors.
I'm working with embedded on a similar industry, and also really enjoy web dev. I don't like the tools and edit-compile-cycle when working with small microcontrollers...
As alleged in the NOV, VW manufactured and installed software in the electronic control module of these vehicles that senses when the vehicle is being tested for compliance with EPA emissions standards. When the vehicle senses that it is undergoing a federal emissions test procedure, it operates in a low NOx “temperature conditioning” mode. Under that mode, the vehicle meets emission standards. At exactly one second after the completion of the initial phases of the standard test procedure, the vehicle immediately changes a number of operating parameters that increase NOx emissions and indicates in the software that it is transitioning to “normal mode,” where emissions of NOx increase up to nine times the EPA standard, depending on the vehicle and type of driving conditions. In other tests where the vehicle does not experience driving conditions similar to the start of the federal test procedure, the emissions are higher from the start, consistent with “normal mode.”
I have not read a substantive response from any part of VW to today's allegations.
Since Porsche do not use small diesels in their cars this looks to me like the scandal is a lot bigger than first thought. It isn't just the team working on the 1.8 litre diesel engine that fiddled the software, there is something more endemic going on as the 3 litre upwards size engines are implicated.
The funny thing is this:
Porsche could sue over £25 a day congestion charge (2008)
> The mayor's office responded saying the threatened legal action was "a double attack on Londoners".
> "First Porsche are trying to deprive Londoners of their democratic right to decide in the mayoral election on 1 May whether they want gas guzzling and polluting cars to drive in London when there is absolutely no need for them to do so. Second they are trying to impose on all Londoners unnecessary levels of pollution and greenhouse gases by a tiny minority," said a spokesman for the mayor.
> "No one is allowed to throw their rubbish in the street and Porsche should not be allowed to impose gas guzzling polluting cars on Londoners who do not want them."
> new tests that were conducted on all diesel car models in the United States by E.P.A., the California Air Resources Board and the regulatory group Environment Canada.
Environment Canada, a "regulatory group"? It's a department of the Government of Canada, yo.
I'd love to get my car tested... Anyone know how I could do that?
I have a 2015 Q5 TDI which while not specifically mentioned has the same size engine as several of the ones that are mentioned (3.0-liter). I would be shocked if they just started using the defeat device in the 2016 model year for the Q5 considering they were already getting pushback from the EPA before the 2016 models even got announced.
I bought the diesel specifically because of the high mileage and supposedly clean emissions. What a crock.
Meta: Earlier in the scandal, I recall reading that the engine management software is from Bosch. I should read further, looking for confirmation of this.
If that is the case, then this is an example of the maxim: When you provide a feature, someone will use it to their own ends -- meaning, "misuse" it.
Another example to wave in front of all those politicians and people advocating for encryption "back doors".
If you put the feature in there, people will use it any way that suits them. Including and especially ways that you did not intend nor want.
If it wasn't Bosch, perhaps there was nonetheless some "legitimate" argument within VW for adding this functionality. Enough to get the software folks -- particularly those not making big bucks off of the deal -- to implement this.
But an observant engineer might nonetheless ask themself, 'what might hypothetically be done with this?' And the engineer with a little more real world experience ("the cynic" ;-) might assume that someone will do it, sooner or later.
That was part of my reputation, for a while: Thinking of what was possible, and assuming -- or insisting -- it needed to be addressed. A year or two later, having done so would prove to have been of benefit. It would sometimes piss Management off, in the short term. But eventually, they came around.
Anyway, looking from the outside or the inside: If it's there and can be used "that way", someone's going to get around to doing so.
I also read about Bosch's involvement but this story was based on a report by the Bild am Sonntag newspaper which is not trustworthy. [1] is an English language source that relays this report. The gist of the story was that Bosch supplied the software for R&D purposes and Volkswagen engineers decided to put it into production cars too.
However, this was later rejected by Bosch who claim that writing additional code is required by the car manufacturer to get the emissions test detection feature [2].
I'm conflicted. What do people think about the ethics of buying (non-diesel) Volkswagen cars at this point? Obviously new cars pad VW's profits and reward bad behavior.
But what about used ones? Seems like prices will be plummeting. And if you like the cars, seems like a great time to buy.
I have one of the cars that will be impacted by the recall. Performance and fuel economy are both top notch. The cabin fit and finish are on par with vehicles three times the price. I'll certainly be holding on to mine. If the used vehicle price plummets I'd definitely consider getting a 2nd one for my wife.
What do people think about the ethics of buying (non-diesel) Volkswagen cars at this point?
Hey, if you want to buy a shit car, I don't think ethics really factor into it.
All kidding aside, despite being in absolutely no danger of ever buying another VW, I would think of it like I think of buying from (to pick a recent popular whipping boy) Amazon: does giving money to the company vote for the world you want to live in? I like workers to be treated fairly and with respect, so no more Amazon for me. I like clean air and products that win because they're great products that don't rely on cheating to make them great, so no more VWs for me.
As for used ones (assuming non-diesel), I'll use another metaphor. I'm a vegetarian. What if my veggie pizza shows up with pepperoni (and assume I otherwise like pepperoni)? I eat it. Cow's already dead, I didn't request that it be killed for my meal, and they're just going to throw it away if I send it back. Might as well eat it. Used VWs? Money's already in the pocket of a crooked company, might as well buy one if it suits your needs. Just take it somewhere other than the dealer when it needs work. :-)
Was looking at a new diesel but will maybe go for the GTE hybrid instead. The Passat GTE the only full size hybrid wagon that I know of(?) and it's very reasonably priced at around $5-10k over the non-hybrid.
I firmly believe in punishing companies that are unethical by personally boycotting them. VW doesn't even sell the best gasoline-powered cars for the price, so there's no way I'd be tempted.
May I suggest that the government raid the ECU engine control unit supplier Bosch and examine their email servers which of their customers has cheated.
"But even so, Bosch still supplied the “defeat” code EDC 16 engine management system at the heart of the #Dieselgate scandal"
"the high temperatures heat the selective catalytic reduction system ("catalyst") and improves the catalyst's ability to reduce tailpipe NOx emissions"
They were experimenting on human subjects without informed consent and using the data to overstate results that favored the regulations they wanted to impose.
These regulations set the particulate limits diesel powered vehicles must meet.
Why doesn't the EPA simply ban the sale of any car manufactured by VW until they start passing the new emissions tests? Even better, begin requiring previously purchased VW cars pass these emissions tests. If the cars don't pass the new emissions tests then VW is required to reimburse the blue book value of the car for false advertising, fraud, and acting in bad faith.
I want VW to be held accountable, but I don't want them to decimate the company.
While I want clean air, and believe in global warming; I'm afraid that the EPA will use VW as a reason to tighten emmission standards to to point where we will need to buy new vechicles in order to drive. I can't afford, nor really want a new vechicle.
I'm at the point now that I pray my vechicles passes smog checks. I'm not working as a mechanic now, but I have been to automotive school. I take those smog results, I get every two years, and tune my vechicles appropriately. My vechicles are tuned properly. I use a secondary 02 sensor attached to secondary bung holes before, and after the catalytic converter. All, so I can get the engines in perfect stoichiometric values. I clean out my erg valves. I change the oil. I double check some emmission sensors with a DVOM. I look for that lazy sensor and replace it if it's not in spec.
Even with all that, it seems like it's a crap shoot on wether my vechicles pass.
(I can't blame the EPA, or CARB totally. I have found gross errors in Motor Emission publications. Off subject, but smog stations are only required to have one emmission's reference on site. Most shops usually just have the current copy of Motor's Emmisson Manual. It's cheap, and that's what they show you if your vechicle fails the visual. The publication is filled with errors. If you failed the visual on your smog test, double check the information with Mitchell Manuals. I have never found an error in a Mitchell manual. Off subject--yes, but when your arguing with a smog tech. you might remember this post.)
Blue book value that recently dropped because of the emissions scandal? ;)
Requiring VW reimburse purchase price would make more sense, but still would not account for time spent having to look for new car etc.
On the other hand, there are numerous reports of owners not wanting to return their cars for software updates out of fear that such fixes would change performance characteristics of their vehicle for the worse.
In terms of banning sales, that would be an effective strategy to 'press' VW into acting faster but is likely to severely impact the co financially - something Germany simply can't allow due to huge numbers of people employed/at stake.
I definitely think management was involved here. However, its entirely possible that some management was not aware of the details. I doubt that more than some couple tens of people at VW group know intimate details of the VW diesel ECU. With badge engineering/platform sharing, its possible that Porsche/Audi/VW or whoever was not directly responsible for delivering the code could be in the dark about it. Im sure they even want it that way so there is plausible deniability. There are plenty of things that go on in the company I work for that I have no knowledge of. There are plenty of things in the software I customize/implement that I do not know about.
Bottom line- VW group cheated, was caught, and Im sure has a few dozen scapegoats lined up in accordance with automotive scandal rules and regulations. People will use this as a platform to get their names in the spotlight as a crusader for the people, the managers that made the decision will go largely unpunished, and the world will move on. I think its crappy behavior, but in the scheme of things they did not directly murder anyone and people willingly participate in much more dangerous activities than breathing excessive NOX fumes... I think that they should just fine them make VW say "We are truly deeply heartfeltly sorry" and then we can worry about the bigger problems in life.
NOX fumes kill tens to hundreds of thousands of people every year, there isn't many "bigger problems in life" than things like that.
Pollution has very diffuse, but very real human and ecological consequences. One of the reasons this fight is so hard is because they are so hard to reason about or even witness. It took systematic study of events like https://en.wikipedia.org/wiki/Great_Smog_of_1952 to realize how many people stuff like NOX emissions kill and disable.
What if doing something took a day off of the life of every single person in the world. In terms of "days of life destroyed" you would have an event unparalleled by any genocide in history, but yet how many people would even care?
> people willingly participate in much more dangerous activities than breathing excessive NOX fumes
What does that have to do with anything? If Alice chooses to wrestle sharks in her spare time, that's completely irrelevant to Bob being forced to breathe toxic fumes.
[+] [-] Johnny555|10 years ago|reply
Though the fact that none of the other manufacturers spoke up about the violation of emissions standards when when they must have done their own tests to see how VW managed to get emissions so low, seems to point to the fact they they were all doing it to some degree. Perhaps not as brazenly as VW, but still no one wants to rock the boat they are in.
[+] [-] bri3d|10 years ago|reply
[+] [-] ChuckMcM|10 years ago|reply
If you presume they are doing their job and not cheating, why would you cross check that your emissions were valid in both the dyno test and on the track? It requires that you assume your fellow workers are cheating, which is hard to do at most places.
[+] [-] amelius|10 years ago|reply
Well, in any case, it shows that ethical codes will probably not work for engineering professions (or any profession for that matter).
[+] [-] azernik|10 years ago|reply
[+] [-] oxryly1|10 years ago|reply
[+] [-] addicted44|10 years ago|reply
[+] [-] dexterdog|10 years ago|reply
[+] [-] hellcow|10 years ago|reply
The CEO of VW was just fired for allowing this scandal on his watch, and he was replaced by a guy guilty of (at the very least) the same exact thing.
[+] [-] monkbroc|10 years ago|reply
Whenever the topic of automotive software comes up on HN there are comments alongs the lines of "global variables bad", but not much construtive feedback.
I want to explain some of the tradeoffs that lead to the architecture used in automotive software to get a better discussion going with HN readers about that architecture.
tl;dr Given the hardware restrictions, real-time requirements and measurement capabilities required in automotive software, shared global variables without locks is a fast and safe way to share state between different software components as long as each variable is only written in one place in the program.
The microprocessor has to run for 10+ years in a wide range of temperatures and be dirt cheap, so you end up with specs like 180 MHz, 4 MB of flash and 128 KB of RAM.
The program must run deterministicly with respect to memory. There is no malloc/new in the code. All variables are statically allocated.
Because the physical world doesn't pause, no code is allowed to block while waiting for resources, especially synchronization primitives like mutexes.
The software architecture is in 2 main parts: basic software containing the real-time OS and hardware drivers, and the application layer which has the domain-specific code for controlling the engine, brakes, etc.
The basic software is implemented using usual C programming techniques. It has an API provided by function calls and structs to hide implementation details of each microcontroller.
The application software is where the programming model is different.
To understand why, you need to know where automotive software comes from and what it is trying to acheive.
Originally all controllers were mechanical: a valve opens proportionally to the vacuum in a part of the system. Then some controllers were implemented in analog electronics: take multiple voltages, feed them through an op-amp and use the output to control a valve.
So automotive software reproduces this: get some inputs, compute the same physical equations at a regular rate and generate outputs.
This is dataflow programming. Blocks of code have inputs and outputs. They are executed at a fixed rate that depends on the physical phenomena (air flow changes fast, temperature changes slowly). Different blocks are conneceted together in a hierachical way to form subsystems. Encapsulation is acheived by viewing these blocks as black boxes: you don't need to care how the block works if you are only interested in knowing which inputs it uses and outputs it produces.
Here's an example component to control a gizmo.
It might be implemented in a visual environment like Simulink by MathWorks, or it implemented by hand from a spec.
It takes some inputs (EngineSpeed, CoolantTemp, Gizmo_PositionDesired, Gizmo_Position), has some intermediate values (Gizmo_Error), and outputs (Gizmo_DutyCycle). Those are implemented as global variables. It also uses some constants (Gizmo_Gain_MAP, Gizmo_Offset_VAL). It has 2 processes, running every 100ms and 10ms. All this information would be specified in an XML file.The header GizmoController_data.h is auto-generated at compile time by a tool from the XML file mentioned above. It will contain global variable definitions for the inputs, intermediates and outputs with the appropriate volatile, const, static and extern storage classes/type qualifiers. This ensures that the compiler will enforce that inputs can't be written to, intermediate values are private to the component and outputs can be read by other modules.
Note that no explicit synchronization is needed to access inter-process variables like Gizmo_Gain or inter-component variables like Gizmo_Position. It's shared memory between 2 processes scheduled in OS tasks that can potentially interrupt each other, but since the write is atomic and happens only in one place, there is no data race. This is huge! Concurrent programming, without locks, with the best efficiency possible, using a simple technique anybody can understand: only one place in the program is allowed to write to any global memory location.
Calibration is another aspect of automotive software. In most software the constants either never change or can be set in some kind of configuration file. For an automotive controller, the value of constants (gains, offsets, limits, etc) depend on the vehicle so they must be configurable at run time during development. This is implemented in the C code by putting all constants in a memory area that is ROM in production units, but RAM in development units. The compiler enforces that application software cannot change constants, but the basic software includes code so that constants can be changed from the outside in development. This process is called calibration and is done by calibration engineers who are usually not the ones who wrote the software. Note that calibration can drastically affect the behavior of the software. What would happen if Gizmo_Gain_MAP is set to all zeros?
Measurement of variables is essential to understanding what's going on inside the embedded controller. Having all that state available in global variables makes it possible for the calibration tool request the value of any variable in the software at a fixed rate and display it in a virtual oscilloscope.
The measurement and calibration tool needs to know how to access the variables and constants. It uses a file that maps from names to addresses for a particular version of software. That file can easily be generated a compile time since all allocations are static.
Going back to the architecture of the application software, let's look at where our gizmo controller fits. It is not the only component needed to make the gizmo work. You also need components to calculate the gizmo position from some external signal (let's say an analog voltage), to route the output signal to the powerstage driver on the PCB, to determine which position the gizmo should currently occupy. These would form the gizmo subsystem package.
When the supplier releases gizmo 2.0 (TM) they upgrade the input signal to be a PWM input instead of an analog input. Modularity in the software allows the software team to simply replace the gizmo position component with one that reads a PWM instead of an analog voltage and keep the rest of the gizmo subsystem the same. In the future, projects that use gizmo 1.0 use one version of the gizmo subsystem and projects that use 2.0 use another.
This is true at any level in the hierarchy: as long as the inputs and outputs are the same, a component or subystem package can be replaced by another.
Version control in automotive software reflects this. Instead of having one tree of versions and releases like a typical software project, each component, subsystem package and software project has its own tree of versions. Each software project will reference the subsystem packages required for their engine type, vehicle platform, sensors and actuators, etc. This is how code reuse is acheived.
Testing is a mix of simulation (the sensor/actuator is simulated in Simulink and connected to Simulink block diagram of the software component), hardware-in-the-loop (a computer simulates the vehicle, but the real electronic control unit is used) and vehicle testing.
Thanks for reading. I hope this improves your understanding of how automotive software is structured.
I'm hoping the discussion will bring examples from other fields like robotics, drones and aeronautics that have similar real-time requirements on how they architect their software.
[+] [-] rtpg|10 years ago|reply
For example, I know there's an embedded systems company that writes everything in Haskell, and running their Haskell program actually generates "safe" C code that they compile down to their chips.
My impression from the oft-cited Toyota report was not only that there was a lot of global variable stuff, but that these "write-once" principles weren't super respected.
[+] [-] jevinskie|10 years ago|reply
EDIT: It basically provided at least two implementations for an ASIC by independent teams. The ASIC guys would implement the real deal in an HDL and test it. However, full system tests where the HDL simulation is hooked up with CPU simulation were too slow. By implementing the same spec in C, you gain simulation performance and get a second implementation and set of eyes that can help find bugs.
[+] [-] STRML|10 years ago|reply
If some of these timers are implemented as separate OS tasks that can interrupt each others, what's to stop one from taking a bit too much time and throwing off the timing of another?
[+] [-] bsimpson|10 years ago|reply
> The microprocessor has to run for 10+ years in a wide range of temperatures and be dirt cheap, so you end up with specs like 180 MHz, 4 MB of flash and 128 KB of RAM.
If I'm paying tens of thousands of dollars for a car, how come they're using the cheapest possible components? If tinkerers can ship the Raspberry Pi for $30 per board, inc. a 900Mhz quad-core chip and 1GB RAM, you'd think GM could get components at least that modern for an insignificant cost relative to the car they are controlling.
[+] [-] ptaipale|10 years ago|reply
When coming to work from university (1989-90), I made a somewhat similar though more primitive design for the embedded software of a redundancy controller of a satellite modem. The various tasks (control of RF and baseband electronics, display and buttons UI, network management interface, etc) would run according to a scheduler, each in turn. No dynamic memory allocation, internal communication via global variables that are written only in one place. There was no mechanism to actually enforce this, it was just the principle in code, and the enforcement was possible because it was a one-man software project so I had no one else to blame.
This was written in PL/M (which looks like Pascal but is more similar to a primitive variant of C) on a 8031 microcontroller, from scratch (no 3rd party SW components at all - that was a long ago).
Later on, I used the same platform for a design we did on a telephone/data cable modem we designed for domestic use (We saw a need for internet enabled cable modem in 1992 and started work on a thing that provided POTS and 128 kbit/s over SLIP). That was also running a 8031, but as we wanted to make it dirt cheap, the board did not have dual-port RAM. The hardware wizards made it run with a hack on the regular RAM access so that the SW could only access RAM if it read any memory location twice: first read would just place the address bits on the bus, and doing another read would actually fetch the content. This enabled us to save several dollars per device.
(This was possible because the 8031 has an architecture which is not strictly von Neumann design: the code and address spaces are different memory spaces and you use different CPU instructions to access the different memories, so your program memory on EPROM works normally but your data memory in external RAM does not. There's also a small internal RAM in yet another overlapping address space which was not impacted by address bus weirdness.)
This is just to show what kind of strange limitations you sometimes have on the hardware in embedded devices. Unlimited virtual memory, large non-volatile storage instead of a puny i2C bus EEPROM... you just dreamed of those. Modern controllers with 4G flash are really huge compared to that, but they are still far below what people have on desktop computers, or even an Android phone.
The limitations in embedded devices indeed come from cost per device as well as heat, vibration, radiation, humidity and similar environmental factors.
[+] [-] paulannesley|10 years ago|reply
[+] [-] oxide|10 years ago|reply
[+] [-] swah|10 years ago|reply
May I ask why you left the industry?
[+] [-] mzs|10 years ago|reply
As alleged in the NOV, VW manufactured and installed software in the electronic control module of these vehicles that senses when the vehicle is being tested for compliance with EPA emissions standards. When the vehicle senses that it is undergoing a federal emissions test procedure, it operates in a low NOx “temperature conditioning” mode. Under that mode, the vehicle meets emission standards. At exactly one second after the completion of the initial phases of the standard test procedure, the vehicle immediately changes a number of operating parameters that increase NOx emissions and indicates in the software that it is transitioning to “normal mode,” where emissions of NOx increase up to nine times the EPA standard, depending on the vehicle and type of driving conditions. In other tests where the vehicle does not experience driving conditions similar to the start of the federal test procedure, the emissions are higher from the start, consistent with “normal mode.”
I have not read a substantive response from any part of VW to today's allegations.
[+] [-] Theodores|10 years ago|reply
The funny thing is this:
Porsche could sue over £25 a day congestion charge (2008)
http://www.theguardian.com/business/2008/feb/19/travelandtra...
> The mayor's office responded saying the threatened legal action was "a double attack on Londoners".
> "First Porsche are trying to deprive Londoners of their democratic right to decide in the mayoral election on 1 May whether they want gas guzzling and polluting cars to drive in London when there is absolutely no need for them to do so. Second they are trying to impose on all Londoners unnecessary levels of pollution and greenhouse gases by a tiny minority," said a spokesman for the mayor.
> "No one is allowed to throw their rubbish in the street and Porsche should not be allowed to impose gas guzzling polluting cars on Londoners who do not want them."
I hope Porsche get taken to the cleaners.
Schadenfreude?
[+] [-] aembleton|10 years ago|reply
[+] [-] alricb|10 years ago|reply
Environment Canada, a "regulatory group"? It's a department of the Government of Canada, yo.
[+] [-] madengr|10 years ago|reply
[+] [-] jonknee|10 years ago|reply
I have a 2015 Q5 TDI which while not specifically mentioned has the same size engine as several of the ones that are mentioned (3.0-liter). I would be shocked if they just started using the defeat device in the 2016 model year for the Q5 considering they were already getting pushback from the EPA before the 2016 models even got announced.
I bought the diesel specifically because of the high mileage and supposedly clean emissions. What a crock.
[+] [-] pasbesoin|10 years ago|reply
If that is the case, then this is an example of the maxim: When you provide a feature, someone will use it to their own ends -- meaning, "misuse" it.
Another example to wave in front of all those politicians and people advocating for encryption "back doors".
If you put the feature in there, people will use it any way that suits them. Including and especially ways that you did not intend nor want.
If it wasn't Bosch, perhaps there was nonetheless some "legitimate" argument within VW for adding this functionality. Enough to get the software folks -- particularly those not making big bucks off of the deal -- to implement this.
But an observant engineer might nonetheless ask themself, 'what might hypothetically be done with this?' And the engineer with a little more real world experience ("the cynic" ;-) might assume that someone will do it, sooner or later.
That was part of my reputation, for a while: Thinking of what was possible, and assuming -- or insisting -- it needed to be addressed. A year or two later, having done so would prove to have been of benefit. It would sometimes piss Management off, in the short term. But eventually, they came around.
Anyway, looking from the outside or the inside: If it's there and can be used "that way", someone's going to get around to doing so.
[+] [-] jonemo|10 years ago|reply
However, this was later rejected by Bosch who claim that writing additional code is required by the car manufacturer to get the emissions test detection feature [2].
[1] http://blog.caranddriver.com/report-bosch-warned-vw-about-di... [2] http://www.reuters.com/article/2015/10/07/volkswagen-emissio...
[+] [-] superuser2|10 years ago|reply
But what about used ones? Seems like prices will be plummeting. And if you like the cars, seems like a great time to buy.
[+] [-] knieveltech|10 years ago|reply
[+] [-] mikestew|10 years ago|reply
Hey, if you want to buy a shit car, I don't think ethics really factor into it.
All kidding aside, despite being in absolutely no danger of ever buying another VW, I would think of it like I think of buying from (to pick a recent popular whipping boy) Amazon: does giving money to the company vote for the world you want to live in? I like workers to be treated fairly and with respect, so no more Amazon for me. I like clean air and products that win because they're great products that don't rely on cheating to make them great, so no more VWs for me.
As for used ones (assuming non-diesel), I'll use another metaphor. I'm a vegetarian. What if my veggie pizza shows up with pepperoni (and assume I otherwise like pepperoni)? I eat it. Cow's already dead, I didn't request that it be killed for my meal, and they're just going to throw it away if I send it back. Might as well eat it. Used VWs? Money's already in the pocket of a crooked company, might as well buy one if it suits your needs. Just take it somewhere other than the dealer when it needs work. :-)
[+] [-] alkonaut|10 years ago|reply
[+] [-] CydeWeys|10 years ago|reply
[+] [-] acd|10 years ago|reply
"But even so, Bosch still supplied the “defeat” code EDC 16 engine management system at the heart of the #Dieselgate scandal"
[+] [-] yuhong|10 years ago|reply
"the high temperatures heat the selective catalytic reduction system ("catalyst") and improves the catalyst's ability to reduce tailpipe NOx emissions"
There has been recently http://www.thestar.com/autos/2015/09/30/canadian-trios-exhau...
I wonder if this tech would work for SCR systems or not.
[+] [-] marvel_boy|10 years ago|reply
[+] [-] tomohawk|10 years ago|reply
http://junkscience.com/2012/10/new-documents-prove-falsifica...
They were experimenting on human subjects without informed consent and using the data to overstate results that favored the regulations they wanted to impose.
These regulations set the particulate limits diesel powered vehicles must meet.
[+] [-] tyho|10 years ago|reply
The real problem is drawing any conclusion from a sample size of 1.
[+] [-] CamperBob2|10 years ago|reply
[+] [-] idbehold|10 years ago|reply
[+] [-] mzs|10 years ago|reply
[+] [-] marincounty|10 years ago|reply
While I want clean air, and believe in global warming; I'm afraid that the EPA will use VW as a reason to tighten emmission standards to to point where we will need to buy new vechicles in order to drive. I can't afford, nor really want a new vechicle.
I'm at the point now that I pray my vechicles passes smog checks. I'm not working as a mechanic now, but I have been to automotive school. I take those smog results, I get every two years, and tune my vechicles appropriately. My vechicles are tuned properly. I use a secondary 02 sensor attached to secondary bung holes before, and after the catalytic converter. All, so I can get the engines in perfect stoichiometric values. I clean out my erg valves. I change the oil. I double check some emmission sensors with a DVOM. I look for that lazy sensor and replace it if it's not in spec.
Even with all that, it seems like it's a crap shoot on wether my vechicles pass.
(I can't blame the EPA, or CARB totally. I have found gross errors in Motor Emission publications. Off subject, but smog stations are only required to have one emmission's reference on site. Most shops usually just have the current copy of Motor's Emmisson Manual. It's cheap, and that's what they show you if your vechicle fails the visual. The publication is filled with errors. If you failed the visual on your smog test, double check the information with Mitchell Manuals. I have never found an error in a Mitchell manual. Off subject--yes, but when your arguing with a smog tech. you might remember this post.)
[+] [-] drglitch|10 years ago|reply
Requiring VW reimburse purchase price would make more sense, but still would not account for time spent having to look for new car etc.
On the other hand, there are numerous reports of owners not wanting to return their cars for software updates out of fear that such fixes would change performance characteristics of their vehicle for the worse.
In terms of banning sales, that would be an effective strategy to 'press' VW into acting faster but is likely to severely impact the co financially - something Germany simply can't allow due to huge numbers of people employed/at stake.
[+] [-] empath75|10 years ago|reply
[+] [-] wnevets|10 years ago|reply
do they have the legal authority to do so? If they did, I doubt the republican led congress would stand by idly.
[+] [-] autobahn|10 years ago|reply
[+] [-] maxerickson|10 years ago|reply
[+] [-] jzd|10 years ago|reply
[+] [-] astrodust|10 years ago|reply
[+] [-] S_A_P|10 years ago|reply
Bottom line- VW group cheated, was caught, and Im sure has a few dozen scapegoats lined up in accordance with automotive scandal rules and regulations. People will use this as a platform to get their names in the spotlight as a crusader for the people, the managers that made the decision will go largely unpunished, and the world will move on. I think its crappy behavior, but in the scheme of things they did not directly murder anyone and people willingly participate in much more dangerous activities than breathing excessive NOX fumes... I think that they should just fine them make VW say "We are truly deeply heartfeltly sorry" and then we can worry about the bigger problems in life.
[+] [-] throwaway2048|10 years ago|reply
Pollution has very diffuse, but very real human and ecological consequences. One of the reasons this fight is so hard is because they are so hard to reason about or even witness. It took systematic study of events like https://en.wikipedia.org/wiki/Great_Smog_of_1952 to realize how many people stuff like NOX emissions kill and disable.
What if doing something took a day off of the life of every single person in the world. In terms of "days of life destroyed" you would have an event unparalleled by any genocide in history, but yet how many people would even care?
[+] [-] PhasmaFelis|10 years ago|reply
What does that have to do with anything? If Alice chooses to wrestle sharks in her spare time, that's completely irrelevant to Bob being forced to breathe toxic fumes.