Do Complex Systems Require Higher Safety Standards From Managers and Engineers? (techcrunch.com) 137
An anonymous reader quotes TechCrunch:
Automotive emissions, nuclear power plants, airplanes, application platforms, and electrical grids all share one thing in common: they are very complex, highly coupled systems... Engineers have matched some of this growing complexity with more sophisticated tools, mostly derived from greater computing power and better modeling. But there are limits to how far the technical tools can help here given our limits of organizational behavior about complexity in these systems. Even if engineers are (potentially) acquiring more sophisticated tools, management itself most definitely is not.... One pattern that binds all of these engineering disasters together is that they all had whistleblowers who were aware of the looming danger before it happened. Someone, somewhere knew what was about to transpire, and couldn't hit the red button to stop the line...
Engineering managers probably have the most challenging role, since they both need to sell upwards and downwards within an organization in order to maintain safety standards. The pattern that I have gleaned from reading many reports on disasters over the years indicates that most safety breakdowns start right here. The eng manager starts to prioritize business concerns from their leadership over the safety of their own product. Resistance of these pecuniary impulses is not enough -- safety has to be the watchword for everyone...
Finally, for individual contributors and employees, the key is to always be observant, to be thinking about safety and security while conducting engineering work, and to bring up any concerns early and often. Safety requires tenacity. And if the organization you are working for is sufficiently corrupt, then frankly, it might be incumbent on you to pull that proverbial red button and whistleblow to stop the madness.... [T]he demise of the ethical engineer doesn't have to be a fait accompli.
Engineering managers probably have the most challenging role, since they both need to sell upwards and downwards within an organization in order to maintain safety standards. The pattern that I have gleaned from reading many reports on disasters over the years indicates that most safety breakdowns start right here. The eng manager starts to prioritize business concerns from their leadership over the safety of their own product. Resistance of these pecuniary impulses is not enough -- safety has to be the watchword for everyone...
Finally, for individual contributors and employees, the key is to always be observant, to be thinking about safety and security while conducting engineering work, and to bring up any concerns early and often. Safety requires tenacity. And if the organization you are working for is sufficiently corrupt, then frankly, it might be incumbent on you to pull that proverbial red button and whistleblow to stop the madness.... [T]he demise of the ethical engineer doesn't have to be a fait accompli.
start prosecuting executives (Score:3, Insightful)
PE powers and more unions! (Score:2)
PE powers and more unions!
Re: (Score:2)
Re: PE powers and more unions! (Score:1)
Bullshit. Most unions are good, you and Fox news just notice the exceptions. CWA has taken very good care of me. AFCME took care of my father in law when he was getting pushed out for being old. The problem its unions such as many of the state education associations that gain political power by keeping their members poor.
Re:start prosecuting executives (Score:5, Insightful)
You will see a very sudden interest in the culture of safety.
Unfortunately, many of the most "accomplished" executives are masters at deflecting blame. A scapegoat will get fired for cause, and the responsible personnel will maybe possibly perhaps "retire in a few months to spend time pursuing other families" or whatever. Snugly strapped in to their golden parachute, ofc.
Executives responsible for all, in law (Score:5, Insightful)
I've experienced, as an electrical worker, the reaction exec's have when they suddenly become, by law, always at fault for "preventable" accidents. "Follow the book" suddenly becomes the only solution. If there isn't one then write it. And if you tell them it'll cost 10x as much that way, they just shrug because they know every other company is in the same boat.
Of course, that may also trigger them into looking for another country to move the factory to.
Re: (Score:3)
And if you tell them it'll cost 10x as much that way, they just shrug because they know every other company is in the same boat.
Of course, that may also trigger them into looking for another country to move the factory to.
That's why we have standards, right? If you demand they follow the standard no matter where they go, then they will only go if they can find trained labor which will work cheaper than the amount it will cost to ship the product back to the market. The standard has to require inspection, of course, to ensure that it's being followed.
Re: (Score:2)
By standards, do you mean "PCI compliance", "SOX compliance", "security polices", and "corporate mandates"? Because for many industries, including airlines and medical safety, many of these standards are not followed. They are merely checklists to note compliance, not actually used to guide practices and policies that may interfere with profit, with quarterly reports of goals met, or which may yield turf to other departments. The bureaucracy or the organization themselves become the goal of the company.
I'v
Re: (Score:1)
"Pursuing other families"
I think this is what sometime actually happens.
Re: start prosecuting executives (Score:2)
How do you allocate guilt among a million shareholders who knew nothing about the decision in question? What if they're mutual fund customers who didn't even know they owned shares in that company?
I do it all the time - while offering solutions (Score:3)
I very often tell the boss that we can't product exactly what they want, with all of the features they want, in the time they want.
HOWEVER I don't use words like "can't". I say something like:
--
Minor feature X probably contributes 10% of the value, and will require 75% of the effort / time. We can complete the project on time and add feature X in phase 2.
--
Or:
--
I understand you'd like it done in 12 weeks.
A barebones MVP will take 8 weeks, without features A, B, C, or D.
Feature A adds two weeks.
Feature B a
Most useful reply award winner*!! Re: I do it all (Score:2)
* I deliver useful working software systems in the same difficult circumstances we all work under. The careful language of this reply may work like magic the next time you face an unreasonable manager. I wish Iâ(TM)d learned this posterâ(TM) approach earlier. Well-spoken developers earn bigger bucks!
They lied to you. They don't really want a unicorn (Score:2)
The trouble with users is what they SAY they want isn't actually what they want.
The trouble with management is that what they SAY they want barely gives any hint as to what they actually want.
When management tells you about a software solution they want, that's a clue that they have some problem they want to solve. That's all it is. The content of the request can *almost* be thrown away.
The first step a proper engineer takes when they receive a request for a system is to thoroughly define the problem to be
"No" (Score:2)
Quoting Betteridge.
Also, no because simple things need it too (Score:2)
Lawn Darts are about as simple as you can get.
A single chunk of heavy pointy metal that kids throw on lawns.
But they're best known for their lacking safety standards.
The problem is more co0mplex (Score:3)
Neural networks (Score:5, Insightful)
This seems like a big problem for automated systems built around algorithms like neural networks. It isn't practical to examine a neural network of reasonable complexity and try to extract information about why it produces the outputs it does. Training is fitting a complex mathematical function to the training data set. Due to the complexity of the neural network (or another algorithm of similar complexity), it seems very difficult to be confident that the mathematical functions haven't been fit in such a way that the inputs at certain points won't lead to undesired behavior. If these algorithms are automating systems like aircraft or self-driving cars, that seems like a big problem. The complexity of the system allows for better performance, but the inability to extract meaningful information about why the system behaves the way it does would seem to be a big problem. I recognize that simpler techniques like decision trees might not provide the skill of a neural network, but at least one can be more confident that they won't behave in an unexpected manner.
Re: (Score:2, Interesting)
This seems like a big problem for automated systems built around algorithms like neural networks.
I believe the oft quoted answer is you don't necessarily rigorously validate the AI/neural network/whatever. You validate the watchdog / verifier bit, and try to prove that it does limit the worst cases within the limits that are required, or that the solution found actually works and meets some definition of correct.
For instance, maybe an AI finds a good path from A to B while avoiding all obstacles, and getting a good reward score internally for however it figured out the path. Perhaps the AI is on a ro
Re: (Score:3)
Safe systems don't rely on unpredictable control systems.
For example, Japanese high speed rail is the safest in the world. Drivers are not allowed to use their own initiative to deal with problems. When there is a fault or unusual situation, they must open the operating manual and follow the instructions precisely.
Their network was the first in the world, starting in 1964, and has never had a single fatality or serious injury due to an accident.
For safe systems AI will be limited to doing things like image
Re: (Score:2)
It's even better. There were two sensors on the airplane. There have always been two sensors, even on the older models as they are used by the autopilot (and correctly).
For some reason MCAS only used one sensor and ignored the other one that was already there. On the next flight, it would switch to that other sensor.
Re: (Score:2)
It was more about a misguided sense that the pilots would know what to do in a rare failure mode, and a significantly higher failure rate than anticipated— likely due to wiring issues. The original theory was that the system was not flight critical so failure just defers to the pilot. Unfortunately, the system re-engaged with bad sensor data, repeating the problem. A poor design to be sure, but the system of logic appears to have been consistent.
Re: (Score:1)
Even electronic accelerator pedals on cars have redundant sensors
Not to my knowledge, no. They merely ensure that the car stops on failure, instead of going full throttle till it hits something. The stopped car is safe, and can be picked up by a tow truck.
This is normally done by using a single potmeter as a sensor. If one wire breaks, you get a max reading or a min reading from that potmeter. A min reading is harmless because the car stops as if the throttle was released. A max reading is also harmless; because full throttle does not reach the max setting on the potmet
Yes. Functional safety is a process. (Score:3)
ISO 26262, DO-178C, IEC 61508, etc, etc.
The problem with American aerospace is they figured out how to game the requirements process so that no one is culpable. At this point I would trust VW's safety record (because the spotlight is already on them) over Boeing's (because they played fast and loose).
Re: (Score:2)
At this point I would trust VW's safety record (because the spotlight is already on them) over Boeing's (because they played fast and loose).
VW has a tolerably but not amazingly good safety record. Several of the brands under their umbrella have had design failure-related unattended vehicle fires and the like. I would expect them to be true to form.
Who came up with this headline ? (Score:1)
This is beyond fake news, it's full on joke news.
Yes, they do... but what about software? (Score:5, Informative)
Many complex systems require greater safety standards when compared to systems that cannot significantly harm people. That is why many of these systems are already subject to safety standards. For example, consider the NFPA standards that are commonly adopted in the USA and regulate electrical wiring, fuel storage, building safety, and many, many more everyday items. There are hundreds and hundreds of other ISO, ANSI, FM standards (to name a few) that also cover safety and standard tests.
Now consider programmers and developers. What standards are used (other than, perhaps, the standards that define various versions of computer languages) to put together very complex systems?
Also consider that there is no such thing as a PE for Software Engineering even though many people's lives depend on software working properly.
Computer science Versus Computer Engineering (Score:5, Insightful)
Engineering is precisley the sceince and training required to manage complex Engineered systems. That's exactly what engineering is. period.
Computer science is unrelated to engineering. Possibly one might study the science of computer engineering but that's going all meta on this and is a distraction from the point
Computer scientists should not be allowed to manage complex safety systems unless they are qualified engineers.
Engineers learn to work in teams where the they manage complexity across interfaces and scales.
If you think git hub takes care of that or you only need to work on your own part of the tool and git merge or an atp-get build will take care of the integration then you are not an engineer.
So yes, people with engineering credentials can manage safety.
Doesn't mean they will do it right. For example, the poster child for engineering safety across every scale used to be Boeing. I've always been awestruck that they can build machines that fly when no one person knows where every nut of bolt goes and how it was certified to be a sufficient fastener. That's not a made up example, many early boeing crashes in world war 2 were attributable to things similar to a nut coming lose and rattling it's way into an electrical junction box. Engineering practice incoporates continuous safety improvements as complex systems (structural like a nut on door) interact with electronic sub systems when the respective engineers don't even work in the same building.
The tragedy of boeing is their last 3 planes have been engineeing duds. Something has happened to that company.
Re:Computer science Versus Computer Engineering (Score:5, Informative)
Turing's Bombe was based on Polish work as was all of Bletchley Parks initial work.
The greatest achievement of the WW2 code breaking effort, Colossus, was NOT built or designed by Alan Turing. It was designed and built by a Post Office Telecomms ENGINEER called Tommy Flowers, whose only claim to fame now is a community centre in London and a BT research centre named after him. There should be a statue to Tommy in Trafalgar Square but he was forgotten about due to the Governments complete secrecy about the work for decades and the fact he wasn't an alumni of Oxford or Cambridge and therefore wasn't the right "chap".
Turing is a renowned hero [deservedly so] and some of the lionisation is because of his death and sexuality. His acclaim crowds out many others who provided just as much, or even more insight into solving the problems [Gordon Welchman for example or Bill Tutte]. Bletchley Park was a huge team project, not the result of one mans efforts.
Re: (Score:1)
None of that is relevant to the claim, which was that computer science just isn't related to "engineering", which Turing [btw used as something called an "example", not a claim that he was the lone superhero who did it all single-handedly] and his team, and many of the other pioneers proves to be a complete lie.
Also, way to move the goalposts. An "engineer" is anyone who applies science, predominantly the laws of mathematics and physics, to solve a problem. The problem could be a simple one or a difficult o
Re: (Score:2)
>An "engineer" is anyone who applies science, predominantly the laws of mathematics and physics, to solve a problem.
That's the starting point, but an engineer (with no quotes) is a member of a professional organization that has been certified to be capable of doing so with human safety as the primary concern. Much like being a doctor is about a lot more than being able to wield a scalpel.
The technical skills are only one aspect of the job. The understanding and acceptance of the overarching importance
Re: Computer science Versus Computer Engineering (Score:2)
an engineer is a member of a professional organization that has been certified to be capable of doing so with human safety as the primary concern.
Personally I have never heard that definition; where did you get it?
Re: (Score:2)
If you're not certified, you can't call yourself an engineer in any professional capacity. Look at the professional ethics manual for the rest.
Re: (Score:2)
Computer science is unrelated to engineering.
I guess it all depends on how you view it and how you define Computer Science. The engineering aspect is very primitive right now, but it does exist, even if code monkeys never put into practice any engineering principals.
Re: (Score:2)
What standards are used (other than, perhaps, the standards that define various versions of computer languages) to put together very complex systems?
As far as software standards goes, there are many, but only a few that are ubiquitous across aerospace and automotive fields. MISRA started as a safety standard for coding automotive applications, but is widely used as a starting point in aero as well. DO-187B is what your going to follow, and test to, if you do any aerospace development.
IEEE-12207 is generally the process your going to follow for developing large applications for aero or auto, but you'll still use MISRA and DO-178B as needed.
If your hardco
Re: (Score:2)
I'm sure Elaine Herzberg would assert that waiting for a company's insurance rate to go up is not the best way to ensure safety. This is an infamous case of a safety supervisor was not able to prevent her death. Do you think that Uber's insurance rates went up very much?
https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
The settlement was likely relatively small. Seeing has her future panhandling earnings weren't a large number.
Safety first means no go (Score:1)
A priority on safety means nothing will work. The safest plane is one that stays on the ground, immobile.
Re: (Score:2)
Very myopic point of view. Planes that only stay on the ground are not planes. If the hard constraint is that the plane needs to fly, not providing this is not meeting the specification.
Incidentally, most aviation mishaps occur when a plane is touching the ground or rapidly becomes immobile.
Re: (Score:2)
Incidentally, most aviation mishaps occur when a plane is touching the ground or rapidly becomes immobile.
I vote that we remove the component causing the failure. If we get rid of the ground, we should eliminate a pretty high percentage of failures.
Re: Safety first means no go (Score:2)
Kind of like saying it's not the fall that kills you but the sudden stop. Though the worst aviation disaster (which hopefully will never be exceeded) was two planes colliding on the ground.
Re: (Score:2)
It isn't. I made one like that once and a truck crashed into it. That couldn't have happened if it was safely in the sky.
Re: (Score:2)
Hardly. Acceptable margins of safety are the first priority, not the only one. Engineering is the art of meeting all the other necessary priorities (functionality, cost, etc) without compromising the first priority.
Nothing will ever be perfectly safe, that must be the accepted starting point of any conversation around safety. But that doesn't mean it's acceptable to throw away acceptable safety margins to get on with the job. "What percentage of users is it acceptable to injure or kill, and with what fr
Re: (Score:2)
Volkswagon faking pollution controls wasn't a safety issue
While it is criminal, it is also a safety issue since they are predicting 1,200 premature deaths in Europe, especially in Germany, Poland, France, and Czech Republic. http://news.mit.edu/2017/volks... [mit.edu]
Re: (Score:1)
> Hire on merit and skill. Never hire on political correctness and for any non academic considerations.
I'm afraid that these are specifically forbidden by many "non-discrimination" policies. Gender, age, marital status, mental illness, and physical disabilities are all legally forbidden from consideration for hiring in many states for many federal positions. This is despite their direct and measurable correlation with work quality.
Re: (Score:2)
Part A worked with system A.
Part A will work with system B as it was tested and passed for work with system A.
No further testing is done and the results are a total fail.
Re: Yes (Score:2)
You can always discriminate on ability to do the job.
Re: (Score:2)
Those factors correlate, strongly, with ability to do the job. It's a compelling reason that they remain factors for hiring and for promotion, despite all attempts to "level the playing field" and to provide "equal outcome" at the expense of "equal opportunity". Hiring based only on merit and skill, including skill associated with age and gender and marital status and religion, can create a real risk of a discrimination lawsuit in many modern workplaces.
I'm afraid that the interplay between desires for fair
If the system is rigged by management pressure (Score:3)
Put together a team to do nothing but rip holes in designs. Separate chain of command with the power to independently veto and halt processes. Make their pay based on how well they find errors, not on how few error come up (i.e., properly oriented incentives).
We used to call them testers and QA.
Re: (Score:2)
Apparently that violates the agile commandments.
Saw a thread on stackexchange about it while looking for something else. Those guys were like religious zealots.
Re: (Score:2)
I m
Re: (Score:1)
" I AM a graduate engineer from the 1960's and we HAD to take courses in engineering ethics back then."
I am a graduate engineer from the 1960s as well. I went to the U of Arizona and there was no mention of engineering ethics that I recall. Could have used it because a few years after graduating I was fired because I would not certify MTBF on an isolation thermocouple module for a nuclear plant. (First look at the schematics found 400 mW on a 250 mW resistor.)
I wonder what schools had courses in engineer
Re: (Score:2)
No. (Score:2)
Don't even think of creating laws to ensure safety. They are job killing regulations and they have no place in the modern free market governed by the
Re: (Score:2)
Process (Score:3)
I have a pal who writes code for medical imaging machines. Each bug he fixes, he creates a new *trunk* of the codebase, adjusts or writes new unit tests, checks the integration tests, fixes the bug, runs the whole unit and integration suite against his change, writes out an explanation of what he did and why, along with failure mode and impact analyses, gets it code reviewed with the whole team, merges back with the trunk, then re-runs all the unit and integration tests *again* That's the process for a simple bug.
So, yeah, mission critical stuff has a more strict development process.
*Branch* (Score:2)
*branch* i meant, but he branches the entire codebase.
Re: (Score:2)
Because that's generally how branches work if you have the whole project in one revision control repository.
If the project is split across different repositories, it's probably a good idea to branch everything anyway so you can be sure you aren't working against something that changes out from under you. This is especially true for systems that interconnect in potentially complex ways.
Re: (Score:2)
> Because that's generally how branches work if you have the whole project in one revision control repository.
With git, not with Subversion or CVS. For other source control systems, branching of individual directories is quite common. The maintainers of larger code bases factor the code into individual modules or components, to ease testing of the individual components and allow their update out of lockstep with the other components. Switching to that model is a necessary factoring as projects grow and a
Trust (Score:2)
IIRC on their VCS, a checkout just checks file timestamps for changes, a branch/merge checks each file's contents via checksum. Basically, you don't trust the VCS, you double-check to make sure *everything* you modify gets checked in. This is part of the documentation process, too.
Re: (Score:2)
Most such systems today check the revision history for timestamps, not the files themselves. Many CI/CD systems are very reliant on the source control history.
Those Quality Safety Standards Exist Now (Score:2)
Unfortunately, failure to meet safety standards may or may not be illegal, depending on the law, rules and regulations governing a specific industry. If metal shards are found in food, there is a recall. If there are more than 50 reports of fault
Sorry, it's the Engineers job. Period. (Score:3)
Yes, management and tools are always important factors. However, it is fundamental to the very concept of being a "professional" engineer to take complete responsibility for one's own work product. It is not permissible to blame others for giving you bad specifications or a bad design or bad tools if you didn't even try to do the due diligence needed to assess and properly perform your own assigned tasks.
I have built aircraft and spacecraft avionics. I have built nuclear reactor instrumentation and control systems. I have built automated X-ray and neutron inspection systems, some of which were for munitions plants where one mistake could kill dozens or hundreds.
I refuse to take assignments where I don't know and understand where and how my piece fits into the whole. I insist on being fully informed, and being given enough time and information to come to my own informed judgments about the project as a whole and the tasks at hand.
I own my work. I also take full responsibility for any and all errors I ship. This attitude and commitment becomes evident in two main areas:
1. The requirements and specifications. Do they make sense? Do they cover all the needs of the product? Do they fully take into account outside regulations and standards? Do they have any gaps? Most importantly, is each and every requirement explicitly testable? Are there ways and methods in place to ensure every requirement is being fully met?
2. Testing. Not only is this needed to ensure requirements are met, but also to ensure the product always works as intended. I test things until they break, not to find weakness in the product, but to ensure strength and completeness both in my testing and in the product definition itself.
Any engineer who thinks they can work with their blinders on, seeking the minimum context needed to get the job done, focusing on the schedule more than thoroughness and quality, is not any kind of engineer in my book. They are droids, working mechanically and blindly, never seeking nor seeing the bigger picture. Droids, not engineers.
They are the folks who help get people killed. No excuses.
I have left employers who tried to keep me in the dark, sometimes with large negative financial impacts. I refuse to have non-engineers try to tell me what engineering is.
Unfortunately, this extends to engineering professors who have never engineered or delivered a safety-critical system. It extends to universities who don't include Requirements Analysis and Design of Experiments in the engineering curriculum. Most importantly, education never exposes students to unsafe situations to illustrate the extreme need for safety, and the correct ways to pursue it. At best, students may see a risky chemistry demo, but that's about it.
Where did I learn it, if not in college? I served in the US Navy before college (to get the GI Bill so I could afford to go), and I saw first-hand what it means to have one's daily job be around extremely hazardous systems, with the need to always practice safety in all its forms, to maintain those systems to maximize safety, even when the maintenance itself has risks. And to be trained and prepared when it all goes wrong, when people get hurt, when systems get destroyed, when the fires burn.
I have seen what happens when safety fails. There is no better education than the real world. Which, unfortunately, to most engineers today is almost a virtual world, an abstraction. It is hard to connect a code editing session to hundreds dead in a crash. Yet that line is present, direct and clear.
I have a silver ring that looks much like a wedding band, but is worn on a different finger. It is based on the Canadian Iron Ring ceremony (https://en.wikipedia.org/wiki/Iron_Ring). Please read the link. Wearing that ring is a visible reminder to both myself and others that failure to do always pursue excellence can cost innocent lives.
It sometimes means standing up to management when situations arise that I believe need
Problem is (Score:2)
Walking away from the project doesn't stop it going ahead. Someone eventually caves to managements request to do it cheaply.
If management doesn't allow the engineers to spend huge up-front safety costs then it doesn't happen. Management can only blame the workers if they can first prove they weren't doing shortcuts themselves.
This applies equally to production too. If a reported failing process is allowed to continue in production just to meet targets, it's not the workers at fault.
The exec's must be to
Re: (Score:2)
> However, it is fundamental to the very concept of being a "professional" engineer to take complete responsibility for one's own work product.
This is often not permitted. For security in a large environment, the tasks are often deliberately fragmented in order to expand the size of the group and limit responsibility, and authority, to increasingly small areas and avoid the risks of a single developer or engineer being a master of the entire system. Network firewalls are in distinct hands from applicati
Re: (Score:2)
I refuse to take assignments where I don't know and understand where and how my piece fits into the whole. I insist on being fully informed, and being given enough time and information to come to my own informed judgments about the project as a whole and the tasks at hand.
It is awesome that you take your responsibility seriously. It is rare to see that, especially with medical doctors... but we are discussing engineering.
The thing is, while we can demand that other engineers have your scruples and morals, the only way to find out that someone is "faking it" is for something to fail. Most things won't fail, so we have a surfeit of engineers who do NOT have the same scruples and morals as you.
How do you intend to solve that problem? It is true, you have accurately assigned bla
Re: (Score:1)
NO!!! The eng manager is already too late (Score:2)
Engineering managers probably have the most challenging role, since they both need to sell upwards and downwards within an organization in order to maintain safety standards. The pattern that I have gleaned from reading many reports on disasters over the years indicates that most safety breakdowns start right here. The eng manager starts to prioritize business concerns from their leadership over the safety of their own product. Resistance of these pecuniary impulses is not enough -- safety has to be the watchword for everyone...
The real problem is that safety is not an explicit business goal that sits right alongside all the other goals for the project at the top level, from day one. If we dare to think that safety is just something an engineering manager needs to remind his/her reports about often enough, the project is already set up to be inherently very risky.
What about false negatives? (Score:2)
"All had whistleblowers" doesn't mean much if we don't know how many safe-enough systems were also whistleblown upon. Can we reasonably review everything all the whistleblowers have whistleblown, without halting the whole economy?