Uncritical adherence to processes and procedures allows diffusion of responsibility,
where many contribute to the problem, but no one is responsible.
“It was all about loyalty” a Boeing engineer said.
“I had a manager tell me, ‘Don’t rock the boat. You don’t want to be upsetting executives.’ ”
These attitudes are deadly.
Early in the development of the 737 MAX, engineers gathered at Boeing’s transonic wind tunnel in Seattle to test the jet’s aerodynamics using a scale model with a wingspan comparable to that of an eagle.
The testing in 2012, with air flow approaching the speed of sound, allowed engineers to analyze how the airplane’s aerodynamics would handle a range of extreme maneuvers. When the data came back, according to an engineer involved in the testing, it was clear there was an issue to address.
Engineers observed a tendency for the plane’s nose to pitch upward during a specific extreme maneuver. After other efforts to fix the problem failed, the solution they arrived at was a piece of software — the Maneuvering Characteristics Augmentation System (MCAS) — that would move a powerful control surface at the tail to push the airplane’s nose down.
This is the story, including previously unreported details, of how Boeing developed MCAS, which played a critical role in two airliners nose-diving out of the sky, killing 346 people in Ethiopia and off the coast of Indonesia.
Extensive interviews with people involved with the program, and a review of proprietary documents, show how Boeing originally designed MCAS as a simple solution with a narrow scope, then altered it late in the plane’s development to expand its power and purpose. Still, a safety-analysis led by Boeing concluded there would be little risk in the event of an MCAS failure — in part because of an FAA-approved assumption that pilots would respond to an unexpected activation in a mere three seconds.
The revised design allowed MCAS to trigger on the inputs of a single sensor, instead of two factors considered in the original plan.
Boeing engineers considered that lack of redundancy acceptable, according to proprietary information reviewed by The Seattle Times, because they calculated the probability of a “hazardous” MCAS malfunction to be virtually inconceivable.
As Boeing and the FAA advanced the 737 MAX toward production, they limited the scrutiny and testing of the MCAS design. Then they agreed not to inform pilots about MCAS in manuals, even though Boeing’s safety analysis expected pilots to be the primary backstop in the event the system went haywire.
In the wake of the two crashes, despite an outcry from the public and from some pilot and airline industry officials, Boeing has defended the processes behind its MCAS design decisions and refused to accept blame.
The grounding of the MAX has entered its 15th week. Safety officials around the world are scrutinizing the changes to MCAS that Boeing has proposed to ensure such accidents won’t happen again. And they are assessing what training pilots may need on the new system.
“Safety is our top priority,” Boeing said in a statement. “Through the work we are doing now in partnership with our customers and regulators to certify and implement the software update, the 737 MAX will be one of the safest airplanes ever to fly.”
This investigation examines what’s known about the origins and operation of MCAS ahead of the final official accident-investigation reports, expected late this year for Lion Air Flight 610 and next year for Ethiopian Airlines Flight 302.
Wind-tunnel and simulator tests
Though Boeing was locked into a plan to revamp its popular 737 model, the Seattle wind-tunnel tests in 2012 revealed a problem.
During flight tests to certify an airplane, pilots must safely fly an extreme maneuver, a banked spiral called a wind-up turn that brings the plane through a stall. While passengers would likely never experience the maneuver on a normal commercial flight, it could occur if pilots for some reason needed to execute a steep banking turn.
Engineers determined that on the MAX, the force the pilots feel in the control column as they execute this maneuver would not smoothly and continuously increase. Pilots who pull back forcefully on the column — sometimes called the stick — might suddenly feel a slackening of resistance. An FAA rule requires that the plane handle with smoothly changing stick forces.
The lack of smooth feel was caused by the jet’s tendency to pitch up, influenced by shock waves that form over the wing at high speeds and the extra lift surface provided by the pods around the MAX’s engines, which are bigger and farther forward on the wing than on previous 737s.
This was verified in early simulator modeling, with planes tested in scenarios at about 20,000 feet of altitude, according to one of the workers involved.
While the problem was narrow in scope, it proved difficult to cope with. The engineers first tried tweaking the plane’s aerodynamic shape, according to two workers familiar with the testing. They placed vortex generators — small metal vanes on the wings — to help modify the flow of air, trying them in different locations, in different quantities and at different angles. They also explored altering the shape of the wing.
Two people familiar with the discussions said 737 MAX chief test pilot Ray Craig preferred such a physical solution to solve the plane’s aerodynamics. Philosophically, Boeing had long opposed efforts to create automated actions such as a stick-pusher — a device used on some aircraft that without pilot action pushes the control column forward to lower the jet’s nose — that would seize control of a situation from the pilot, according to one of the people.
But the aerodynamic solutions didn’t produce enough effect, the two people said, and so the engineers turned to MCAS.
It was simple in concept but powerful in effect, quickly solving the issue.
In the midst of a wind-up turn, the software would automatically swivel up the leading edge of the plane’s entire horizontal tail, known as the horizontal stabilizer, so that the air flow would push the tail up and correspondingly push the nose down.
As the pilot pulled on the control column, this uncommanded movement in the background would counter the jet’s tendency to pitch up and smooth out the feel of the column throughout the maneuver.
An engineer recalled Craig testing MCAS for the first time in the simulator.
“Yeah! This is great,” Craig gushed after seeing how MCAS responded, according to the engineer. (Craig left Boeing before the operation of MCAS was revised.)
This original version of MCAS, according to two people familiar with the details, was activated only if two distinct sensors indicated such an extreme maneuver: a high angle of attack and a high G-force.
Angle of attack is the angle between the wing and the oncoming air flow. G-force is the plane’s acceleration in the vertical direction.
How much MCAS moved the tail when activated was a function of the angle of attack and the jet’s speed, said one of the people familiar with the MCAS design who, like many of the sources in this story, asked for anonymity because of the sensitivity of ongoing investigations.
The fix didn’t stir much controversy.
Another Boeing plane, the KC-46 Air Force tanker, has a software-driven system that similarly moves the stabilizer in a wind-up turn and even has the same MCAS name, though the design is very different.
Boeing’s failure analysis
When Boeing was ready to certify the 737 MAX, it laid out its plan for MCAS in documents for the FAA.
Under the proposal, MCAS would trigger in narrow circumstances. It was designed “to address potentially unacceptable nose-up pitching moment at high angles of attack at high airspeeds,” Boeing told the FAA in a proprietary System Safety Assessment reviewed by The Times.
In a separate presentation made for foreign safety regulators that was reviewed by The Times, Boeing described MCAS as providing “a nose down command to oppose the pitch up. Command is limited to 0.6 degrees from trimmed position.”
Two people involved in the initial design plans for MCAS said the goal was to limit the system’s effect, giving it as little authority as possible. That 0.6-degree limit was embedded in the company’s system safety review for the FAA.
The Boeing submission also included an analysis that calculated the effect of possible MCAS failures, with each scenario characterized as either a minor, a major or a hazardous failure — increasingly severe categories that determine how much redundancy must be built in to prevent the event.
Virtually all equipment on any commercial airplane, including the various sensors, is reliable enough to meet the “major failure” requirement, which is that the probability of a failure must be less than 1 in 100,000.
A “major failure” is not expected to produce any serious injuries and is defined more as something that would increase the cockpit crew’s workload. Such systems are therefore typically allowed to rely on a single input sensor.
Boeing analyzed what would happen if, in normal flight mode, MCAS triggered inadvertently up to its maximum authority and moved the horizontal stabilizer the maximum 0.6 degrees.
It also calculated what would happen on a normal flight if somehow the system kept running for three seconds at its standard rate of 0.27 degrees per second, producing 0.81 degrees of movement, thus exceeding the supposed maximum authority.
Why three seconds? That’s the period of time that FAA guidance says it should take a pilot to recognize what’s happening and begin to counter it.
Boeing assessed both of these failure modes as “major.” Finally, the analysis looked at the inadvertent operation of MCAS during a wind-up turn, which was assessed as “hazardous,” defined in a cold actuarial analysis as an event causing serious or fatal injuries to a small number of people, but short of losing the plane (that’s called “catastrophic”).
Hazardous events typically demand more than one sensor — except when they are outside normal flight conditions and unlikely to be encountered, such as a wind-up turn.
According to a document reviewed by The Seattle Times, Boeing’s safety analysis calculated this hazardous MCAS failure to be almost inconceivable: Given the improbability of an airliner experiencing a wind-up turn, compounded by the unlikelihood of MCAS failing while it happened, Boeing came up with a probability for this failure of about once every 223 trillion hours of flight. In its first year in service, the MAX fleet logged 118,000 flight hours.
So even though this original version of MCAS required two factors — angle of attack and G-force — to activate, Boeing’s analysis indicated that just one sensor would be acceptable in all circumstances.
In flight test, MCAS changes
About a third of the way through flight testing in 2016, as first reported by The Seattle Times in March, Boeing made substantial changes to MCAS.
The flight-test pilots had found another problem: The same lack of smooth stick forces was also occurring in certain low-speed flight conditions. To cover that issue too, engineers decided to expand the scope and power of MCAS.
Because at low speed a control surface must be deflected more to have the same effect, engineers increased the power of the system at low speed from 0.6 degrees of stabilizer nose-down deflection to 2.5 degrees each time it was activated.
On the stabilizer, maximum nose down is about 4.7 degrees away from level flight. So with the new increased authority to move the stabilizer, just a couple of iterations of the system could push it to that maximum.
Because there are no excessive G-forces at low speed, the engineers removed the G-force factor as a trigger. But that meant MCAS was now activated by a single angle-of-attack sensor.
One of the people familiar with MCAS’s evolution said the system designers didn’t see any need to add an additional sensor or redundancy because the hazard assessment had determined that an MCAS failure in normal flight would only qualify in the “major” category for which the single sensor is the norm.
“It wasn’t like it was there to cover some safety or certification requirement,” the person said. “The trigger isn’t a safeguard. It tells (the system) when to operate.”
While the changes were dramatic, Boeing did not submit documentation of the revised system safety assessment to the FAA.
An FAA spokesman said the safety agency did not require a new system safety analysis because it wasn’t deemed to be critical.
“The change to MCAS didn’t trigger an additional safety assessment because it did not affect the most critical phase of flight, considered to be higher cruise speeds,” he said.
The person familiar with the details of MCAS’ evolution said Boeing did the extra analysis of the new low-speed, higher-authority changes. He said the effect of the potential failures at low speed was less, and so didn’t add any risk to the prior analysis. So the documents sent to the FAA with the failure analysis were not revised.
“You turn in the answer,” he said. “You don’t have to document all your work.”
MCAS as it was actually implemented differed in another way from what was described in the safety analysis turned in to the FAA.
The failure analysis didn’t appear to consider the possibility that MCAS could trigger repeatedly, as it did on both accident flights. Moving multiple times in 0.6 or 2.5 increments depending on the speed, it effectively had unlimited authority if pilots did not intervene.
Discussions around this new MCAS design appear to have been limited during flight testing.
Two former Boeing test pilots described a culture of pressure inside the company to limit flight testing, which can delay projects at a time when orders are stacking up, costing the company money.
Matt Menza, a different pilot who did test flights on the MAX, recalled times when test pilots at Boeing would have the chance to thoroughly examine systems in what he called a “system-safety murder board” to explore all the potential failures. But he reported that the general corps of test pilots didn’t have a lot of technical details about the MCAS design, such as the single-sensor input.
Boeing never flight-tested a scenario in which a broken angle-of-attack sensor triggered MCAS on its own, instead relying on simulator analysis, according to a person familiar with the process. One of the former test pilots expressed bewilderment that the angle-of-attack failure was never explored in the air.
A variety of employees have described internal pressures to advance the MAX to completion, as Boeing hurried to catch up with the hot-selling A320 from rival Airbus.
Mark Rabin, an engineer who did flight-testing work unrelated to the flight controls, said there was always talk about how delays of even one day can cost substantial amounts. Meanwhile, staff were expected to stay in line, Rabin said.
“It was all about loyalty,” Rabin said. “I had a manager tell me, ‘Don’t rock the boat. You don’t want to be upsetting executives.’”
Do pilots need more training?
Boeing’s system safety analysis of MCAS, in working out the failure probabilities, assumes that the pilots will take steps in response to anything that arises, and will do so quickly.
The pilots’ struggles to control their planes before both MAX crashes suggest that the FAA’s three-second guidance for expected pilot response time, upon which part of Boeing’s system safety analysis was based, needs to be carefully reassessed.
“If the three seconds is not an appropriate amount of time to be able to catch a runaway stabilizer, and it actually takes seven seconds, then … we need to understand that,” said the person familiar with the details of MCAS.
When MCAS is activated in the cockpit and moves the horizontal stabilizer, a large wheel beside each pilot that’s mechanically connected to the stabilizer begins to spin. This is the manual trim wheel. As a last resort to stop a stabilizer moving uncommanded, a pilot can grab and hold the wheel.
The person familiar with MCAS said the wheel will spin noisily and fast, 30 or 40 times, for each activation. Meanwhile the stabilizer movement will increase the force needed to hold the control column, by about 40 to 50 pounds for a 2.5 degree movement. Such uncommanded movement that won’t stop is referred to as a “runaway stabilizer.”
Boeing has said that to deal with this, pilots need first to have basic hand-flying skills — pull the nose up to where you want it, then use the thumb switches on the yoke that connect electrically to the stabilizer to neutralize the forces — and then shut off MCAS with a pilot checklist procedure on how to handle a “runaway stabilizer.”
However on both accident flights, the angle-of-attack sensor failure set off multiple alerts causing distraction and confusion from the moment of takeoff, even before MCAS kicked in.
On the Ethiopian Airlines flight, for example, a “stick shaker” noisily vibrated the pilot’s control column throughout the flight, warning the plane was in danger of a stall, which it wasn’t; a computerized voice repeating a loud “Don’t sink!” warned that the jet was too close to the ground; a “clacker” making a very loud clicking sound signaled the jet was going too fast; and multiple warning lights told the crew that the speed, altitude and other readings on their instruments were unreliable.
Exactly what pilot training for MCAS is appropriate has become a big issue that threatens to prolong the grounding of the MAX.
While the FAA and U.S. airlines seem ready to clear the plane to fly with just iPad training for American pilots on the MCAS fixes, some foreign regulators want more intensive simulator training for all pilots on how to handle a runaway stabilizer.
Early in the process of selling the MAX, according to two people familiar with the discussions, Boeing promised to give Southwest Airlines a substantial rebate for every plane if the MAX required simulator training.
One former MAX worker, Rick Ludtke, said the rebate reported to him by managers was $1 million per plane, a figure another Boeing employee indicated is roughly accurate.
A Southwest spokesperson said, “We do not discuss publicly the specific details of our contractual agreements,” but added that “the purchase of an aircraft is a significant investment, and guarantees for various items … are incorporated into every 737 contract.”
Ludtke and two other former workers described internal pressures during the MAX certification to avoid any changes to the design of the plane that might cause the FAA to lean toward a simulator mandate.
It became a significant point of attention for Michael Teal, the 737 MAX program manager, and Keith Leverkuhn, vice president and general manager of the 737 MAX program, according to a person involved in the discussions. They felt confident based on past experience that the MAX would be approved without simulator training, but they were wary, according to the worker.
Meanwhile, Boeing’s chief technical pilot on the MAX, Mark Forkner, was also facing pressure, according to another person involved in the project. The person recalled Forkner as frequently anxious about the deadlines and pressures faced in the program, going to some of his peers in the piloting world for help.
As first reported by The New York Times, Mark Forkner suggested to the FAA that MCAS not be included in the pilot manual, according to a person familiar with the discussions.
“Mark never dreamed anything like this could happen,” said Forkner’s attorney, David Gerger. “He put safety first – at this job and in the Air Force.”
U.S. pilot unions have expressed concern at the omission of MCAS from the manual. One reason is that when MCAS activates, it changes somewhat the response of the airplane.
For example, there is a cutout switch in the control column so that when a pilot pulls or pushes in the opposite direction to a runaway stabilizer, it cuts electric power to the stabilizer. When MCAS is active, this cutout switch doesn’t work, which could surprise a pilot who didn’t know about the system.
Boeing ultimately won the FAA’s approval to give pilots just an hour of training through an iPad about the differences between the MAX and the previous 737 generation. MCAS was not mentioned.
The FAA, after internal deliberations, also agreed to keep MCAS out of the manual, reasoning that MCAS was a software code that operates in the background as part of the flight-control system, according to an official familiar with the discussions.
A single sensor
Boeing has avoided accepting direct blame in public, saying MCAS was only one link in a chain of events. Its leaders have also said MCAS was designed according to the standard procedures it has used for years.
“The 737 MAX was certified in accordance with the identical FAA requirements and processes that have governed certification of previous new airplanes and derivatives. The FAA considered the final configuration and operating parameters of MCAS during MAX certification, and concluded that it met all certification and regulatory requirements,” Boeing said in a statement.
The most controversial detail of the MCAS design has been the reliance on a single angle-of-attack sensor. On both of the deadly flights, everything started with a faulty sensor. In the second crash in Ethiopia, the data trace strongly suggests that the sensor was destroyed in an instant, likely by a bird strike.
There are two such sensors, one on either side of the fuselage. Why didn’t Boeing, especially after discarding the G-force as a trigger, use both angle-of-attack sensors?
The thinking was that requiring input from two angle-of-attack sensors would mean that if either one failed the system would not function.
That has implications not only for safety but for airline costs. If the system is down, a pilot might fly into a situation where it’s needed and find it unavailable. Or the airline might have to take the plane out of service and lose money.
Both factors point toward a principle of not adding complexity: Keep a system as simple as possible.
“You don’t want to disrupt your customer’s operations,” said the person familiar with the MCAS details. And you don’t want to “increase the risk that the system fails when you need it.”
In this case, as simple as possible meant as minimal as the safety regulations allow. Since Boeing’s system safety analysis concluded that one sensor was acceptable, that’s what it went with.
But that’s not the logic followed for a system on the KC-46 Air Force tanker, also called MCAS.
Boeing says the MCAS systems on the MAX and on the tanker share only a name and a similar function, and have completely different avionics.
But they both move the horizontal stabilizer to smooth the pilot stick forces in a wind-up turn. Their basic design architecture can be compared to some extent.
Air Force spokeswoman Ann Stefanek says “MCAS on the KC-46 has two sensors and the system compares the two readings.”
Boeing’s proposed update to MCAS for the MAX will have the same.
Last Sunday at the Paris Air Show, Boeing CEO Dennis Muilenburg reiterated the company’s position that while the original MCAS was properly designed, “we know we can improve it.”
The fixes include relying on two sensors rather than one, limiting MCAS to one rather than multiple activations, and revising the software.
“We are confident that they will result in a safe airplane, one of the safest airplanes ever to fly, and that MCAS will not contribute to a future accident,” he said.
You must be logged in to post a comment.