Scrutinizing SpaceX, NASA overlooked some Boeing software problems
By Kenneth Chang
After a couple of rounds of investigation and some soul-searching, NASA and Boeing believe that they have identified what went wrong during a troubled test flight of an uncrewed Boeing spacecraft designed to carry NASA astronauts. In addition to the software errors that slipped through undetected and were not fixed before the spacecraft was launched in December, NASA officials admitted that decades of working with Boeing gave them a level of trust. As a result, the agency might not have been paying attention to the company as closely as it should have while it was also placing more scrutiny on SpaceX, which also built a capsule for carrying people to the space station.
“We were, I would say, a little more used to the Boeing process,” Steve Stich, manager of NASA’s commercial crew program, said during a news conference Tuesday. “It’s one that we have used in the past on successful NASA programs like space shuttle and the International Space Station.” Instead of building and operating its own spacecraft to take astronauts to space as it has in the past, NASA has hired two companies — Boeing and SpaceX, the aerospace newcomer started by Elon Musk — to provide transportation to and from the space station. “We may have been focused a little more on SpaceX because they use a bit of a nontraditional approach to their software development,” Stich said.
“And so we may have had a few more people looking at that.” Last month, SpaceX successfully launched two NASA astronauts, Robert Behnken and Douglas Hurley, to the space station in its Crew Dragon capsule. The two astronauts are in orbit working on experiments and maintenance of the space station and will attempt a return trip in the SpaceX vessel later this year. If Boeing’s December test flight of its spacecraft, called Starliner, had gone as planned, a demonstration flight with two astronauts aboard similar to Behnken and Hurley’s voyage would most likely have occurred by now. Instead, Boeing will now repeat the uncrewed test, perhaps late this year, with a crewed flight delayed until next year.
In addition to an investigation of what went wrong technically, NASA officials also declared what is known as “high visibility close call” to examine whether there were any blind spots within NASA that led to its overlooking Boeing’s issues. Kathy Lueders, who was recently appointed associate administrator of the agency’s human exploration and operations directorate, said that NASA, as part of efforts to reduce costly bureaucratic overhead, had not asked Boeing or SpaceX for a high-level management plan for how complex pieces of engineering would be put together and tested. But that lack of knowledge meant that NASA did not fully understand how Boeing was designing the Starliner’s software and the testing process for verifying that it would work as intended. “We thought we understood it, but we ended up finding out that over time, that kind of changed,”
Lueders said. She also said that for the December test flight, NASA had focused on the highest priorities, in particular ensuring that Starliner did not pose any danger to the space station as it approached, and that might have caused software engineering to receive less attention. “Where do you apply the resources to make sure that you’re getting kind of the biggest bang for your buck, to be able to really flesh out where you have problems in your systems?” she said. NASA is now overseeing software development more closely at both SpaceX and Boeing. The Starliner spacecraft, launched on top of an Atlas 5 rocket Dec. 20, encountered two major software problems during its flight.
The first occurred minutes after the spacecraft had separated from the rocket, because the capsule’s clock had been set wrong. That caused the spacecraft to squander its propellant, and a planned docking at the International Space Station was called off. Starliner also experienced a communications problem that prevented mission controllers from quickly regaining control.
An investigation revealed that the spacecraft’s radio receiver had been listening to too wide a swath of frequencies, which led to interference from other transmissions from Earth. Boeing engineers have added a filter to limit the frequencies. The second software flaw would have fired the wrong thrusters as Starliner was preparing for reentry.
As Boeing engineers hastily combed through the Starliner software in the aftermath of the clock problem, they found that problem and fixed it. If it had not been fixed, two pieces of Starliner — the capsule that returns to Earth and the service module, which is discarded — might have collided. The capsule might have tumbled and burned up in the atmosphere instead of landing safely in White Sands, New Mexico. A review panel made 80 recommendations that Boeing is working to implement before the next uncrewed test flight.