RiskIt: NASA’s High Risk Commercial Cargo Strategy
A massive explosion occurred right after the Antares rocket hit the ground.
Commercial Cargo’s Lower Costs Brought Higher Risks
By Douglas Messier
Managing Editor
In October 2014, NASA engineers were deeply worried about Orbital Sciences Corporation’s upcoming Orb-3 commercial resupply mission to the International Space Station (ISS).
An Antares booster was set to send a Cygnus cargo ship loaded with 2,215 kg (4,883 lb) of supplies to astronauts aboard the orbiting laboratory. It would be the third of eight Cygnus flights to the station under a Commercial Resupply Services-1 (CRS-1) contract worth $1.9 billion.
The engineers were worried about the reliability of the launch vehicle’s first stage AJ26 engines. The 40+-year old motors — leftovers from the Soviet Union’s failed manned lunar program — had been refurbished by Aerojet Rocketdyne. And lately, they had been showing their age during ground tests.
“NASA engineering personnel expressed significant concerns about the Orb-3 launch vehicle’s engines and the recent failures Orbital had experienced on test stands, characterizing the likelihood of mission failure for Orb-3 as ’50/50,'” according to an audit issued on June 28 by NASA’s Office of Inspector General (OIG). “In contrast, the ISS Program’s risk matrix reflected the risk of Orb-3 engine issues as ‘low’ and assigned a subjective risk of ‘elevated but acceptable.’”
Officially, NASA placed the odds of a CRS-1 flight failing catastrophically at 1-in-6 — high by traditional agency standards, but much lower than the engineers involved in the program were giving. The 1-in-6 odds covered CRS-1 flights by Orbital and SpaceX, which had a separate resupply contract worth $1.6 billion.
Normally, a 50-50 chance of failure would be reason to stop a launch until the engineers’ concerns could be addressed. But, this was not a launch under direct NASA control; the space agency was buying a contracted service. It’s also not clear how far up the chain of command the engineers’ concerns reached within the space agency.
“Although according to some ISS Program officials NASA management is generally willing to accept heightened risk for cargo missions, it is unclear whether senior NASA management clearly understood the increased likelihood of failure for the Orb-3 mission,” the report stated. “Even so, the disparity between 50/50 and 1-in-6 for the same mission raises questions about the adequacy of communication between the engineers and top program management.”
When the launch went forward on Oct. 28, 2014, it took all of 15 seconds for the engineers’ worst fears to be realized. The Antares exploded in a spectacular nighttime fireball, destroying the Cygnus with $51 million worth of cargo aboard and causing $16.2 million worth of damage the the launch complex.
An aerial view of the Wallops Island launch facilities taken by the Wallops Incident Response Team Oct. 29 following the failed launch attempt of Orbital Science Corp.’s Antares rocket Oct. 28. (Credit: NASA/Terry Zaperach)
Under terms of the CRS-1 contract, NASA ended up paying 80 percent of Orbital’s $240 million launch fee despite the mission being a complete failure. Orbital was made whole by an insurance policy that covered the $48 million in milestone payments the company forfeited for failing to deliver the cargo.
It was a costly failure. Assuming the cargo could be replaced at the same price, the cost of the accident can be estimated at:
- Launch Cost: $192 million
- Value of Lost Cargo: $51 million
- Cost of Launch Complex Repairs: $16.2 million
- Total: $259.2 million
The bulk of the $259.2 million came out of NASA’s budget. The space agency had most of the cargo on board. NASA also ended up paying $5 million for repairs to the launch pad. Private companies and researchers also lost satellites set to be deployed from and experiments to be conducted aboard the station.
Under the CRS-1 contract, NASA could not recover the $192 million in milestone payments it made before Antares exploded. Nor could the agency recover any of the other costs it incurred.
“The CRS-1 contracts do not require SpaceX or Orbital to re-fly failed missions or carry upmass from a failed mission on future flights, nor do they make the companies liable for any cargo destroyed as a result of a launch failure or other anomaly,” the OIG audit stated.
This practice of privatizing profits while leaving taxpayers to pay for failures might seem unfair. However, it “is not unusual for Government contracts relating to space operations given the associated expense and risks, and the limited number of capable contractors,” the report added. “Due to the relationship between risk and price, shifting more risk to the contractor would likely increase contract price.”
Investigators determined that a turbopump of one of the first stage engines failed. Orbital maintained that the turbopump was defective; Aerojet Rocketdyne claimed that it was likely damaged by debris that entered it from another part of the rocket.
Aerojet Rocketdyne agreed to pay $50 million to Orbital ATK — as the company is now known — as part of an agreement to end its contract to supply rocket engines for Antares. Orbital decided to refit the booster with modern Russian RD-181 engines.
More than 20 months later, Antares remains grounded as that process continues. A critical return to flight is likely in August. The company will not conduct a flight test with the new engines. Instead, it will place a Cygnus aboard and hope all goes as planned.
In the meantime, Orbital ATK has partly fulfilled its supply contract by launching Cygnus cargo ships on United Launch Alliance’s (ULA) Atlas V boosters.
A String of Accidents
The Antares failure was just beginning of the space station’s supply problems. In April 2015, a Russian Progress freighter tumbled out of control in orbit after launch from the Baikonur Cosmodrome. The mission was declared a total loss.
Dragon capsule separated from Falcon 9 launch vehicle.
Two months after that accident, a SpaceX Falcon 9 rocket blew up after launch from Cape Canaveral, sending a Dragon resupply ship to the bottom of the Atlantic Ocean with cargo worth $118 million.
SpaceX forfeited 30 percent of its launch fee, which averages $133.3 million per flight. Parabolic Arc estimates the loss of the failed mission at:
- Launch Cost: $93.3 million (estimate)
- Value of Lost Cargo: $118 million
- Total: $211.3 million
Add all these figures together, and the two failures cost an estimated $470.5 million. That cost does not include any additional flights NASA will have to order to fly replacement cargo.
Falcon 9 was grounded for six months. SpaceX investigators believe that a single defective strut supplied by a contractor caused the destruction of their rocket. A separate NASA investigation found that other factors in addition to the faulty strut likely contributed to the loss.
[See NASA Investigation into SpaceX’s Falcon 9 Explosion Questions Single Strut Theory]
Three failures in eight months strained supplies aboard the space station. Progress’ return to flight in July 2015 and a Japanese H-II resupply mission the following month eased the supply situation in orbit. There have been no subsequent cargo mission failures since the Falcon 9 accident.
Faster Better Cheaper — Chose Two
An Orbital Sciences Corporation Antares rocket lifts off with the company’s first contracted cargo delivery flight to the space station for NASA. (Credit: NASA/Bill Ingalls)
There were more than just rockets and spaceships falling out of the sky when Antares and Falcon 9 exploded. The failures also brought a highly-touted NASA program back down to Earth, exposing the risks the agency had taken to make it a reality.
Following the loss of the space shuttle Columbia in 2004, the Bush Administration decided to stop flying the three remaining orbiters after they completed flights required to finish construction of the space station.
Officials also decided that NASA would no longer risk astronauts’ lives to deliver supplies to the ISS. Crew and cargo flights would be separated. NASA launched the Commercial Orbital Transportation Services (COTS) program, under which Orbital Science and SpaceX developed their cargo systems.
Today, COTS is a widely viewed as a tremendous success story. For an investment of about $800 million, NASA was able to fund the development of two new launch vehicles and cargo ships. Such thrift would not have been possible under traditional NASA contracting methods.
Private companies have built launch vehicles and spacecraft for NASA since the beginning of the Space Age. However, the space agency has typically been deeply involved in the design, construction and operation of the vehicles.
In the case of the space shuttle, Apollo and other human programs, NASA served as owner and operators of the systems. For planetary missions such as the Juno probe, the space agency maintains tight control over spacecraft production by private contractors and operates the missions. The agency purchases launch services from commercial providers such as ULA, Orbital ATK and SpaceX.
Atlas V liftoff (Credit: ULA)
Developing new launch vehicles and spacecraft with close government involvement and oversight is a costly proposition. The U.S. Air Force’s Evolved Expendable Launch Vehicle (EELV) program is a case in point.
Under the program, which was launched in 1994, the Air Force worked with Lockheed Martin and Boeing to develop two new launch vehicles, Atlas V and Delta IV. Although the government provided funding, the companies maintained ownership over the boosters and marketed their services commercially to the government and private customers.
At the time, large companies were planning large constellations of satellites that would bring communications to all corners of the globe. The assembly lines and the tooling for the boosters were sized for high-volume launch rates.
Frequent launches were important for two reasons: they would help keep prices competitive by spreading fixed costs over high booster outputs, and they would aid in ensuring reliability. Or, at least that was the plan.
The program was launched “under the assumption that mission assurance would be achieved through a high commercial launch rate,” according to the NASA OIG report. “However, in the late 1990’s there were several commercial and USAF launch failures and the commercial launch market collapsed, which caused USAF to transition from the original commercial-like approach to the increased role of a Government launch readiness verification and certification process.
“The creation of the launch verification matrix process – a process in which launch readiness verification activities are planned, executed, and recorded – is an example of this increased role,” the report stated. “Currently, the USAF has a comprehensive insight role into their contractors’ activities and the launch verification matrix is reviewed for efficient and effective mission assurance with each launch provider.”
Delta IV Heavy lifts off with the NROL-37 satellite. (Credit: ULA)
The comprehensive insight and oversight have given Air Force officials a great deal of confidence in the two boosters — which has borne out by their launch histories. Although Atlas V and Delta IV have suffered in-flight anomalies, they have flown a combined 94 missions without a catastrophic failure since their debuts 2002 and 2003, respectively.
This level of reliability is crucial to the Air Force and national security agencies. The satellites they operate are crucial to national defense. Some of the spacecraft cost a $1 billion or more. Making sure they are safely placed into orbit is the top priority.
This approach has a high cost, however. The FAA estimates the cost of an Altas V launch ranges from $110 to $230 million, depending upon the variant used. The price for a Delta IV launch ranges from $164 and $400 million.
A Commercial Approach
As NASA looked around for a way to supply the ISS with cargo after the space shuttle retired, it wanted some less expensive options. The agency also looked beyond ISS resupply toward a commercial future in low Earth orbit.
“One of the goals of the CRS-1 contract was to achieve reliable, cost effective access to low Earth orbit while creating a market environment in which commercial space transportation services are available to Government and private sector customers,” according to the OIG report.
To keep costs low, NASA took a much more hands-off approach oversight than it had used on previous programs. SpaceX and Orbital worked under Space Act Agreements, which were subject to far fewer government regulations than traditional contracting.
The companies were given much greater latitude in they developed their vehicles. For example, NASA was willing to accept the greater risks that went with using rocket engines that had been sitting in storage for 40 years. The loss of a cargo ship was far less serious than a crew vehicle.
A successful acceptance test of the AJ26 engine on Aug. 8, 2013. (Credit: NASA)
While NASA has implemented a comprehensive certification process for Commercial Crew vehicles that will carry astronauts to the space station, nothing so detailed was put in place for cargo flights.
Cygnus and Dragon vehicles were certified for operations in proximity to the space station only. Nothing else about the spacecraft or their boosters underwent certification before commercial resupply missions began. (Falcon 9 has been since certified to carry defense and other NASA spacecraft.)
Reassessing Risk
For resupply flights, NASA deviated from the normal criteria it uses to evaluate mission risks. According to the OIG’s report, the space agency generally uses the following processes:
“Risk Classification for Payloads. This process categorizes payload risk as class A (high) through class D (low) and provides a structured approach for defining a hierarchy of risk combinations for payloads by considering such factors as availability of alternative research or reflight opportunities, success criteria, and magnitude of investment.
“Launch Services Risk Mitigation. This certification process categorizes launch vehicle risk as 1 (high), 2 (medium), or 3 (low) in conjunction with the payload classification and sets parameters for using a particular launch vehicle, such as flight experience and testing, verification, and risk management activities.”
The space agency has “informally treated CRS-1 cargo as class D payloads” regardless of what a payload was, how much it cost, or how difficult it would be to replace. Under NASA risk management policies, “high-risk” launch vehicles can only carry class D payloads.
“This approach results in nebulous risk classifications not defined in NASA policy,” the OIG audit concluded. “In our view, using a more formal risk categorization approach for CRS-1 missions would better inform Agency management about the risk level of particular missions and allow for consideration of possible ways to mitigate associated risks such as requesting additional testing or, as suggested to us by a former program engineer in relation to the Orb-3 flight, that the company adjust the throttle to exert less force on the engines.”
International Docking Adapter. (Credit: NASA)
The failed Falcon 9 flight carried a crucial piece of equipment that was not exactly a class D payload. The International Docking Adapter (IDA) was the first of two new mating systems to be installed on the space station for future commercial crew missions.
NASA wanted both adapters installed on station by May 2017 when the first of four flight tests by SpaceX and Boeing are set to begin. Instead, only one adapter will be installed, raising the risk of a failed flight test if one of the spacecraft has difficulty docking with it.
A replacement IDA is being built, but it is unlikely to be attached to ISS until after the commercial crew flight tests are set to be completed in early 2018.
“Although the ISS Program has spares of each of the parts, several key items with the longest lead times to manufacture, including the metal shielding that wraps around the Adapter, need to be fabricated,” the OIG report stated. “Moreover, even if NASA is able to meet its planned schedule, the Station likely will have only one Adapter when the commercial crew demonstration missions are scheduled to arrive in May, August, and December 2017, and February 2018.”
Outsourcing Risk Assessment
SpaceX Dragon freighter at ISS. (Credit: NASA)
In addition to changing how it evaluated risk, NASA has also outsourced part of the process to the very commercial cargo contractors it employs. “The ISS Program heavily relies on SpaceX and Orbital to assess and mitigate risk for launches,” the OIG audit found.
The agency does have an “insight clause” in the contract that allows it to gain information about the risks associated with launches. NASA also conducts a technical assessment of the readiness and risk posture prior to a launch.
“However, ISS Program officials told us there is no integrated presentation or package that documents all risk areas for a given launch,” according to the OIG report. “Instead, separate presentations are used to determine the ‘acceptable’ risk posture – a term that evolves frequently. An acceptable risk may be based on such factors as the level of reserves and supplies aboard the ISS, the need to deliver or return research, or the timing of upcoming scheduled flights.”
Cygnus approaches ISS (Credit: NASA)
The OIG report said that while flexibility in determining risks has some benefits, it “may also introduce confusion into the process.” The report cites the differing assessments of the doomed Orb-3 flight where management believed the risk of failure to be 1-in-6 while engineers closer to the program put the odds at 50/50.
“In our judgment, the absence of a multi-disciplined approach to launch readiness, such as identifying and understanding all launch vehicle and payload issues and assigning a more objective launch rating to the mission to aid in communication of the risk, hampers successful risk mitigation efforts,” the OIG report read.
OIG Recommendations
The OIG felt NASA could benefit from examining the much more thorough approach the U.S. Air Force took for the development and operation of the Atlas V and Delta IV launchers.
“USAF officials told us that after a series of launch failures in the late 1990s, they applied a more disciplined approach to launch mission assurance,” the report stated. “Adjustments to the depth and priority of the required insight in specific areas happened only after the contractors had a proven track record of success….
“ISS Program officials and officials in NASA’s Office of Safety and Mission Assurance agreed that a more regimented approach to communicating risk would benefit the ISS Program,” the report added.
The audit made two specific recommendations to NASA concerning risk. The first was that the agency incorporate the risks associated with the IDA into its risk management processes.
NASA agreed with the recommendation, saying it has already taken to steps to mitigate the impact of losing the adapter. The second adapter is set for launch later this month aboard a Dragon spacecraft. The replacement docking unit is on schedule for delivery in March 2017, the agency said. Both IDAs will be on station in time for the first commercial crew mission to follow the flight tests.
The second recommendation was for NASA to “quantify overall mission risk ratings and communicate the risks for upcoming launches early and in coordination with varying levels of engineering and management.”
NASA management did not concur with this recommendation. The lengthy response boils down to NASA believing it already has sufficient processes in place in this area.
“The risk for individual cargo launches, both foreign and domestic, are managed by the program through well-established control board processes….The ISS program manages the overall risk posture by assessing the risk of the cargo vehicle/launcher and by managing the manifest of the cargo vehicle in the context of ISS resupply needs and traffic model,” the agency wrote.
Conclusions
NASA’s commercial cargo effort is broadly seen as a successful private-public partnership that has been a relative bargain for the space agency.
However, those programs have come at a cost of NASA accepting much greater risks and suffering the financial losses and disruptions that have resulted from two launch failures. The losses cut into safety margins for astronauts aboard the station, and have added risk to upcoming commercial crew flight tests.
So far, the failures have not resulted in any critical situations for the ISS program. NASA officials have judged the problems to be an acceptable price to pay for what commercial cargo has provided the agency.
49 responses to “RiskIt: NASA’s High Risk Commercial Cargo Strategy”
Leave a Reply
You must be logged in to post a comment.
Please give careful thought to your postings. I believe your intention is to do a quality investigative journalism, but your mistakes overshadow your good work and undermine your credibility.
You cannot count the cost of the lost cargo twice.
If I order a load of gravel for my driveway and the driver of the truck takes off with it so I have to buy a second load of gravel, only the cost of one load is attributable to the theft. The second load is attributable to the cost of the driveway and always would have been.
The incremental cost attributable to the theft of the load is only the cost of ONE load of gravel. You can’t count it twice!
You are correct. Story has been modified to reflect new numbers. Thanks for pointing out my error.
The bigger cost — which is really unknown — is what it costs NASA to refly the payloads lost. It’s a tricky question I’m going to deal with in a future post.
https://media.giphy.com/med…
Doug, these concerns should certainly be in the public domain. The question remains, however, will we ever allow _enough_ risk to develop ways of reducing cost substantially? Particular cases may be particularly risky (I was never thrilled about retreading the 40 year old engines) but absent active “X-plane” incremental testing by some research agency, what should we do?
There are many and sometimes subtle institutional and political realities that have constrained our options. And, E.g,. besides the X-plane issue, ideally we should have had MORE vehicles fly for COTS and the CCP.
In any case, if the conclusion is always that cargo (including people) are too precious to fly without the all-up traditional NASA techniques, we never will get out of the 50 year cost rut.
The question I always have is what are we learning here.
1. Don’t use engines that have been in storage for 40 years
2. Don’t launch rockets with 50/50 chance of success
3. Take the time and extra money required to do QA on all those struts you’re using.
4. Clear the test stand of workers before doing anything with nitrous oxide.
5. Don’t store anything flammable next to a pair of nitrous oxide tanks.
6. Don’t trust pilots not to make mistakes during high-pressure times in a test flight.
7. Don’t grant waivers on hazard analysis to companies seeking experimental permits to fly people in spaceships.
Don’t get me wrong. I see a lot of innovation out there. Companies taking risks and pushing the technology. SpaceX and Blue Origin. It’s all good stuff.
It’s just….and I’m not sure how to explain this exactly….the hands off approach tends to empower a lot of people with no idea what they’re doing. They end up relearning things people learned a long time ago. Perhaps that’s the cost of progress.
Then there’s this from the report:
“The CRS-1 contracts place much of the risk associated with an unsuccessful mission on NASA.”
It’s true. Private profit — the companies get paid despite failed missions — while the public picks up the cost of their failures. In a truly commercial arrangement, the formula would be reversed. We’re not there yet.
>>>”The CRS-1 contracts place much of the risk associated with an unsuccessful mission on NASA.”
It’s true. Private profit — the companies get paid despite failed missions — while the public picks up the cost of their failures. In a truly commercial arrangement, the formula would be reversed. We’re not there yet.<<<<
To me, this is the crucial point. Even at a higher price as you indicated in the article, the cost of failures should rest on the launch company. Whether they incur the cost in-house or have to spend more on insurance, it gives the company more incentive to succeed without the vastly increased cost of NASA detailed oversight. If the total cost of these failures had rested on SpaceX and Orbital, Much of this discussion would be moot.
When I do a construction job and have problems, it’s my problem and I don’t get paid until I get it right. The customers second guessing is not needed or welcome.
“When I do a construction job and have problems, it’s my problem and I don’t get paid until I get it right.”
You can’t really compare a construction job to the CRS program.
Any particular reason why?
This is a public-private partnership where NASA and the companies share both the costs and the risks of development and delivery. Now, how they decide to apportion those costs and risks is open to negotiation. And I believe that they have tweeked those agreements for future missions. That works for me.
Fair enough if lessons go forward. In my case, delivery before payment. I tend to think of it as a healthy model.
Yeah, that seems to be NASA’s end game. But they need a healthy commercial industry in place before they can pick up that phone and place that order. I guess that we’re all looking forward to the day when we’ve got Bigelow and other habitats up there waiting for your delivery. But you may have to trade in your truck for something a little more sporty. https://uploads.disquscdn.c…
Wimpy thin, no power, so I REWIRED it.
You don’t get progress payments? Most construction contracts I’ve ever seen are a hell of a lot more complex than payment-upon-turnkey. Often the supply costs are passed on, likewise some subcontractor costs, all in advance. Progress and milestone payments, usually on a fixed schedule. With huge amounts of variation, especially when the client wants something that’s unusual or unique.
We never take payment in advance. You’d be amazed how much conflict is avoided and how much leverage that gives me to tell them to get out of the way. It’s when the customer gets hooks into you when they can yank the line. Progress payments on phase completion is the same thing. Change orders are verbal with most of our clients, written with new or shaky ones.
The cost plus type jobs you are referring to we only take under certain circumstances such as customers with known integrity, and when work is so tight that we are willing to put up with the crap rather than sit home. Fixed price is easier and more profitable for a company that performs.
You have no leverage if they declare bankruptcy and you have footed the entire project cost upfront. Get paid for materials as they are delivered to the job site..
Paul had it. No more than 90 days and it is concrete. In 30 years I’ve has two customers go broke and both of them gave me equipment to offset the loss. Part of me getting my price and staying busy is a low hassle factor. Contractors like being able to pay at the end when they get their draws without a lot of detail in the middle. Also, when you itemized your bid, you are open to nitpicking that cost you time and annoyance even if you stick to your guns. I’m not doing bids for the competition to look at to see if they missed something.
I always had the materials paid for on arrival to the job site. Labor part of the bill was broken into parts.. based on amount of job completed.. I never did turn key projects and foot the entire project cost myself upfront..
My Dad’s company (as a subcontractor) would cover the cost of materials, payment only after completion, but the primary cost (concrete) occurred in the last few days before completion, so was always covering in the current or next billing month. Formwork supplies and other consumables were covered by his business as ongoing costs (although some types of formwork were supplied by the builder). However the builder did pass on incremental costs to the job-owner on a specific schedule. And that was for fairly routine jobs, such as cookie-cutter housing subdivisions. Large projects were much more complex.
Likewise, with manufacturers I’ve done the books for, clients ordering new product lines pay for development, not just per-unit costs once full manufacturing begins. With dies costing upwards of a $million each, it doesn’t take long for costs to destroy cashflow for most businesses.
I can only assume Redneck’s business is small and primarily doing short term jobs (no more than 90 days between costs and payment.)
(BTW, to Redneck, periodic payments doesn’t have to mean “cost-plus”. It can still be fixed price for each milestone.)
Correct me if I’m wrong here, but aren’t SpaceX/OATK/and-soon-to-be-SNC contracted to lift a certain amount of mass to the ISS. How many launches they use is negotiable. So getting paid a proportion of the contract value even when a launch fails is not really “getting paid for failures”, since the shortfall in mass lift will have to be made up on later launches – as OATK are doing. The cost of the payload is perhaps more the point, and I don’t know if insurance pays for this or if the launch company will have to recoup NASA via future discounts, as has reportedly been the case with SpaceX. Regardless, the space launch industry is clearly an infant by any useful measure, and simplistic statements like “the public picks up the cost of their failures” is neither helpful nor fully accurate. Encouraging the expansion of new areas of social and economic endeavour is exactly in the purview of governmental responsibility, since we, “the public”, are ultimately the beneficiaries.
The issue of upmass is really interesting. It’s something I should have discussed here. I am working on a separate post looking at this area. As with everything else associated with commercial cargo, it’s complicated.
Here’s part of your answer relating to upmass and the company’s responsibility to fulfill their contracts:
http://www.parabolicarc.com…
I won’t spoil it for you, but with SpaceX, the answer is probably not what you thought.
It goes to the cost of obtaining each innovation. When you look back at the auto industry and the amount of loss of life and property that was considered acceptable, and is still acceptable today, 30,000 auto deaths loss of billions in productivity, I do not see any difference.
Space flight accidents may cost more, but are a lot more rare than an auto accident. Each “hands off” accident leads to more knowledge on how to mitigate that accident. Not that I am promoting accidents… I just do not worry about them, they ARE going to happen.. there will never be any accident free systems as long as humans are involved… do we run from the risk or just keep correcting
well it may be better to understand the “Why” of these failures more then the What… What is easy to figure out, why is harder.
Why is overconfidence and a belief that there is nothing to learn from the past or from others.
And its not new. John Kittinger discusses in his book “Come up and get me”, how the USAF warned NASA of the dangers of a 100% oxygen atmosphere based on their experience in aerospace medical research. NASA ignored them and learned the lesson the hard way in the Apollo 1 accident.
So this isn’t just a space commerce issue, its instead the classic example of hubris. You see it now with the ISS and the belief that it will last until the ISS partners decide it will be de-orbited.
Doug, it’s interesting that we are talking about rockets, but so many of these discussions come back to a discussion of psychology! How do you make someone learn? What are you trying to teach them?
In aviation and automotive sectors, you can pull people with decades of experience from many different companies that have worked on low volume luxury products or low cost mass market products with different design and production philosophies.
In launch we don’t have a “deep bench” to draw from. You can pick people from the NASA/AF way from NASA, the AF or ULA. That’s about it. There’s just one way and the products are the same.
I’ve led projects in aerospace where we intentionally turn suppliers into design agents: it’s not easy. You have to allow more budget and time for them to learn things you already know. You try to do it on projects that have lesser consequences and are training-wheel type projects.
Doug, certainly keep doing what you’re doing, but the question comes back to: how do we teach these companies these lessons? I don’t know a better why than what NASA is doing: give the learners more latitude, be patient and give them low-ish impact projects where a screw up can be salvaged.
Yes, the old saying applies – “Good decisions are based on experience. Experience is based on bad decisions.”
Does a list of changes made to fix faults exist?
Young engineers can read it to learn what to avoid. Experienced engineers as points to support their use of alternative designs in meetings and quality inspector to demonstrate why they are suspicious about a design.
If such a list does not exist it is easy to start one.
Things like this do exist, but they are fragmented and kept by each individual company/agency. The design guides are considered either highly proprietary or are even controlled by ITAR. There’s a great story that Joe Sutter (designer of the 747) in his book, tells about how at the end of the Cold War a Soviet delegation went to Washington to negotiate for a copy of the Boeing airliner design guide. Suffice it to say, Boeing considers it to be their version of the “Coca Cola recipe”.
This industry has been around for 60ish years, but the first decent textbook on manned spacecraft design was only released LAST YEAR! Compare that with a book like Roark’s Formulas for Stress and Strain (in print since 1938) or Grey’s Anatomy (since 1858). If you are in a design review and someone asks how you did a structural analysis, you can say “I used Roark’s” and the conversation is over! It is a proven and accepted resource, that’s not true for the spaceflight industry – where knowledge is guarded to protect nuclear secrets.
Even with investigative journalists like Doug, the full story doesn’t come out until after the company goes belly up, or the knowledgable folks retire and tell the inside story. Look at a book like Space Systems Failures – it was written almost entirely on Aviation Week articles and contains no analysis or deeper understanding.
Commercial cargo has been far cheaper than if NASA had done this themselves (using typical cost plus contracts to develop their own vehicle to fly on ULA’s more costly EELVs). Would this have been more reliable? Most likely, but we can’t be certain. Even ULA has had issues with some of its “successful” flights. And the cargo ship might not have been successful every time either.
At any rate, NASA’s own cost models have shown that it would also have been far more expensive to do it this way. A future article on this topic should note just how much more expensive it would have been.
My point is, even factoring in the two failures so far, NASA is still coming out ahead in terms of overall cost. And even better, there are currently two competing commercial providers. Competition is a good thing.
Fair enough.
What astounds me is they went ahead with the Orb-3 launch when the engineers who understood the real risks involved put the odds of success at 50/50. Yes, it’s just cargo. But, this was a quarter billion failure that has put Antares out of commission for nearly 2 years and strained supply reserves on the station. And it affected not just NASA but the private companies and researchers with cargo on board.
Now they’re going to put a Cygnus on board what is essentially a new launch vehicle without a previous flight test.
“Although according to some ISS Program officials NASA management is
generally willing to accept heightened risk for cargo missions, it is unclear whether senior NASA management clearly understood the increased likelihood of failure for the Orb-3 mission. Even so, the disparity between 50/50 and 1-in-6 for the same mission raises questions about the adequacy of communication between the engineers and top program management.”
Well, yeah. WTH was going on there? Inadequate seems to be an inadequate way to describe that disconnect.
Did NASA really have a choice? This is the program that they’re stuck with, now that the shuttles are history. The engines were put through all the standard pre-flight tests. I don’t see how postponing that particular flight for some extra head-scratching would have changed the outcome. (I always enjoy your investigative reporting, by the way.)
Other engines were also put through standard pre-flight tests and were failing regularly. Hence, the 50/50 estimate from engineers close to the program.
Yeah. I’m just wondering if you have an opinion on what NASA could or should have done at that point. Did the engineers express opinions on a workable solution?
Good questions. I don’t know what the engineers did. It’s a question the OIG audit didn’t fully answer. The whole who knew what when question relating to upper management is unclear. What upper management would have or should have done is uncertain.
As to options. We know that Orbital had a limited supply of the AJ26 engines sufficient to fulfill the original CRS-1 contract as well as modest additional orders. They were evaluating replacement engines already when Orb-3 crashed.
Would it have made sense for NASA to suspend the contract until those options were ready? Or asked Orbital to launch Cygnus on an Atlas V? What were the terms of the contract? Or was it just better to risk the cargo and hope for the best?
These aren’t easy questions to answer in retrospect. The OIG report is unclear whether the concerns reached a high enough level to be debated.
That’s exactly what I was thinking. Thanks for the thoughtful reply.
The Columbia disaster had a far bigger effect on the ISS than both the commercial cargo failures and the shuttle was a far safer rocket. It not only stopped Construction but forced a crew of two due to lack of supplies.
In the case of the Antares it would take 2 years to redesign the rocket failure or not.They were also able to use ULA to launch 2 flights and so Cygnus was only out for a year or so. Orbital knew it’s choice was risk the flight while they try to fix it or suffer big costs and delays. If successful they would have avoided the costs and if unsuccessful the costs remain about the same(NASA cargo for the most part is low value–with some exceptions). NASA faced a similar problem 25 years ago with Challenger but the difference this time is that lives were not at risk. If the station had run out of supplies we could simple evacuate it in an organized manner.
The station’s reserves were strained but that is what reserves are for. Space X was able to send two cargo flights while Cygnus was down before suffering a failure. Cygnus was able to ride Atlas a few months after Space X went down and Space X returned to flight this year. Orbital has an option to buy a third flight of Atlas if Antares fails.
Risk is something best managed and can not be advoided.
> Now they’re going to put a Cygnus on board what is essentially a new launch vehicle without a previous flight test.
Cygnus is now a standard component and can be replaced by simply ordering an extra one. So this becomes a cost trade off:
(Price of Cygnus / probability of failure) Vs. (price of a dummy payload and second flight)
The other trade is the payload being carried by the Cygnus. This appears to be supplies which are a good choice. Food, water, oxygen and clothes are cheap and can be replaced quickly.
Whereas say a science experiment that took 5 years to develop will take a significant time (years?) to replace.
NASA may wish to have a veto on the cargo chosen for this experimental flight.
Sounds exactly like NASA’s own failures to listen to their engineers for the Columbia loss. Which repeated the exact mistake they’d made leading up to the Challenger loss. (To treat previous survived anomalies as the new baseline after a simplistic analysis (usually under pressure from senior management to rush through approval.) And dangerously assuming that previous testing (decades earlier) on subsystems is valid for flights that are well outside the criteria of the tests.)
So let’s say the argument is correct and SpaceX/Orbital are terrible at understanding risks and NASA has erred tremendously in giving them too much say… Who do you suppose should take over that risk-approval process? NASA?
All good points. Antares is a bit of an odd duck, given the prior experience of then Orbital Sciences Corporation with launch vehicles based on solid stages. Before Antares, I do not believe that Orbital ATK had any experience with developing and flying large liquid fueled stages, let alone launch vehicles.
The article has both the cost and reliability risks involved here almost backwards. It’s important to step back a bit.
NASA and DoD have already lost multi-billion dollar payloads on multi-billion dollar systems. I would think that if you looked for a co-relation of costs (as effort) to reliability in launch you’d find some weak relationship at best, or more likely none at all. Add to this the incalculable loss of crewed missions.
Down to numbers, the government has reduced it’s cost risk, even with the Antares and Falcon mishaps. Consider the alternative as a cost-plus set of spacecraft (think Orion-like) given to a couple of companies like Lockheed and Boeing, with launchers like Atlas and Delta. These alternate life cycle costs would be greater even with all the losses, by an ORDER OF MAGNITUDE.
The reliability that launch systems might achieve is also ultimately low so long as the government flies on what are basically government only paradigms. The move to favor companies that are seeking to grow business, non-government business, will ultimately create flight rates that will favor reliability in real and measurable ways.
Cost and reliability risks to the government are measurably LOWER with the COTS/CRS commercial approach, even with these 2 losses, vs. any alternative. The article is seeming to miss this broader reality.
These are excellent points. However, there are some caveats.
The CRS-1 contract with Orbital relied upon a limited supply of 40-year old Soviet lunar program engines. That turned out to be an affordable idea in the short term (cheap engines) but a bad one for long-term reliability.
Orbital was in the process of evaluating replacement engines at the time of the accident. The crash speeded up that process. The choices included reopening the assembly line for the engine they were using as well as other choices. They eventually went with the Russian RD-181 engine.
The consequence of that is that next month Orbital will be risking another Cygnus mission on what is essentially a brand new launch vehicle with no flight history. They’re starting back at zero, at least when it comes to the all important first stage that gets you off the planet. The upper stage and the Cygnus remain the same.
The other consequence of this is a limited launch market. After all the ruckus over the use of Russian RD-180 engines in the Atlas V, the military will not certify a booster using a Russian agency to carry vital defense payloads. So, the Antares is limited to whatever it gets from NASA for cargo and satellites and any commercial business it can win.
What you are saying about high launch rates and reliability applies much more to SpaceX. As mentioned in the article, the U.S. Air Force originally took this approach with the EELV development until the market for comsats collapsed and the government had suffered several launch failures in the late 1990’s.
Here, however, there are caveats. SpaceX never launches the same rocket twice. It’s made continual upgrades and changes to the boosters since the beginning. Not modest upgrades to systems that have flown many times. But, major ones.
This can be seen in the CRS-1 program. The first two commercial flights to the station were well under capacity because the Dragons were heavier than expected and booster was not as powerful as planned. The engines got upgraded. Then after they started landing first stages, there was another major upgrade to make up for some of the lift capacity they lost.
The concern I’ve heard is that with a constantly changing booster, it’s difficult to get the multiple flights with a set configuration you need to develop reliability and gain insights into how the systems are performing. Every major upgrade to the booster, and you have to start counting over again.
Then there’s the investigation into the Falcon 9 accident. The SpaceX one found a pretty simple cause and solution: faulty strut broke, brought down the rocket, replace the strut. The NASA investigation found a whole bunch of factors in addition to the strut that could have contributed to the crash. That led to a letter from Charlie Bolden and Bill Gerstenmaier expressing concerns about SpaceX’s QA practices.
The other thing is beginning next year (if the schedule doesn’t slip further) SpaceX will start launching crew Dragons. NASA wants and needs a lot of insights into how those boosters are operating. They will want to know that the version of the booster they fly crew on has a certain track record in that configuration. (Allowing for changes to accommodate crew Dragon, of course.) The accident may have been a positive thing in that NASA got SpaceX to address some issues relating to QA processes.
So, yes, the hands off let the commercial sector launch a lot and we’ll get great reliability is not bad in theory. But, it’s all in how things are implemented. Orbital’s approach was conservative; it assembled a rocket from parts adapted from other uses (engines — lunar program, first stage Zenit, etc.) SpaceX has been almost hyper-innovative in its constant upgrades to the Falcon 9.
You don’t see the contradiction here? If SpaceX didn’t have a program of continuous upgrades, the next couple of years of Falcon/Dragon flights would have been underpowered and overweight. How is that an improvement?
I particularly disagree with the second paragraph:
IMO, it’s the complete opposite. Incremental upgrades are the only way to learn your craft. The lack of such incremental development is precisely why so much of space exploration has stalled so much.
Operations teaches you about the current design. Development teaches you how to design.
The 20+ years of Shuttle operations did not teach NASA about the capabilities and limits of the Shuttle system. Quite the contrary, the longer they went with a single system, the more knowledge they lost. By the end, they were terrified of their own vehicle. Like your grandmother using a computer, they did the same thing over and over that someone else wrote down for them, with no understanding of why, and when something unexpected popped up they just clicked “ok” and hoped for the best.
Re the Shuttle – “Quite the contrary, the longer they went with a single system, the more knowledge they lost. By the end, they were terrified of their own vehicle.”
You get it!
The NASA objections to SpaceX’s procedures weren’t particularly QA-centric. QA was in there, to be sure, but the IG report refers to systems engineering and management practices as well as assembly and repair practices. One can take this in one of two general ways.
1) SpaceX is a parvenu company that has no idea what it’s doing and needs a firm hand from “older and wiser heads” who’ve been doing this stuff forever and know all there is to know about it. NASA’s opinions are taken to be canonically correct. You seem to tilt in this direction.
2) NASA has time-honored ways of doing things, but the last working large launch system it designed was done in the pre-PC era, had at least two very dangerous faults that somehow escaped detection in all the sacred Design Reviews and caused the loss of two complete vehicles and 14 lives. No one currently on NASA’s payroll has ever been involved with a successful launch vehicle program from the ground up. There are people at NASA who don’t like SpaceX because they think NASA should always be in charge of anything space-related. Some of these people took the opportunity of an IG investigation to throw all their accumulated shade on SpaceX in the hope something would stick. They don’t really have any idea what caused the loss of CRS-7 and they don’t really care very much. They just availed themselves of an opportunity to get all their complaints of SpaceX’s heretical divergence from “How We Do Things Around Here” into the written record.
I, needless to say, hew toward explanation 2.
As others have pointed out, lack of progress is not equivalent to quality. The Japanese are generally considered to build the highest quality mass-market cars in the world. They didn’t always. They got to where they are now by adopting a process of continuous improvement as a basic strategy. They did this in the late 1960’s, about the time the Apollo program was reaching its high-water mark. NASA’s and the legacy contractors’ fetish for doing everything exactly they way it was done in the sainted 60’s has had the U.S. space industry stuck in neutral for 40 years. SpaceX, Blue Origin and other organizations who’ve come up since then have other ideas, even if some of them are only comparatively new, like continuous process improvement.
Agreed, the SpaceX experience to date takes the reliability and cost advantage further than Orbital, and Orbital’s approach has negated much of the intent of the commercial paradigm (using existing systems not owned, with no path to significant improvements in reliability, not having a price low enough to attract non-government business and grow volume, etc.)
Nonetheless, I’d say the changes from one Falcon 9 design to another are actually consistent with rapid learning and the overall direction of learning faster, with a potential to increase reliability to a greater degree over time. More launch and space systems should be doing this, not less. The Shuttle program had huge upgrades budgets, multi-billion dollar budgets, but was so slow to change and make improvements that ultimately most of the billions just scratched the surface on improving reliability. (NASA’s own documents show they were relating upgrades to significant reliability growth; an overly optimistic view in retrospect. Most upgrades were really all about performance, an indirect result of these huge inefficiencies.)
One day, a six-sigma day for aerospace, we’ll need to slow down the change, much like the mature aircraft / airline / airport industry. That will be natural then (diminishing returns and all that).
Now is not the time for launch and space systems to think 20 flights and then a tweak. That was Shuttle. Atlas/Delta are near there. That tweaking at great expense happens when a system in so un-affordable in the first place, in just a steady state, that making change for learning is near impossible. Stability then is not a feature, but a flaw.
Doug, excellent article.. Really fantastic research. Some of the best i’ve seen…
I would point out that the original 1980’s plan was principally based upon using STS as the logistics supply stream. With both losses of an Orbiter causing a 30 month down time, that was a serious if underweighted risk. Soyuz and Progress have both had problems with near misses on the vehicles and a station collision. ATV is no longer available… What has reduced program risk has been a diversity of logistics supply. STS was backed up by Soyuz/Progress. Soyuz/Progress was backed up by ATV. ATV has been Backed up by CRS/Antares/Falcon… The CRS program has not been without bumps but it’s worked in interesting ways. The Cygnus/Atlas reflight showed there was more capacity out there then envisioned.
Similar to a thought I had. Did these unnamed engineers make their 50/50 prediction just once, before Orb-3, or did they make the same prediction before every Orbital CRS test or delivery mission? Without knowing things like that, we have no way to distinguish between habitual kvetching and prescience.
While I hated to see atk lose a Cygnus, I have to say that the top pix is pretty cool.
Just saw this article referenced over on NASASpaceFlight, and I think it the article is a great summary of things.
I am a fan of commercializing the routine things NASA does so that NASA can focus on those things that are not yet routine.
Delivering cargo to the ISS is a routine task, in that all the elements of getting cargo from Earth to the ISS have been done and the technology and techniques for doing it repetitively are understood.
However, as the article documents, just because something is understood doesn’t mean there still isn’t a degree of risk around it. Commercial air travel is very routine, but we all understand that airlines still crash every year, so perfection is just a goal, not mandatory.
But if handled correctly, the losses early on can lead to much more robust future systems, and my hope is that both Orbital ATK and SpaceX will have far more reliable transportation systems now. Not necessarily perfect, just more reliable.