NASA Investigation into SpaceX’s Falcon 9 Explosion Questions Single Strut Theory
Dragon capsule separated from Falcon 9 launch vehicle.
While SpaceX blames a faulty strut supplied by a contractor for the explosion of a Falcon 9 rocket in June 2015, an independent investigation by NASA Launch Services Program (LSP) concluded there were several “credible causes” for the accident, including poor quality control at Elon Musk’s launch company.
“In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure,” according to a report published on Tuesday by the NASA Office of Inspector General (OIG). “LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.”
The information is contained in a new OIG audit, “NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station.” The report says LSP failed to find a probable cause for a failure that sent a Dragon supply ship carrying cargo for the International Space Station to the bottom of the Atlantic Ocean.
SpaceX’s investigation concluded the “most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle….The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, ‘out of specification’ materials, and improper heat treatment.”
LSP’s findings alarmed officials at NASA, which awarded commercial cargo and crew contracts to SpaceX to service the International Space Station.
“In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure,” the report reads.
The OIG reports that SpaceX has taken a number of corrective actions to address concerns about the strut and its processes.
“The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards,” the report reads. “In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware.
“To administer its updated quality control process, SpaceX has reorganized into three teams called ‘Design Reliability,’ ‘Build Reliability,’ and ‘Flight Reliability.’ Besides monitoring corrective actions taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel mixture and corrective actions on software implementation plans, which are both rated as low risks by the ISS Program,” according to the report.
The relevant excerpt from the report follows.
NASA Office of the Inspector General
Excerpt from
NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station
SpaceX’s Return to Flight Plan
Following the SPX-7 failure, SpaceX recovered parts of the Falcon 9 rocket and, through telemetry analysis and other testing, determined the most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle. SpaceX completed an extensive analysis of the SPX-7 failure, consulted with NASA and the United States Air Force (USAF) regarding their analysis, and provided a mishap report and Return to Flight Plan to the FAA and NASA in November 2015. The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, “out of specification” materials, and improper heat treatment.23
NASA’s Launch Services Program (LSP) conducted a separate, independent review of the failure, briefing its results to senior NASA leadership on December 18, 2015.24 LSP did not identify a single probable cause for the launch failure, instead listing several “credible causes.” In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.25
SpaceX has taken action to correct the deficiencies that led to the failed strut assembly and to address NASA’s concerns by conducting inspections, replacing suspect parts, and conducting additional testing. The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards. In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware.
To administer its updated quality control process, SpaceX has reorganized into three teams called “Design Reliability,” “Build Reliability,” and “Flight Reliability.” Besides monitoring corrective actions taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel mixture and corrective actions on software implementation plans, which are both rated as low risks by the ISS Program.
___________________
23 A casting defect is an irregularity that occurs when molten metal is poured into a mold and cooled. An “out of specification” material has a technical attribute (e.g., chemical composition, mechanical property) outside of the prescribed values for the type of metal specified for a particular use. Heat treatment at accurate temperatures strengthens metal parts while improper heat treatment can cause deviations or weaknesses.
24 LSP purchases commercial launch services for NASA customers, including missions of the Agency’s Science Mission Directorate. LSP had a contract with SpaceX to use the Falcon 9 to deliver a science mission payload.
25 In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure.
72 responses to “NASA Investigation into SpaceX’s Falcon 9 Explosion Questions Single Strut Theory”
Leave a Reply
You must be logged in to post a comment.
They didn’t discount the strut, they simply stated there could be other causes, which is true in any complex failure analysis. As for standing on items, sure – but I wonder how much real stress a 75kg guy puts on a part supposedly good for 2 orders of magnitude more than that.
Something clearly caused the helium tank to break free and shoot up into the dome of the fuel tank. SpaceX’s answer is defective strut from contractor. Case closed.
NASA doesn’t think that alone would have done it and is concerned about deficiencies in SpaceX’s processes. Hence the high-level letter from the administrator and associate administrator. I’d been hearing the same for months from sources that they thought it was a more complicated failure.
In any event, I’m hoping these reports will be released. There was a report released after the Antares accident.
> NASA doesn’t think that alone would have done it
No, NASA does not say anything like that.
What NASA says is that *it’s possible* that the reason was different. NASA is not sure.
To be fair, spacex was not sure either. They just felt that it the most likely cause and was backed by finding that other struts had been built wrong ( outsourced American company had outsourced to China who cut all sorts of corner ).
Thanks for the last sentence, I had never understood how this could have come about. Presumably outsourcing by the contracted company had not been addressed in the contract. Bad procurement failure by someone!
American companies regularly cheat. About a decade ago, a company in San Diego who was supposed to be making all sorts of titanium parts for the DOD had quietly shipped to CHina who was purposely sending back bad parts. Once the DOD figured out, W’s admin fined the company a few bucks and then went right back to them.
We really need to address how to deal with these companies that cheat like this.
Thankfully, Musk is doing the right thing.
I am HOPING that he will soon create a defense unit, whose purpose is to land contracts for doing parts work for the defense industry. IOW, they can lower their costs by doing honest work for other projects.
I’d like to see the complete report too. From the summary information provided thus far it’s impossible to tell whether any of NASA’s objections are substantial or based on differences in culture. NASA is used to dealing with legacy contractors. SpaceX doesn’t operate like a legacy contractor. This seems to give a lot of old NASA hands the heebie-jeebies. It would be nice to have enough information to rationally judge whether the objections made are substantive or are simply based on “That’s not the way we do things.”
It’s a good point. But, if you read the full report, you will find that NASA gave a lot of leeway to SpaceX and Orbital ATK in terms of the risks the agency was willing to take on cargo flights. Forty year old Soviet era engines? Hey, why not?
Losing cargo is one thing. The stakes are about to go a lot higher next year with commercial crew. Everybody’s knows that. So, it’s could be a very good thing that NASA is pushing SpaceX to up its game.
And here you are uncritically assuming SpaceX’s game needs to be “upped”.
Considering NASA/SLS is using solids on a crewed system again, I’m sure it’s not SpaceX’s game that needs upping…
…And I see no evidence the incentives which produced the criminally corrupt phrase, “Take off your engineering hat and put on your management hat.”, are incentives which have changed.
Oh good lord We’ve now changed the subject to the Challenger explosion.
Oh good Lord.
Now we’re ignoring the feet of clay on NASA, and every point raised critical of the report or the reporting.
“NASA doesn’t think that alone would have done it “
Where is your skepticism? Some committee at NASA is willing to claim in print it could have been something else, and they provide no evidence you have reported it was anything else.
NASA. Said. Exactly. That. Tom. OK?
“LSP did not identify a single probable cause for the launch failure, instead listing several “credible causes.” In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.”
This is merely a summary of two reports that haven’t been released. I would love to see the full reports.
THIS. GROUP. AT. NASA. HAS. NO. BASIS. YOU. HAVE. REPORTED. FOR. THE. CLAIM. IT. MAKES.
It’s no “public acknowledgement” as much as a publicly made baseless insinuation.
You’ve concluded that without seeing the report? Really?
Don’t you think you should maybe wait to see the results before declaring that LSP’s conclusions are wrong and have no basis?
Well you’re the guy pretending to be a journalist who made a post about nothing.
You haven’t seen the report, what do you pretend you have to report that’s worth reading then?
As it stands, the notion the SpaceX’s conclusions are insufficient is a baseless one to make.
Even if it’s NASA.
I never wrote that SpaceX’s conclusions were insufficient. I wrote the LSP’s conclusions differed from SpaceX’s conclusions. That’s based on a credible source (NASA OIG) summarizing the results of the two investigations that have not been publicly released.
I did not make a post out of nothing. I wrote a post based on the OIG’s findings.
And what you failed to point out is that NASA has not in any way justified it’s insinuations.
It doesn’t quite rise to the point of burying the lede, but yeah it’s close.
That’s BS.
Which?
That they did justify their insinuations and you didn’t report that?
Or you did bury the lede?
If proper reporting had been done on the ALAR “scandal” it would’ve fizzled promptly.
What interests are served by not reporting that NASA is making claims it doesn’t back up? A lie goes around the world before the truth puts it’s boots on–it doesn’t need journalists tying it shoestrings (or failing to point out they are tied).
Releasing unsupported claims is bullsh!t. Reporting them uncritically is bullsh!t.
Where’s the beef Doug?
I have no idea. I do know that SpaceX fanboys will say and do anything to defend the company against bias and unfairness, real or completely imagined.
Somehow a story about SpaceX losing a rocket and payload, the subsequent investigation, and NASA’s serious concerns about SpaceX’s QA processes turned into a discussion about alleged media bias and my reporting.
That would be because you reported accusations without reporting any evidence for them.
Hence the “reporting baseless accusations” part.
Can’t report on documents that NASA hasn’t released and that I don’t have.
“about alleged media bias”
The unintentional humor is the best part.
No humor, intentional or unintentional, could be found in this discussion.
So, here we have a start-up that dares to step outside to the status quo. I am surprised that it has not had more failures to date. All post accident flights of the F9 have been trouble free and I assume SpaceX upgraded the struts in question. I say good for SpaceX because they quickly identified the problem and corrected it in a timely fashion, which is something that the government in any respect knows is impossible.
The strut gave way well below its design limit – so it’s not an upgrade issue, it’s a quality control issue. Basically, SpaceX is blaming the supplier for a deficient part. Given that this particular component error has been fixed, and no doubt they have also improved their assembly processes as well, and there have since been a bunch of trouble-free flights, does appear to exonerate SpaceX. Given that SpaceX are far more sensitive to a repeat failure than anyone else, the insinuation that they may be a bit slap dash in their processes seems a pointless conclusion for this report to make. This report smells more like NASA covering its ass, since they are responsible for spending tax payer’s money. Though it’s ironic that they felt the need to spend several million dollars to produce a report that says SpaceX may or may not have fixed the problem that has not subsequently reoccurred in the upgraded full-thrust version of the F9.
Actually, I think this report was useful.
Basically, it is trying to point out all possible sources of errors which spacex may/may not have since corrected.
“possible sources of error”…”may/may not”
So possible sources of error as opposed to definite sources of error, that may have definitely been fixed or and may definitely not have been fixed. You’re right, quite useful.
so, you think that looking at a couple more issues is not worth solving?
Hmmm.
I dunno. NASA contracted to SpaceX to deliver cargo to the station. SpaceX’s rocket blew up and sent said cargo to the bottom of the Atlantic.
Why do you assume that NASA is covering its ass here? What mistake did NASA make that it would have to cover its ass for? Hiring SpaceX in the first place?
A possible “mistake” according to public opinion: hiring an “imperfect” company to do stuff instead of doing it in house or contracting a more traditional and “mature” contractor like ULA.
I really don’t read that report (or summary of reports) as “SpaceX’s conclusion is wrong or questionable”, it even reinforces the probably failure scenario.
There is a fundamental difference however: SpaceX seems to have been focused on finding and fixing the technical cause of the problem. The NASA panel looked at the incident from a more broad perspective and points to possible (not proven) problems regarding SpaceX’s procedures. AKA “how could this happen in the first place?”
Namely:
Selection of suppliers, inspection/review/control of suppliers,
quality control of parts, handling of parts, etc.
As you pointed out, those are the relevant points where the full report needs to be released to determine if those are just unspecific concerns (someone pointed out that differences in cultures could be at the source of those concerns) or if some specific problems/shortcomings have been identified that need to be addressed.
This wasn’t a NASA report. It’s the final report from the OIG of an audit of NASA’s handling of the investigations into both the Orbital and the SpaceX launch failures, it’s packed full of interesting details on the commercial cargo program, and is well worth reading for that alone. It’s an analysis of what NASA did well and what NASA did not do so well, and it has recommendations for what NASA could do to improve investigations in the future.
Hold on. Spacex did not say that strut was the cause. They said that we can not find anything else with as much issue as these struts, so it is likely the case.
SpaceX is not a start-up any longer. They are a lead contractor for several NASA programs. The report is highlighting the fact that they have several QA and process issues that need to be addressed. You say they quickly identified the problem, but NASA says they might have only identified one possible cause, and that there may be several other issues that need to be addressed.
Yeah, that’s exactly right. A succinct summary of what’s important here. Lose a cargo ship, OK, you can fix that. Lose a crew, much more serious. The game is about to move to a much higher level.
I would like to know how much of this is simply second guessing people that are doing the work. Somewhat akin to me bringing up a spelling error in one of Dougs’ posts when I couldn’t have written the post to begin with, much less the thousands more that he did in the last year.
That’s a rather simple view of accident investigation processes. Seriously.
Having dealt with people that indeed operate that way, I think it is s legitimate question. It doesn’t imply SpaceX is perfect or blameless,just that some people and organizations are something less than unbiased.
“how much of this is simply second guessing people that are doing the work”
As Mr. Messier has reported it, it is all second guessing, done on the basis of nothing.
No evidence at all is cited it is not exactly as SpaceX described.
This is merely a briefly summary of two reports nobody outside of SpaceX and NASA has seen. All the more reason to release the results of the investigations.
Accident investigations are meticulous involve a great deal more than guessing. Theyr’e not based on “nothing.”
In any event, the much more interesting elements involve NASA’s concerns about SpaceX’s QA processes. It would be great if NASA releases the February 2016 letter to SpaceX.
I did not mean to imply based on nothing. I meant that there are always things visible later on, especially to nit pickers. Wrench dropped on foot and “was he wearing his hard hat” type stuff.
My wife owns a Tesla. One of the issues that you find watching this company is that all of the recalls and nearly all issues involves third party parts that have to be replaced. IOW, the parts coming in have QA issues. Oddly, all of these either came from China or Germany( surprising ). It shows a real lack of QA in manufacturing.
And it appears that spacex has same issue.
IOW , outsourcing means cutting corners in order to lower prices.
There’s always an outside supplier who will do a better job at higher cost. If third party parts are being replaced due to low quality then it’s down to SpaceX or Tesla for making that vendor selection in the first place.
Normally, it is just cheaper and higher quality to do it yourself, unless it is very few items.
And struts are needed EVERY WHERE.
Understand the gut reaction to this but no it is not always the case. A specialist in that particular field is usually better and cheaper than investing the resources in to an area that is not your specialty, and cos it isn’t your thing usually you don’t do as good a job.
There are always exceptions but here it looks like either bad supplier selection (as I said a bad procurement failure) or the contracted firm outsourced (possibly against the contract terms) to an inadequate supplier and passed off non specification parts. Which means either SpaceX procurement get fired or some one gets their arses sued!
As for the strut, it bothers me that the safety of the entire flight is dependent on this single, seemingly trivial component (The helium tank has two, but if one lets go, the other one gets twisted off immediately afterwards).
Methane Raptor rockets won’t need helium tanks, nor will Blue Origins Methane BE-4.
Simpler, cheaper, safer, better
If had been the old way SpaceX would still be grounded.
??? Why not. The Helium is for the LOX tank. Yes?
It’s stored in the LOX tank to keep it cold, allowing the helium tank to hold more helium. It’s warmed as its drawn off and used to pressurize other rocket components.
No idea if the LOX tank itself was autogenously pressurized or used the helium. Seems like there might be issues maintaining pressure in a tank full of subcooled LOX and gaseous O2, and the same would go for subcooled CH4.
The strut was over-designed by a factor of 3. It should never have failed.
You’re right, but I’m wondering if there was a better way to mount the tank that had more redundancy.
Figuring that out would require detailed engineering schematics for the tank wall attach points, the helium tank, and the struts, and we definitely aren’t going to get any of that. My guess is the way they did it is the lightest way possible to attach the helium tank to the tank wall, and for safety they overdesigned the strut. Multiplying struts means multiplying weight and there are, according to Elon Musk, “hundreds” of these struts on the rocket already.
This is the crowd that let Shuttle fly with how many possible LOM modes.
My understanding is that SpaceX followed a rigorous root cause failure identification process. NASA was involved all along the way. If they didn’t like the conclusion, SpaceX wouldn’t be flying NASA missions now. This report is simply a rehash of what has already been discussed and decided. IOW a waste of time and resources.
Cheers
One thing that the nasa naysayers are right about, is that NASA has far too many groups. Yeah, a NASA group DID help SpaceX, but it was not this one.
And I am guessing that this group DOES have an axe to grind.
But, I do not recall hearing that ppl were walking/jumping on tanks, etc.
You’ve concluded that the Launch Services Program has an axe to grind against SpaceX based on a brief summary of two investigations that haven’t been publicly released? That’s a helluva conclusion.
I think that you missed my point.
I am saying that OIG has an axe to grind with new private space. They have always kept quiet on the shuttle, ULA, Boeing, Ball, Atlas V, Delta IV, etc. yet, scream murder about new space, esp. SpaceX. TO be fair, it is probably helpful to spacex, and foolish not to be that harsh on old space as well.
Not sure which group it is that is directly helping SpaceX, ATK, etc with COTS/CCDev/etc. but that group has given invaluable help to them. It is for that reason that I maintain that NASA is so very valuable, even though it is soo large and soo top heavy, that groups are fighting each other.
I don’t think OIG has a axe to grind. The full report both praised and criticized NASA. Some of the recommendations NASA has greed to implement. Others it disagreed with. That’s not unusual.
This is the first public acknowledgment indicating that a poorly manufactured strut may not have been the only cause of the accident.
You can’t rehash something that’s never been hashed before. That’s the whole point of the “re-” prefix. Sheesh.
The writers at NASA have presumably rehashed this after SpaceX hashed it the first time.
Curious they have no evidence for you to report to go along with their speculation.
Unless it’s only intended to be baseless speculation.
It’s the final report from the OIG of an audit of NASA’s handling of the investigations into both the Orbital and the SpaceX launch failures, it’s packed full of interesting details on the commercial cargo program, and is well worth reading for that alone. Far from a rehash, it’s an analysis of what NASA did well and what NASA did not do so well. Not a waste of time and resources, as it has recommendations for what NASA could do to improve investigations in the future.
If that’s what the report says, why is Mr. Messier obsessing with and only writing about the unsupported claims SpaceX did not do a thorough enough examination, and that it changed practices found to be unsatisfactory?
Cause he doesn’t talk about what you’re talking about.
What he says isn’t really supported by the report. I’d recommend reading it in full.
There’s a link to a PDF of the report here:
http://forum.nasaspacefligh…
Tom’s comments here have become abusive. He’s banned.
I don’t like banning people. I generally try not to do it. But, he’s made this about me and the comments just get nastier.
I mean, I can’t write in detail about reports neither NASA nor SpaceX has released.
It is difficult to report on the details of unreleased documents that I do not have copies of. Why don’t you take all the energy you’ve directed toward this argument to trying to convince NASA to release them? Yelling at me isn’t going to do that.
*edit* Statement retracted, as it seems clear that this comment was not intended to be directed at me.
So a part of NASA is at the best accusing SpaceX of making an insufficiently factually grounded, conclusory determination that an improperly made strut caused the loss of the flight. That’s a damaging accusation. Whose interests does that serve?
Are those interests served by floating the accusation even if the actual reports are as big a pile of nothing burger as this first uncritical take on the blurb that the full reports will be released?
No, not at all. I strongly recommend you read the entire report. It does not have an accusatory tone in any way.
It basically says that improper manufacturing may not be the only reason the strut broke. It does not second-guess that the strut broke, simply says that other factors may have been at play, such as installation practices and QC procedures. It also does note that spaceX has taken steps to improve in those areas, as well.
When something goes wrong in someting as demanding as rocketry, there’s usually more than one factor that comes together to cause a failure.
Footnotes are really interesting. You find things like this:
The SpaceX accident board included 11 SpaceX employees and a lone FAA representative.
“In comparison, the seven-member contractor board that investigated the Orb-3 failure included four Orbital employees, two NASA employees, and one third-party expert.”
Then there’s this:
“Only the 11 SpaceX board members signed the final accident investigation report.”
While I am not a SpaceX fanboy, I am certainly strongly opposed to bureaucratic and political games. While I would welcome some sort of safe way to double check the company’s findings, I can not think of a way that would not devolve into finger pointing and chest beating and committee forming. I am thus forced to believe that a company that has just started a multi billion business would be the most motivated to find the true reason for a technical failure and a solution on how to avoid it.
Really, the first question that comes to my mind is; Why are accidents of space-going transportation craft not subject to a National Transportation and Safety Board investigation, the same as every other machine involved in transport? Privatization of space travel is no different than privatization of any other travel industry, the rest of which are investigated by a board not answerable to anyone. You can’t even use NTSB reports in lawsuits, that’s how separate they keep themselves. Their reports are excruciatingly thorough, as well as publicly available, right down to the video of the final hearing these days. NTSB, after all, is the group that makes recommendations to the FAA, not the other way around. Compare what’s been released on this accident so far to the NTSB’s report on the SpaceShipTwo accident. Night and day as far as completeness and speed go.
BTW Siderite, I liked your comment as a whole. But unfortunately history tends to indicate the business itself is the least motivated to implement a solution, even when it bothers to find the root cause. Significant numbers have instead reasoned that paying damages or fines is cheaper than re-engineering. Heaven forbid anything should cut into profits.