Sudden Management Change at IV&V

Keith's 23 Dec note: IV&V had an unexpected visit from NASA's Chief of Safety and Mission Assurance Officer Bryan O'Connor today. He relieved Butch Caffall immediately of his duties as Facility Director of NASA IV&V and re-assigned him to NASA HQ "to work some technical issues for NASA starting early in January" according to an internal memo. Greg Blaney is acting IV&V Program Manager and IV&V Facility Director. IV&V employees had been expecting something to happen but this action was more abrupt and sudden than had been expected.

According to a NASA IV&V engineer: "The immediate re-assignment of the NASA IV&V director represents the end of modeling of NASA projects as part of our IV&V analysis. We have endured four years of spending IV&V funds on, what has been from the onset, an R&D effort to create a new method of doing IV&V using an independently built system reference model (UML based) of a space vehicle design based on Project artifacts. Its been costly in dollars and in performance. This R&D has taken countless man-hours away from our directive to find issues/problems with S/C FSW development."

"Critics of NASA IV&V would argue that we have never been effective or productive to justify the $35M annual budget. The argument I make is that the intent of IV&V is not as a lead effort to debug and assure mission success but rather as a final step in assuring mission success. We receive documents not in draft but primarily first and later revisions after peer review, V&V and/or I&T have analyzed artifacts. This is the ideal circumstance. We do receive, in fact, drafts and documents lacking project V&V and under such circumstances we do find more issues.

But the bottom line is that we cannot be expected to find numerous high severity issues or many issues overall. On this basis, the NASA IV&V funding level is well spent. NASA IV&V has delivered high severity issues to many projects that have saved development time and/or prevented serious events from unfolding during a mission's operation. Additionally, IV&V has functioned also as a watchdog and has kept Project developers, V&V and I&T more on their toes. This latter point is a hard to measure return on investment but it is significant.  One other question that arises is whether, IV&V needs to reside off-site, as presently, and remote from all Projects. It is not necessary to be remotely situated to maintain independence of the V&V.

However, I strongly disagree that NASA IV&V should be disbanded, removed from WV and distributed to NASA centers. The present IV&V facility has acquired an excellent group of analysts who, if given a proper directive, method and also cooperation from the Projects they support, will deliver the analysis and issues that cost-effectively raises the mission assurance of every project they review. Our existence is the result of the efforts of our Senator and Congressman to bring technology jobs to West Virginia. The nation's capital metropolitan area, FL, TX, CA, AL, MS, OH have benefited greatly from the presence of NASA Centers.

NASA IV&V remains a critical core group in the Technology Park developing in Fairmont and it is a very small price to pay to assist the West Virginia economy. Given the proper support from HQ and from the Projects we assist, the existing IV&V personnel in Fairmont can function effectively and fulfill the directive and fill the needs that have were found lacking after review of the Challenger and Columbia disasters and the string of Mars mission losses in the 1990s. As with the question of continued funding of major NASA Centers that have been on the chopping block over the years, politics is a major factor. The whole NASA budget is only ~1.4% of the Federal Budget.

NASA IV&V is 2/10ths of 1% of the NASA budget - a small sum expended on added assurance. Added expenditures and over-runs due to poor project management and design errors amounts to on the order of 100 times more than the cost of the IV&V budget. Our divulges involving modeling do not represent a catastrophic event or loss of mission. We argue that our mission be righted and our funding maintained. The engineers in Fairmont very eager to make needed changes."


Advertise Here

13 Comments

| Leave a comment

I am not trying to be snarky, just wondering if you mean Bryan O'Connor, or if there is another NASA official with a similar name?

Wow. What a relief. What a colossal waste of three years of our lives we'll never get back. Those who don't believe in life after death should have been in the halls after it was announced that Butch had gotten the ax. Butch scored the rare quadra-fecta of mismanaging the funding, organization, personnel and technical aspects of the IV&V facility. He had also screwed-up the building itself, when he began construction on a Taj-Mahalesque suite for himself. I don't even want to contemplate the millions of dollars he wasted using the IV&V facility as his personal pseudo-intellectual post-doc lab boondoggle. Beware people with shiny new PhDs.

When he didn't get results he changed the organization. When that didn't work, he changed metrics. Still no results, form tiger teams and working groups. Butch's pipe dream was for IV&V to find the high-level home run error. The running joke was that we'd review Mars mission documents, and save the government millions when we discovered a typo that would change the mission to the Moon. Thank goodness HQ finally woke up.

Nasaboy,
I'm confused, in private industry modeling is used in this area with success and saves money. Does the fact this guy bungled it so badly mean the technique itself has been discredited?
Charlie03m

It's been a few years since my time at the IV&V facility, all before the current (now ex) director, but I really think although these is value at having a consolidated IV&V facility, Fairmont is exactly the wrong place to have it to attract the best to the effort. That is not to say there aren't many good folks there (many of them friends), but finding and keeping good people in West Virginia is a constant challenge which could be overcome by collocating the facility with Goddard. This would provide access to one of the strongest concentrations of software talent in the country, many major universities, and a comfort level for staff who are always thinking "where will the next job come in West Virginia if my project is canceled?"

IV&V is in West Virginia because Senator Byrd is in West Virginia.

IV&V is organizationally part of Goddard, though I am not sure who they report to out in Greenbelt.

And there are really great bright people working in Fairmont.

Let's be blunt - Butch made huge changes at the IV&V facility in Fairmont, WV. As a result of these changes, the value of the facility to NASA has been significantly reduced.

Butch was an incompetent manager and a clueless engineer. He had no clue as to how to marry modeling (using UML) with V&V. This is not to say that modeling has no place in V&V. On the contrary. Merging modeling with traditional V&V techniques would enhance the value that the IV&V facility provides to NASA. But Butch wanted nothing to do with traditional V&V techniques. I know this from conversations I had with him over the past two years. Instead he threw out everything the facility had learned over 12 years, and started from scratch with modeling techniques. This would have been fine if Butch had brought with him a methodology for using modeling with V&V, but the man had no idea how to do this. Instead he brow beat his staff to figure out how to make modeling work with V&V, and as a result, the facility floundered for three years. Essentially he turned the entire facility of over 200 people into a huge research project; a project that after three years has been almost a total failure.

I am surprised that NASA management did not remove this clown sooner. Frankly, the man should be fired.

Do I sound bitter? Yes I am!! Working for NASA has always been a dream. To see this dream turned into a nightmare by a fat clown was very sad.

I hope that NASA management will now realize the error of their ways, and appoint a competent manager as director of the IV&V facility. Is this too much to hope for?

I'll give you two words why co-locating IV&V at Goddard won't ever happen.
The first is Byrd, the second Mollohand....

Charlie03 - There is a place for *some* modeling in V&V, certainly in the area of developing/using truth models for requirements validation. But in Butch's bizarre reasoning, everything had to be related to a behavioral model (he specifically forbade consideration of performance requirements - try assessing an architecture without considering performance). Looking at a system through the soda straw of behaviors yields an incomplete picture.

Butch didn't have a vision so much as a notion followed by a comedy of edicts, white papers and "business models." More than anything, Butch lacked a scientific mind. Even the rankest amateur researcher, after tweaking every variable, will go back and reconsider his approach. Butch never did. His approach was sacrosanct. In that respect, he was stupid. There's no way to sugarcoat it. Oh yeah, that, and handing-out spot-awards in the bathroom. Genius.

The IV&V facility has its share of useless civil servants, but also more than its share of experience-rich engineers who have consistently provided valuable technical feedback to the development projects over the years. While no one particularly looks forward to having their shortcomings pointed out (our highest praise is generally a quiet "thank you"), under Butch's reign of terror the usual project skepticism quickly devolved to open derision. Butch never traveled to reviews to present his fantasies, instead victimizing his staff (ok, a select few kind of had it coming) to be torn apart by development project management. The carnage was like nothing I've seen in 25 years of working in aerospace. Thankfully, I suppose, IV&V became so irrelevant that reviewers stopped paying attention for the most part, sparing critical review of some of the Butch-inspired inanities.

I forwarded one of Butch's memos regarding "goodness," to a colleague at a different center, and he refused to believe it was serious.

If there's a lesson learned, it's not so much that Butch was an unqualified loser, incapable of directing a rummage sale, much less a 200-person facility, but that the IV&V facility essentially ran open-loop for nearly 3 years, and that can't happen again. I don't know if HQ woke up on its own, people at the facility blew the whistle, projects complained or all of the above. Either way, Goddard would be wise to conduct some sort of debrief (in addition to the ding-dong-the-witch-is-dead all-hands on Wednesday) for the CS and contractor staff to calm people down.

I hope Butch enjoys his new job of modeling the system behavior of paper clips.

Having great familiarity with IV&V in WV I am not at all surprised at the recent turn of events. The only reason the facility exists at all is WV politics and pork barrel earmarks. The facility provides marginal value-added at best. Historically, good managers and good people have been ground down, removed or simply chosen to leave out of sheer frustration. Cong. Mollihan is the protector and inforcer of the status quo, meaning, "don't make any changes I don't approve of". The staff know this very well and complain to the Congressman directly when ever they don't like what is happening. This is an example of the worst form of political patronage and tax payer wasted money.

I have seen good value in the past from the IV&V Facility--not always mind you but enough to make it valuable and worthwhile. This should impress anyone who understands the huge hurdle of being an outsider organization pointing out problems in other people's products. It remains to be seen if those who hunkered down during the past few years or those who return will be able to bring new life to the Facility. Just before everything was thrown away to start afresh, IV&V had added engineering rigor to their work where only human experts had served in the past. The separate facility located off center is a very good thing in my opinion. The independent part of IV&V is important and requires distance, though I will not try to detail such arguments here. Just to give the full disclaimer, I have worked as a DoD contractor, IV&V engineer and manager, and in NASA "projects".

Captain,
Perhaps I'm misunderstanding... are you saying the staff is politically connected, and therefore complained to their congressman to get Butch (allegedly a good man) fired? That's exactly backwards. Politically, Butch was a huge success. He actually got MORE money for the facility, thus making a palace revolt against his moronic financial, technical, organizational and personnel leadership even more difficult. How do you remove the darling of the congressman? The guy who's hiring WV people left and right?

It was Butch who had the ear of the pork purveyors. He was competent in two areas - flimflamming his bosses and sucking-up to politicians; creating a technically retarded approach, then claiming victory when creative people somehow circumvented his byzantine directives (no real direction, just edicts) to produce something, anything presentable to a project.

The pork barrel machine couldn't care less about the actual performance of the technical work, as long as that work is being done in WV and the constituents are happy. Butch was not a good man ground down - he was a buffoon who fooled a lot of people to get that gig - and then proved himself so utterly incompetent it was close to criminal. NASA finally woke up and removed him. Took long enough.

Nasaboy: If you are correct that NASA HQ did this without first checking with the Congressman I would be surprised (flabbergasted in fact!!). They know how sensitive Mollihan is to any actions that could adversely effect IV&V. Generally, they walk on egg shells when it comes to upsetting the good Congressman! It would be a breath of fresh air if HQ acted on fact and principle alone.

My previous comments were in no way meant to defend the now ex Director. My comments were directed at the value added of the IV&V facility in general and over the past decade or more. Theoretically, the idea of a fully independent and highly competent IV&V activity for NASA is a great idea. Unfortunately, the reality is far far different!

Captain Kirk,

Nasaboy is correct. He snowed everyone including our government reps and NASA HQ. It kept everyone in-the-know off balance and unwilling to take the risks necessary to halt the destruction. No doubt, there were numerous personnel that sent messages to GSFC and HQ informing them of the problems and it did take too long for them to take the corrective measure.

And to call the IV&V facility pork is simply the pot calling the kettle black. How do you figure the Johnson or Glenn Centers got their names? Placement of government facilities involves give and take and hopefully not everything ends up in sunny California. NASA IV&V should remain here. We provided added assurance that justifies our existence. The DoD has had IV&V for decades and recognizes the necessity. Look at how the Constellation program has progressed and one quickly realizes that unnecessary faults and risks will continue to creep into NASA development and checks and balances of many sorts remain necessary.

At NASA IV&V, identifying severity 1 or 2 issues is rare, very few per year but just one alone could spell the loss of a $200M or $2B+ NASA project. And just one such issue represents multiple years of the NASA IV&V budget. Now consider the hundreds of lesser issues. Most would be found by the Project developers in time and fixed but how many would not? Because we inform the Projects, we do not know how many would fall through the cracks; of those, some would be compounding issues. Nearly every Severity 1/2 issue would be captured in time but you just do not know for sure.

Having seen hundreds of issues delivered I am sure our presence, here in Fairmont, is more than paying its way and at the same time we are now poised to make corrections to our procedures and management structure and improve our deliverables.

Leave a comment



Monthly Archives

Mortgage Lead

online bingo latest online bingo game reviews, bonuses and bingo news

Play online bingo at the top bingo sites.

Interested in Space Travel, try the next best thing, name your own star.

About this Entry

This page contains a single entry by Keith Cowing published on December 28, 2009 2:07 PM.

Sen. Cornyn Was For NASA Before He Was Against It was the previous entry in this blog.

Good Things In Small Packages is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.



- RADWIN's broadband access enabels cellular carriers to connect users everywhere.

- Looking for great prices on Burton Snowboards? Visit PortersTahoe.com

-