Jebin Cherian M1020 SIMSREE

Five years and $405 million into the FBIs effort to create an all-digital casemanagement system for its special agents, its clear how far the Sentinel project has slid off the rails. According to the inspector general at the Department of Justice, many promised functions remain undelivered and defective code inhibits the use of some of the pieces of the system that are already in the FBIs hands.

But FBI CIO Chad Fulgham insists Sentinel, which he inherited when he joined the bureau in December 2008, isnt doomedeven though independent reviewers estimate it could take another eight years to complete, at an additional cost of $453 million. To save it, the former Lehman Brothers IT executive is battling what he calls old guard thinking and a federal bureaucracy that he says is nuts.

Sentinel is a big problem, but its just one of several Fulgham faced when he arrived. He was surprised to also find outdated technology, 200 open IT positions and a convoluted federal purchasing process.

Like any new CIO, in government or the private sector, Fulgham had to assess what was working and what wasnt, triage immediate problems and build relationships with colleagues who could support his longterm agenda. Congress and various government oversight groups want action, just as a corporate CEO and board of directors would, observes Charlie Feld, CEO of the Feld Group, an IT turnaround consultancy.

Feld, who has been CIO at Delta Airlines, Burlington Northern-Santa Fe railroad and Frito-Lay, says that upon starting a new job, a CIO carries the most political capital hell have for the next two years. And he needs to spend it even if it pisses people off, Feld advises. Senior executives can take bad news. They just cant take not knowing or continuous streams of bad news that build uncertainty.

In the past two years, Fulgham has drawn heavily on his political capital, using his corporate experience to bring the latest IT-management techniques and new technologies to the FBI. This includes collaboration software, mobile applications and a modern network infrastructure. Nowhere has Fulgham spent more of his political capital, however, than on his remediation plan for Sentinel.

With support from FBI Director Robert Mueller, Fulgham has restructured parts of the original project agenda. He has, for example, traded traditional sequential software development for agile methods and will replace a Sun Solaris platform with virtualized Red Hat Linux servers.

He also hopes that throttling further participation by contractor Lockheed Martin will get Sentinel done, within budget, by September. Thats when Muellers 10-year term ends. When we release this, FBI will declare victory on Sentinel, Fulgham says.

The inspector general, however, calls Fulghams plan optimistic, noting the FBIs inexperience with agile development. Delays, these overseers warn, are apt to pile up as the bureau changes course. The most difficult phases of the project are ahead, they say.

Fulgham, 36, doesnt flinch. FBI agents and staff, he says, love what were doing. But the old guard thats used to doing things one way, as Fulgham calls his critics, theyre not exactly happy with the Chadster.

FBI technology hasn't ever been much to brag about. The agency has failed to build a fully electronic case-management system since coming up with the idea 11 years ago, with efforts so far costing taxpayers at least $575 million, plus an inestimable toll in lessefficient crime fighting.

After the 9/11 terrorist attacks, critics faulted the FBI and intelligence agencies for not sharing information effectively. Technology failures were a core problem.

There have been some improvements: FBI employees can process some forms digitally, for example, and the CIOs of the largest intelligence agencies meet regularly to devise ways to share information and trade project advice. But lacking a fully digital case-management system means FBI agents still spend a lot of time shuffling paper.

Sentinels electronic workflow would speed up and improve analysis of case notes and evidence, as well as facilitate better coordination with fellow law enforcers.

We must make sure agents and analysts have the technology tools they need to do their jobs. I wont accept any more technoboondoggles, Sen. Barbara Mikulski (D-Md.) said in an interview. Mikulski chairs the Senate subcommittee that oversees the FBIs $8.3 billion budget.

In 1995, while corporate CIOs were building distributed applications running on PC servers and diving into Web technology, the FBI launched a system for managing case files and evidencethe Automated Case System (ACS)that used green-screen, mainframe software.

By 2000, the FBI was planning to replace ACS as part of a larger modernization project called Trilogy. Within Trilogy, the Virtual Case File was to have been a set of applications that eliminated paper forms that agents had to fill out by hand, route around the office in file folders and store in metal drawers.

After 9/11, Mueller accelerated Trilogy and got approval from Congress to create a CIO position. But in the next three years the bureau went through four CIOs, seeking the right fit in terms of experience and management style.

In 2005, the FBI canceled the Virtual Case File amid reports from the Government Accountability Office that the bureau wasnt watching its contractors well enough and couldnt account for 1,200 pieces of missing equipment worth $7.5 million. Worse, the GAO said, the unfinished project, which to that point had cost $170 million, was poorly designed and relied on custom code that would be difficult to maintain.

The FBI began planning Sentinel in 2006, awarding a six-year, $451 million contract to Lockheed Martin. In Senate hearings last April to review the FBIs fiscal 2011 budget request, Mikulski recalled the agencys checkered IT past, telling Mueller she was concerned that now Sentinel was in jeopardy of becoming technojunk, just as Virtual Case File had.

When you have a project that goes over four or five years, Mueller responded, some forms of delays are, I wouldnt say inevitable, but need to be identified and addressed and contained. I think weve done it here.

Before joining the FBI, Fulgham was a senior vice president at Lehman Brothers, directing its worldwide identity management, messaging and unified communications teams. He helped to manage projects using well-known agile principles, such as short development cycles with frequent prototyping. He also worked directly with vendor developers to speed projects along.

In 2008, as Lehman collapsed into bankruptcy, Fulgham interviewed for the FBI CIO position. Then-CIO Zalmai Azmi was set to leave in November after five years in the job. Fulgham had doubts about Sentinel.

Corporate IT executives learn to avoid such megaprojects because they have high failure rates, he says. From the outside, he recalls telling Mueller, we dont think you can pull it off.

He took the job anyway. Fulgham spent his first few months at the bureau examining its IT procedures and infrastructure, although it didnt take him long to find out that more than Sentinel needed fixing. On his first day, his PC wouldnt boot. Nor was his BlackBerry configured.

On his second day, he turned on his PC to reveal that the standard desktop image at the FBI was Windows 2000 with Corel WordPerfect 8. I really was disheartened, he says.

But he saw the chance to make fast improvements that played to his strengths. At Lehman, he had planned and implemented unified communications tools from Microsoft for 28,000 employees worldwide.

One project integrated Microsoft Communicator instant messaging, voice-over-IP (VoIP) and desktop videoconferencing tools with Ciscos Call Manager and Unity to give Lehman employees one dashboard for managing their communications. At IBM Global Services and JP Morgan Chase before that, he had led teams responsible for securing networks and mitigating security risks.

Accordingly, he spent much of his first year replacing the FBIs 1990s-era ATM and frame relay network with revamped cryptography and new Cisco gear that supplies 45 times the bandwidth to 800 local offices nationwide. More than 27,000 agents, analysts and other employees got BlackBerry smart phones. RSA identity-management tools from EMC let authorized FBI employees use those BlackBerrys to access the secure National Crime Information Center database of criminal records while working in the field.

Meanwhile, 30,177 desktops and laptops were replaced and almost 100,000 are being upgraded to Office 2007. Unified communications software, also from Microsoft, includes SharePoint for collaboration via instant messaging and document sharing. The so-called next-generation workspace sports 24-inch monitors and advanced cameras and headsets for peer-to-peer VoIP videoconferencing.

Helping to smooth the rollout of these new technologies was a reorganized IT staff. Fulgham filled the open positions, embedded IT staff in user groups and established a chief marketing officer position to advertise new features and capabilities to FBI employees.

The modern infrastructure has already improved coordination and workflow, he says. In October, the bureau arrested 133 public officials and others involved in a massive corruption case in San Juan, Puerto Rico. One thousand agents and staff fanned out across the island in the largest single operation in FBI history. The takedown was coordinated in real time using SharePoint and the other new UC and mobile tools, he says.

After the raids, Mueller demonstrated the new technology to Attorney General Eric Holder. I had smile on my face like the Cheshire cat, Fulgham recalls.

Such quick IT wins can supply a new CIO with the momentum needed to set a troubled project right, says Brad Brown, director in the IT practice at McKinsey. And when its time to tackle the problem project, Brown advises CIOs not to halt the whole thing, if possible. Instead, find one piece that can be finished right away while senior leaders work with the CIO to dissect the larger project plan.

That approach accomplishes two things, he says. First, keeping at a clearly defined task that is important to the project prevents the organization from falling into chaos. Second, delivering that piece improves morale and gives the team a success to build on. Youre trying to build trust at the same time youre making change, he says.

The CIO must emit self-confidence and confidence in the organization, he adds. If a projects in trouble, theres angst. People look for a lifeline and a way forward.

Fulghams call to action came in late 2009, when Lockheed missed its first major deadline under his watch, then had trouble resolving problems with electronic forms it had built. Immediately, Fulgham demanded regular demonstrations of the in-progress code for senior executives from both Lockheed and FBI.

He also began quarterly meetings with Lockheed CEO Robert Stevens and Linda Gooden, the executive vice president of Lockheed Martin Information Systems and Global Services, along with Mueller and other FBI leaders. Having top executives in those meetings escalated the issues, he says, so they could see firsthand what my customers would see.

When trouble continued, Fulgham issued a stop-work order to curtail Lockheeds remaining work on Sentinel. He also directed Lockheed to correct defects in the pieces of Sentinel it delivered in 2009 and 2010. Among them, according to the inspector general, was a security setup that could allow a user to create a fake identity to electronically sign documents, which resulted from a disagreement between Lockheed and the FBI about how to interpret a digital signature requirement.

Meanwhile, Fulgham formed plans to switch to agile development, which required a smaller team than the group of 200-odd Lockheed personnel assigned to Sentinel. Fulgham instead wanted developers who were intimately familiar with the key products Sentinel would use, including Microsoft search, Oracle databases, EMC document-management software and Adobe document-creation tools.

Fulgham dialed the vendor CEOs, including Steve Ballmer at Microsoft and Joe Tucci at EMC, to explain his plight and enlist their help. Not many people turn down phone calls from the FBI, Fulgham notes. They were ecstatic because they werent able to make their products successful in Sentinel the way the project had been managed.

A fresh team of about 50 vendor developers and support staff, along with the bureaus own IT staff, now deliver a piece of Sentinel every two weeks. Fulgham and others at the FBI see demos every other Friday. Fulghams point man on Sentinel, CTO Jeff Johnson sits with the developers. Some pieces of Sentinel now have been successfully rolled out, such as a module for writing interview reports and sharing case leads.

A new CIO has two years to prove himself, according to Feld, the turnaround expert. Thats enough time to have completed at least one big innovative project and have corrected the trajectory of inherited problems, he says. By three years, hes either really solid or gone.

As Fulgham enters year three, he counts several successes and has set a new course for the mess he now owns. This year he plans to finish Mobile FBI Net, a project to allow secure access to the bureaus secret network from outside FBI facilities. For example, a field agent who finishes an interview related to an open case would be able to hold a secure videoconference about it with the agent managing the case at headquarters.

Meanwhile, some of the most substantive parts of Sentinel remain to be done. For example, only four of the 18 forms routinely used by FBI employees have been recreated digitally. Yet fully computerized workflow for investigation approvals, evidence handling and other tasks cant be deployed until all the right forms are available, Fulgham says.

Perhaps most critical is making the 8.3 million FBI case histories that reside in the ACS mainframe system fully accessible to special agents using their new workstations. The original plan called for Lockheed to migrate ACS data to Sentinel.

Now Fulgham has decided Sentinel users will tap that data using enterprise search technology from Microsoft. The FAST search product will index and search the FBIs proprietary mainframe data, as well as newer Oracle databases and the FBIs new EMC Documentum documentmanagement system.

Fulgham thinks that enterprise search and analytical tools will allow special agents to look at relationships among different kinds of data, including the FBIs case histories, geospacial information, link analysis and repositories from other intelligence and lawenforcement sources.

The inspector general is doubtful Sentinel will be done by September, within budget, as Fulgham has promised. An October report notes that two of the four original project phases have yet to be built, and the parts of Sentinel deployed so far are used by only one-third as many employees as regularly use the old ACS mainframe system. Of Fulghams decision to reduce the size of the development team, the report notes wryly: while useful in keeping costs down, [the smaller staff] may not be sufficient to complete the system.

In a written response, FBI Associate Deputy Director Thomas Harrington said the report doesnt accurately reflect how the FBI is handling Sentinel now. It includes, he says, an inflated cost estimate for completing Sentinel that is based on a worst case scenario for a plan that we are no longer using.

The inspector generals criticism is misplaced but understandable, Fulgham says. The agile method is very new to the federal government. The IG is going to look at any new strategy with skepticism. He has offered to train auditors in agile so they can better understand what [CTO] Jeff [Johnson] and I and others take for granted.

Troubled projects are difficult enough to mend in the corporate realm, but would-be fixers at the federal government face many more obstacles, says Feld. Once expensive, multiyear endeavors get underway, federal procurement rules and other regulations make them tough to stop.

For example, the FBI doesnt yet know how it will resolve its agreements with Lockheed, an FBI spokesman says. And the bureau hasnt revealed any estimates for the cost of getting out of the contracts, the inspector general says.

Meanwhile, circuitous procedures govern everything from how products can be bought to how staff can be organized. Fulgham was shocked to learn that he cant negotiate contracts with vendors directly: Thats done by contracting officers.

I thought, Are you nuts? Why did you hire me? he says. You pay executives to make these decisions. I know what things are worth and can play one vendor against another.

John Reece, former CIO of the Internal Revenue Service and now a consultant to the federal government, sympathizes with Fulgham. Reece joined the IRS in 2001, having recently retired as CIO of Time Warner, to straighten out an IT modernization project begun two years earlier. I was facing exactly the same barriers, he says. Processes take forever.

Fulgham, Reece says, has every reason to be frustrated. He cites one recent study by the Department of Homeland Security which found it takes 250 days, on average, to get an IT procurement request approved.

Fulgham says hes shared his views with federal CIO Vivek Kundra, who has pledged to reform IT management processes, including procurement, so government IT projects can proceed faster and make better use of emerging technology.

Changes announced in December call for Congress to work with federal CIOs and the Office of Management and Budget to overhaul budgeting models for large projects to allow for modular developmentjust the approach Fulgham is now taking with Sentinel. Kundra also plans to work with federal CIOs to redefine their roles, so they have, for example, more control of commodity IT purchases.

Despite the shivers they produce for many CIOs, megaprojects will never go away, even in the private sector, says McKinseys Brown. At some point, every organization must undertake a major platform or architecture upgrade. But constructing these large projects well, he says, is the hardest thing we do as technology professionals.

Fulghams five-year strategic plan doesnt include any such monsters; important projects are broken down into manageable pieces. He intends to increase the FBIs use of smallteam agile development. Other goals: consolidate data centers, replace a proprietary network security system with an offthe-shelf product, and redesign the unclassified, secret and top-secret networks for a single look and feel girded with multiple levels of access controls.

IT is all about helping us conduct investigations faster, Fulgham says, and successful deployment of Sentinel will go a long way toward that goal. But September is coming up fast.