The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

The Seven Stages of Expertise in Software Engineering

Meilir Page-Jones
Wayland Systems Inc. All rights reserved. (In this article, an indefinite he means he or she.) For almost two decades we at Wayland Systems Inc. have toiled to transfer the latest developments in software engineering to practitioners at large and small companies around the world. However, until recently we didnt do a very good job of quantifying the long-term field results of our efforts. So we decided to ask a question reminiscent of the rhetoric of Ed Koch (the former mayor of New York City): Howre we doin? We surveyed some of our clients to whom wed provided education and consulting over the years and discovered an interesting but very disturbing fact. Over 50% of the shops into which wed introduced Software-Engineering techniques had either abandoned the use of the techniques or had let their band of active practitioners shrink to a dwindling core of diehards. What was going wrong here? We turned to self-examination and reviewed our courses and consulting practices. But, whatever the problem was, it didnt seem limited to Wayland Systems: Informal discussions with our competitors revealed that they too were having similar problems. The attrition rate across our whole industry was painfully high. External examination revealed several factors at work, many of which were to do with corporate management habits. Examples of these were: unrealistic management expectations; a lack of commitment to quality; an obsession with coding (which we termed WIMP Why isnt Marvin Programming? management); pernicious deadlines; and a mis-allocation of skills to tasks. Other factors included: engineers resistance to change; engineers reluctance to participate in team approaches; and the then-unavailabilty of useful CASE tools. The above factors were significant and worthy of articles in their own right. However, there was another factor, which is more important and fundamental than the others. This factor is the way that people are able to absorb sophisticated new techniques and then to apply them to their work in other words the stages of expertise through which we all pass as we learn and use new and powerful disciplines. At first we were aware only that there were two types of person: novice and expert. That was hardly earth-shattering news. However, further research revealed that there are actually seven stages of expertise through which a person may pass on the journey from total ignorance to world-class knowledge. With devastating acumen we numbered these seven stages Stages 1 through 7. (We also gave them names: Innocent, Exposed, Apprentice, Practitioner, Journeyman, Master and Expert.) As Ill discuss below, these seven stages have a deep impact on the successful or otherwise introduction of software engineering into an organization. A participant at a conference asked me how universal these seven stages are. Very universal, I answered.

1 of 7

3/28/2011 2:53 AM

The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

You mean that I could even apply them to the skill of bear-hunting? he retorted unexpectedly. Yes, I said. So, if youll bear with me, I should like to illustrate the Seven Stages of Expertise with reference to hunting (although Im most definitely not a fan of such an activity). Following that illustration Ill outline the application of the Seven Stages to our own business of Software Engineering.

Stage 1: Innocent
A Stage 1 person has never seen or heard of bears. It would not occur to a Stage 1, if he encountered a bear, that the bear could be hunted. Neither would he realize that a bear is a potential source of danger.

Stage 2: Exposed
A Stage 2 person has seen an occasional bear and has read articles in airline magazines suggesting that bears may be hunted. Moreover, a Stage 2 probably has friends that have hunted bears and has learned some desultory but intriguing facts about bears and their habits. He is motivated to learn more.

Stage 3: Apprentice
A Stage 3 has attended a 5-day seminar on bear hunting. During this seminar, the participants form into teams of three or four and practice hunting very small bears under the ever-watchful eye of the instructor. After a few interim setbacks, by Friday afternoon all the teams have successfully hunted their bears. They fill out evaluation forms attesting that bear hunting is very useful and relevant to my job. However, they are barely prepared for the world of real bears.

Stage 4: Practitioner
The Stage 4, having completed formal bear-hunting education, is full of confidence. Hes ready to transcend the minuscule bears of the 5-day workshop and go out for real bears, larger bears, fierce bears. Hes ready for Ursa Major. His manager is also keen to send him out with the latest bear-hunting techniques because the users want fur and they want it yesterday. Unfortunately, in the resulting scramble the budding bear hunter may be sent out without a map and with the wrong caliber arrow in his longbow. In the heat of ursine confrontation, the Stage 4 may also forget or misinterpret his classroom instruction and precipitate disaster. Its typical that some Stage 4s get some bears; but its also typical that some bears get some Stage 4s.

Stage 5: Journeyman
The Stage 5 has survived the traumas of Stage 4 and has bear hunting down cold. The Stage 5 uses modern bear-hunting techniques naturally and automatically; in fact, he cant imagine how he ever got along without them. He is accurate and productive: The Steering Committee merely points out the bear and he hunts it within both budget and deadline. The Stage 5 is the exemplary modern hunter that salespeople of bear-hunting seminars refer to in their brochures.

2 of 7

3/28/2011 2:53 AM

The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

Stage 6: Master
Stage 6 bear hunters have internalized not only the mechanics of bear hunting but also the principles underlying the techniques. Stage 6s know more than rules: They know why the rules exist and even when its permissible to break them. For example, a Stage 3 or 4 may stand upwind of a bear accidently and scare off the bear. However, a Stage 6 may know that by wearing Yogi-Spray Deodorant he can stand upwind without being detected and can thus surprise the bear from an unexpected quarter. Because of their deep knowledge, Stage 6s are very capable of training others in hunting techniques.

Stage 7: Researcher
Stage 7s are asked to write books and give talks at bear-hunting user groups. They are also engaged in extending and generalizing bear-hunting techniques to solve new problems. For example, a Stage 7 may extend bear-hunting to work also on Big Foot or he may even develop the ultimate Yeti-Oriented Hunting Methodology. Now lets come back to the world of software engineering and see how the Seven Stages of Expertise apply to us.

Stage 1: Innocent
A Stage 1 may not have heard of Software-Engineering techniques. Or more likely nowadays he may be vaguely aware of their existence but may not see their possible relevance to his situation. Indeed, he may be only dimly aware that there are any software-development problems in his shop. You may find it incredible that Stage 1s could exist in the 1990s, but they do. And the reason stems from the way in which software complexity evolved. Software became insidiously more and more complex in the 1970s and 1980s as users demanded more and more sophisticated systems be installed on the more and more powerful hardware that became available. Yet there was no sharp transition. The earth was not hit by a Complexity Asteroid in 1975 that suddenly made software three orders of magnitude more complex and cast our reptilian development techniques into extinction. I call the way in which software complexity actually increased Frog in the Pan. This is because although a frog will jump out of a pan of hot water, a frog that is placed in a pan of cold water and slowly heated will fail to leap forth and will actually boil to death. The temperature gradient is so gradual that there will never be a point at which the frog declares: Boy, its suddenly gotten hot in here! I think I should hop out. (Before I get into deeper trouble from animal-rights folks, I hasten to add that this analogy is apocryphal. Ive never tried the experiment and I dont recommend that you do so either !) Many Stage 1s are experiencing Frog in the Pan and are trying to tackle problems of the 1990s with approaches of the 1960s and 1970s, without realizing that the problems theyre facing are the very ones that modern Software-Engineering techniques were created to alleviate.

Stage 2: Exposed

3 of 7

3/28/2011 2:53 AM

The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

Stage 2s have noticed that the water is getting decidedly warm, if not downright hot. So they are actively seeking Software-Engineering techniques that will get them out of the pan or at least reduce the heat. Stage 2s may survey magazines, confer with colleagues or attend one-day overviews of the techniques. Their interest level is high but their knowledge level is low, being limited to a few terms and definitions and not based on any practical Software-Engineering experience.

Stage 3: Apprentice
Stage 3s have attended one or two 5-day workshops on Software-Engineering techniques. In these workshops they tackled small but realistic case studies that resembled their own projects in miniature. The case studies provided valuable kinesthetic reinforcement of the formal lecture material and were thus indispensable. However, the case studies apparent realism conveyed to the Stage 3 a confidence that is often unwarranted. If a Stage 3 absorbs everything from a seminar, then he is minimally equipped to tackle a true, full-sized project in the corporate jungle. Usually, however, a Stage 3 does not grasp everything or has difficulty scaling the techniques up from a case study to a real project. You could say that most Stage 3s know just enough to be dangerous!

Stage 4: Practitioner
The rite of passage to Stage 4 is the use of Software-Engineering techniques on at least one significant project. Achieving Stage 4-hood is for many people the most difficult transition of the six transitions between stages. The fledgling Stage 4 is asked to take untried (by him) techniques and apply them to a corporate project with the usual demonic cocktail of politics, deadlines and changing requirements. At the same time, he is attempting to recall what he learned in class and scale up the examples 10- or 100-fold. He continually needs consulting advice, without which he will encounter a series of minor setbacks or major failures. Since many people throw up their hands at this point and revert to their old mediocre but familiar ways, a large proportion of Stage 3s never make it to Stage 4. If an entire project is peopled with Stage 3s, then its highly likely that the project itself will fail and the SoftwareEngineering techniques will be publicly pilloried and then abandoned.

Stage 5: Journeyman
Stage 5s have made it. Their experience of Software-Engineering is latched in place and there is little risk of their reverting to the past. In the Stage 5 the SoftwareEngineering techniques yield for the first time the productivity the marketing folks promised; and on each successive project a Stage 5 further hones his skill and enhances his productivity. A Stage 5 is self-sufficient - more often the source of Software-Engineering advice than its recipient.

Stage 6: Master
The Stage 6 not only is an adept technician, but also possesses a profound methodological foundation. Beyond the whats and hows, the Stage 6 knows the whys of Software Engineering. This depth allows him sometimes to break a surface rule, while adhering to a more fundamental methodological principle. The Stage 6 is

4 of 7

3/28/2011 2:53 AM

The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

a good instructor because his theoretical and practical knowledge give him the wherewithal to tackle difficult student questions.

Stage 7: Researcher
The Stage 7 is concerned with delivering the latest developments in Software Engineering to a wider audience, via books, articles and conference appearances. The Stage 7 looks out for flaws in contemporary Software-Engineering techniques and for consequent ways to improve the techniques. He also scans the horizon for new problems towards whose solution Software Engineering can be extended and generalized. The Stage 7 is at a methodological pinnacle. These Seven Stages of Expertise are valuable in their own right. You might think about a Software-Engineering technique that you know and consider which of the stages you are in with respect to that technique. You may also decide what (if anything) you should do to progress to the next stage. However, there are organizational implications behind the Seven Stages. Below I discuss four of these: The Productivity Curve, Pilot Projects, The Critical Consulting Core and Ephemeral Technology.

A. The Productivity Curve

Vendors who sell Software-Engineering techniques do so by convincing management of their boon to productivity. Although vendors rarely say so explicitly we at Wayland Systems never used to they compare the productivity of Stage 5s to that of Stage 2s. Unfortunately, this presentation (see Fig. 1) is not a complete picture. Between Stage 3 and Stage 4 theres a hair-raising J-curve of productivity (see Fig. 2), which represents the difficult rite of passage of the incipient Stage 4. Fig. 1 (to come) Fig. 2 (to come) This J-curve has led more than one manager to abandon all hope in SoftwareEngineering techniques. Such a manager is not merely impatient; no-one warned him of the J-curve and so when it manifests itself he can only assume that the much-touted Software-Engineering techniques were mere hokum. A couple of our clients measured the length of time from Stages 3 to 5. Depending on the individual person, Stage 3 to Stage 4 took 6 to 18 months (about the length of a project or phase) and Stage 4 to Stage 5 took 18 to 36 months. (These times are very approximate because of the subjective nature of the stages.) Except for a few outstanding people, the shortest time from Software-Engineering seminar to exquisite software productivity is 24 months. When managers tell me: We just cant wait that long, I retort: The 2 years will pass anyway, whether you do anything or not. Act now! (By the way, notice that Stage 7s have lower project productivity than Stage 6s. This is because a portion of Stage 7s energy is directed away from the specific project and towards general methodological research.)

B. Pilot Projects

5 of 7

3/28/2011 2:53 AM

The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

But the J-curve has a facet nastier than simple delay in productivity. This is the Who will bell the cat? problem. In the fable of that title, a group of mice whod been persecuted by a cat decided to hang a bell around the cats neck that would warn them of its approach. All agreed that this was a wonderful idea ... but there was one minor difficulty: None of the mice wanted to be the one to risk walking up to the sleeping cat to tie a bell around its neck. This impasse is known formally as the Technology Trap. Most project managers would agree that Software-Engineering techniques are a Good Thing. But which of them with bone-crushing deadlines and ax-wielding users wants to be the first to experience the depths of the J-curve. (Me neither!) Thus many shopfuls of intelligent people appear to be dinosaurs because the Technology Trap holds them in the 1970s. If who will bell the cat? is a problem, then I recommend that you try SoftwareEngineering techniques on a pilot project first. The pilot will uncover all manner of pitfalls that occur during the Stage 3/4 transition as well as problems in the techniques themselves. The pilot should be short (6 months) and should yield a non-critical system. You should learn as much as you can about applying the new technology from the pilot project; this is one project where the learning experience is more important than the end-product. A pilot project will not eliminate J-curves on subsequent projects. People will still have to go from Stage 3 to Stage 4. But the pilot will make future J-curves more shallow, as some causes of the curves nadir are removed. Indeed, although the delivered system is the most important product of most projects, you should never forget the learning by-product of every project. Review what you discovered about doing projects when you complete each project and share that information with others. In that sense, treat every project as a pilot.

C. Core of Internal / External Consultants

One of the surest recipes for disaster is to staff a project with only Stage 3s. Yet many managers do exactly that, pushing a group of analysts through a seminar one week and letting them loose on a crucial project the next week. (Jerry Weinberg refers to this as the Sheep Dip approach.) Every project should have access to a Stage 5, or - better still have a Stage 5 on the project team. Cultivate your own Stage 5s and Stage 6s in-house as part of your long-term staff-development policy and make them available to project teams. If you dont yet have such people on board, then call in external consultants from Wayland Systems or one of our competitors. However, dont become chronically reliant on external consultants, but instead continue to work on building up a pyramid of internal expertise with Stage 6s at the top that will ensure a lasting mastery of Software Engineering.

D. Learning Latency vs Lifetime of Technique

One manager of a major organization recently complained to me that by the time a critical mass of his people had learned a Software-Engineering technique, that technique had been left behind in the mud of obsolescence by the relentless march of progress. Although there is, of course, some truth in this complaint, it is also true that plus a change, plus cest la mme chose. If you buy a power drill to replace a

6 of 7

3/28/2011 2:53 AM

The Seven Stages of Expertise in Software Engineering

http://www.waysys.com/ws_content_al_sse.html

manual drill, the deep principles of drilling will be largely unchanged. Similarly, most new Software-Engineering methods are basically founded on familiar underlying methodological principles (such as the principle of Safe Deferral or of Partitioning to Minimize Interfaces). Develop a large capacitance of Stage 6s in your organization in order to smooth the apparent high-frequency ripples in technology. These people will be able quickly to absorb a new technique and to assess its applicability to the shops needs. They will also be able to compare and contrast the new technique to current techniques and to steady the transition from old to new, because many of the new ways of addressing software are simply refinements of ways already known to Stages 3, 4 and 5.

Finally Some Recommendations

Be aware of the Seven Stages of Expertise and their effects on productivity under a specific Software-Engineering technique. Reconcile your expectations for Software Engineering with the realities of the Seven Stages. Take stock of where you are. Have each person assess his stage with respect to each Software-Engineering technique that your shop uses. Have each person state their medium- and long-term goals for expertise in each relevant technique and suggest a plan for achieving those goals. Establish similar medium- and long-term goals for the shop as whole. Determine the most appropriate blend of stages for the work that you do and set up a plan to achieve that blend. Never attempt a crucial project solely with Stage 3s and below. Seed each project with Stage 4s and allow access to Stage 5s and 6s if possible. If you dont have a Stage 5, beg, borrow, steal, buy, kidnap, ... one. External consultants are valuable as a short-term measure. Stabilize the effects of technological change by nurturing an in-house expertise (Stage 6) in fundamental methodological principles. Try to formulate shop strategies (as much as you can) in non-technological terms. Superimpose technology on to the strategy in order to yield tactics. Have a mission that will never go away. For example, the mission of Wayland Systems is Mastery in Software Engineering. In other words, we seek to assist our clients in becoming self-sustaining at at least the Stage 6 level of Software Engineering. This mission will not evaporate for many decades yet, for software is becoming ever more complex and ever more critical. There will be much to learn even for Stage 6s and Stage 7s as we pass into the new century.

Acknowlegments
I should like to thank my colleague Claude Maley for inspiring me in the first place to go out and search for the Seven Stages at our client sites and my colleague Steven Weiss for helping me to name and define the characteristics of each of the Seven Stages.

7 of 7

3/28/2011 2:53 AM