This paper discusses the role of case studies in systems engineering and systems manage- ment, especially case studies that involve systems acquisition. We first provide a brief overview of case studies, including some of the analysis techniques useful for the conduct of case studies. Next, we discuss a two-dimensional framework for systems engineering and management case studies. The framework is in the form of a 9 row by 3 column matrix. We present a number of vignettes of case studies, at least one for each of the 27 cellular entries in this matrix.
The hope is that this will be a stimulus and precursor of additional systems engineering and management case study efforts, both in terms of appropriate frameworks for these and in the actual conduct of case study research. © 2003 Wiley Periodicals, Inc. Syst Eng 7: 84–97, 2004 Key words: case studies; case framework; systems engineering taxonomy 1. INTRODUCTION Systems engineering concepts—including methods, processes and systems management to enable trustwor- thy implementation of processes and methods to ensure *Author to whom all correspondence should be addressed (e-mail: quality products and services—are very important for [email protected]). success today.
The material to follow was brought to- Systems Engineering, Vol. 7, No. 1, 2004 gether with two major considerations: © 2003 Wiley Periodicals, Inc. 84 CASE STUDIES OF SYSTEMS ENGINEERING AND MANAGEMENT IN SYSTEMS ACQUISITION 85 1. In the teaching of systems engineering, case stud- longer period of time. In the social sciences, case stud- ies are potentially very instructive in that they ies became less popular a half century ago as the social relate aspects of the real world to the student science disciplines became more “scientific” and quan- through exposure to realities in the world of titative.
Case studies were seen as suffering from fun- professional practice. Frequently, the bad exam- damental problems of external and internal validity, and ples are even more valuable than the good ones, sometimes even from construct validity and reliability since they emphasize the penalties for not follow- (Yin, 2003a): ing the proper concepts and processes of systems engineering. • Internal Validity —Were the findings actually 2. In teaching systems engineering , there has pre- justified by the research, or were there problems viously been little distinction between the duties of researcher bias?
Has the case study researcher and responsibilities of public sector, government, demonstrated a causal relationship between fac- and private sector, industry, activities. There is tors by showing that other plausible factors could also a third sector, comprised of nonprofit organi- not equally well or perhaps better explain the observed relationships? zations, which may often need to be distin- • External Validity —Could the research findings guished from for-profit private sector be generalized? organizations.
• Construct Validity —Do the measures used in the case study make the concepts involved opera- Understanding of all three roles is needed. Many, if tional? In other words have we used multiple not most, of the textbooks available, with the exception evidence sources, have we sufficiently estab- of strictly government publications, stress the private lished chains of evidence and have those provid- sector aspects of systems engineering. However, in ing evidence to the case study been allowed to future curricula, the distinction should be made clearer review the case report before finalizing it.and the government aspects deserve much more empha-
• Reliability —To what extent would other re- sis, as does the need for and role of joint government- searchers who are studying the same case in contractor effort. Of course, the private sector industrial exactly the same way arrive at equivalent conclu- aspects should not be ignored; government and non- sions? profit sector professionals should be expected to be- come smart buyers of private sector contractor activities Each of these is particularly important in case study and appreciate their many problems and challenges. research.
For example, relative to researcher bias, it is It is usually important to define a concept, even those particularly important to avoid even the appearance of that seem to be familiar. For our purposes (Yin, 2003a), researcher bias. At a minimum, this suggests strongly a case study is empirical inquiry that: that the case study research team members should not all be stakeholders of the programs being reviewed. • Investigates a contemporary phenomenon within A recent revival of interest in case study research has its real-life context, especially when occurred, esp ecially in evaluation research in many • Boundaries between phenomenon and context areas such as enterprise management.
This has led to are not clearly evident, and in which the recognition that case study research can fill impor- • Multiple sources of evidence are generally used. tant needs. In particular, it is often insufficient to know that X can cause Y. We also need to know the how and Many other authors infer similar definitions. Case stud- the why X causes Y, and for what specific X and Y.
Case ies have has a long history in many disciplines, notably, study research can potentially answer these interroga- sociology, psychology, medicine, history and political tives and thereby help us become more contextually science, and business. The history of case studies shows aware. Modern case study research is able to deal with periods of intense use and also periods of considerable issues of internal and external validity if the case study disuse. is well defined, developed, and deployed, with appro- priate formulation, analysis and assessment, and inter- 2.
AN OVERVIEW OF CASE STUDY pretation across these three phases of the case study RESEARCH effort. These are the three basic phases and steps in a two-dimensional framework for systems engineering Case studies have had a long history, and formal ones (Sage, 1992, 1995; Sage and Rouse, 1999), and it is have been conducted for approximately a century.
In- fully appropriate and, we believe, desirable that a sys- formal case studies have, of course, existed for a much tems engineering process be used to identify a suitable 86 FRIEDMAN AND SAGE framework for case study research in systems engineer- ences about events.
They can result in false clues, ing. especially in the hands of inexperienced case A p rototypical characteristic of case studies is that study researchers, and this is one criticism of case study research, leading to the observation that a they support a holistic understanding and interpretation case study researcher should be a “critical ob- of the systems of action, or interrelated activities en- server” to avoid being misled.
gaged in by the participants or actors in the case situ- 2. Archival Records can be organizational docu- ation subject to study. Selecting cases must be accomplished in such a way as to maximize what can ments, official lists of names, survey data, and be learned in the period of time available for the study. other such records. A case study researcher must be careful in evaluating the accuracy of record s Case studies often are selective, focusing on one or two before using them.
Even if the reco rds are quan- issues that are believed most fundamental to under- titative, they might still not be accurate. standing the issue under consideration. Case studies 3. Interviews represent a most important source of should be multiperspective in nature, where the case case study information. There are several forms study researcher considers not only perspectives of the of interviews: open ended, focused, and struc- actors, but also of the interaction between them. What tured or survey. is “most fundamental and important” in a given case 4.
Direct Observations are obtained by such activi- depends greatly upon the viewpoint or perspective of ties as field visits to a case study site, and other the case study participants and observers. efforts to obtain data that range from the formal Yin (2003a) has summarized the major strengths and to the casual. It is thereby possible to cover events limitations inherent in case study designs. First, case and contexts in real time. The disadvantages are studies are useful for addressing questions regarding that it may be a time consuming and costly activ- how and why phenomena behave the way they behave.
ity, discriminatory results may be obtained unless These case studies more often lead to hypotheses about the observations are broadly based, and direct behavior rather than being useful for validating general observation may cause events to proceed differ- claims about behavior. Second, case study research ently than the otherwise would. often reveals a rich detail of information that highlights 5. Participant Observations have many of the the critical contingencies that exist among the variables same strengths and weaknesses as direct obser- in the case study. Finally, the case study method is vations.
They may lead to insightful observation especially useful for exploration of topics when there is of interperson al behavior but are subject to bias not a strong theory to which one can appeal. Even when if the case study researcher manip ulates events in there is a strong normative theory, case studies can often some ways. lead to valuable insights, and potentially even to revi- 6. Physical Artifact Observations may provide sion of the normative theories. much insight into cultural features and technical Case studies can involve either single or multiple operations. The weaknesses of this source of case designs.
Single case designs are generally used to evidence are selectivity and availability of obser- confirm or challenge a theory, or to represent a unique vations. or extreme case of behavior. Single case designs require very careful investigation in order to avoid misrepresen- The current rebirth of interest in case study research tation and to allow maximum access by the researcher highlights the central role of theory and the importance to relevant evidence. Multiple case studies follow a of having a framework within which to address case replication logic in which each individual case study study research.
We address this latter subject in our next consists of a “whole” study in which relevant informa- section of this paper. Theory should guide design and tion is gathered from various sources and conclusions analysis of case studies, and the results of a case study are drawn based on this information. should assist in the development of theory. Therefore, Yin (2003a) has identified six sources of evidence in identification of boundaries for a case study should be case studies. strongly influenced by the theoretical perspective taken by the case study researcher to the purpose and subject 1.
Documentation , which could be letters, memo- matter fo r the study. randa, agendas, administrative documents, news- Yin (2003a) recommends three general analytical paper articles, or any information believed strategies for establishing a case study: relevant to the investigation. Documents are communications between actors in the study and 1. Relying on theoretical propositions. The first and often serve to corroborate or refute evidence from generally more preferred strategy is to follow the other sources. Documents are used to make infer- theoretical propositions that led to the case study.
CASE STUDIES OF SYSTEMS ENGINEERING AND MANAGEMENT IN SYSTEMS ACQUISITION 87 in the first place, at least initially. These proposi- which is suggested in Figure 1, and associated tions necessarily shape data collection and allow analysis involves breaking case study material us to focus attention on certain data and to ignore down into these categories and then linking them other data. They ideally help us to organize a together such as to make structure, function, and complete, or entire, case study and to identify purpose of cases and their explanation more ex- alternative potential explanations that should be plicit.
Janssen and Sage (2000) discuss the use of examined. The “how” and “why” are especially Toulmin logic in a knowledge management and relevant here. conflict resolution support system that may well 2. Identify thoughts about contending explanations. have case studies as suitable inputs. This allows us to examine and accept or reject 4. Time Series Analysis is used in an attempt to alternative premises concerning case behavior. match observable patterns in data to an underly.
3. Developing a case description. In this often-used ing model proposed for explanatory purposes. approach, the researcher writes a narrative that There are three major situations in time series describes the case and its history. analysis. The first and simplest situation is when there is a single endogenous or dependent vari- There are at least five specific techn iques for analyz- able that may or may not be affected by unusual ing case studies: pattern matching, explanation build- events. The second situation extends the first by ing, logic models, time-series analysis, and cross-case inclusion of input or causal variables that may synthesis.
These approaches are not mutually exclusive have a role in the prediction or modeling of the and use of multiple approaches will generally be very assumed single dependent variable. The third useful, whenever it is possible to do so. situation provides for simultaneous modeling of multiple dependent time-series. Very interesting 1. Pattern Matching. Pattern matching contrasts situations occur when the data is such as to sug- and compares an empirically observed pattern gest chaotic behavior patterns and the potential with one that is predicted.
This might be used, for need to involve complex adaptive system proper- example, to compare ou r case study to one found ties as case explanations. in the literature. Here a researcher develops a 5. Cross-Case Analysis and Synthesis is particu- prediction about patterns expected to be seen in larly useful when case studies are themselves collected data. The associated analysis attempts comprised of more than a single case. to determine the pattern that best corresponds to observed data.
The major difficulty with this The works of Yin (2003a) and Stake (1995) concerning approach, assuming, of course, that it is possible case study research are especially recommended. Yin to collect the relevant data, is to decide upon how (2003b) provides several illustrative examples of case close is the “fit” of the data to a predicted pattern. studies in the social sciences. Tien (1999) provides a Much researcher interpretation and judgment useful discussion of the evaluation of systems and Adel- may be needed. 2. Explanation Building.
Here, the researcher analyses data by building up an assumed expla- nation about the case. The purpose of this is to analyze the case study data by establishing an explanation about the case. This is usually ac- complished by linking events and issues causally. This approach goes beyond pattern matching in that specification of the nature of the links, usu- ally causal, between elements that make up the pattern is a hoped for resu lt. The researcher then tests the evidence for the relationships. 3. Toulmin Logic Models. Toulmin argumentation.
(Toulmin, Rieke, and Janik, 1984) is based on the notion that the statements that make up an expla- nation fall into six basic types: data or grounds, inference warrants, backing, modal qualifier, re- buttals or reservations, and resulting claims or conclusions. Toulmin logic, the structure of Figure 1. Toulmin logic structure. 88 FRIEDMAN AND SAGE man (1992) discusses evaluation in decision support approaches as aggregate statistics, peer review, and the laws of bibliometrics. Each approach is useful and and expert systems..
The Harvard Business School is among the many development of newer case study techniques has not business schools that emphasize case studies in their resulted in the obsolescence and retirement of the ear- educational program (Barnes, Christensen, and Han- lier approaches. sen, 1994). Shapiro (1988) suggests that the total case In this section, we have discussed the completeness and rigor of case studies descriptions.
However, in process in an academic study environment, based upon applying case studies to systems engineering research the premise that management is primarily a skill-based and education, there is a symmetrical issue: that of activity rather than one based on a collection of tools potential overdescription of the elements and environ- and techniques, is comprised of four steps: ment associated with crucial events of th e systems engineering process. It will often be the minimum es- 1. Individual analysis and preparation of and for the sential set of measurements and data that most clearly case; illuminates the workings of newly investigated phe- 2. Optional informal small group discussions; nomena.
Excessive descriptions and elaborations may 3. Classroom discussions; and tend to cloud the inner workings of the judgmental 4. End of class generalization relative to the learn- process needed for effective case study research. ing accomplished. There are two generic types of case studies, the first of 3. A FRAMEWORK FOR SYSTEMS which is retrospective analysis, which can be further ENGINEERING CASE STUDY RESEARCH categorized as historical descriptions, research events studies, and matched comparisons.
The second type To support the objectives identified in Section 1, we represents a combination of retrospective assessment identify a two dimensional framework for systems en- with prospective approaches and use of such analysis gineering case study research. Figure 2 represents the Figure 2. A framework of key systems engineering concepts and responsibilities. [Color figure can be viewed in the online issue, which is available at www. interscience. wiley. com. ]
CASE STUDIES OF SYSTEMS ENGINEERING AND MANAGEMENT IN SYSTEMS ACQUISITION 89 initially suggested framework. It is constructed as a 9 (family, federation, coalition) that is comprised of a set ? 3 matrix with nine general SE concept areas as rows, of systems each of which are fully fo rmed and func- A–I and three columns, 1–3, depicting the domain of tional, and which possesses such properties as manage- the principal activity: primarily contractor, shared be- rial independence, geographic and/or temporal tween contractor and government, and primarily gov- distribution, emergence, evolution, emergence, and ad- ernment.
This is intended to provide the general aptation. structure o f a framework for systems engineering con- Thus, while the three column structure of “Contrac- cepts and their illustration through related case studies. tor Responsibilities,” “Shared Responsibilities,” and This two dimensional framework is central to the case “Government Responsibilities” is very appropriate; the study efforts suggested here.
specific nature of these will vary greatly depending Six of the nine concept domain areas in Figure 1 upon whether the system being engineered is a mono- represent phases in the systems engineering lifecycle: lithic (stand alone) system or a System of Systems (or Family or Federation of Systems). In a great many A. Requirements Definition and Management contemporary efforts that involve the engineering of a B. Systems Architecting and Conceptual Design system, we are really dealing today with engineering a C. Detailed System and Subsystem Design and Im- System (or Family or Federation or Coalition) of Sys- plementation tems.
When the government (or a private developer, for D. Systems and Interface Integratio that matter) is acquiring a stand-alone system, it may E. Validation and Verification often contract directly with a contractor that offers F. System Deployment and Post Deployment “maximum value added” to the government. But when the government is acquiring a System of Systems, there Three of the nine concept areas represen t necessary are at least three rather different approaches that might process and systems management support: be chosen (Carlock and Fenton, 2001): G. Life Cycle Support a. Option (a) is that of letting a prime SOS engineer- H.
Risk Management ing contract to a single systems engineering (and I. System and Program Management integration) firm. This prime contractor is then responsible for the total systems engineering ef- While other concepts could be identified, we believe fort to bring about the SOS. There are a number that these nine are the most relevant to systems engi- of pitfalls here. For example, the government neering in that they cover the essential life cycle proc- may feel, after contract award, that the prime esses in systems acquisition, and the systems contractor is selecting dismal quality firms as management support of these.
Most other concept areas subcontractors for various systems that comprise that were identified appear to be subsets of one of these. the SOS. As noted, the three columns of this two-dimensional b. Option (b) is that where the government assigns concept framework represents the responsibility do- separate contracts for the engineering of each of main and perspectives of government, and contractor, the component systems in the SOS and then the and shared responsibilities of government and contrac- government acts as the integrator and SOS pro- tor. This could be expanded much further.
For example, gram manager. each of these groups will be concerned with being c. Option (c) is that where the government assig ns responsive to abstractions or contextual awareness in- separate contracts for the engineering of each of terrogatives of why, how, what, who, when, and where. the component systems in the SOS and then also We might also disaggregate the contractor responsibili- contracts with another firm that acts as an inde- ties into nonprofit contractor responsibilities and for- pendent SOS integrator and SOS program man- profit contractor responsibilities.
Our belief, at this ager. point, is that this additional complexity is not warranted as the notion of “contractor” and “government” n eeds Thus, delineation of responsibilities in the three col- natural modification when there are multiple contrac- umns may be expected to vary considerably between tors and multiple government agencies involved. This acquisition of a stand alone system and acquisition of leads naturally into consideration of “system family” an SOS and the program policy and management op- (Sage, 2004) considerations needed to engineer a sys- tions (a, b, c) chosen.
What is contractor responsibility, tem of systems, a family of systems, a federation of shared responsibility, and government responsibility systems, or a coalition of systems.
These expressions depends on which of these three options is chosen. What are in common contemporary use to describe a system this suggests is the great importance of considering the 90 FRIEDMAN AND SAGE three basic responsibility domains in the systems engi- bornly refused to be quantified, program man- neering concept framework suggested here; and devel- agement “incentivized” their determination by oping familiarity with the roles of the two major coupling pay raises with how many TBDs were players, government, and contractor, as well as shared eliminated.
The response by the engineers was to roles, in the contemporary en gineering of systems. “temporarily” eliminate these TBDs by remov- ing the paragraphs in which they were imbedded, thereby creating “silent specs” with the hope to 4.
CASE STUDIES ILLUSTRATING fill them in when the analysis was comp lete. SYSTEMS ENGINEERING CONCEPTS Time passed, and the engineers were transferred It would be highly desirable to write and present here to other programs without warning their succes- one or two case studies, with each illustrating the wealth sors of the silent specs. Some of these specs of information and knowledge that would fill all of the should have been passed on to a key subcontrac- cells in the matrix of Figure 2.
While ultimately this is tor but were not. The result of this was that the possible, we present here, for conceptual purposes, systems-level test failed, the prime contractor simpler studies or vignettes illustrating cell-wise facets lost schedule and over $100M in profit. The of good and bad systems engineering practice. We can program was eventually cancelled. obtain a synthesis of lessons learned from each of th ese cell wise representations and do present this here. From 4. 1. 2.
Shared (A2) experiences gained in doing this and in understanding SE concept: Customer and contractor shall share these, one is in a better position to attempt larger more with one another their knowledge of the state of holistic efforts involving complete cases. technical maturity relative to the new, unprece- What follows is a list comprising at least one systems dented systems being engineered. engineering concept within each of the 27 elements of Case study: A government customer desired to pro- the case study concept framework matrix.
These con- cure two new navigational systems: one stellar cepts are written in the spirit of being requirements for inertial and the other pure inertial. The most good systems engineering; thus they are written as capable contractor in the pure inertial field de- “shall statements. ” Each of the concepts is associated cided to expand the portion of the market sector with at least one illustration or vignette of a situation, under their purview and substantially underbid in this instance drawn from a multitude of different the proposal for the stellar inertial system.
This cases, which would in an actual application be charac- contractor won; so the stellar inertial contractor teristic of a specific case history. Essentially, then, the counter attacked and won the pure inertial con- trajectory of these composite situations across the rows tract. Both contractors, unfamiliar with the new and columns of the matrix would form the skeleton of (to them) technologies to be used in the product a full case history. These are intended for augmentation to be engineered, did poorly both technically and and revision as experience with this effort is gained.
financially. They suffered—as did the cus- Some of these situations are decades old, and the current tomer—from the procurement policy that they focus on evolutionary acquisition and system families could not be negative about the technical aspects should provide even more current and broad scope of contractors who had submitted reputable pro- illustrations. Perhaps this list might ultimately become posals in the past. a set of useful Heuristics for a Body of Knowledge of Systems Engineering Concepts and Practices.
4. 1. 3. Government (A3) SE concept: The government shall integrate the 4. 1. Requirements Management (A) needs of its user organizations with the manage- ment activities of its developmental organiza- 4. 1. 1. Contractor (A1) tions. Often, the user and development SE concept: Requirements shall flow down in a organizations are separated by gulfs of culture coherent and traceable manner from the top level and language. The development organization to all lower levels of the system being engineered.
sponsors new technology and contracts out pro- Case study: The flow-down of requirements was grams; the user organization—the ultimate cli- often slowed by the lack of early lifecycle sys- ent—operates the new systems, trains the tems modeling capabilities on the part of the personnel, fights battles, and should be inti- contractor, so there were many paragraphs with TBDs (“to be determined” elements instead of mately involved in writing the requirements for quantitative specifications). As the TBDs stub- new programs.
CASE STUDIES OF SYSTEMS ENGINEERING AND MANAGEMENT IN SYSTEMS ACQUISITION 91 Case study: An advanced state of the art navigational However, unless the total system arch itecture system was specified for an attack aircraft. Sev- implications are well understood, COTS can be eral years after the program start, the operational extremely disappointing regarding its incom- arm of the government added more requirements plete characterization, testing and maintenance, in the crew station interface area. They claimed and logistic aspects.
Frequently, many upgrade that they had insufficient voice in the construc- generations of software occur during a single tion o f the o riginal specification. Results: sub- government development cycle, and the producer stantial financial and schedule impact. of the software—who view the government as a small customer compared to their commercial 4. 2. Systems Architecture (B) markets—is unwilling to continue supporting the old versions. In many cases, the government 4. 2. 1. Contractor (B1) pendulum of employing excessive specifications SE concept:
The systems baseline architecture of swinging to relying entirely on commercial best complex programs shall be established early in practices has injured the government and indus- every program and shall involve all dimensions try’s ability to manage the integrity of new de- of technical issues, as well as such enterprise signs. architecture issues as customer needs and satis- Additional case study result:
The effectiveness of faction, political pressures and continuity of guided missiles could be greatly improved by funding. A properly executed systems architec-integrating the functions of guidance and fusing; ture activity provides benefits of effectiveness far that is, the information regarding the terminal in excess of its costs. In dealing with system kinematics gathered by the guidance system can families, it is to be anticipated that the systems be transferred to the fusing system, substantially architecture will evolve and emerge from the improving the probability of kill.
However, these baseline architecture as the system is engineered. two systems are generally kept separate by tradi- Case study: If the allocation of functions to equip- tional procurement policies and divisions. ment and crew is performed without deep under- standing of human/machine interactions and 4. 2. 3. Government (B3) their respective strengths and limitations, the SE concept:
A total systems architecture shall be overall systems performance will suffer. In a employed by the government for the reasons particularly stressful case of a combat aircraft stated in 4. 2. 1 in order to provide a sound basis required to perform aggressive maneuvers while of effectiveness across the broadest spectrum of attempting to acquire and classify a low contrast contractors and operations.
target, the task of aircraft controls was delegated Case study: Joint developments in aircraft and mis- to the pilot, and the task of classification was siles have been undertaken across the services in delegated to an automatic pattern recognition order to attain envisioned economies. In most pro gram.
The effectiveness of the system would cases, these have been met with several difficul- have been substantially improved if the maneu- ties in extended schedules, higher costs, compro- vers were under automatic control (supervised bymises in operational effectiveness, and awkward the pilot) and the task of pattern recognition were integration within diverse logistical support sys- given to the far superior cognitive capability of tems. In the early stages of program planning and the crew member.