Beruflich Dokumente
Kultur Dokumente
2006
Soft Systems Approach for Managing Free and Open Source Software Quality
Anas Tawileh, Wendy Ivins and Omer Rana
Abstract
Quality in the free and open source software (F/OSS) community has been an explored area for a long while. This can be attributed to the lack of consensus about the notion of quality among different stakeholders. In this paper, we propose the use of Soft Systems Methodology as a radically different approach to software quality that could accommodate the systemic nature of the quality problem. The application of SSM was combined with UML to design and implement a supporting information system that would facilitate the integration of users' needs and expectations early in the development process.
Introduction
Software development practices have received considerable research attention and many initiatives have been proposed to improve the quality of their resultant artefacts. However, quality within relatively new development paradigms, such as free and open source software development, is still an area of ambiguity and concern. This can be attributed partially to the very nature of F/OSS development practices, which are based on the contribution and distributed collaboration of large numbers of developers from all over the world, and the highly informal information sharing structures of the community. However, we are of the opinion that it is necessary to reassess the very definition of quality associated with software artefacts, especially when viewed with reference to the F/OSS development process. The F/OSS community argue that the software they develop is technically superior to many commercial efforts and that their open and transparent development practices would provide for higher quality levels than those of proprietary software. End users, on the other hand, oppose these claims by describing the lack of appropriate end user participation mechanisms within the F/OSS community, and the consequent lack of regard of user needs and requirements. This issue is not exclusive to F/OSS software, which is evident in the increasing demand for more involvement of users in the development process -- proposed by recently evolving paradigms such as prototyping and extreme programming. However, the particular nature of open interaction and lack of authority in the F/OSS community magnifies the users feeling of alienation in the development process. The apparent lack of any consensus about the notion of quality at large further complicates the situation. In the literature, different definitions have been proposed to explain quality. The International Standards Organisation (ISO, 2002), for instance, defines quality as: "The totality of features and characteristics of a product or service that bear on its ability to satisfy specified or implied needs."
-1-
Soft Systems Approach for Managing Free and Open Source Software Quality
Another definition of quality is proposed by Juran as being "fitness for user" (Juran, 1979), and Crosby considers quality to be "conformance to requirement" (Crosby, 1979). Placing quality in the context of F/OSS software development provides even more difficulty to developing a coherent definition. As software has mixed characteristics of both a product and a service, and being developed in an open environment, the expectations and perspectives of different stakeholders are more susceptible to conflicts and collisions. In addition, the lack of end user involvement makes it difficult to claim quality in reference to requirements such as the definition proposed by ISO. It was suggested that in the context of software development, the process of defining quality is as problematic as the management of quality (Vidgen et al., 1993). A systemic approach to enhancing quality in the F/OSS community is proposed. We suggest that by investigating the systemic mechanisms underlying software quality, the perspectives and expectations of different stakeholders may be accommodated, and a set of mechanisms could be devised and implemented to i prove the problematic situation of software quality in an open environment. The m paper aims to test the validity of systems thinking ideas, and the Soft Systems Methodology (Checkland, 1999) in particular, in tackling ill defined and complex problem situations, and which contain humans with differing perspectives and free will. The use of Soft Systems Methodology was supplemented by other modelling techniques such as UML to compensate for the limitations in the methodology in developing an eventual solution. This approach also demonstrates the potential gain of combining both techniques (Bustard et al., 1999), along with mechanisms to evaluate the proposed approach.
Background
Development practices within the F/OSS community are fundamentally different from those adopted by the proprietary software industry. The main principle underlying the F/OSS movement is the freedom of the end user to access the software source code, modify it and redistribute it for third party use, as long as s/he respects the terms and conditions of the original license of the software thereby guaranteeing the transfer of similar modification privileges to any subsequent users. These principles encourage interested developers to engage in developing, promoting and initiating F/OSS software projects. The high levels of collaboration and communication required to tackle the complexity of software development projects, coupled with similar ideals of the participating volunteers, has resulted in the formation of the largest virtual community in human history. This community has several distinct characteristics; it contains different schools of thought, ranging from pure idealists to extreme pragmatists. It spans geographical and cultural boundaries more than any other human community or business enterprise. However, the benefits associated with this diversity come at a price. While the F/OSS community has developed novel mechanisms to effectively tackle coordination and collaboration difficulties resulting from its open, informal and highly distributed nature; some areas were, perhaps unintentionally, not given sufficient consideration. Members of the F/OSS community usually claim credit for producing, in a very informal and decentralised process, software that outperforms centrally coordinated, heavily financed counterparts. They believe that the evolved open and transparent development practices constitute a sufficient guarantee of the quality levels attainable by the community (Raymond, 2000). However, users of F/OSS software do not seem to agree much with such claims. They believe that the claimed open access to the community is restricted by barriers of difficult language and technical jargon. Users do
-2-
-3-
Soft Systems Approach for Managing Free and Open Source Software Quality Modelling
SSM suggests an intellectual construct called "Root Definition" (RD), which aims to provide a clear and unambiguous textual definition of the system to be modelled. It provides a way to capture the essence (root) of the purpose to be served (Checkland, 1999). Different root definitions could be developed to accommodate the different viewpoints or activities of the system. The following two root definitions were developed to capture the essence of the software quality situation from two different perspectives: end users and developers:
RD-1: A system to provide guarantees acceptable by end-users that developed artefacts within the F/OSS community will meet users' expectations by deriving appropriate mechanisms to assess these artefacts and produce such guarantees, while considering the resource constraints inherent in the F/OSS community.
RD-2: A system to ensure that artefacts developed within the F/OSS community will meet determined quality criteria by deriving mechanisms to evaluate and approve submitted contributions to these artefacts against the acceptance criteria.
Based upon the formulated root definitions, the conceptual model that represents the system described in these root definitions was developed. The conceptual model illustrates any required activities present within such systems, in addition to any logical relationships between these activities. These relationships represent logical dependencies between activities, and would allow one to determine the inputs and outputs of each activity. The resulting conceptual model is reproduced in Figure 1. One weakness of the Soft Systems Methodology is that although it structures highly complex and messy problems and provides valuable aid in investigating possible intervention in situations involving humans, it only provides insight into what should be done to improve the problematic situation. SSM does not provide any guidance on how to build the required support system or implement the proposed solutions. In order to describe the proposed solution in such a way that facilitates its implementation, different tools and techniques should be used in conjunction with the artefacts of SSM. Several studies have attempted to bridge this gap, including the work of Bustard, Zhonglin & Wilkie (1999) and Wade & Hopkins (2002) who argue that SSM lacks the detailed information required to develop the intended systems, and that the use of the Unified Modelling Language (UML) would compensate for this weakness and provide the level of detail required to facilitate the design and implementation of the desired systems. Mingers proposed embedding structured methods into SSM (Mingers, 1992). Two primary processes could be identified within the conceptual model, the boundaries of each process are illustrated in Figure 1: 1. The F/OSS development process. 2. The guarantees' provision process.
-4-
Each process was identified by grouping appropriate activities relevant to the achievement of the process' goals. Links between activities determine the logical progression of action and the required resources and expected results for the achievement of each activity. To enable implementation of any proposed interventions that resulted from the SSM analysis, it was decided that UML should be used. This was undertaken in order to create models that describe the proposed solutions in sufficient detail to enable subsequent design and implementation of the supporting information system. Using UML's standardised notation assures the clarity of representation and ease of communicating the designed systems. The synergy of complementing SSM and UML will enhance the final outcome, and provide a logically defensible development route from high level system analysis down into specific system implementation details (Wade and Hopkins, 2002) (Bustard et al., 1999). The UML activity diagram for each process can be developed by mapping each activity within the process boundary in the conceptual model (Figure 1) to an activity in the UML diagram using its standardised notation. Flows in the UML diagram should match the dependencies represented in the conceptual model and the logical sequence of activities. The activity diagram also illustrates the actor who should perform each activity in addition to the resource requirements and outputs generated by each activity. Using UML modelling makes actors and outputs more explicit and therefore provides the
-5-
Soft Systems Approach for Managing Free and Open Source Software Quality
basis for the design of the supporting information system. Figure 2 shows the UML activity diagram developed for the guarantees' provision process.
Atkinson (2000) proposes the SISTeM approach for informing debate about the development of an information system focusing on human/machine activities. The approach is based on the use of two "cycles": (i) the first deals with the overall strategic decision making; (ii) the second deals with translating some of these strategic decisions into operational actions that lead to some change within an organisation. The SISTeM approach is more comprehensive than SSM, taking account of social, political and market impacts on a problem situation, and involves the use of multipe forms of models: conceptual, expressive and matrix. As a consequence, the approach is also more complex to use, requiring a much more significant involvment of the actors and their understanding of the context in which they are evaluating a particular problem situation. The approach is therefore useful for understanding the likely impact an information system will have on an organisation, and the subsequent debate that is necessary to fully appreciate how the information system impacts a particular actor and his/her role. However, the usefulness of the approach to realise an information system is limited in scope. The second cycle in SISTeM, which focuses on particular operational decision making, also does not fully indicate the set of activities that are required to implement the eventual information system. This aspect of the final realisation of the information system is still open to the interpretation of a developer. The SISTeM approach therefore provides a useful starting point to better evaluate requirements and relate these to the wider context within which a F/OSS software may be deployed. Its usefulness as a means to translate these requirements into an information system is limited in scope.
q q
Does the activity currently exist in reality? Is the activity in the real world done the way it should be done?
-6-
This mapping leads to recommendations for possible actions that could be implemented by the F/OSS community to improve the uptake of F/OSS artefacts. The assessment of activities in the conceptual model in order to derive any relevant recommendations is presented in Table 1.
Activity
By Whom
Outputs
Process
Inputs
Judgement
Recommendations
End User
No formal exists
system
Implement measures to collect user's expectations Improve current requirements determination practices to consider user's expectations Implement measures to determine guarantees to be provided to customers based on their expectations and the artefact's requirements Implement measures to obtain user's acceptance of determined guarantees Improve the process of deriving mechanisms to produce guarantees to develop mechanisms that would produce acceptable guarantees for users Implement measures to enable users to assess different mechanisms for guarantees production Implement measures to enable users to select the appropriate guarantee production mechanism Implement measures to assess developed artefacts against the selected mechanisms Implement mechanisms guarantees selected produce
Partially
Maintainer
No
Maintainer
No formal exists
system
No
End User
No formal exists
system
to
Partially
M aintainer
No
End User
No formal exists
system
Select mechanism
appropriate
No
End User
No formal exists
system
against
No
Maintainer
No formal exists
system
Produce guarantees
No
System
No formal exists
system
to
-7-
Soft Systems Approach for Managing Free and Open Source Software Quality
Analysis of the conceptual model indicates a significant lack of end user involvement within the current development processes within the F/OSS community. In order to improve the quality levels of the F/OSS artefacts, end user requirements and expectations should be considered from the early stages of the project development process. Quality may be perceived differently by different parties (Michlmayr, 2005), and our proposed approach suggests a systematic method to consider and integrate the different perspectives of developers and end users. Software development projects within the F/OSS community are initiated by a developer or a group of developers to fulfil their own personal requirements. This contrasts with the traditional software development process which is mainly initiated to satisfy specific customers' requirements. The significant share of server side applications within the F/OSS community in comparison to desktop and end user applications is a direct consequence of this characteristic. However, the different advantages provided by the F/OSS software stimulated the wide adoption of these applications, which in turn increased the size of both its users' and developers' communities. F/OSS community members were urged to adapt to the new situation by trying to involve the end users more in their software development process in order to meet their specific requirements and expectations. Nevertheless, the evolutionary nature of this change prevented the development of clear, systematic procedures to involve users and incorporate their needs and expectations. Many approaches were based on simple, unstructured web based communication and collaboration techniques such as mailing lists and discussion forums. Each developer, after initiating his project, will try to define the expectations from the project. In the current tradition within the F/OSS community, the developer usually determines the project's direction based merely on his own expectations and needs. The process suggested by the conceptual model analysis recommends implementing measures to include user's expectations in the early stages of the project development. Upon determining the maintainer's and user's expectations, the maintainer will try to decide whether the user's expectations are inline with his own. At this point, the maintainer may make some compromises on his own expectations in order to align them with those of the users or if there are substantial differences between the two, the maintainer may not alter his expectations, and the user might look for another project that better matches his expectations. After a balance is struck between the maintainer's and user's expectations, the maintainer will translate those expectations into detailed requirements. In the current processes, the maintainer translates his own expectations into requirements, without considering those of the user. This process will be improved by considering the user's expectations, in accordance with the findings of the process analysis. Requirements are descriptions of "required features of the proposed system". They may be functional (describing what the system should do) or non-functional (describing how a facility should be provided, or how well, or to what level of quality) (BSI, 1994). F/OSS projects usually lack an explicitly formulated requirements specification (Bartkowiak, 2005), because most of the projects depend upon the evolutionary development of requirements. Determining project's requirements does not conflict with this concept, and the recommendation simply suggests formulating the translated expectations in clearly defined requirements that the developed project should meet. The initial formulation may capture only the essential required functionality, and may advance through the same current evolutionary development mechanisms. For the developer to assure the users that the artefact she is developing matches their initial expectations, she has to devise and suggest a set of guarantees that would serve this function. Users
-8-
-9-
Soft Systems Approach for Managing Free and Open Source Software Quality
In the case of software, the guarantees to be provided are d efined as a set of functional and nonfunctional test results that should satisfy the established acceptance criteria. Therefore, the mechanism to produce these guarantees should provide appropriate test results as its output. If the provided test results satisfy the acceptance criteria, then the software has met the users' expectations, and those results could be used to guarantee this claim. Transparency and openness throughout the process will enable users to verify the guarantees by themselves, which will increase their level of confidence in the software under consideration and in the development process as a whole. Deriving appropriate mechanisms for this scenario then melts down into the selection of possible testing techniques that could provide the intended test results in a transparent and repeatable manner. Selection of appropriate testing techniques largely depends upon the implementation technology (programming language, operating system, etc) and the availability of testing tools for that partic ular technology. The selected tool should be able to perform tests that would generate results which can be validated against the approved acceptance criteria. The final activity in the process as illustrated by the SSM conceptual model is the assessment of the developed artefact against the selected validation mechanisms in order to ensure the proper functionality of the artefact and to produce guarantees that assure the end user of the conformance of the artefact to his expectations. If the project maintainer validates the artefact against the selected mechanism and the artefact passes the proposed tests, she can release the artefact with greater confidence that it will attract better adoption by end users. Test results and the acceptance criteria should be bundled with the released artefact to enable the authentication of the guarantees by any potential user and to ensure accurate linking between different releases and their associated guarantees. Although project maintainers can tweak their own projects' collaboration platforms to perform the proposed activities and processes, this will result in different implementations of the same concepts which might cause confusion to users. Building a central platform that provides these services in a standardised way and which project maintainers could use to host information about their projects would eliminate this confusion. This approach also offers the advantage of comparing different projects against each other to select the most suitable option. A simple prototype was developed to demonstrate the applicability of the proposed interventions and the functionality of the supporting information system that could support the implementation of the recommended courses of action. The prototype acts as an integrated pla tform to allow communication between developers and end users at the different stages of the project development lifecycle, and an easy and convenient way to communicate test results and quality information. It could be improved by integrating test automation into the system's interface. Results produced by the assessment methodologies proposed by previous initiatives can be supplemented by the results stored within the system to provide for better and more extensive evaluation of F/OSS artefacts.
Conclusions
Quality in the free and open source (F/OSS) community is a sensitive issue that has been the subject of much debate. Although the F/OSS community has proved to be highly effective in managing massively complex distributed development efforts, it has not provided proper consideration to various quality aspects of its development processes. The main issue of the quality problem lies in the lack of consensus on what software quality actually is, and whose interpretation of quality should be considered. Different initiatives were proposed to evaluate and measure quality in an open source
- 10 -
Acknowledgement
We would like to thank Steve McIntosh for his suggestions on the use of SSM in this work.
- 11 -
Soft Systems Approach for Managing Free and Open Source Software Quality References
Al-Humaidan, F. and Rossiter, N. (2004), 'Business Process Modelling with OBPM: Combining Soft and Hard Approaches', Workshop on Computer Supported Activity Coordination (CSAC), 6th International Conference on Enterprise Information Systems, Porto, 13-14 April, 2004, pp 253260. Atkinson C. J., "The Soft Information Systems and Technology Methodology (SISTEM): A Contingency Approach to Integrated Decision Making and Development", pp 71-76, Proceedings of First International Conference on Systems Thinking in Management, 2000. Bach, J. (1996), 'The Challenge of "Good Enough" Software', http://www.di.ufpe.br/~hermano/cursos/calcprog/good-enough-software.htm, ST Labs, (accessed April 7, 2006). Bartkowiak, I. (2005), 'Quality Assurance of Open Source Projects', Seminar - Open Source Software Engineering, Wintersemester 2005. BS 7738-1:1994 (1994), 'Specification for information systems products using SSADM (Structured Systems Analysis and Design Method). Implementation of SSADM version 4', British Standards Institute. Bustard, D. W., He, Z., and Wilkie, F. G. (1999). 'Soft Systems and Use-Case Modelling: Mutually Supportive or Mutually Exclusive?' Proceedings of the Thirty-Second Annual Hawaii international Conference on System Sciences, Volume 3 (January 05 - 08, 1999). HICSS. IEEE Computer Society, Washington, DC, 3055. Checkland, P. (1999) Systems Thinking, Systems Practice. Wiley, UK. Crosby P.B., (1979), Quality is Free: The Art of Making Quality Certain. McGraw-Hill. Duijnhouwer, F. and Widdows, C. (2003), 'Capgemini Expert Letter Open Source Maturity Model', Capgemini, http://www.seriouslyopen.org/nuke/html/modules/Downloads/htmldocs/osmm1.html, (accessed April 7, 2006). Juran, J. (1979), Quality Control Handbook, 3rd edition. McGraw-Hill. International Standards Organisation,(2002), http://www.iso.ch/iso/en/ISOOnline.frontpage, (accessed April 7, 2006). Michlmayr, M., Hunt, F. and Probert, D. (2005) "Quality Practices and Problems in Free Software Projects" in Proceedings of the First International Conference on Open Source Systems. Genova, Italy, 2005, pp 24-28. Mingers, J. (1992), 'SSM and Information Systems: An Overview', Systemist, Vol 14 (3), August 1992. Raymond, E. S. (2000), The Cathedral and the Bazaar, http://www.catb.org/~esr/writings/cathedralbazaar/cathedral-bazaar/, (accessed April 7, 2006). OpenBRR.org (2005), Business Readiness Rating for Open Source, http://www.openbrr.org/docs/BRR_whitepaper_2005RFC1.pdf, (accessed April 7, 2006). Open Source Maturity Model (2003), http://www.seriouslyopen.org/, (accessed April 7, 2006). Vidgen, R., Wood-Harper, T., and Wood, R. (1993). 'A Soft Systems Approach to Information Systems Quality'. Scand. J. Inf. Syst. 5 (Aug. 1993), pp 97-112. Wade, S. and Hopkins, J. (2002), 'A Framework for Incorporating Systems Thinking into Object Oriented Design', Seventh CAiSE/IFIP8.1 International Workshop on Evaluation of Modeling Methods in Systems Analysis and Design (EMMSAD02), Toronto, Canada, 27-28 May, 2002. Wilson, B. (1984): Systems Concepts, Methodologies, and Applications. Wiley, UK.
- 12 -
- 13 -