Master's Thesis - Akash Manjunath

Decision-making and Cognitive Biases in Designing Software Architectures

Introduction

This thesis explores the different decision-making processes relevant from the context of designing software architectures and relates the processes with the OODA loop decision cycle. Additionally, it explores which cogntive biases influence the software architects when making architectural decisions. 

Problem diagnosis and relevance of the thesis

Designing software systems involves continuous decision-making and is an iterative process. Different decision-making models (DMMs) can concretely represent this process. Every DMM is comprised of a series of steps specifying the course of actions taken to reach a decision. Furthermore, each DMM has its own set of advantages and limitations. The limitations of the DMMs adversely affect quality of the resultant software.

This thesis focuses on one specific limitation when making software architecture design decisions(SADD) – the cognitive limitation of software architects leading to the designing of sub-par software architectures.

The cognitive limitation is due to the limited capacity of the human brain in dealing with complexity and manifests itself in the form of cognitive biases. By definition, a cognitive bias is a systematic pattern of deviation from rationality in judgement. Cognitive biases limit objective reasoning resulting in biased decision-making. Thus, there is a need to avoid cognitive biases or at least reduce their impact to make quality SADD.

Core Concepts

OODA Loop

It is a popular decision-making tool originated from a military background and presently used by decision-makers in other fields as well.

Decision-making processes

The process of decision-making can be formally represented by using different models. Three decision-making models are examined in detail during the course of the thesis - : the rational economic model, the bounded rational model and the recognition-primed decision(RPD) model (naturalistic decision-making framework).

Each of the three DMMs is presented as a sequence of steps through separate process models and are the three horizontals in the skeleton matrix respectively as shown in the following figure :

Cognitive Biases

The third and final area of research is on cognitive biases and its different variations. A list of cognitive biases is created by consolidating information from previous research. From an aggregated list of over two hundred types of cognitive biases, thirty-three were chosen as relevant from the context of making architectural design decisions.The biases are then classified in two levels. The first level classifies the cognitive biases under one or more of the observe, orient, decide or act phases. In the second level, a custom classification under each phase of the OODA Loop is made to further enhance understandability through increased modularity of information. The results are presented in the form of a cognitive bias catalogue.

Research questions

The following research questions are addressed in this thesis:

1. Which decision-making models are relevant in the context of making software architecture design decisions?
2. What is the relationship between the decision-making models and the OODA loop?
3. Which cognitive biases influence software architects when designing architectures?

Preview on what the readers will find in this Master´s Thesis

The following items can be found in this thesis: A catalog of biases commonly found in decisions related to software architecture along with associated examples

Documentation and Short Paper

After completing the above analysis, the thesis documentation will contain the write up of the completed work. In addition to the thesis document, a short paper will be submitted for the ICSA and ECSA conferences.

Target Group

The research conducted in this thesis is focused towards the community of software architects. However, decision-making is a continuous process and often other actors such as software developers, testers, product owners and others are involved in making decisions which impact the quality of a software. Thus, the end results of the thesis could potentially be beneficial to all those actors involved in the SADD making process.

Thesis Contents

I.    Acknowledgment
II.    Abstract
1    Introduction
1.1    Problem Statement
1.2    Core Concepts
1.3    Research Goals
1.4    Research Questions
1.5    Target Group
1.6    Using the Thesis Artifacts
2    Related Work
2.1    OODA Loop
2.1.1    The need for understanding the OODA Loop
2.1.2    Observe Phase
2.1.3    Orient Phase
2.1.4    Decide Phase
2.1.5    Act Phase
2.1.6    Setting the Tempo
2.1.7    Conclusion
2.2    Decision-making
2.2.1    Modeling the Decision-making Process
2.2.2    Normative Approach
2.2.2.1    Rational Economic Model
2.2.2.2    Brunwick’s Lens Model
2.2.2.3    The Cynefin Framework
2.2.3    Behavioral Approach
2.2.3.1    Incrementalism
2.2.3.2    Naturalistic Decision-making
2.2.3.3    Bounded Rationality
2.2.4    Cognitive Biases
2.2.4.1    Types of Cognitive Biases
3    Thesis Contribution
3.1    OODA Loop and DMMs
3.2    Selection Criteria for DMMs
3.3    Rational Economic Model
3.4    Recognition-Primed Decision Mode
3.5    Bounded Rational Model
3.6    Cognitive Biases in the context of SADD
3.7    Classification of Cognitive Biases
3.8    Cognitive Bias Catalogue Template
3.9    Cognitive Bias Catalogue
4    Results and Feedback
4.1    Reflections on Research Questions
4.2    Using the Thesis Artifacts
4.2.1    Understanding the DMMs
4.2.2    Using the Cognitive Bias Catalogue
4.3    Evaluation through Expert Feedback
5    Future Work and Conclusion
5.1    Future Work
5.2    Conclusion
III.    Bibliography
IV.    Appendix I List of Cognitive Biases
V.    Appendix II Feedback Form Template
VI.    Abbreviations

Bibliography

[1] “A Cognitive bias cheat sheet – Better Humans”. Better Humans, 2016-09-01.
[2] Arnott, D. (2006), Cognitive biases and decision support systems development: a design science approach. Information Systems Journal, 16: 55–78. doi:10.1111/j.1365-2575.2006.00208.x
[3] Arnott, D. (1998). A taxonomy of decision biases. Monash University, School of Information Management and Systems, Caulfield.
[4] A. Jansen and J. Bosch, "Software Architecture as a Set of Architectural Design Decisions," 5th Working IEEE/IFIP Conference on Software Architecture (WICSA'05), Pittsburgh, PA, USA, 2005, pp. 109-120.
[5] Barros, Gustavo. (2010). Herbert A. Simon and the concept of rationality: boundaries and procedures. Brazilian Journal of Political Economy, 30(3), 455-472.
[6] Benson, Buster, "Cognitive bias cheat sheet", Better Humans, Sep 1.2016.
[7] Burge, Janet E. Rationale-Based Software Engineering. Springer, 2010.
[8] Crowder, M. (2013). Decision-making: Two schools of thought or one? In V. Sheppard (Ed), Proceedings of the Salford Postgraduate Annual Research Conference (SPARC) 2012 (pp 47-63). Salford: University of Salford.
[9] C. Zannier, F. Maurer, "Social Factors Relevant to Capturing Design Decisions", In Proceedings of the Second Workshop on Sharing and Reusing Architectural Knowledge (SHARK-ADI07), 2007.
[10] David G. Ullman. “OO-OO-OO!” the sound of a broken OODA loop. Crosstalk,April 2007.
[11] Davide Falessi, Giovanni Canton, Rick Kazman, Philippe Kruchten, Decision-making techniques for software architecture design: A comparative survey, ACM Computing Surveys (CSUR), v.43 n.4, p.1-28, October 2011.
[12] Durdik, Zoya. Architectural Design Decision Documentation through Reuse of Design Patterns. KIT Scientific Publishing, 2016.
[13] G. Pedraza-García, H. Astudillo and D. Correal, "Modeling Software Architecture Process with a Decision-Making Approach," 2014 33rd International Conference of the Chilean Computer Science Society (SCCC), Talca, 2014, pp. 1-6.
[14] Hammond, Grant Tedrick. The Mind of War: John Boyd and American Security. Smithsonian Books, 2012.
[15] Hans van Vliet , Antony Tang, Decision making in software architecture, Journal of Systems and Software, v.117 n.C, p.638-644, July 2016  
[16] Kahneman, D; Tversky, (1972). “Subjective probability: A judgement of representativeness”. Cognitibe Psychology/
[17] “List of Cognitive Biases”, n.d.
[18] McKay, Brett & Kate. “OODA Loop: A Comprehensive Guide.” The Art of Manliness, 8 Jan. 2018, www.artofmanliness.com/2014/09/15/ooda-loop/.
[19] Munier, B., Selten, R., Bouyssou, D. et al. Marketing Letters (1999) 10: 233.
[20] Osinga, Frans P. B. Science, Strategy and War: the Strategic Theory of John Boyd. Routledge, 2007.
[21] Stephen T. Hassard, Ann Blandford, Anna L. Cox, Analogies in design decision-making, Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology, p.140-148, September 01-05, 2009, Cambridge, United Kingdom
[22] S. Bhukya and S. Pabboju, "Software architecture central concern, key decisions," 2016 International Conference on Electrical,k Electronics, and Optimization Techniques (ICEEOT), Chennai, 2016, pp. 2451-2457.
[23] S. Dasanayake, J. Markkula, S. Aaramaa and M. Oivo, "Software Architecture Decision-Making Practices and Challenges: An Industrial Case Study," 2015 24th Australasian Software Engineering Conference, Adelaide, SA, 2015, pp. 88-97.
[24] Tang, Antony and Man F. Lau. 2014. Software architecture review by association. J. Syst. Softw. 88, C (February 2014), 87-101.
[25] Tang, Antony, M. Razavian, B. Paech and T. M. Hesse, "Human Aspects in Software Architecture Decision Making: A Literature Review," 2017 IEEE International Conference on Software Architecture (ICSA), Gothenburg, 2017, pp. 107-116.
[26] Tang, Antony. "Software designers, are you biased?" Proceedings of the 6th International Workshop on Sharing and Reusing Architectural Knowledge. ACM, 2011.
[27] Tang. Antony, M. Razavian, B. Paech and T. M. Hesse, "Human Aspects in Software Architecture Decision Making: A Literature Review," 2017 IEEE International Conference on Software Architecture (ICSA), Gothenburg, Sweden, 2017, pp. 107-116.
[28] T. D. LaToza, E. Shabani and A. van der Hoek, "A study of architectural decision practices," 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), San Francisco, CA, 2013, pp. 77-80.
[29] van der Ven J.S., Jansen A.G.J., Nijhuis J.A.G., Bosch J. (2006) Design Decisions: The Bridge between Rationale and Architecture. In: Dutoit A.H., McCall R., Mistrík I., Paech B. (eds) Rationale Management in Software Engineering. Springer, Berlin, Heidelberg
[30] Zalewski Andrzej, Ratkowski Andrzej, On Cognitive Biases in Architecture Decision Making, 2017.