Handling of Unknown Unknowns in Risk Management Essay

Introduction

The concept of unknown unknowns has been gaining increasing importance within both scholarly literature and empirical efforts on risk management within the recent decade. In essence, unknown unknowns refer to the situation when certain types of risks that are not readily predicted within the project management plans and therefore represent a fraction of risk which cannot be effectively quantified.

For more clarity in conceptualizing unknown unknowns, it is important to mention that this term implies unknowns unknown specifically by the one or a group of decision-makers of a project, as the same risks can be identifiable by either outsiders or some other stakeholders. In the recent news, we can see the impact of unknown unknowns on a wide variety of topics –from the detrimental effects of hurricanes and the events on stock exchanges to political election results.

In each of these cases, one single unidentifiable event can challenge the well-being of countries and regions and undermine the course of trends which are commonly considered to be determinate. One of the good examples of the unknown unknowns is the 2014 plane crash in Brazil which killed one of the most probable presidential election winners of the time Eduardo Campos (Bonato, 2014). As a result, the local financial markets suffered a temporary downturn, and the level of political volatility was increased. This specific unknown might also be having an impact on the course of political and economic development of Brazil in the subsequent years.

While the majority of the existing models are not reliable in handling unknown unknowns, there exist methodological solutions suitable for finding them out in specific situations. These methods and tools are mostly project-specific and should be therefore applied to the risk management process on an ad hoc basis, taking into account the operating environment of the project and each of its elements.

The structure of the subsequent paper will be as follows. First of all, some of the traditional models of risk assessment and management will be explored to set the context for further discussion. This will help to see how the case of dealing with quantifiable risks is different from the handling of unknown unknowns. Secondly, the concept of the unknown unknowns will be explored in details, with a further section explaining the key two types of such risks – the knowable and unknowable unknown unknowns. Then, a short section will highlight how the majority of the traditional models are not suitable for identifying unknown unknowns.

The largest sixths section of the paper will be concentrated on the variety of methods and tools from the literature which were found useful in the process of identification of unknown unknowns. The paragraphs covering some theoretical papers will be followed by two sections of case studies – one from the past and one ongoing, together with the attempts to see these two risk management challenges through the lenses of the methods suggested in theoretical papers. Lastly, the discussion will be finalized with the reflections on the topic contained in the conclusion section.

Key Types and Characteristics of the Traditional Risk Management Models

The set of the prevalent traditional risk management models are usually relying on specific assumptions aimed at estimating the riskiness of projects basing on the input characteristics of impact, timing, associated probabilities and alike (Muriana & Vizzini 2017). In many cases, these assumptions are built upon the data obtained from past events and statistical analysis of historical trends.

As noted by Raghunath, Devi, and Patro (2018), in most general terms risk in its traditional sense can be defined as the chances for variation associated with the different states of the anticipated outcomes. Exposure to risk can also be alternatively conceptualized as the probability of the expected loss or gain multiplied by the strength of its effect on the considered outcomes. For the project management context, it is usually relevant to not eliminate all risk, but rather make sure that the level of inevitable risks associated with the project plans is kept at the pre-defined acceptable level.

One of the traditional models of risk assessment is the model of Value at Risk (VAR). This model is commonly used to get a precise measurement of the levels of risk exposure attributable to some considered projects. The Value at Risk methodology can be applied to the project as a whole, as well as to its various structural elements. The fundamental logic of VAR is in determining the amounts of potential losses associated with a certain project based on the assessment of the estimated loss amount and the probability of its occurrence. The output of the model of Value at Risk indicated the probability that a certain risk amount exceeds a pre-defined level within a specific period.

One of the practical applications of using the Value at Risk model in project management is described in an article by Kang, Batta, and Kwon (2014). The authors focus on modelling the framework of VAR application to the transportation of hazardous material. Given that the model suggested by the authors delivered different optimal transportation paths on different assumed confidence levels, it was concluded that the VAR solution is a function of the level of risk tolerance of the managers of the considered project.

In their recent survey of the risk assessment and management models, Ghaleb et al. (2014) note the frequent use of such types of frameworks as the Bayesian Belief Networks, risk assessment based on project metrics and risk management based on classification and prioritization. Risk assessment based on software metrics suggests constructing a risk management framework basing on the pre-defined questionnaire given over to experts in a certain field or industry who are specifically in the sphere of risk assessment. Collecting and analyzing the data from collected in this way can help to construct a set of project metrics which should be closely tracked.

Risk assessment models based on classification and prioritization attempt to classify various risk-related events depending on the degree to which they are mutually dependent or independent. As for the Bayesian Belief Networks, this is an artificial intelligence-based framework which aims to model the set of existing probabilistic relationships among certain variables basing on the past probabilities of the project outcomes. This type of technique is traditionally applied to solving some of the most complex risk management problems.

The Concept of Unknown Unknowns

Some of the major challenges for modern risk managers and project managers are related to handling the risks and uncertainties of certain projects which generate a set of outcomes that can either not be defined or can be defined only in an imprecise way. These types of undefinable uncertainties can have one or more positive or negative impacts on the goals of the developed project. In this context, beneficial uncertainties of the kind can become a source of opportunities, while the detrimental ones can serve as the sources of undesirable risks.

As noted by Ramasesh and Browning (2014), unknown unknowns can be generally defined as the “unrecognized uncertainties” which the project managers are unable to predict. Whenever there occurs an unexpected and surprising turn of the project (usually with detrimental or catastrophic consequences), they are being categorized as an unknown unknown. However, it is also important to note that once this type of negative outcome was predicted from the beginning but considered too unlikely or costly to mitigate, the event cannot be theoretically interpreted as an unknown anymore.

Given that by definition unknown unknowns are the set of unanticipated outcomes in risk management, they are frequently connected to the spheres of project safety, project reliability and the probability of accidents. This is specifically relevant for the example of hazardous industries, which have been theorized in the literature through the influential sociological constructs such as the theory of high-reliability organizations (HRO) and the normal accident theory (NAT) (Gong et al. 2014).

Within the context of the unknown unknowns, whenever the characteristics of a high-reliability organization are present within a company, the probability of encountering unknown unknowns can be reduced. As for the influence of NAT, a certain rate of unknown unknowns is considered inevitability within hazardous systems (such as power plants). This happens due to the high degree of organizational complexity within such systems and the fact that they are more often than not tightly coupled. According to NAT, by accepting the wide use of technology in the modern world, the global societies are granting their simultaneous consent to the occurrence of unknown unknowns which can be at times detrimental or even catastrophic.

In considering the occurrence of unknown unknowns, it is also essential to take into account some of the factors which are particularly stimulating for the emergence of this phenomenon. The two key environment characteristics that are correlated to a higher probability of unknown unknowns are the equivocality and dynamism. They, in turn, contribute to the four major drivers of unknown unknowns within modern projects – mindlessness, complicatedness, complexity and project pathologies (Ramasesh & Browning 2014).

In this context, mindlessness can be interpreted as the set of behavioural characteristics attributable to the individual level factors of various stakeholders of a project, such as company employees, top management, project recipients or local communities. These behavioural characteristics can be arising from the propensity to miss weak signals, purposeful ignorance, prejudiced mindset, and excessive psychological intensity. The complexity of the project, in turn, can be caused by the excessive numbers of elements, increased internal complexity of every separate element or the increased variety and complicatedness of the relationships between the elements of a project.

Complicatedness of a project can be caused by the low number of standardized inter-element interactions, lack of observer abilities of the project management and the lack of intuitiveness within the organization of an infrastructure of a project. Lastly, the factors stimulating the occurrence of product pathologies can be the existence of mismatched infrastructural elements or subsystems of a project, prevalence of fragmented expertise among the specialists and managers, the prevalence of vague expectations of stockholders and the existence of dysfunctional operational and risk management cultures. In case of each of these four main factors is present extensively in various elements of a project’s infrastructure, close attention should be dedicated to the unknown unknowns occurrence potential of such projects.

Even though the cost of prevention and identification measures can be frequently high, the degree of avoided loss can also be very significant and ever critical for the case of complex or hazardous industries. However, close monitoring of the factors of complexity, complicatedness, mindlessness and project pathologies of projects can prove to be a beneficial strategy due to its ability to identify the weak and early signals of unknown unknowns.

The positive takeout of the concept of unknown unknowns is that even though these factors or events are currently not predictable, the knowledge of their potential existence can improve the quality and precision of the utilized risk management tools. This happens because the very fact that something is an unknown today doesn’t mean that it will stay in this status in the future. This type of logics stimulates further research and investigation and increases the transparency of sets of factors influencing any project’s long-term strategic success. The following section will shed some light on the differences between the unknown unknowns that can be investigated and those which are out of the scope of reach of the risk management analytics.

Knowable and Unknowable Unknown Unknowns

To get a better understanding of the concept of the unknown unknowns, it is also essential to distinguish between the two key types of this phenomenon – the knowable unknown unknowns and the unknowable unknown unknowns. As defined by Ramasesh and Browning (2014), the unknowable unknown unknowns are the types of situations or circumstances which cannot in any way be anticipated by a risk manager or a project manager.

As a result, there is no actionable plan or algorithm which can help the manager to make a transition from the unknown unknowns to known unknowns. Some of the examples of such events are the natural catastrophes which cannot be envisaged with the help of modern technologies. The authors specifically mention the case of a tsunami which occurred on the territory of the Indian Ocean in 2004 and had an extremely negative influence on a large number of local projects of the time.

For instance, some of its negative outcomes impacted construction and infrastructural projects in Southeast Asia and specifically the countries such as Thailand, India, and Indonesia. As it can be clear from the context, the risk managers of these construction projects had no way of preparing to such extreme adverse events or have a reasonable plan to counter their consequences.

In contrast to the unknowable unknown unknowns, knowable unknown unknowns can be theoretically predicted by risk managers of enterprises theoretically, but are for some reason not envisaged in practice (Ramasesh & Browning 2014).

One of the possible reasons for this can be the existence of barriers to a cognition of the individuals in charge of the company’s decision making. The authors suggest that a large fraction of the existing unknown unknowns can be anticipated under the condition of due diligence on the side of risk managers or project managers.

One of the quoted examples to support the case of knowable unknown unknowns is the case of the failure of the automated baggage system which happened in 1995 in the Denver International Airport. According to the investigation of the related evidence, the failures of the system could have been predicted, but this did not happen until the project started experiencing significant delays in completion.

Traditional Models and Unknown Unknowns

One of the key ways in which traditional risk management models are not reliable is their failure to predict the unknown unknowns of projects. In general, the majority of the traditional models are built on quantifying a project’s risks through a set of relevant indicators such as the probability, impact, and timing derived from the historical data and some estimates derived from the trends therein. However, with the unknown unknowns that are not readily predictable, relying on past statistics can prove to be completely irrelevant.

The most typical classifications of risks, as it was also discussed previously, are based on two key inputs – the patterns of the degree occurrence and their impacts. In this context, the key barrier to properly addressing the unknown unknowns within the traditional risk management models is that they are very frequently represented by the patterns or events which are hard to imagine, define and quantify. Alternatively, some of the specific consequences of the known unknowns can be conceptualized as unknown unknowns, and at this point, the limitations of the risk assessment models defining them come into place.

One of the examples of this type of unknown is the impact of a hurricane on a certain region. While the occurrence of a hurricane itself can be predicted by modern technology, its set of secondary effects is frequently unknown and only possible to predict on ad hoc basis and only in a certain part. For that end, a separate set of instruments and models is necessary to predict the occurrence of unknown unknowns, and some of them will be discussed in the following section.

Ways and Models to Handle Unknown Unknowns

One of the key points which should be taken into account while dealing with the unknown unknowns is that the set of undefinable risks and opportunities encountered by risk managers in real life are strictly project-specific. Therefore, there is no one-size-fits-all solution for determining the unknown unknowns. To successfully deal with this problem, modern risk managers should be aware of the best practices described both in theoretical research and empirical case studies and search for familiarity with their projects at hand. Some of the frequently cited ways of handling the risks of unknown unknowns in projects will be discussed in this section.

A paper by Kim (2017) suggests a model based on separation principles and the Theory of Inventive Problem Solving (TRIZ) as one of the most suitable ways of handling the emergence of unknown unknowns in risky projects. The authors suggest five major patterns of separation to solve the early identification task. These patterns include the perspective separation, space separation, time separation, condition separation and the separation between individual elements and the whole.

The approach suggests that within the described procedure, some of the unknown unknowns can be effectively recognizable by risk managers before they occur. One of the drawbacks of this method is the fact that it cannot eliminate or mitigate the risks connected to the existing knowledge gaps. Some of the further details on the utilization of this model in practice will be given further in the case study section.

Another recent paper by Ramasesh and Browning (2014) suggests that there are two major frameworks of reducing the influence of unknown unknowns on the project outcomes and identifying them on early life stages of a project. These are the project design approaches and behavioural approaches. Project design approaches focus on the optimization of various stages of operations and conceptualizing them from different perspectives, while the behavioural approaches are largely focused on different stages of cognition and communication.

The optimal choice of any of these recommended models or instruments is highly dependent on specific characteristics and nodes of operations of the considered projects. In some situations, it can be optimal to utilize the project design approach to scenario planning. Unlike forecasting, which essentially searches for some form of a most expected outcome of the considered project, scenario planning is focused on accepting uncertainty, making attempts to include it into the reasoning and exploring its various possible outcomes.

The fundamental element of scenario planning in this context is making up as many as possible “alternative stories” for the future of the project operations and exploring the resulting dynamic sequence of interrelated events and outcomes. This helps to identify the spheres of highest hazard and some high value / low-cost solutions for risk management and loss aversion.

The framework of scenario analysis can be further strengthened by the incorporation of the risk cascades, which attempt identifying some of the potential indirect impacts of direct threats to a project related to cohorts of the key stakeholders such as customers, employees, and suppliers. Scenario analysis can be specifically relevant to the projects characterized by high levels of complexity and mindlessness, as it helps to build more detailed mind maps of potential project outcomes which take into account a variety of extraordinary events.

One more relevant project design approach of predicting and handling unknown unknowns is exploring different project elements and stages with the help of predefined risk categories and checklists (Vrhovec et al. 2015). This method is specifically suitable for the projects with history or those which represent the replications of previous similar undertakings. Eventually, checklists of possible unknown unknowns encountered at previous stages can be considered prompts for generating ideas on the future potential, unpredictable risks.

Scrutinizing the approved project plans and inviting independent consultants can be considered as some more of the useful instruments in handling the unknown unknowns. In some cases, niche or industry experts can provide valuable insights on some of the reliability engineering techniques, such as robust design and failure mode and effect analysis. Engaging external experts and scrutinizing projects can be especially helpful in the environments subjected to a prejudiced mindset of management and a high degree of pathological intensity. An additional way of handling the process of uncovering the unknown unknowns can also be approached through the method of long interviews.

With the help of this tool, the interviewers have to scrutinize the assumptions of the approved plan by asking the project management some untypical and unexpected questions. This instrument has the potential to uncover some weak points of the project which are not easily articulated by the project stakeholders and thus transfer some of the unknown unknowns to the category of known unknowns.

Looking at the behavioural methods of uncovering the unknown unknowns, it is important to note the necessity for establishing an appropriate incentive system, which would stimulate the risk managers in charge of the project to uncover new unknown unknowns. One of the workable ways to do so is to give priority to the long-term success indicators of the project over achieving some short-term intermediary goals.

This would encourage the project managers to prefer the fulfilment of strategic goals over the less important tactical ones and indirectly give a higher priority to identifying the set of unknown unknowns. Also, Ramasesh and Browning (2014) suggest cultivation of a culture of alertness as an additional behavioural approach to identifying more unknown unknowns in each given project. This type of culture should include such elements as the developed attitudes to system thinking, learning from unexpected outcomes and building up the ad hoc experiential expertise.

Another possible solution to handling some fraction of the newly emerging unknown unknowns is implementing the methods and models of strategic agility. Teece, Peteraf, and Leih (2016) note that organizational agility has the potential to promote the dynamic capabilities of projects. Consistently with the previously mentioned research, the authors recommend the use of scenario planning for optimal identification of unknown unknowns, in this case – in the broader context of the strategic agility framework. Hall, Rowland, and Stokes (2015) and Du and Chen (2018) are some of the other authors relating the concepts of flexibility and strategic agility to the handling of the unknown unknowns.

As it can be seen from the above account of methods of predicting and handling the unknown unknowns, no one of them is either comprehensive or relevant to all the types of projects and their related potential hazards. For that end, modern risk managers should adopt the perspective of constant learning and always learn from the real-world cases of ad hoc risk mitigation associated with the emergence of unknown unknowns. Some of this evidence can be collected from research literature and other –from real-life cases, such as presented in the two subsequent sections.

The Case of Deepwater Horizon Oil Spill

The case of the Deepwater Horizon is one of the biggest and most detrimental sea oil spills in the history of the gasoline industry. This case of the interference of the unknown unknowns is described in much detail in a recent paper by Kim (2017). This event happened in 2010 in the Gulf of Mexico and involved an explosion happened at the oil rig operated by BP, which was followed by a release of close to 4.9 barrels of oil into the water within three months before being eventually capped.

Some of the key damages of this real-life manifestation of the unknown unknowns were the substantial damages to the local marine and wildlife, as well as the losses suffered by the economy through the disruptions in such industries as tourism and fishing. Kim (2017) suggests that the case of Deepwater Horizon can be optimally analyzed regarding its inherent unknown unknowns through the utilization of separation principles and the Theory of Inventive Problem Solving. Separation principles are applied to some of the initially identified elements of risk within the project, to explore the broader landscape of the applicable unknown unknowns.

The time separation principle first addresses the issue of the existence of the response plans (known), which were structured to address the possible negative consequences of the oil spills (unknown).

In this context, the search of the potential of the unknown should be executed through time separation – namely, looking for some specific points in time in which the known instrument can fail to address the unknown circumstance, thus creating an unknown within the considered project. After executing the time differentiation, it can also be useful to consult the relevant industry experts, to make sure that the differentiation in time separation is relevant. For the case of the Deepwater Horizon, a useful type of differentiation in time could be achieved through separately considering the time of the storm season in the Gulf of Mexico.

Another separation principle relevant to the connection of the oil spill response plans and the occurrence of the actual spill is the separation of space. Through pursuing this specific principle, it would have been possible to determine that unexpected difficulties may occur when applying the response plans in deep water areas. Some other types of space separation can include the separate consideration of the implications of oil spills at the level below seafloor or in different parts of the reservoir.

Also, the separation principles of system levels could have been used to predict the unpreparedness of the government to address this specific catastrophe at a timely and efficient manner. In this context, the separation of system levels could also consider the unpreparedness of the industry, the unpreparedness of the company’s top management or unpreparedness of the local facility specialists to react to the known unknown of the possibility of the oil spill, thus transferring it to the category of the unknown unknowns.

Similarly, Kim (2017) suggests applying the appropriate separation principles to some other elements of the projects which arise from the interaction of the known actions and instruments connected to the unknown events. This strategy helps to uncover some completely new sets of unknown unknowns which could be indeed transformed into known unknown if due diligence was in place. This transformation could be facilitated significantly by applying the concept of separation principles and the Theory of Inventive Problem Solving to the oil spill case, which happened in the Gulf of Mexico in 2010.

The Case of GMO

One of the most impactful recent cases of trying to manage the influence of unknown unknowns on the global level is the case of GMO. Given that producing genetically modified products is the relatively new practice of the most recent decades, many risks associated with it might be not possible to either identify or quantify. For that end, the policymakers are most frequently conceptualizing this process as a certain form of cost-benefit analysis.

In this analysis, the positive side of more food for the population and lower use of pesticides is prioritized over the risks of potential human mutations or other negative spill-overs which may take dozens of years to arrive and take the forms not imaginable today. For this situation, it is relevant to mention the previously described normal accident theory, which implies that the use of progressive technologies comes at a cost, some of which is the occurrence of potentially detrimental unknown unknowns (Gong et al. 2014).

Last week’s article by Splitter (2018) published in Forbes covers some of the recent developments of the GMO industry. The authors note that while ten years ago the GMO industry was dominated mainly by transgenic technologies, which implies adding the genes from one species to the others, nowadays the focus of attention has shifted towards higher utilization of gene editing. The article also quotes the potential of GMO to improve food safety as an important opportunity. This is specifically relevant for fruits and vegetables consumed as whole foods, such as eggplants in Bangladesh.

Given that for now, a tangible amount of risks associated with consuming GMO might be unidentifiable, it may be worth applying some of the tools and models of treating unknown unknowns to this currently evolving situation.

For instance, it can be beneficial to employ the culture of alertness to unknown unknowns, as it was suggested in the article by Ramasesh and Browning (2014). A certain part of this function is performed directly and indirectly by the Non-GMO movement, which is keeping an eye on the existing and emerging empirical evidence related to the utilization of GMO. Also, the instrument of establishing the system of incentives for identifying unknown unknowns for GMO scientists and industry practitioners should be established.

This can potentially balance at least some part of the conflict of interest existing between the funding of non-GMO studies and the high profits collected by the GMO-producing companies that could have the financial resources to support them. Similarly, the method of scenario analysis can be applied to exploring the potential set of GMO-related unknown unknowns. Focus groups of experts could gather in collective events and frame some of the relevant uncomfortable and out-of-the-box questions that could present a broader context for the GMO utilization.

Lastly, some of the separation principles suggested by Kim (2017) can also be applied to exploring the unknown unknowns of the GMO development. For instance, space separation principle could help to distinguish between different regions of the world in which the GMO are currently being used. Some of the questions to figure out could be, for instance, whether the soils and the environments of different regions of the world react differently to the utilization of GMO, and how these differences can trigger various currently unpredictable unknown unknowns.

Also, it could be useful to figure out if human organisms are giving different reactions to GMO depending on the location of the habitat of the affected individuals. Separation upon condition could also attempt to distinguish how the use of GMO can have a different impact depending on whether the gene-editing or transgenic technologies are emphasized in science and the industry, and what can be the possible implications for the related unknown unknowns.

Similarly to the case of the Deepwater Horizon, a separation between the parts of the system and the whole could help to identify the unknown unknowns related to the complications in the future use of GMO. For instance, some of the structures which can be slow to react to the risks are the governments, international organizations or the companies producing GMOs. Careful analysis of the unknown unknowns’ potential with the help of each of these instruments can help to eliminate or mitigate the crisis related to the use of unknown unknowns.

Conclusion

Summing up the above discussion, it can be seen that the traditional risk management models are in most cases not able to predict the unknown unknowns, the risks which cannot be identified by the decision-makers of a project in advance. This mainly happened because traditional risk management models are mostly quantitative and are thus based on the historical data on systematic behaviours observed in the past. However, for the case of unknown unknowns, this type of data is usually absent by the very definition of these types of risks.

Some of the characteristics of projects which have a higher probability of containing many unknown unknowns include mindlessness, complexity, complicatedness and the existence of project pathologies. Depending on the characteristics which are prevalent in the project’s structure, theoretical literature suggests the application of different unknown unknowns’ prediction tools. For instance, scenario analysis technic can be useful for projects with a high level of complexity and mindlessness.

Scenario analysis can help to model the project outcomes accounting for the occurrence of various unforeseeable events, in this way broadening the landscape of possibilities considered by risk managers. Some of the other mentioned tools of addressing the project complexity, mindlessness and inherent pathologies were the methods of long interviews, promotion of the culture cautious of unknown unknowns and establishing the set of incentives stimulating the project stakeholders to see and report the weak and early signs of unknown unknowns at different stages of project execution. One more tool relevant to resolving the challenges of unknown unknowns was the model of separation principles.

It implies separating the possible future states of a project basing on the factors of time, space, structural elements and conditions. Depending on the specific types of project environments and unknown unknowns, different separation principles can be most relevant to model the future states of the processes in question.

As it was shown in the two case study sections, suggested tools and models can also apply to practice, which allows reducing the future exposure to adverse negative risks. For instance, the case of Deepwater Horizon shows that the application of the structural level, time and space separation principles could help to identify the pain points of the Mexican Gulf oil rag of BR before the explosion and leakage happened.

Similarly, the case of today’s global trends in GMO production showed that there is a clear potential to reduce future exposure to unknown unknowns by implementing the techniques of scenario analysis. This can be achieved by using the models based on separation principles and adopting the systems of incentives stimulating the science and industry stakeholder to effectively identify and report the emerging possibilities for unknown unknowns.

Given that within the field of risk management, there is no one-size-fits-all model to address the challenges related to unknown unknowns, ad hoc approach is more relevant in addressing the associated problems. For that end, risk management professionals should be well aware of the accumulated theoretical evidence on the unknown unknowns, as well as of the existing successful and unsuccessful empirical case studies. This can help to adhere to the best practices of the unknown unknowns risk management in the future.

Reference List

Bonato, G 2014, Brazil presidential candidate Campos killed in plane crash. Web.

Du, J & Chen, Z 2018, ‘Applying Organizational Ambidexterity in strategic management under a “VUCA” environment: Evidence from high tech companies in China’, International Journal of Innovation Studies, vol. 2, no. 1, pp. 42-52.

Ghaleb, T, Alsri, A, Shabaneh, L & Niazi, M 2014, ‘A survey of project risk assessment and estimation models’, In Proceedings of the World Congress on Engineering. Web.

Gong, L, Zhang, S, Tang, P & Lu, Y 2014, ‘An integrated graphic–taxonomic–associative approach to analyze human factors in aviation accidents’, Chinese Journal of Aeronautics, vol. 27 no. 2, pp. 226-240.

Hall, R, Rowland, C & Stokes, P 2015, September. ‘New thoughts on leadership in turbulent times’, In 8th Annual Conference of the EuroMed Academy of Business. Web.

Kang, Y, Batta, R & Kwon, C 2014. ‘Value-at-risk model for hazardous material transportation’, Annals of Operations Research, vol. 222, no. 1, pp. 361-387.

Kim, S.D 2017, ‘Characterization of unknown unknowns using separation principles in the case study on Deepwater Horizon oil spill’, Journal of Risk Research, vol. 20, no.1, pp. 151-168.

Muriana, C & Vizzini, G 2017, ‘Project risk management: a deterministic quantitative technique for assessment and mitigation’, International Journal of Project Management, vol. 35, no.3, pp. 320-340.

Raghunath, K, Devi, S. & Patro, C. 2018, ‘Impact of Risk Assessment Models on Risk Factors: a Holistic Outlook’, In Research, Practices, and Innovations in Global Risk and Contingency Management (pp. 134-153). IGI Global.

Ramasesh, R.V & Browning, T.R 2014, ‘A conceptual framework for tackling knowable unknown unknowns in project management’, Journal of Operations Management, vol. 32, no. 4, pp. 190-204.

Splitter, J 2018, How the Non-GMO Project is adapting to A gene-edited world. Web.

Teece, D, Peteraf, M & Leigh, S 2016, ‘Dynamic capabilities and organizational agility: risk, uncertainty, and strategy in the innovation economy’, California Management Review, vol. 58, no. 4, pp. 13-35.

Vrhovec, S, Hovelja, T, Vavpotič, D & Krisper, M 2015, ‘Diagnosing organizational risks in software projects: stakeholder resistance’, International journal of project management, vol. 33, no. 6, pp. 1262-1273.