9_Ahmed23_Recommending Root-Cause and Mitigation Steps for Cloud Incidents using LLMs

Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models Toufique Ahmed * § , Supriyo Ghosh † , Chetan Bansal † Thomas Zimmermann ‡ , Xuchao Zhang † , Saravan Rajmohan † * UC Davis † Microsoft ‡ Microsoft Research Abstract —Incident management for cloud services is a complex process involving several steps and has a huge impact on both service health and developer productivity. On-call engineers require significant amount of domain knowledge and manual effort for root causing and mitigation of production incidents. Recent advances in artificial intelligence has resulted in state-of- the-art large language models like GPT-3.x (both GPT-3.0 and GPT-3.5), which have been used to solve a variety of problems ranging from question answering to text summarization. In this work, we do the first large-scale study to evaluate the effectiveness of these models for helping engineers root cause and mitigate production incidents. We do a rigorous study at Microsoft, on more than 40,000 incidents and compare several large language models in zero-shot, fine-tuned and multi-task setting using semantic and lexical metrics. Lastly, our human evaluation with actual incident owners show the efficacy and future potential of using artificial intelligence for resolving cloud incidents. Index Terms —Incident Management, Service Quality, GPT-3.x, Large Language Models I. I NTRODUCTION Large IT enterprises such as Amazon, Google, Microsoft, and Salesforce have replaced the traditional shrink-wrapped software and moved towards deploying applications and ser- vices on cloud platforms. In today’s cloud systems, production incidents (e.g., outage or performance degradation, unplanned interruptions) adversely impact the customers and can be expensive in terms of penalty associated with service level agreement violations and engineering efforts required to mit- igate the incidents. For example, one hour of downtime is estimated to cost Amazon US$100 million on major shopping days [1]. Despite continuous reliability efforts over the years, cloud services still experience inevitable severe incidents. Artificial Intelligence (AI) for IT Operations, also known as AIOps, has increased in popularity. Data-driven and AI techniques have been leveraged for automating parts of the incident life-cycle, for example, incident prioritization [2], retrieval of incidents with similar symptoms [3], and reducing the time to mitigate incidents [4], [5]. However, on-call engineers (OCEs) still spend a significant amount of manual toil through multiple rounds of back and forth communication for identifying root causes and mitigation steps . Motivated by the recent successes of leveraging GPT-3 models for non- trivial tasks [6], [7] and code generation [8], we apply such § This work is done during the author’s internship at Microsoft Research. models to incident management. We identified the following two scenarios: 1) Find the incident’s root cause. Diagnosing incidents typically requires significant time and communication be- fore engineers can identify the root cause of the incident. We investigate how effective large language models are at suggesting root causes for incidents (RQ1). 2) Suggest the mitigation steps for the incident. After a root cause has been located, engineers take actions to mitigate the problem. We investigate how effective large language models are at recommending the mitigation steps for incidents (RQ2). When applying large language models several considera- tions and decisions need to be taken. Since the models were not trained with incident management data, is fine-tuning of the models necessary (RQ3)? Is it more effective to build one model for each task ( single-task ) or one combined model that supports both root causes and incidents ( multiple task ) (RQ4)? Does the root cause help language models to find better mitigation steps (RQ5)? Do the models perform better for certain types of incidents (RQ6)? We address these questions with a rigorous large-scale evaluation of 44,340 incidents from 1,759 services of Microsoft. In addition to lexical and semantic evaluation metrics that are typically reported for such experiments, we present the results from a human validation, where we asked incident owners to assess the correctness and readability of suggested root causes and mitigation steps. The original incident owners are the most qualified to assess the performance of the models on incidents. In this paper, we make the following contributions: 1) This is the first work to demonstrate the usefulness of state-of-the-art large language models (LLMs) such as GPT-3.x (both GPT-3.0 and GPT-3.5) for resolving production incidents in a real world setting. (Section III) 2) We present a rigorous and large-scale study in Microsoft on over 40,000 incidents from 1000+ cloud services with six semantic and lexical metrics. (Section IV) • Fine-tuning significantly improves the effectiveness of LLMs for incident data. • GPT-3 and GPT-3.5 models significantly outperform encoder-decoder models in our experiments. • Metrics such as BLEU-4 are useful to measure relative 1 arXiv:2301.03797v2 [cs.SE] 9 Feb 2023

performance of models in different settings. How- ever, manual inspection and validation with experts is needed to assess the actual performance. 3) Our human study with the actual incident owners of pro- duction incidents helps prove the efficacy of the proposed approach. (Section V) II. O VERVIEW A. Incident management Production incidents are inevitable in large-scale cloud services and often severely affect the customer experience. Also, they can be extremely expensive in terms of engineer- ing resources required to root cause and mitigate them. An incident life-cycle typically has the following four stages: (1) Detection: The first step in the incident life-cycle is detection where the incidents are either reported by internal or external customers of a given service after they notice anomalous behavior. Also, incidents can also be reported via automated monitors which are configured by the service owners. (2) Triaging: Once an incident is reported, a team of OCEs analyze the problem and route the incident ticket to appropriate engineering team. This process is often referred as incident triaging. (3) Diagnosis: The incident diagnosis and root cause identification process requires multiple iterations of back and forth communication between engineers inspecting the different aspects to understand the broad nature of the incident and identify the root cause. (4) Mitigation: Based on the identified root causes, actions are taken to mitigate the problem so as to recover the service health and minimize the impact on the service users. Lately, AIOps (AI for IT Operations) has gained popularity for automating various parts of the incident life-cycle by combining data-driven and AI techniques with data-sources like application logs, time series performance metrics and service traces [2], [4], [5], [9]. Albeit significant efforts, incident management in large cloud systems still requires a huge amount of engineering effort and cost. More specifically, even with plethora of historical incident data, root cause iden- tification and mitigation remains notoriously challenging and time consuming tasks. In this work, we propose to use large language models such as GPT-3.x to automatically recommend root causes and mitigation for new incidents by leveraging historical incident data. B. The promise of LLMs/GPT-3.x models Large language models (LLMs) such as GPT-3.x [7] have emerged as one of the hottest trends in natural language processing over the last few years. With 175 billion parame- ters, the GPT-3.x language models, which held the record for being the largest neural network ever developed, is an order of magnitude larger than prior language models. Using this massive model architecture, GPT-3.x were trained using almost all accessible data from the Internet, including CommonCrawl [10], WebText [11], Wikipedia [12], and a corpus of books. Title: Attach vm fails with connection timeout Summary: The workspace is not associated with any vnet. Cus- tomer has a vm which is already running inside a vnet. They like to attach that vm into [product omitted]. We tried the UI and CLI route, but still fails with same connection timeout error. Error points that it resolves to some public ip [...] Reference root cause: It is not supported to attach a private vm to a public workspace directly. Reference mitigation: Open a task to provide better official docu- ment for customer on the topic of virtual machine. Fig. 1: A sample production incident. GPT-3.x models surpass the state-of-the-art models in a va- riety of NLP tasks, including machine translation, question- answering, and close tasks. Furthermore, the GPT-3.x models achieved a significant milestone by showing that unsupervised language models trained with adequate data can multi-task to the same level of fine-tuned models using just a few examples of the new tasks. As a result of its powerful text generation capabilities in new tasks, GPT-3.x are used in a wide range of categories and industries, from productivity and education to creativity and gaming. For instance, GPT-3.x are used to produce creative writing, including blog posts, advertisements, and poetry, that mimics the literary style of well-known writers like Shakespeare. C. Root-causing and mitigating incidents Incident root-causing and mitigation is a complex process which requires significant amount of manual effort and, also, domain knowledge about the services. Incidents can be caused by various kind of issues such as code bugs, dependency failures, infrastructure issues, configuration bugs, etc. Due to the vast number of possibilities, it is non-trivial for the OCEs to root cause the incidents. Similarly, once the root cause is identified, there can be various mitigation steps which can be taken such as code rollback, hotfix, infrastructure changes, configuration update, etc. Identifying the correct mitigation step is again non-trivial and requires domain knowledge and experience. Human errors in root causing or mitigation of incidents results in not just more effort and human toil but also impact on the customers and the revenue. Fig. 1 shows a real incident from a service where we can see the title and summary provided by the customer along with the actual root cause and mitigation. In this study, we evaluate the effectiveness of large lan- guage models like GPT-3.x and Codex for root causing and mitigating production incidents. When an incident is created, the author would specify a title for the incident and describe any relevant details such as any error messages, anomalous behavior and other details which could potentially help with resolution. Once the OCE starts investigating the incident, they might get more details by communicating with the incident author or by looking at telemetry and logs. During the course of the investigation, the OCE might often updates the incident details. For our evaluation, we use the title and the summary of a given incident at the time of incident creation as input 2

We collected data for incidents from the database that has the creation date between January 1, 2018, to July 15, 2022. Initially, we collected 123,953 instances for root causes and 23,544 mitigations from the “Resolved” or “Mitigated” incidents with severity levels 0-3 (most severe incidents belong to level 0). The samples for mitigation are low because they can be found in the postmortem of the incident, and post- mortem are not done for every incident. After collecting the data, we observe many incidents with duplicate root causes and mitigations. Some severe incidents/ denial of service trigger hundreds of incident reports for the same event, all of which have the exact root causes and mitigations. To fairly evaluate the model, we remove the exact duplicates for root causes and mitigation plans and end up with 57,520 root causes and 8,300 mitigation plans. The average root causes and mitigations lengths are 87 and 12 tokens, respectively. Some root causes are very long, and it is difficult for the models and human evaluators to generate and evaluate the models’ output. We kept the root causes up to 100 tokens, allowing us to keep 73% of the instances for root causes. We also discarded root causes and mitigation plans with less than three tokens because those are not informative. After deduplication and filtering, we sorted the data accord- ing to the creation date to use only historical data for training the model. We selected 35820, 3000 and 2000 root causes for training, testing and validation. We have fewer instances for mitigations. Hence, the training, test and validation sets for mitigations have 5455, 2000 and 500 data, respectively. Even after this rigorous filtering and deduplication of data, some root causes and mitigations do not carry any useful information ( e.g., root cause in a different link, transient, and auto-mitigated incidents). We manually went through 3000 root causes and 2000 mitigation plans from test sets and selected 2,621 root causes and 1,780 mitigation plans. 1 B. OpenAI models and baselines The recent advancement of the deep neural network models is greatly influenced by the introduction of Transformer mod- els [14]. Prior approaches ( i.e., LSTM [15] and GRU [16]) modeled the sequential dependencies of the generated text us- ing recurrent architecture. These recurrent models use “Back- Propagation through Time” (BPTT) to recursively propagate loss values over gradients within the same recurrent units pro- hibiting the possibility of parallel computation while capturing the long-distance dependencies of the tokens in the sequence. Bahdanau et al. introduced an attention mechanism that works on top recurrent architecture and improves the performance of recurrent neural models by providing an attention vector that indicates the relevant part of the input to the target output [17]. Transformer model completely removes the recurrence unit and entirely relies on the attention mechanism. It uses a multi- layer, multi-head self-attention architecture where the attention mechanism can relate different positions of a single sequence to compute a sequence representation. 1 We cannot share the dataset because incident data can contain confidential and private data and sharing such data would violate the terms of service. Pre-trained models are currently achieving state-of-the-art performance for various natural language and code tasks. These pre-trained models work in 2 stages ( i.e., pre-training and fine-tuning). In the pre-training stage, we train the model to learn statistics of language (or code) in a self-supervised fashion from large-scale corpora. After that, we use a smaller labeled dataset to fine-tune the model for specific tasks. It is nearly infeasible to have sufficient labeled data to train such high-capacity deep learning models. Pre-trained models enable us to train such big models with the unlabeled data in a self-supervised way in the pre-training stage. All the recent pre-trained (encoder-only and encoder-decoder) models ( e.g., BERT [18], RoBERTA [19], BART [20], T5 [21]) and decoder-only generative models ( e.g., GPT [22], GPT-2 [23], GPT-3 [7], OPT [24]) are basically Transformer models of various capacity trained with different pre-training objectives. The following subsections briefly discuss the baselines and OpenAI models we used for our experiments. 1) Baselines encoder-decoder models: We can apply the encoder-decoder models for both root cause and mitigation. The encoder will encode the input, and the decoder will generate the root cause or mitigation using the encoded representation provided by the encoder. Pre-trained NLP models ( e.g., BERT [18], RoBERTa [19], BART [20], T5 [21]) use different self-supervised pre- training objectives to learn robust language representa- tions. NLP models have programming language counterparts ( e.g., CodeBERT [25], GraphCodeBERT [26], PLBART [27], CodeT5 [13], NatGen [28]) where the models are initialized with the NLP models’ weights and continued pre-training with code and associated natural language comments in most cases. Though root cause and mitigation are natural language descriptions, the vocabulary ( e.g., identifiers) overlaps more with the comments used in code models. Therefore we picked both NLP and code models from OpenAI and baseline criteria to see if the performance differs depending on the domain used for pre-training. For baselining, we pick RoBERTa [19] and CodeBERT [25] models because of two reasons: i) these two models are architecturally identical with 125M parameters, ii) Both models are widely used as baselines (in fact, CodeBERT is the primary baseline model of the CodeXGLUE [29] dataset, which is a popular benchmark of 10 SE tasks including encoder-decoder tasks like code summarization and code trans- lation). Note that many transformer-based encoder-decoder models can be applied to this problem. However, comparing with all the models is beyond the scope of the paper. RoBERTa: BERT is the first model that introduced the pre- training strategy that outperforms the traditional Transformer models. It applied two pre-training strategies: Masked Lan- guage Modeling (MLM) and NSP (Next Sentence Prediction). In MLM pre-training, we randomly mask out 15% of the tokens and ask the model to recover those tokens, whereas, in NSP, we train the model to learn to predict the next sentence following an input sentence. Liu et al. [19] propose RoBERTa (A Robustly Optimized BERT Pre-training Approach), which outperforms the BERT model with a few changes, such as 4

dynamic masking and dropping NSP, achieves better perfor- mance. We apply RoBERTa as NLP baseline model. CodeBERT: CodeBERT is architecturally identical to RoBERTa model that uses two pre-training objectives: MLM and Replaced Token Detection (RTD) [30]. We can define RTD as a binary classification problem where two data generators (i.e., NL and PL) generate plausible alternatives for a set of randomly masked positions. A discriminator is trained to determine whether a word is the original one or not. CodeBERT is pre-trained on CodeSerachNet [31] dataset. 2) OpenAI generative models: Radford et al. introduced general task-agnostic generative pre-training of language mod- els (GPT) and outperformed 9 out of 12 discriminatively trained models that use architectures designed for the spe- cific task [22]. In generative pre-training, we autoregressively predict the probability of a token given the previous tokens moving from left to right. This left-to-right autoregressive training prevents the model from retrieving information from future tokens. All the subsequent generative models ( e.g., GPT- 2, GPT-3) use very similar pre-training objectives but have a higher capacity than previous ones and are pre-trained on a much larger dataset. Very large language models (LLMs) like GPT-3.x have 175 billion parameters and are found to be effective with few-shot learning replacing the need for fine-tuning for a specific set of tasks. However, fine-tuning GPT-3.x models are still beneficial for some tasks [7]. This paper evaluates our approach using four OpenAI [32] GPT- 3.x models: Curie, Codex, Davinci, and Code-davinci-002. Curie: Curie is the fastest GPT-3 model with 6.7B parameters. This model is trained with natural language data and performs well on language translation, complex classification, text sen- timent, and summarization tasks. This is the smallest model we use for our experiments. Codex: The Codex models are also GPT-3 models trained for understanding and generating code. The training data contains both natural language and billions of lines of public code from GitHub. We use one model, Codex-cushman from Codex family, with 12 billion parameters. Though the models are pre-trained for code-related tasks, it somehow relevant to incident management. Root cause and mitigation contain a lot of terminology ( e.g., filenames, identifiers) which relate more to comments used in software development projects. Davinci: Davinci is the biggest GPT-3 model (175 billion parameters) we use for our experiments. It can perform tasks with fewer instructions than other GPT-3 models. Davinci usually performs better at understanding the content or creative content generation task. It is also very good at solving logic problems. However, training the 175 billion parameters model is costly and requires a much longer period (almost four times compared to Curie with the same dataset) and more resources. Davinci is not trained to understand or generate code. Code-davinci-002: Code-davinci-002 is the 175 billion pa- rameters GPT-3.5 model we use for our experiments. Code- davinci-002 is an upgraded and more capable version of Codex model that was trained on a more recent dataset of text and code corpus. C. Model configuration One of the limitations of pre-trained encoder-decoder mod- els is that they can only encode 512 tokens. We observe that several samples from our test set truncated in GPT-3 model even though GPT-3 models support from 2048 tokens ( e.g., Curie, Codex) to 4000 tokens ( e.g., Code-davinci-002). Therefore, we can assume that the traditional encoder-encoder models do not have enough capacity to encode our sequences. Encoder-decoder models have been successful for problems like code summarization [13], [25], [27], code translation [29], and natural language translation [14], [20], [21]. We usually generate one sample using beam search for each input and compare the results with the reference. Generating one sample is sufficient for these problems because the target text is less open-ended. Besides, most of the information needed for successful generation can be found in the input for this set of problems. The models need to learn the syntactic alignment between two programming languages for code translation. Learning to transform conditional statements and loops from one programming language to another may be enough to do a successful translation, which is learnable from a few thousand samples. For natural language translation learning the mapping between the words from different natural languages is essential to generate good quality translation. Code summarization is slightly different from these two, where the input is much longer than the output. However, Ahmed and Devanbu found that all the necessary information for code summarization is extracted from the identifiers, and obfuscating the identifiers hurts the models [33]. Generating root causes and mitigation plans is much more complex than these problems, where the input may not contain handy information. The models need to be able to generate more diverse and creative solutions to answer the question. Our problem is more aligned with code generation problems where the input does not carry most information. For these types of problems, it is found that instead of using the encoder-decoder model, decoder-only models ( e.g., GPT-3.x) are more successful where we only focus on the following tokens considering the prior tokens generated by the models. It is well-established that encoder- decoder models are not as successful as decoder-only models in code generation tasks. However, we still apply encoder- decoder models to our problems and discuss our findings in the following sections. For RoBERTa [19] and CodeBERT [25] we use the exact setup that is used for the code summarization task [31], [34]. We adjust the length to 512 tokens with a batch size of 8 to provide as much as information to the model. Full fine-tuning that retrains all the parameters is very costly and challenging for the OpenAI models with billions of parameters. We use LoRA (Low-Rank Adaptation), a novel approach that significantly reduces the number of trainable parameters by freezing the pre-trained model weights and injecting trainable rank decomposition matrices into each layer of the Transformer architecture [35]. Even though LoRA reduces trainable parameters, it performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT- 5

TABLE I: Effectiveness of fine-tuned GPT-3.x models at finding root causes of the incidents Model BLEU-4 ROUGE-L METEOR BERTScore BLEURT NUBIA Top1 Top5 Top1 Top5 Top1 Top5 Top1 Top5 Top1 Top5 Top1 Top5 RoBERTa 4.21 NA 12.83 NA 9.89 NA 85.38 NA 35.66 NA 33.94 NA CodeBERT 3.38 NA 10.17 NA 6.58 NA 84.88 NA 33.19 NA 39.05 NA Curie 3.40 6.29 9.04 15.44 7.21 13.65 84.90 86.36 32.62 40.08 33.52 49.76 Codex 3.44 6.25 8.98 15.51 7.33 13.82 84.85 86.33 32.50 40.11 33.64 49.77 Davinci 3.34 5.94 8.53 15.10 6.67 12.95 83.13 84.41 31.06 38.61 35.28 50.79 Davinci-002 4.24 7.15 11.43 17.2 10.42 16.8 85.42 86.78 36.77 42.87 32.3 51.34 %gain for Davinci-002 23.26 13.67 26.44 10.90 42.16 21.56 0.61 0.49 12.72 6.88 -8.45 1.08 underperform at correctness criteria. Among OpenAI models, GPT-3.5 (i.e., Code-davinci-002) model significantly outper- forms all GPT-3 models as well as other baselines in terms of all the 6 automated metrics. Though the automatic metrics fail to detect the weaknesses of the encoder-decoder models, these metrics are still widely used. Human evaluation is hard to perform in every scenario, and these metrics can be useful to find the models’ relative performance. Therefore, even though we achieve a low score on these metrics, these are useful while trying to capture the relative performance of the model in different settings. Also, getting a lower score with lexical metrics is not surprising because lexical metrics only consider token overlaps and root cause and mitigation are open-ended, and the same root cause/mitigation can be written differently. In Section V, from the interviews with OCEs, we found that suggestions with lower BLEU-4 or other metrics are still helpful. B. How effective are fine-tuned GPT-3.x models in recom- mending mitigation plans for an incident? (RQ2) Table II shows that we achieved a slightly higher mitigation score (4.44-6.76 BLEU-4) than the root cause recommendation (3.38-4.24 BLEU-4). We observed a similar and consistent pattern (Table III) of the output as observed with root causes. The encoder-decoder models generate generic comments ( e.g., “the issue is self-mitigated”, “fix deployed to all regions”) like before, and those recommendations are mostly useless for the OCEs. For both RQ1 and RQ2, the fine-tuned Davinci model (even with 175 Billion parameters) is significantly un- derperforming other baseline methods according to automatic metrics. However, the Davinci and Code-davinci-002 models are the best performing models according to the incident owners (see Section V) C. How much fine-tuning improves over zero-shot learning performance of GPT-3.x models? (RQ3) As discussed in Section II-D, we will investigate the per- formance of OpenAI models in the zero-shot setting. Table IV presents the performance of the OpenAI models for root cause and mitigation. As expected, the model did not perform well in this setting since the models were not trained on confidential data from the incident management space. The models achieve 0.80-2.18 BLEU-4 for the top candidate, which is much lower (210%) than what we achieved with fine-tuning the models (5.47-6.76) while recommending mitigation steps. Though we achieved a higher score for mitigation than root cause during fine-tuning, in the zero-shot setting, the numbers for root cause are slightly high (1.18-2.83 for the top candidates). The model tries to complete the sequence depending on the given input. Copying a few tokens from input may help the model because the root cause is usually longer than mitigation and tends to share more tokens with the input. Because of unigram overlaps METEOR is doing better compared to other metrics (BLEU-4 and ROUGE-L) because it looks for the unigram precision and recall, making it lenient compared to BLEU-4 and ROUGE-L. We observe another interesting phenomenon here. Though the Davinci model was underperforming in RQ1 and RQ2, it significantly outperforms the other OpenAI models at zero-shot settings for both root cause and mitigation. This is because the model has higher parameters and is trained on more data enabling it to infer better without explicit training. D. Does multi-task learning improve the performance of GPT- 3.x models at finding root causes and mitigation plans? (RQ4) To evaluate the results of multi-task training in the root cause recommendation and mitigating planning tasks, we com- bine the training set of the two tasks for GPT-3.x models. The models are then individually tested using the corresponding test sets. Table V shows the results of root cause and mitigation with multi-task training. Overall, we observe that multi-task training does not significantly outperform training for a single task. The performance of Curie and Codex models has fallen by an average of 2.8% for BLEU-4, 2.0% for Rouge-L and 7.2% for Meteor. Only the Davinci model is marginally 6.2% better than single task training in terms of BLEU-4 metric. The performance of Code-davinci-002 is almost always lower across all lexical metrics in a multi-task setting. Similar to this, the results of mitigation generation reveals a 4.1% performance decline in average for all the four models. The lack of connection between the root cause and mitigation is what mostly contributes to the decline in performance. It is challenging to transfer knowledge from one task to the other because of the distinct distribution in their answer spaces, such as the variations in root cause and mitigation length and concreteness. E. Do GPT-3.x models get better at proposing mitigation plans if the root cause is given? (RQ5) We assess the performance of the mitigation generation while the root cause is being revealed. Our training set of mitigation is reduced from 5,455 to 2,973 as a result of the missing root causes in the incidents, and we have 166 test 7

TABLE II: Effectiveness of fine-tuned GPT-3.x models at finding mitigation plans of the incidents Model BLEU-4 ROUGE-L METEOR BERTScore BLEURT NUBIA Top1 Top5 Top1 Top5 Top1 Top5 Top1 Top5 Top1 Top5 Top1 Top5 RoBERTa 4.44 NA 7.10 NA 4.52 NA 86.33 NA 26.80 NA 14.90 NA CodeBERT 6.02 NA 4.40 NA 3.37 NA 86.83 NA 28.44 NA 27.89 NA Curie 5.47 10.62 8.03 16.31 6.22 12.75 85.65 87.13 27.20 37.23 15.30 25.46 Codex 5.53 10.62 8.15 16.23 6.19 13.15 85.68 87.35 28.43 37.92 15.77 26.33 Davinci 5.54 10.66 8.10 15.96 6.08 12.49 85.72 87.19 27.15 37.00 15.71 25.61 Davinci-002 6.76 11.66 10.22 18.14 8.23 15.13 86.17 87.65 30.19 38.96 17.58 28.81 %gain for Davinci-002 22.02 9.38 25.40 11.22 32.32 15.06 0.52 0.34 6.19 2.74 11.48 9.42 TABLE III: Uniqueness of the models’ suggestions Model Root cause Mitigation # of unique recommendations In % of total # of unique recommendations In % of total RoBERTa 160 6.10 4 0.22 CodeBERT 437 16.67 2 0.1 Curie 2612 99.65 1669 93.76 Codex 2614 99.73 1743 97.92 Davinci 2587 98.70 1731 97.24 Davinci-002 2614 99.73 1696 95.28 TABLE IV: Effectiveness of OpenAI models for recommend- ing root causes and mitigation steps at zero-shot setting Objective Model BLEU-4 ROUGE-L METEOR Top1 Top5 Top1 Top5 Top1 Top5 Root cause Curie 1.26 2.01 4.75 7.80 7.94 13.30 Codex 1.18 1.94 3.80 7.07 6.58 12.20 Davinci 2.83 4.37 6.11 11.55 6.04 11.87 Davinci- 002 1.35 2.5 4.89 8.58 7.65 13.55 Finetuned- Davinci- 002 4.24 7.15 11.43 17.2 10.42 16.8 % gain for Finetuning 49.82 63.62 87.07 48.92 31.23 23.99 Mitigation Curie 0.81 1.50 2.45 4.59 5.33 9.40 Codex 0.80 1.57 1.97 4.05 4.56 8.55 Davinci 2.18 3.67 3.84 7.84 4.99 10.44 Davinci- 002 0.92 1.89 2.31 4.52 4.92 9.2 Finetuned- Davinci- 002 6.76 11.66 10.22 18.14 8.23 15.13 % gain for Finetuning 210.1 217.7 166.2 131.4 54.4 44.9 samples to evaluate the model. Despite the sample reduction in the training set, Table VI reveals a considerable performance gain with the additional root cause information: the average for all three metrics is improved by 9.8% for the Curie model, 8.3% for the Codex model, 5.4% for the Davinci model and 26% for the Code-davinci-002. Nevertheless, we observe that the performance gain of the Code-davinci-002 model’s Top-5 recommendations is modest compared to the improvement of the Top-1 results. Despite this, the overall promising results highlight the significance of root cause information in generating mitigation plans. F. Do the models better propose mitigation plans for machine- detected incidents than human-detected ones? (RQ6) We analyze the mitigation generation performance of GPT- 3.x models for both machine and human detected incidents in Table VII. We employ the same training set but separate the test samples by the categories of human and machine detected incidents. The testing samples consist of 592 incidents rec- TABLE V: Effectiveness of multi-task learning Objective Model Multi- tasking? BLEU-4 ROUGE-L METEOR Top1 Top5 Top1 Top5 Top1 Top5 Root Cause Curie No 3.40 6.29 9.04 15.44 7.21 13.65 Yes 3.30 6.13 8.66 15.51 6.60 12.97 Codex No 3.44 6.25 8.98 15.51 7.33 13.82 Yes 3.42 6.11 8.64 15.24 6.53 12.81 Davinci No 3.34 5.94 8.53 15.10 6.67 12.95 Yes 3.60 6.27 9.11 15.66 7.31 13.64 Davinci-002 No 4.24 7.15 11.43 17.2 10.42 16.8 Yes 4.24 7.09 11.32 17.14 10.32 16.34 Mitigation Curie No 5.47 10.62 8.03 16.31 6.22 12.75 Yes 5.49 10.89 7.98 16.14 5.92 12.54 Codex No 5.53 10.62 8.15 16.23 6.19 13.15 Yes 5.15 10.88 7.49 15.87 5.55 11.85 Davinci No 5.54 10.66 8.10 15.96 6.18 12.49 Yes 5.64 10.74 7.88 15.97 6.13 12.99 Davinci-002 No 6.76 11.66 10.22 18.14 8.23 15.13 Yes 6.58 11.36 10.04 17.76 7.91 14.36 TABLE VI: Effectiveness of GPT-3 models at proposing mitigation plans given root causes Model Root-cause given? BLEU-4 ROUGE-L METEOR Top1 Top5 Top1 Top5 Top1 Top5 Curie No 5.92 11.29 9.46 17.76 7.34 13.35 Yes 6.59 12.40 10.25 18.61 8.24 16.00 Codex No 6.25 11.23 8.94 17.62 6.46 13.00 Yes 6.23 12.03 9.32 18.48 7.73 15.96 Davinci No 6.35 12.05 8.75 18.21 7.28 15.07 Yes 7.02 11.47 9.49 18.20 8.40 16.17 Davinci-002 No 6.8 12 9.48 17.37 8.15 15.53 Yes 8.6 13.28 11.56 19.46 10.9 18.08 %gain 26.47 10.21 21.94 12.03 33.74 16.42 ognized by machines and 1188 incidents detected by humans. Table VII demonstrates that machine-recognized incidents can outperform those detected by humans by a factor of 9.5% for BLEU-4, 20% for ROUGE-L and 23% for METEOR in the context of Top-1 recommendations of Code-davinci-002 model. It is due to the fact that machine detected incidents usually adhere to certain patterns, which are easier for machine learning models to recognize. V. L OOKING THROUGH THE I NCIDENT O WNERS ’ EYES A. Methodology From our test sets for root causes and mitigation plans, we selected the incidents with both root causes and mitigation, so that each incident owner could evaluate both the models in the same interview. Incident resolution is a complex task requiring significant context and domain knowledge about the service and also about the specific incidents. Hence, we conducted this human evaluation with the actual owners who root caused and mitigated the incidents. We chose 50 recent incidents which occurred in the last two months, to 8

TABLE VIII: Correctness and readability scores assigned by the incident owners Objective Criteria RoBERTA CodeBERT Curie Codex Davinci Davinci-002 Max OpenAI Mean Median Mean Median Mean Median Mean Median Mean Median Mean Median Mean Median Root cause Correctness 1.56 1 1.72 1 2.40 2 2.40 2 2.88 3 2.56 2 3.52 3 Readability 3.56 5 3.68 5 3.08 4 3.52 4 3.56 5 3.8 4 4.52 5 Mitigation Correctness 1.6 1 1.52 1 2.28 2 2.28 1 3.04 3 3.16 3 4.04 4 Readability 2.88 2 3.04 4 2.52 2 2.8 3 3.52 4 4.08 4 4.64 5 TABLE IX: Usefulness of LLMs for incident resolution Score # of incident owners In percent (%) of total 5 2 7.41 4 9 33.33 3 8 29.63 2 6 22.22 1 2 7.41 These textual dissimilarities also show why the OCEs found the GPT-3.x models promising, but the automatic evaluation metrics failed to do that. VI. D ISCUSSION & T HREATS A. Do automatic metrics reflect human perception? Automatic evaluation metrics are known to be representative of human perception and are widely used in problems like nat- ural language translation [14], [20], [21]. Though some recent works looked into the effectiveness of these metrics in code summarization and reported many pitfalls and weaknesses of these metrics [44]–[47], researchers are still using them for benchmarking. The best possible alternative to automatic metrics is human validation or some form of automatic test case evaluation (done in code generation tasks). The main challenge in incident management is that even experts face difficulties evaluating the incidents if they are not involved in resolving particular incidents. In some cases, the OCEs could not clearly remember the incidents if they happened two months ago. Thus conducting a large-scale study is quite challenging in this area. However, we interviewed 25 incident owners and found that the models perform pretty well even after achieving lower scores with automatic metrics. We calculated the Pearson coefficient for all three lexical metrics ( i.e., BLEU-4, ROUGE-L, and METEOR) with the correctness and readability score assigned by the OCEs. We observed that the co-efficient varies from -0.42 to +0.62, preventing us from getting specific patterns in the value. That also indicates that these automatic metrics may not be coherent with human perception for resolving cloud incidents. However, more sample cases are needed to reach any concrete resolution. B. Natural language or code? Which family of models are better for incident management? While choosing the models, we selected both natural lan- guage ( i.e., RoBERTa, Curie, Davinci) and code models ( i.e., CodeBERT, Codex-cushman, Code-davinci-002) to see which family of models is beneficial for incident management. We did not find any winners from these two groups. Davinci and Code-davinci-002 models are found to be producing correct and readable suggestions compared to other models. Note that both of them have 175 billion parameters. We leave fine-tuning larger code models or pre-training a model from scratch with incident data for future research. C. How the models’ performance can be improved? We received several recommendations from the incident owners. The main recommendation is to incorporate the dis- cussions among the OCEs into the model. This will guide the model to locate better suggestions. We also dropped many incidents with summaries that written or updated at the time of incident resolution. To fairly evaluate the model and prevent possible data leakage (root cause and mitigation can be written in summary if updated later), we discarded them from our dataset. Incorporating them into our dataset after preventing data leakage may improve the performance of the models. We also lost some critical information while cleaning the summaries ( e.g., discarding images and tables). Incorporating that information may also help. D. Threats to Validity There are several threats to our study. The semantic metrics use pre-trained models at the core, and we use the default, natural language models for the evaluation. A model pre- trained with incident management text may result in some changes in the performance evaluation. Also, we train and evaluate the models with the services available within our organization. These models may show unexpected behaviors if evaluated on a different set of services from other organi- zations. Some incidents owners expressed concerns about the models’ efficacy with rare incidents, and rare incidents are frequently reported at Microsoft. Another threat to our study is the sample size of our human subject study. It is difficult to achieve statistical significance on correctness and readability scores with such small samples. However, it is challenging to scale depending on the nature of the study. VII. R ELATED W ORK A. Incident management Incident management in large cloud services has become a popular topic of research in the Systems and Software Engineering communities. Prior work in this space has focused on two main directions. First, there has been several empirical studies on analyzing incidents and outages in production systems which have focused on studying incidents caused by certain type of issues [48]–[51] or issues from specific services and systems [52]–[54]. Second and more related to our work is the use of machine learning and data driven techniques for automating different aspects of incident life- cycle such as triaging [55]–[57], diagnosis [58]–[60] and 10

TABLE X: Examples exhibiting the effectiveness of GPT-3.x model for incident resolution Examples Task Original text Generated by DaVinci model I Root cause The customer was sending very large amounts of data as their payload. The size in addition to the rate at which these transactions were being sent led to the initial network crash. Large payloads that exceed the max payload size limit of the ledger caused the ledger to crash. Mitigation A code upgrade with higher payload size solved the problem. Republished the ledger with a higher payload size II Root cause An unexpected power loss caused multiple VMs to go down in East US causing DB services. Datacenter outage impacting multiple services Mitigation DB team mitigated the upstream issue in the West US region DB team mitigated the issue by restarting the service. mitigation [5]. Different from prior work, this is the first effort on leveraging state-of-the art language models for assisting OCEs with incident resolution. We hope that this work will also motivate future work which will merge traditional task- specific discriminative models with LLMs to do end-to-end automation of production incidents. B. LLMs in Software Engineering Even though this is the first work leveraging LLMs for AIOps, several works in Software Engineering have tried to solve other challenging problems with LLMs. Github Copi- lot uses GPT-3 for automated code generation from natural language inputs [8]. Several researchers have addressed code generation [8], [36], docstring generation [8], [61], and code repair [62], [63] problems. Bareiß et al. [64] show how few- shot learning can be effective at (i) code mutation; (ii) test oracle generation from natural language documentation; and (iii) test case generation task. Jain et al. propose an approach to augment large language models with post-processing steps based on program analysis and synthesis techniques and achieve better performance [65]. However, unlike code gener- ation where we have both lexical and structural information along with massive amount of training data, we explore the problem of incident resolution using state-of-the-art LLMs which has not been done before. VIII. C ONCLUSION With this work, we show that state-of-the-art large language models such as GPT-3 and GPT-3.5 are effective to help with incident management, specifically, to identify root causes and mitigation steps. To compare the effectiveness of the models, we conducted a rigorous and large-scale study at Microsoft, on over 40,000 incidents. To assess the actual usefulness of the approach, we involved the actual owners of production incidents. We expect that this paper is the first of many studies that leverage LLMs to make incident management more effective. Our next steps are to deploy the models in production to assist the OCEs with incident resolution. We are also planning to explore other usage scenarios for LLMs such as incident summarization. IX. A CKNOWLEDGEMENTS We would like to thank the engineers who participated in the validation of root causes and mitigation steps. We would like to also acknowledge the contributions of the following people across Microsoft: Oleg Losinets, Jim Jernigan and Jim Kleewein. R EFERENCES [1] S. Wolfe, “Amazon’s one hour of downtime on prime day may have cost it up to $100 million in lost sales,” 2018. [Online]. Available: https://www.businessinsider.com/ amazon-prime-day-website-issues-cost-it-millions-in-lost-sales-2018-7 [2] J. Chen, S. Zhang, X. He, Q. Lin, H. Zhang, D. Hao, Y. Kang, F. Gao, Z. Xu, Y. Dang et al. , “How incidental are the incidents? characterizing and prioritizing incidents for large-scale online service systems,” in Pro- ceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering , 2020, pp. 373–384. [3] A. Saha and S. C. Hoi, “Mining root cause knowledge from cloud service incident investigations for aiops,” arXiv preprint arXiv:2204.11598 , 2022. [4] J. Chen, X. He, Q. Lin, H. Zhang, D. Hao, F. Gao, Z. Xu, Y. Dang, and D. Zhang, “Continuous incident triage for large-scale online service sys- tems,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2019, pp. 364–375. [5] J. Jiang, W. Lu, J. Chen, Q. Lin, P. Zhao, Y. Kang, H. Zhang, Y. Xiong, F. Gao, Z. Xu et al. , “How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 1410–1420. [6] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” arXiv preprint arXiv:2201.11903 , 2022. [7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language mod- els are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020. [8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021. [9] Z. Chen, Y. Kang, L. Li, X. Zhang, H. Zhang, H. Xu, Y. Zhou, L. Yang, J. Sun, Z. Xu et al. , “Towards intelligent incident management: why we need it and how we make it,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 1487–1497. [10] “Common Crawl.” [Online]. Available: https://commoncrawl.org/ [11] S. Kulkarni, A. Singh, G. Ramakrishnan, and S. Chakrabarti, “Collective annotation of wikipedia entities in web text,” in Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , 2009, pp. 457–466. [12] “Wikipedia.” [Online]. Available: https://www.wikipedia.org/ [13] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021. [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008. [15] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997. [16] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555 , 2014. [17] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014. [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018. 11

monitoring system for detecting and diagnosing service issues,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2015, pp. 2029–2038. [59] C. Bansal, S. Renganathan, A. Asudani, O. Midy, and M. Janakiraman, “Decaf: Diagnosing and triaging performance issues in large-scale cloud services,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , 2020. [60] C. Luo, J.-G. Lou, Q. Lin, Q. Fu, R. Ding, D. Zhang, and Z. Wang, “Correlating events with time series for incident diagnosis,” in Proceed- ings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , 2014, pp. 1583–1592. [61] T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–5. [62] Z. Fan, X. Gao, A. Roychoudhury, and S. H. Tan, “Improving automat- ically generated code from codex via automated program repair,” arXiv preprint arXiv:2205.10583 , 2022. [63] H. Joshi, J. Cambronero, S. Gulwani, V. Le, I. Radicek, and G. Ver- bruggen, “Repair is nearly generation: Multilingual program repair with llms,” arXiv preprint arXiv:2208.11640 , 2022. [64] P. Bareiß, B. Souza, M. d’Amorim, and M. Pradel, “Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code,” arXiv preprint arXiv:2206.01335 , 2022. [65] N. Jain, S. Vaidyanath, A. Iyer, N. Natarajan, S. Parthasarathy, S. Ra- jamani, and R. Sharma, “Jigsaw: Large language models meet program synthesis,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1219–1231. 13