Service Operations

1Introduction 2Serv. Mgmt. 3Principles 4Process 5Activities 6Organization 7Consideration 8Implementation 9Issues AAppendeces

4. Service Operation Processes


4.4 Problem Management

ITIL defines a 'problem' as the unknown cause of one or more incidents.

4.4.1 Purpose, Goals and Objectives
Problem Management is the process responsible for managing the lifecycle of all problems. The primary objectives of Problem Management are to prevent problems and resulting incidents from happening, to eliminate recurring incidents and to minimize the impact of incidents that cannot be prevented.

4.4.2 Scope
Problem Management includes the activities required to diagnose the root cause of incidents and to determine the resolution to those problems. It is also responsible for ensuring that the resolution is implemented through the appropriate control procedures, especially Change Management and Release Management.

Problem Management will also maintain information about problems and the appropriate workarounds and resolutions, so that the organization is able to reduce the number and impact of incidents over time. In this respect, Problem Management has a strong interface with Knowledge Management, and tools such as the Known Error Database will be used for both.

Although Incident and Problem Management are separate processes, they are closely related and will typically use the same tools, and may use similar categorization, impact and priority coding systems. This will ensure effective communication when dealing with related incidents and problems.

4.4.3 Value to Business
Problem Management works together with Incident Management and Change Management to ensure that IT service availability and quality are increased. When incidents are resolved, information about the resolution is recorded. Over time, this information is used to speed up the resolution time and identify permanent solutions, reducing the number and resolution time of incidents. This results in less downtime and less disruption to business critical systems.

Additional value is derived from the following:

4.4.4 Policies, Principles and Basic Concepts
There are some important concepts of Problem Management that must be taken into account from the outset. These include: Problem Models
Many problems will be unique and will require handling in an individual way - but it is conceivable that some incidents may recur because of dormant or underlying problems (for example, where the cost of a permanent resolution will be high and a decision has been taken not to go ahead with an expensive solution - but to 'live with' the problem).

As well as creating a Known Error Record in the Known Error Database (see paragraph to ensure quicker diagnosis, the creation of a Problem Model for handling such problems in the future may be helpful. This is very similar in concept to the idea of Incident Models already described in paragraph, but applied to problems as well as incidents.

4.4.5 Process Activities, Methods And Techniques
Figure 4.4 Problem Management process flow
Figure 4.4 Problem Management process flow

Problem Management consists of two major processes:

The reactive Problem Management process is shown in Figure 4.4. This is a simplified chart to show the normal process flow, but in reality some of the states may be iterative or variations may have to be made in order to handle particular situations. Problem Detection
It is likely that multiple ways of detecting problems will exist in all organizations. These will include:

Frequent and regular analysis of incident and problem data must be performed to identify any trends as they become discernible. This will require meaningful and detailed categorization of incidents/problems and regular reporting of patterns and areas of high occurrence. Top ten' reporting, with drill-down capabilities to lower levels, is useful in identifying trends.

Further details of how detected trends should be handled are included in the Continual Service Improvement publication. Problem Logging
Regardless of the detection method, all the relevant details of the problem must be recorded so that a full historic record exists. This must be date and time stamped to allow suitable control and escalation. A cross-reference must be made to the incident(s) which initiated the Problem Record - and all relevant details must be copied from the Incident Record(s) to the Problem Record. It is difficult to be exact, as cases may vary, but typically this will include details such as: Problem Categorization
Problems must be categorized in the same way as incidents (and it is advisable to use the same coding system) so that the true nature of the problem can be easily traced in the future and meaningful management information can be obtained. Problem Prioritization
Problems must be prioritized in the same way and for the same reasons as incidents - but the frequency and impact of related incidents must also be taken into account. The coding system described earlier in Table 4.1 (which combines impact with urgency to give an overall priority level) can be used to prioritize problems in the same way that it might be used for incidents, though the definitions and guidance to support staff on what constitutes a problem, and the related service targets at each level, must obviously be devised separately.

Problem prioritization should also take into account the severity of the problems. Severity in this context refers to how serious the problem is from an infrastructure perspective, for example: Problem Investigation and Diagnosis
An investigation should be conducted to try to diagnose the root cause of the problem - the speed and nature of this investigation will vary depending upon the impact, severity and urgency of the problem - but the appropriate level of resources and expertise should be applied to finding a resolution commensurate with the priority code allocated and the service target in place for that priority level.

There are a number of useful problem solving techniques that can be used to help diagnose and resolve problems - .and these should be used as appropriate. Such techniques are described in more detail later in this section.

The CMS must be used to help determine the level of impact and to assist in pinpointing and diagnosing the exact point of failure. The Know Error Database (KEDB) should also be accessed and problem-matching techniques (such as key word searches) should be used to see if the problem has occurred before and, if so, to find the resolution.

It is often valuable to try to recreate the failure, so as to understand what has gone wrong, and then to try various ways of finding the most appropriate and cost-effective resolution to the problem. To do this effectively without causing further disruption to the users, a test system will be necessary that mirrors the production environment.

There are many problem analysis, diagnosis and solving techniques available and much research has been done in this area. Some of the most useful and frequently used techniques include:

From this chart it is clear to see that there are three primary causes for network failure in the organization. These should therefore be targeted first.

CausesPercentage of totalComputationCumulative
Network Controller350+35%35
File corruption2635%+26%61
Addressing conflicts1961%+19%80
Server OS680%+6%86
Scripting error586%+5%91
Untested change391%+3%94
Operator error294%+2%96
Backup failure296%+2%98
Intrusion attempts198%+1%99
Disk failure199%+1%100
Table 4.2 Pareto cause ranking chart of Network failures
Figure 4.5 Important versus trivial causes
Figure 4.5 Important versus trivial causes Workarounds
In some cases it may be possible to find a workaround to the incidents caused by the problem - a temporary way to overcoming the difficulties. For example, a manual amendment may be made to an input file to allow a program to complete its run successfully and allow a billing process to complete satisfactorily, but it is important that work on a permanent resolution continues where this is justified - in this example the reason for the file becoming corrupted in the first place must be found and corrected to prevent this happening again.

In cases where a workaround is found, it is therefore important that the problem record remains open, and details of the workaround are always documented within the Problem Record. Raising a Known Error Record
As soon as the diagnosis is complete, and particularly where a workaround has been found (even though it may not yet be a permanent resolution), a Known Error Record must be raised and placed in the Known Error Database - so that if further incidents or problems arise, they can be identified and the service restored more quickly.

However, in some cases it may be advantageous to raise a Known Error Record even earlier in the overall process - just for information purposes, for example - even though the diagnosis may not be complete or a workaround found, so it is inadvisable to set a concrete procedural point exactly when a Known Error Record must be raised. It should be done as soon as it becomes useful to do so!

The Known Error Database and the way it should be used are described in more detail in paragraph Problem Resolution
Ideally, as soon as a solution has been found, it should be applied to resolve the problem - but in reality safeguards may be needed to ensure that this does not cause further difficulties. If any change in functionality is required this will require an RFC to be raised and approved before the resolution can be applied. If the problem is very serious and an urgent fix is needed for business reasons, then an Emergency RFC should be handled by the Change Advisory Board Emergency Committee (CAB/EC) to facilitate this urgent action. Otherwise, the RFC should follow the established Change Management process for that type of change - and the resolution should be applied only when the change has been approved and scheduled for release. In the meantime, the KEDB should be used to help resolve quickly any further occurrences of the incidents/problems that occur.

Note: There may be some problems for which a Business Case for resolution cannot be justified (e.g. where the impact is limited but the cost of resolution would be extremely high). In such cases a decision may be taken to leave the Problem Record open but to use a workaround description in the Known Error Record to detect and resolve any recurrences quickly. Care should be taken to use the appropriate code to flag the open Problem Record so that it does not count against the performance of the team performing the process and so that unauthorized rework does not take place. Problem Closure
When any change has been completed (and successfully reviewed), and the resolution has been applied, the Problem Record should be formally closed - as should any related Incident Records that are still open. A check should be performed at this time to ensure that the record contains a full historical description of all events - and if not, the record should be updated. The status of any related Known Error Record should be updated to shown that the resolution has been applied. Major Problem Review
After every major problem (as determined by the organization's priority system), while memories are still fresh a review should be conducted to learn any lessons for the future. Specifically, the review should examine:

Such reviews can be used as part of training and awareness activities for support staff - and any lessons learned should be documented in appropriate procedures, work instructions, diagnostic scripts or Known Error Records. The Problem Manager facilitates the session and documents any agreed actions.

The knowledge learned from the review should be incorporated into a service review meeting with the business customer to ensure the customer is aware of the actions taken and the plans to prevent future major incidents from occurring. This helps to improve customer satisfaction and assure the business that Service Operations is handling major incidents responsibly and actively working to prevent their future recurrence. Errors Detected In The Development Environment
It is rare for any new applications, systems or software releases to be completely error-free. It is more likely that during testing of such new applications, systems or releases a prioritization system will be used to eradicate the more serious faults, but it is possible that minor faults are not rectified - often because of the balance that has to be made between delivering new functionality to the business as quickly as possible and ensuring totally fault free code or components.

Where a decision is made to release something into the production environment that includes known deficiencies, these should be logged as Known Errors in the KEDB, together with details of workarounds or resolution activities. There should be a formal step in the testing sign-off that ensures that this hand-over always takes place (see Service Transition publication). Experience has shown if this does not happen, it will lead to far higher support costs when the users start to experience the faults and raise incidents that have to be re-diagnosed and resolved all over again!

4.4.6 Triggers, Input And Output / Interprocess Interfaces
The vast majority of Problem Records will be triggered in reaction to one or more incidents, and many will be raised or initiated via Service Desk staff. Other Problem Records, and corresponding Known Error Records, may be triggered in testing, particularly the latter stages of testing such as User Acceptance Testing/Trials (UAT), if a decision is made to go ahead with a release even though some faults are known. Suppliers may trigger the need for some Problem Records through the notification of potential faults or known deficiencies in their products or services (e.g. a warning may be given regarding the use of a particular Cl and a Problem Record may be raised to facilitate the investigation by technical staff of the condition of such CIs within the organization's IT Infrastructure).

The primary relationship between Incident and Problem Management has been discussed in detail in paragraphs 4.2.6 and Other key interfaces include the following:

4.4.7 Information Management CMS
The CMS will hold details of all of the components of the IT Infrastructure as well as the relationships between these components. It will act as a valuable source for problem diagnosis and for evaluating the impact of problems (e.g. if this disk is down, what data is on that disk; which services use that data; which users use those services?). As it will also hold details of previous activities, it can also be used as a valuable source of historical data to help identify trends or potential weaknesses - a key part of proactive Problem Management (see Continual Service Improvement publication). Known Error Database
The purpose of a Known Error Database is to allow storage of previous knowledge of incidents and problems - and how they were overcome - to allow quicker diagnosis and resolution if they recur.

The Known Error Record should hold exact details of the fault and the symptoms that occurred, together with precise details of any workaround or resolution action that can be taken to restore the service and/or resolve the problem. An incident count will also be useful to determine the frequency with which incidents are likely to recur and influence priorities, etc. It should be noted that a Business Case for a permanent resolution for some problems may not exist. For example, if a problem does not cause serious disruption and a workaround exists and/or the cost of resolving the problem far outweighs the benefits of a permanent resolution - then a decision may be taken to tolerate the existence of the problem. However, it will still be desirable to diagnose and implement a workaround as quickly as possible, which is where the KEDB can be of assistance.

It is essential that any data put into the database can be quickly and accurately retrieved. The Problem Manager should be fully trained and familiar with the search methods/algorithms used by the selected database and should carefully ensure that when new records are added, the relevant search key criteria are correctly included.

Care should be taken to avoid duplication of records (i.e. the same problem described in two or more ways as separate records). To avoid this, the Problem Manager should be the only person able to enter a new record. Other support groups should be allowed, indeed encouraged, to propose new records, but these should be vetted by the Problem Manager before entry to the KEDB. In large organizations where Problem Management staff exist in multiple locations but a single KEDB is used (recommended!), a procedure must be agreed between all Problem Management staff to ensure that such duplication cannot occur. This may involve designating just one staff member as the central KEDB Manager.

Figure 4.6 Service Knowledge Management System
Figure 4.6 Service Knowledge Management System

The KEDB should be used during the Incident and Problem Diagnosis phases to try to speed up the resolution process - and new records should be added as quickly as possible when a new problem has been identified and diagnosedR.

All support staff should be fully trained and conversant with the value that the KEDB can offer and the way it should be used. They should be able readily to retrieve and use data.

The KEDB, like the CMS, forms part of a larger Service Knowledge Management System (SKMS) illustrated in Figure 4.6. More information on the SKMS can be found in the Service Transition publication.

4.4.8 Metrics
The following metrics should be used to judge the effectiveness and efficiency of the Problem Management process, or its operation:

All metrics should be broken down by category, impact, severity, urgency and priority level and compared with previous periods.

4.4.9 Challenges, Critical Success Factors and Risks
A major dependency for Problem Management is the establishment of an effective Incident Management process and tools. This will ensure that problems are identified as soon as possible and that as much work is done on pre-qualification as possible. However, it is also critical that the two processes have formal interfaces and common working practices. This implies the following:

In addition it is important that Problem Management is able to use all Knowledge and Configuration Management resources available.

Another CSF is the ongoing training of technical staff in both technical aspects of their job as well as the business implications of the services they support and the processes they use.

[To top of Page]

Visit my web site