Martello BLOG : 2020

Before we dive into incident management, let’s get on the same page about some common terminology.

ITSM (IT service management) is a common approach to creating, supporting, and managing IT services. The core concept of ITSM is the belief that IT should be delivered as a service. And one of the core practices of ITSM is incident management.

Incidents are unplanned events of any kind that disrupt or reduce the quality of service (or threaten to do so). A business application going down is an incident. A crawling-but-not-yet-dead web server can be an incident, too. It’s running slowly and interfering with productivity. Worse yet, it poses the even-greater risk of complete failure.

A problem is the not-yet-known root cause behind one or more incidents. In the incident above where the network is creeping and a business application is down, a reconfigured router could be the underlying problem behind both.

What is Incident Management?

Incident Management is a key area of focus of IT departments as how the process is created to restore “normal” service operations as quickly as possible. This will remove an adverse impact to the business operations or the end user. To be successful, IT teams must promptly and effectively deal with all Incidents reported by user.

One of the most common mistakes of busy, growing IT organizations is to try and reinvent the wheel and create processes from scratch, or build their own tools for fielding tickets.

Incident Management Workflow

Incident Management has a simple workflow that goes from an Incident occurring, to a ticket being raised and identifying the problem, restoring service, and finally reporting to look for possible ways to prevent problem in the future.

Best Practices to Improve Incident Management

Create Robust Workflows

Establish a workflow for a clear process that encourages rapid resolution time. This includes identifying major incidents, communicating to impacted stakeholders, assigning to the right individuals to fix the problem, following and documenting the lifecycle of the incident.

Provide Multi-Channel Support

Allow users to raise tickets easily through various channels including email, chat, portal, etc. With several employees working remotely, they may find some forms of communication to be easier than others. Offering several channels that they can access from both their desktop and mobile will help improve the overall satisfaction and user experience.

Automate Where Possible

Automation can save a lot of time in a busy IT department. Start by automating ticket assignments to the right members of the team for quick resolution. Consolidate the same alerts into one incident for less noise and to avoid redundancy.

Prioritize Incidents

Ensure business critical issues are addressed first with proper classification and assignment. Knowing how each incident effects the overall business is important to ensure the best use of resources within an IT department.

Communicate Effectively

Share the status of tickets with members of the team and the user that logged the incident. Ensuring the right stakeholders are aware of a potential outage will help increase efficiencies in the workplace, reduce frustrations and provide a better overall user experience.

Review and Report on Significant Incidents

Analyze major incidents with the goal of finding areas of improvement. This will help you take a proactive approach to your IT system. Data you will want to collect includes:

-Number of major incidents raised and closed each month

-Average resolution time for major incidents

-How much downtime the major incidents caused

-The problems and changes linked to major incidents

Learning from these experiences will help you proactively address potential issues, resulting in less downtime for the business.

Martello iQ Makes Incident Management Simple

Pulling in alerts and health state information from various monitoring tools, Martello iQ consolidates the information into a single pane of glass for efficient management.

Automation

If you have integrated an ITSM system with Martello iQ, you can automate the creation of incidents. With this feature enabled, iQ creates an incident when an alert is raised for a board, a business service or saved search. Any subsequent alert will be consolidated into one incident on your ITSM.

Martello iQ also lets you send notifications and alerts of incidents to a specific team automatically, for a faster resolution time.

SCOM Reporting is often neglected due to lack of time, knowledge or awareness. SCOM is a feature-rich system, and navigating through these details to produce meaningful reports can be discouraging at times, especially when reports turn out empty.

With many stakeholders in the company requesting frequent reports, to know basic metrics about the servers for capacity management, SCOM reporting is not something that can be ignored. Get the most actionable data out of your SCOM reporting with these best practices.

We sat down with Martello’s Justin Boerrigter and Managing Consultant, SCOM and System Center MVP from TopQore, Bob Cornelissen to discuss SCOM reporting and the future of SCOM.

SCOM pulls in a ton of data, so it can be difficult to filter through the results. What are some of the best practices you recommend in creating actionable data and meaningful reports? 

Bob: Well, first of all I would recommend to start using SCOM Reporting. Think about what you need to know and report on for the business. Think about if there is data in there which can help the SCOM Admin to improve the SCOM environment.

Some of the best practices I am thinking about are also around how to avoid empty reports. I have discussed some major points on this subject in a blog post series we have put online on the TopQore blog earlier in the week.

And finally a best practice would be to save your reports for later runs when you have built something useful, and schedule some of them if you know they are asked for regularly.

What are the most common reports SCOM administrators are required to produce?

Bob: The most common reports would be the performance and availability statistics. Managers and auditors generally want to know about the availability of their systems, especially if they are customer based. This can also be internal customers for high priority applications. Aside from that there are often Service Level Agreements there or KPI the application or the service or the machine needs to comply to.

There is much more data in the database to work through for SCOM admins and System Admins, which I will in part discuss in my blog post series.
Therefore I think the most common requested reports by the business have to do with KPI performance reports, usually for a few counters + Availability and SLA reports for Services and applications.

Empty reports can be an issue. What are some of the best way to prevent reports from coming up empty?

Bob: To be honest, many times the problem is targeting. Targeting has to do with the Class structure which SCOM depends upon for all monitoring and performance collection. To say it bluntly, if you request to see a SQL database free space counter and ask this counter from a Linux website… you are not going to get an answer. Or it is like asking me how Sandra’s weekend was to say it in human terms, and keep in mind she lives 6000 km away from me. You must target your question to the object where the performance counter is collected from – or one of its parent objects.

Other reasons why your report may be empty have to do with performance collection rules maybe have been turned off by default or by override to tune down data collection. You can check this from Performance views within the console or your dashboarding tool. It could also be there is a problem with the SCOM DataWarehouse database, preventing the data from being put in the database or from being processed correctly into the tables the reports are looking at.

Those are some of the common reasons.

How can you use these reports to demonstrate how IT departments should be investing in their IT?

Bob: First of all, if you look at core performance reports, you could use them for capacity management. You can discover machines who are underutilizing their assigned resources, and those which are often running into capacity issues for being too busy for the assigned hardware to keep up. You could also target the same reports to Services or Application groups within your SCOM environment. That way you can focus the capacity planning to each Service specifically and determine if there are any reasons to adjust capacity planning for these or use availability and SLA reports to check if there are regular problems having to do with this specific application. This way you can target your efforts to each business service and improve. What is important in this respect is that you have a sense of what a business service is made up of, either by creating a group of servers in its most basic form to report on, or by extending that into SCOM Distributed Applications or even better by using Live Maps. You have to see, this sea of servers in your datacenter not as one whole thing, but it is a group of applications, sometimes spread across a few servers to provide this service.

Monitoring is paramount to optimizing IT resources and as such, it is vital that organizations keep one step ahead and have a way of proactively tracking and notifying business leaders on infrastructure or code issues and address them before they cause further downstream damage.  SCOM has always been the “go-to” solution to monitor on-prem infrastructures but now many businesses are moving to a hybrid solution or full Cloud environments. Is there a concern about SCOM sunsetting and if so, how are current SCOM users to modify their reporting while using Azure Monitor? 

Bob: We get this question asked a lot, basically the question if SCOM is dead or not. I can tell you that it is very much alive and supported. I have very regular chats with Microsoft product team members for the System Center teams and they are working hard on new things with SCOM. The current main version is 2019 and supported to 2029, but there will be a new version of SCOM coming with the next version of Windows as far as we can tell. So there is no need to step away from SCOM.

However, the world is turning more and more hybrid and cloudy. Most of the times if they are existing companies, I think it fair to say Hybrid is what we can expect to see most. So there have to be ways to monitor in a hybrid world. I have done a few recorded presentations about this subject lately. SCOM can connect to the cloud services and pull in a nice amount of data and you can report on those all the same.

There will also be items in the cloud or more cloud native which are most likely being monitored by an Azure Monitor related system. And are more suited to be monitored there. Reporting from the Azure Monitor Log Analytics system works differently from SCOM. You will have to write some queries in the Azure Monitor native language of KQL. Some queries are provided as examples and defaults, and the rest you have to create yourself. This is often used to create an ad-hoc insight or overview on a live dashboard in Azure Monitor. Or to generate alerts from.

What we see more often is that these queries that pull up the data you are interested in, are used by PowerBi to generate more insights and more human readable and acceptable formats for viewing and analysis.

This is a long answer to a complex world of hybrid monitoring. You will use SCOM to pull in data, but you will also use Azure Monitor to pull in data and report on both. With both having different methods. And of course the world is even more complex than this.

If you had to compare, which product has the steeper learning curve SCOM or Azure?

Bob: That is a difficult question. SCOM is not the easiest product, because it has the complexities needed to be so flexible in what it can do. But with a number of days SCOM training you can get to grips with a lot of it. TopQore does provide trainings by the way for SCOM admins and operators.
Now Azure is of course also a very large ecosystem with so many components. And there is a lot to learn about infrastructure and security and ways to do things in the cloud. You can get started in Azure in 5 minutes literally, but to do things right on a medium sized company way it requires more training. There are certification training and exams for multiple parts of Azure. For Azure Monitor, which is a part of the monitoring infrastructure in Azure I also say it takes you a minute to start. But it will take you longer to delve deeper in what is possible. And again, if you are in for example a medium or large sized company you will find that you need to monitor and view and report on very specific things and you need to create those insights yourself, based on the wealth of information gathered in Azure Monitor.

When speaking with SCOM administrators, they’re very attached to their tools. What is there number one concern about moving from SCOM to Azure?

Bob: Well, a concern is that they are different tools, and you need to learn new things in order to do something similar. And as with all these things, you can not expect Azure Monitor to be built the same as SCOM and act the same as SCOM. Neither will any other tool. So there is a learning curve there, and the views and dashboards and reports will be different, while you might be asking the same questions to different tools. And what is important for the business is that you can somehow still connect the different tools, so you can still try to see part of this story come together in a joint dashboard and alerting space.

With Martello iQ you are able to integrate Azure and expand your service map beyond your SCOM data. This means you can incorporate any data in your IT environment, which can allow you to work proactively. Can you share with us some of the strategies IT managers can use to take a proactive approach to their network with iQ?  

Justin: We have two majors ways to represent data that comes from multiple systems. iQ is essential, to coin a term from Bob, a data lake that pulls in data. We’re not as big as the lakes that are sitting in Azure, as they pull in a lot of data. What we do is we will in specific data for a specific purpose which is alerts, health states and component CIs. Our purpose is then to grab monitoring data from the various monitoring tools and enable someone to see and visualize them in real time and then be able to act on them in real time.