Synthetic Transaction Monitoring for Microsoft 365

Originally posted on gsx.com

Now organizations are using Digital Experience Platforms (DXPs) to provide complete multi-channel digital experiences for their employees and partners at all their sites. Gartner defines a digital experience platform as “an integrated set of technologies, based on a common platform, that provides a broad range of audiences with consistent, secure and personalized access to information and applications across many digital touchpoints.”

Microsoft is one of the most prominent DXP vendors. It offers a broad range of capabilities, across Azure, Dynamics Business Solutions and Office 365, that can be used to address DXP requirements. In particular, Microsoft offers deep and readily deployable capability for B2E DXP scenarios, as part of Office 365, SharePoint Online and Microsoft Teams are the focus of most DXP efforts by Microsoft customers; it provides content services, a front end for building and managing sites and portals and the collaboration features through chat, MS Teams channel and even Voice.

Satisfactory user experience is the reward if all the parts of DXP are working together smoothly. But what does “satisfactory” mean? Does it only mean that no users are calling to complain or filing tickets? These are all subjective measures that do not include any data to describe what is going on.

The way to keep track of DXP is with digital experience monitoring (DEM). Gartner defines DEM as “an availability and performance monitoring discipline that supports the optimization of the operational experience and behavior of a digital agent, human or machine, as it interacts with enterprise applications and services. For the purposes of this evaluation, it includes real-user monitoring (RUM) and synthetic transaction monitoring (STM) for both web- and mobile-based end users.”

DEM is a newly defined category of application performance monitoring (APM). Users want to see their applications on all their screens as running smoothly with a singular, recognizable experience. It is essential to ensure an optimal user experience at every contact point. DEM is a real-time monitoring tool that keeps track of that.

Synthetic transaction monitoring is a method to monitor your performance by simulating users. It works by issuing automated, simulated transactions from a robot client to your application in order to mimic what a typical user might do. The results are measured, collected and reported as required. This provides hard data to respond to the user who says, “Office 365 is responding slowly today in Manila.”

The Office 365 end-user experience

There’s a lot involved between a user request made on a local workstation and the response by Office 365; local network connectivity, DNS, Internet routing, latency, authentication, etc. all can go wrong, resulting in a bad user experience. And IT is responsible, no matter where an issue lies. But by measuring the user experience – putting objective metrics to the end-to-end experience had by users – organizations can quantify productivity.

There are three objectives in deploying and using Office 365:

• Improving organizational productivity
• Improving service quality
• Lowering the total cost of ownership

It is important to be able to measure the user experience to determine to what extent the three objectives are being met.

There are many factors that can influence the user experience. These make it a challenge to determine, if there’s a problem, what the root issue is. These factors include:

• User Hardware
• WiFi/Network Connectivity
• Security Solutions (Firewalls, SSL decryption, etc.)
• Internet Access & Routing
• Authentication (e.g., Azure AD Sync, ADFS)
• Hybrid Components (e.g., Exchange Server, Edge)
• Microsoft Infrastructure
• Network Latency

Microsoft offers a health status for all its services at status.office365.com. This provides the status of all services that are a part of your subscription, as well as any known incidents and advisories. But this only covers the status within the “walls” of Microsoft 365, which is just one part of the user experience tale.

Client-side monitoring also shows only part of the path. Because we’re talking about monitoring users interacting with a SaaS application, the measurement must represent the entire path from the client to the service and back.

Digital experience monitoring (DEM) with synthetic transaction monitoring is the way to measure the Microsoft 365 end-user experience.

Real user interaction with Microsoft 365 is mimicked– including authentication, file upload/download, and use of individual services. Measured experiential metrics that can be used to identify performance issues and root causes of problems. Because the transactions occur within the very same paths that users themselves take, they offer a real-world view into what users are experiencing and provide metrics before end users notice any issues.

Go Monitor Your System Right Now!

There are a few points in the journey to Microsoft 365 where monitoring the user experience via DEM synthetic transaction monitoring can be useful. The first point is right now, before you do anything else! This ensures that your monitoring is working properly and establishes a baseline for all the changes you and others may make in the future. Many set out without recording where they began. This makes it impossible to calculate any deltas, including the top-level value ROI. No matter how eager you are to make changes, taking the time to measure now will be vastly rewarded later.

Monitor the Architecture Before You Deploy

Now that you have monitored the existing status, you can think about the architecture to connect your IT network to Microsoft 365. Microsoft has published guidance for connectivity to Microsoft 365. By all means follow their best practices, but then you should test the user experience with your intended proxy servers, VPNs, firewalls, etc.

Consider performing tests on a site-by-site basis to ensure consistent performance. Also, if possible, consider simulating a workload representative of the planned workforce. You can use this to determine what the experience will be like. You can see the impact of removing any parts of the current network path on the speed of interaction between the user and Microsoft 365. By testing the user experience using synthetic transactions before any actual deployment occurs, IT can prevent unexpected issues and confirm the architecture.

Now Monitor Your Deployment

The IT department has a plan of how a deployment “should” go. Testing the user experience for each phase of the deployment ensures that the plan’s assumed performance for that phase is actually met. It also allows IT to ensure a continuous level of service quality is maintained throughout the deployment. If not, issues can be resolved before proceeding to the next phase.

It is imperative that the IT department understands whether users who have recently made the switch to Microsoft 365 are enjoying equivalent levels of perceived performance to their previously on-premises service counterparts (such as utilizing Exchange on-premises to Exchange Online). Monitoring user experience at this stage helps to ensure adoption and provide a feedback loop to make IT aware of any performance issues users are experiencing – in this case – well-before users even know that what they are experiencing isn’t “normal.” Users may not even know what to expect and may think it is convenient to just work around longer response times.

Monitoring during deployment helps to ensure the projected costs remain under budget.

Monitor Continuously During Operation

Ongoing monitoring is necessary. This is how you can ensure service delivery and service quality. Through monitoring you can spot issues before they impact users. If an issue arises, monitoring provides IT staff with context and detail so that they can better understand the problem and formulate a potential solution.

Monitoring is about putting objective metrics in place to represent what is otherwise a subjective “good” or “bad” user experience. It is not just “up” or “down.” Ongoing Microsoft 365 synthetic transaction monitoring helps IT organizations in several ways:

• Understanding the performance impact any network issues have on the Microsoft 365 end-users
• Providing service quality data to Microsoft, if they need to be involved in resolving an issue
• Calculating the performance gains and resulting ROI when making improvements that positively impact Microsoft 365 users

IT organizations can lower the overall cost of their Microsoft 365 investment through improved usage and reduced support costs.

“Outlook seems slow in Berlin” – monitoring helps you to troubleshoot

Imagine a user calling the helpdesk who says, “Outlook seems slow in Berlin”. This is merely a personal perception with no quantifiable data. Is Outlook really slow in Berlin?  There is no directional guidance on where helpdesk teams can start. If Outlook really is slow in Berlin, there may be many more users experiencing the same issue that didn’t call the helpdesk. Maybe even at other locations.

Microsoft support isn’t going to go much farther than “our services are working.” If the client itself is not the issue, the IT staff only knows that each end of the interaction is functioning.

Using synthetic transactions to simulate the Microsoft 365 user experience can help your IT department to look beyond basic network-related issues and narrow down the root cause of the issue. By testing the user experience using different conditions such as location, use of security solutions or network services that cause indirect routes to Microsoft 365, and authentication methods, organizations can work to isolate where exactly the issue lies.

For example. If you believe a proxy server is the culprit, you can simulate a user that bypasses the proxy and produce its connectivity metrics. This would be nearly impossible without synthetic transaction monitoring.

Optimize Your Network with the Help of Monitoring

You can assume that your organization will change over time. There may be new locations, acquisitions, new security and compliance initiatives, or improvements in network connectivity. Your IT department needs to understand exactly how these changes affect Microsoft 365 use by testing their impact on performance.

Maybe your organization has moved to the cloud before upgrading the local network that was initially designed for local service delivery. This may result in slower cloud performance – because of the high reliance on Microsoft 365 to be productive. Change then comes in adopting the network connectivity best practices outlined by Microsoft.

Whether change comes through evolution or a planned optimization, monitoring the user experience via synthetic transactions helps ensure that each change intended to improve the connection between the user and Microsoft 365 performs as expected.