This September, Broadcom (formerly CA) released version 20.2 of its DX Operations Intelligence (DX OI) solution. This product is positioned in the market as an umbrella monitoring system. The system is capable of collecting and combining data from monitoring systems across various domains (network, infrastructure, applications, databases) from both CA and third-party vendors, including open-source solutions (Zabbix, Prometheus, and others).

The primary function of DX OI is to create a comprehensive resource-service model (RSM) based on configuration items (CIs) that populate the inventory database when integrated with external systems. DX OI implements Machine Learning and Artificial Intelligence (ML and AI) functionalities on the incoming platform data, allowing for the assessment/prediction of the failure probability of a specific CI and determining the impact of that failure on the business service that relies on that particular CI. Furthermore, DX OI serves as a single point of collection for monitoring events and, consequently, integrates with the Service Desk system, which is an undeniable advantage of using the system in unified monitoring centers for organizations' duty shifts. In this article, we will discuss the functionality of the system in more detail and show the user and administrator interfaces.

Architecture of the DX OI solution

The DX platform has a microservices architecture and operates under the management of Kubernetes or OpenShift. The following diagram shows the components of the solution that can be used as standalone monitoring tools or replaced with existing monitoring systems with similar functions (the diagram includes examples of such systems) and subsequently connect to the DX OI umbrella. In the diagram below:

• Monitoring of mobile applications is performed in DX App Experience Analytics;
• Application performance monitoring is carried out in DX APM;
• Infrastructure monitoring is handled in DX Infrastructure Manager;
• Network device monitoring is conducted in DX NetOps Manager.

DX components operate under the control of a Kubernetes cluster and scale easily by launching new PODs. Below is a high-level overview diagram of the solution.

Administration, scaling, and updating of the DX platform is performed through the administrative console. From a single console, you can manage a multi-tenant architecture that can encompass multiple enterprises or several business units within a company. In this model, each enterprise can be individually configured as a tenant with its own set of configurations.

The administration console is a web tool for managing operations and systems, providing administrators with a coherent unified interface to perform cluster management tasks.

New tenants for business units or enterprises within a company are deployed in just a few minutes. This provides an advantage if you want to have a unified monitoring system while delineating monitoring objects between departments at the platform level (rather than access rights).

Resource-service models and monitoring of business services

DX OI includes built-in mechanisms for creating services and developing classic RSMs by assigning impact logic and weights between service components. There are also mechanisms for exporting RSMs from external CMDBs. Below is the built-in RSM editor (note the weights of the connections).

DX OI provides a holistic view of key performance indicators for business services or IT services with detailed information, including service availability and risk failure forecasting. The tool can also provide insights into the impact of performance issues or changes in the structure of IT components (applications or infrastructure) on business services. Below is an interactive dashboard displaying the status of all services.

Let's take a closer look using the Digital Banking service as an example. Clicking on the service name takes us to the detailed RSM of the service. We see that the status of the Digital Banking service depends on the state of infrastructural and transactional sub-services with varying weights. Working with weights and displaying them is an interesting advantage of DX OI.

Topology is a crucial element of operational monitoring in the enterprise, allowing operators and engineers to analyze the relationships between components, find root causes, and understand impacts.

DX OI Topology Viewer is a service that utilizes topological data from domain monitoring systems that collect data directly from monitored objects. The tool is designed for searching through multiple layers of topology storage and displaying a context-dependent relationship map. For problem investigation, you can navigate to the problematic Backend Banking subservice and view the topology and problematic components. Additionally, for each component, you can analyze incident reports and performance metrics.

When analyzing the transactional components of Payments (user transactions), we can track the values of business KPIs that are also considered when calculating the service's availability status and health. An example of a business KPI is provided below:

Event Analytics (Alarm Analytics)

Algorithmic noise reduction through clustering of incidents

One of the key features of DX OI in event processing is clustering. The mechanism works on all notifications coming into the system to identify patterns based on various contexts and group them together. These clusters are self-learning and do not require manual configuration.

Thus, clustering allows users to combine and group a vast number of events and analyze only those that share a common context. For example, a set of events representing an incident impacting application or data center operations. Situations are created using clustering algorithms based on machine learning, utilizing temporal correlation, topological relationships, and natural language processing. The images below show examples of clustered message visualizations, known as Situations Alarms, and Evidence Timeline, displaying key grouping parameters and the process of reducing noise events.

Root cause analysis and incident correlation

In a modern hybrid environment, a user transaction may involve multiple systems that are used dynamically. As a result, several alerts may be generated from different systems, all related to the same issue or incident. DX OI employs patented mechanisms to suppress redundant and duplicate alerts and correlate related alerts to enhance the detection of critical issues and facilitate faster resolution.

Let's consider an example where numerous emergency messages are received by the system regarding different objects (KE) underlying a single service. In case of impact on the availability and functionality of the service, the system will generate a service alarm, indicating and marking the probable root cause (the problematic KE and the emergency message for KE) that contributed to the degradation or failure of the service. The image below visualizes the emergency situation for the Webex service.

DX OI allows handling events through intuitive actions in the system's web interface. Users can manually assign events to a responsible employee for troubleshooting, reset/confirm alerts, create tickets, or send email notifications, and run automated scripts for remediation (Remediation Workflow, more on this later). Consequently, DX OI enables duty operators to focus on the root emergency message and helps streamline the process of sorting messages into clustered arrays.

Machine algorithms for metric processing and performance data analysis.

Machine learning enables the tracking, aggregating, and visualizing of key performance indicators over any specific period, providing users with the following benefits:

• Detection of bottlenecks and performance anomalies;
• Comparison of multiple metrics for the same devices, interfaces, or networks;
• Comparison of identical metrics across multiple objects;
• Comparison of various metrics for one or several objects;
• Comparison of multidimensional metrics across several objects.

To analyze incoming metrics, DX OI uses machine analytics functions employing mathematical algorithms, which helps reduce the time required to configure static thresholds and generate alerts when anomalies occur.

The result of applying mathematical algorithms is the construction of so-called probabilistic distributions of the metric value (Rare, Probable, Center, Mean, Actual). The figures above and below illustrate these probabilistic distributions.

The two graphs above display the following data:

• Actual data. Actual data is represented on the graph as a solid black line (indicating no alerts) or a solid colored line (indicating an alert condition). The line is calculated based on the actual data for the metric. By comparing the actual data with the median value, you can quickly see variations in the metric. When an event occurs, the black line changes to a solid colored line corresponding to the severity of the event, displaying corresponding criticality indicators above the graph. For example, red for critical anomalies, orange for significant anomalies, and yellow for minor anomalies.
• Mean value. The mean or average value for the metric is shown on the chart as a gray line. The mean value is displayed when historical data is insufficient.
• Center value. The median line represents the midpoint of the range and is shown as a green dashed line. Areas closest to this line are most typical of the metric values.
• Common value. The data from the common zone tracks the values closest to the central line or the norm for your metric and is displayed as a dark green bar. Analytical calculations place the common zone one percentile above or below the norm.
• Probabilistic data. The data from the probabilistic zone is shown on the graph as a green bar. The system places the probabilistic zone two percentiles above or below the norm.
• Rare data. Information about rare zones is shown on the graph as a light green band. The system places a zone with rare metric values three percentiles above or below the norm and signals when the indicator's behavior is outside the normal range, generating what is known as an Anomaly Alert.

An anomaly is a measurement or event that is inconsistent with normal metric indicators. Detecting anomalies to identify problems and understand trends in infrastructure and applications is a key feature of DX OI. Anomaly detection allows for recognizing unusual behavior (for example, a server responding slower than usual, or unusual network activity caused by a breach) and responding accordingly (creating an incident, initiating an automated Remediation script).

The anomaly detection feature of DX OI provides the following benefits:

• There is no need to set threshold values. DX OI will independently match data and identify anomalies.
• DX OI includes over ten algorithms for artificial intelligence and machine learning, including EWMA (Exponentially Weighted Moving Average) and KDE (Kernel Density Estimation). These algorithms enable quick root cause analysis and forecasting of future metric values.

Predictive analytics and failure alerts

Predictive Insights is a feature that leverages machine learning capabilities to identify patterns and trends. Based on these trends, the system forecasts events that may occur in the future. These messages indicate the need for proactive action before metric values exceed normal levels, potentially impacting critical business services. Predictive Insights are depicted in the figure below.

This is a visualization of predictive alerts for a specific metric.

Forecasting computing capacity load with the capability to set load scenarios

The Capacity Analytics feature helps manage IT resources by ensuring the right size of resources to meet current and future business needs. It allows you to optimize the performance and efficiency of existing resources, plan, and justify any financial investments.

The Capacity Analytics feature in DX OI provides the following benefits:

• Capacity forecasting during peak seasons;
• Determining when additional resources will be required to ensure the quality functioning of the service;
• Purchasing additional resources only when necessary;
• Effective management of infrastructure and networks;
• Eliminating excessive energy costs by identifying underutilized resources;
• Assessing resource load in the event of a planned increase in service or resource needs.

The Capacity Analytics DX OI page (illustration below) contains the following widgets:

• Resource Capacity Status;
• Monitored Groups/Services;
• Top Capacity Consumers.

The main Capacity Analytics page displays resource components that are overburdened and running out of capacity. This page helps platform administrators identify excessively used resources and assists them in resizing and optimizing resources. Resource status can be analyzed based on color codes and their corresponding values. Resources are classified according to the degree of overload on the resource capacity status page. You can click on each color to view the list of components within the selected category. A heat map will then display all objects and forecasts for 12 months, allowing the identification of resources that are about to be exhausted.

For each metric in Capacity Analytics, filters can be specified that DX Operational Intelligence uses to generate forecasts (illustration below).

The following filters are available:

• Metric. The metric that will be used for forecasting.
• Base on. Selection of the volume of historical data to be used for future forecasts. This field is used for comparison and analysis of trends from the last month, trends from the last 3 months, yearly trends, etc.
• Growth. Expected workload growth rate that you want to use for power forecast modeling. This data can be used to predict growth beyond forecasts. For example, resource usage is expected to increase by another 40 percent due to the opening of a new office.

Log Analysis

The log analysis feature in DX OI provides:

• collection and aggregation of logs from various sources (including those obtained through agent and agentless methods);
• parsing and normalization of data;
• analysis for compliance with specified conditions and event generation;
• event correlation based on logs, including events obtained from monitoring IT infrastructure;
• data visualization based on analysis in DX Dashboards;
• conclusions about service availability based on analysis of logs.

Log collection using agentless methods is performed by the system for Windows Event logs and Syslog. Text logs are collected using agent-based methods.

Automated Incident Remediation Feature

Automated remediation actions (Remediation Workflow) allow resolving issues that triggered the event in DX OI. For example, a CPU load issue generates an incident alert, and the remediation process resolves the issue by restarting the server where the problem occurred. Integration between DX OI and the automation system allows remediation processes to be triggered from the event console in DX Operational Intelligence and tracked in the automation system console.

After integration with the automation system, automated remediation actions for any incident can be initiated in the DX OI console from the context of the incident alert. You can review recommended actions along with information about the probability of successfully resolving the issue by executing the action.

Initially, when there is no historical data on the results of Remediation Workflow execution, the recommendation mechanism offers potential options based on keyword searches, followed by machine learning results, and the mechanism begins to recommend remediation methods based on heuristics. As you start evaluating the results of the provided suggestions, the accuracy of the recommendations will improve.

User feedback example: the user selects a liked or disliked proposed action, and the system takes this choice into account for future recommendations. Like/dislike:

Recommended corrective actions for specific alerts are based on a combination of feedback that determines whether the action is acceptable. DX OI comes with built-in integration with Automic Automation system.

Integration of DX OI with third-party systems

We will not delve into the integration of data from Broadcom's native monitoring products (DX NetOps, DX Infrastructure Management, DX Application Performance Management). Instead, we will consider how data from third-party systems is integrated, analyzing an example with one of the most popular systems — Zabbix.

For integration with third-party systems, the DX Gateway component is used. DX Gateway consists of 3 components — On-Prem Gateway, RESTmon, and Log Collector (Logstash). You can install all 3 components or only the one you need by modifying the common configuration file during the DX Gateway installation. Below is the architecture of the DX Gateway.

Let's look at the purpose of the DX Gateway components separately.

On-Prem Gateway. This is an interface that collects alert signals from the DX platform and sends incident events to third-party systems. The On-Prem Gateway acts as a poller, periodically gathering event data from DX OI using API requests over HTTPS protocol, and then sends alerts to a third-party server integrated with the DX platform using webhooks.

DX Log Collector receives syslog from network devices or servers and uploads them to OI. The DX Log Collector allows separating the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is tagged with an object code indicating the type of software generating the message and is assigned a severity level. All this can then be viewed in DX Dashboards.

DX RESTmon integrates with third-party products/services via REST API and transmits data to OI. Below is a diagram illustrating the operation of DX RESTmon with integration examples with Solarwinds and SCOM monitoring systems.

Key features of DX RESTmon:

• Connect to any third-party data source to receive data:
  • PULL: connect and extract data from public REST APIs;
  • PUSH: stream data into RESTmon via REST.
• Support for JSON and XML formats;
• Receiving metrics, alerts, groups, topology, inventory, and logs;
• Ready-made connectors for various tools/technologies, with the possibility to develop a connector to any source with an open API (a list of out-of-the-box connectors is shown below);
• Support for basic authentication (by default) when accessing the Swagger interface and API;
• Support for HTTPS (by default) for all incoming and outgoing messages;
• Support for incoming and outgoing proxies;
• Powerful text parsing capabilities for logs received via REST;
• Customizable parsing with RESTmon for effective log analysis and visualization;
• Support for extracting information about device groups from monitoring applications and uploading it to OI for analysis and visualization;
• Support for regular expression matching. This can be used for parsing and matching log messages received via REST, as well as for generating or closing events based on specific regular expression conditions.

Now let’s discuss the process of setting up DX OI integration with Zabbix via DX RESTmon. The out-of-the-box integration pulls the following data from Zabbix:

• inventory data;
• topology;
• issues;
• metrics.

Since the connector for Zabbix is available out of the box, all that needs to be done to set up the integration is to update the profile by specifying the Zabbix API server IP address and account, and then upload the profile through the Swagger web interface. An example is shown in the next two figures.

After setting up the integration, the analytical features of DX OI described above will be available for data coming from Zabbix, namely: Alarm Analytics, Performance Analytics, Predictive Insights, Service Analytics, and Remediation. The figure below provides an example of performance metrics analysis for objects integrated from Zabbix.

Conclusion

DX OI — a modern analytics tool that will provide significant operational efficiency for IT departments, enabling faster and better decision-making to enhance the quality of IT services and business services through cross-domain contextual analysis. For application owners and business units, DX OI will calculate service availability and quality not only in terms of technological IT metrics but also business KPIs derived from transactional statistics of end users.

If you would like to learn more about this solution, please request a demonstration or pilot project. in a way that is convenient for you on our website.

Source: habr.com