InterSystems IRIS is a universal real-time AI/ML platform

Author: Sergey Lukyanchikov, InterSystems Consulting Engineer

Real-time AI/ML Computing Challenges

Let's start with examples from the experience of Data Science practice at InterSystems:

• The “loaded” buyer portal is connected to an online recommender system. Restructuring of promotions on the scale of the retail network is coming (for example, instead of a “flat” line of promotions, the “segment-tactics” matrix will now be used). What happens to recommenders? What happens with the submission and updating of data to the recommender mechanism (the volume of input data has increased by 25000 times)? What happens to the development of recommendations (the need for a thousandfold decrease in the filtering threshold of recommendatory rules due to a thousandfold increase in their number and “assortment”)?
• There is a system for monitoring the probability of development of defects in equipment nodes. A process control system was connected to the monitoring system, transmitting thousands of process parameters every second. What happens to the monitoring system that previously worked on “manual samples” (is it capable of providing second-to-second probability monitoring)? What will happen if a new block with several hundred columns appears in the input data with the readings of sensors recently introduced into the process control system (whether and how long will it be necessary to stop the monitoring system to include data from new sensors in the analysis)?
• A set of AI/ML mechanisms (recommendatory, monitoring, predictive) has been created, using the results of each other's work. How many man-hours are required monthly to adapt the operation of this complex to changes in the input data? What is the overall “slowdown” supported by the decision-making complex (frequency of occurrence of new supporting information in it relative to the frequency of occurrence of new input data)?

Summarizing these and many other examples, we came to the formulation of the challenges that arise when moving to the use of machine learning and real-time artificial intelligence mechanisms:

• Are we satisfied with the speed of creation and adaptation (to the changing situation) of AI / ML developments in our company?
• To what extent do the AI/ML solutions we use support real-time business management?
• Are the AI/ML solutions we use able to independently (without developers) adapt to changes in data and business management practices?

Our article is a detailed overview of the capabilities of the InterSystems IRIS platform in terms of universal support for the deployment of AI / ML mechanisms, assembly (integration) of AI / ML solutions and training (testing) of AI / ML solutions on intensive data flows. We will turn to market research, case studies of AI/ML solutions, and conceptual aspects of what we refer to as a real-time AI/ML platform in this article.

What we know from surveys: real-time applications

The results surveyconducted among about 800 IT professionals in 2019 by Lightbend speak for themselves:

Figure 1 Leading consumers of real-time data

Let us quote the fragments of the report on the results of this survey that are important for us in our translation:

“…Trends in the popularity of dataflow integration tools and, at the same time, support for containerized computing provide a synergistic response to the market demand for a more efficient, rational, dynamic proposal of effective solutions. Data streams allow information to be transferred faster than traditional packet data. Added to this is the ability to quickly apply computational methods, such as AI/ML-based recommendations, to create competitive advantages through increased customer satisfaction. The race for agility also impacts all roles in the DevOps paradigm – making application development and deployment more efficient. … Eight hundred and four IT professionals provided information on the use of data flows in their organizations. Respondents were predominantly located in Western countries (41% in Europe and 37% in North America) and were almost evenly distributed among small, medium and large companies. …

… Artificial intelligence is not hype. Fifty-eight percent of those who already use dataflow processing in productive AI/ML applications confirm that their use in AI/ML will see the biggest gains next year (compared to other applications).

• According to the majority of respondents, the use of data flows in AI / ML scenarios will receive the largest increase in the next year.
• Applications in AI/ML will grow not only through relatively new types of scenarios, but also through traditional scenarios, in which real-time data is increasingly used.
• In addition to AI/ML, the level of enthusiasm among users of IoT data pipelines is impressive - 48% of those who have already integrated IoT data say that the implementation of scenarios on this data will receive a significant increase in the near future. … »

From this rather interesting survey, it can be seen that the perception of machine learning and artificial intelligence scenarios as leaders in the consumption of data streams is already “on the way”. But the perception of real-time AI / ML through DevOps optics becomes an equally important observation: here we can already begin to talk about the transformation of the still dominant culture of “one-time AI / ML with a fully accessible data set”.

Real-time AI/ML platform concept

One typical application for real-time AI/ML is in manufacturing process control. On its example and taking into account previous reflections, we will formulate the concept of a real-time AI / ML platform.
The use of artificial intelligence and machine learning in process control has a number of features:

• Data on the state of the technological process is received intensively: with a high frequency and over a wide range of parameters (up to tens of thousands of parameter values ​​transmitted per second from the process control system)
• Data on the detection of defects, not to mention data on their development, on the contrary, are scarce and irregular, characterized by insufficient typing of defects and their localization in time (often represented by records on paper)
• From a practical point of view, only the “relevance window” of the initial data is available for training and applying models, reflecting the dynamics of the technological process for a reasonable sliding interval, ending with the last read values ​​of the process parameters

These features force us, in addition to receiving and basic real-time processing of intense “broadband input” from the process, to perform (in parallel) the application, training and quality control of the results of the work of AI / ML models - also in real time. The “frame” that our models “see” in the sliding window of relevance is constantly changing – and with it, the quality of the results of the work of AI / ML models trained on one of the “frames” in the past also changes. If the quality of the results of the work of AI / ML models deteriorates (for example: the value of the “alarm-norm” classification error has gone beyond the boundaries we have defined), retraining of models should be automatically started on a more relevant “frame” - and the choice of the moment to start retraining of models should take into account how the duration of the training itself, as well as the dynamics of deterioration in the quality of the current version of the models (since the current versions of the models continue to be applied while the models are being trained, and until their “newly trained” versions are generated).

InterSystems IRIS has key platform capabilities to enable real-time AI/ML solutions for process control. These possibilities can be divided into three main groups:

• Continuous deployment (Continuous Deployment / Delivery, CD) of new or adapted existing AI / ML mechanisms into a productive solution that operates in real time on the InterSystems IRIS platform
• Continuous Integration (CI) into a single productive solution of incoming process data flows, data queues for the application / training / quality control of AI / ML mechanisms and data / code / control exchanges with mathematical modeling environments, which are orchestrated in real-time InterSystems IRIS platform
• Continuous (self-) learning (Continuous Training, CT) of AI / ML mechanisms performed in mathematical modeling environments using data, code and control actions (“decisions made”) transmitted by the InterSystems IRIS platform

The classification of platform capabilities in relation to machine learning and artificial intelligence in precisely such groups is not accidental. Let us quote the methodological the publication of Google, which provides a conceptual basis for this classification, in our translation:

“… The DevOps concept, which is popular these days, covers the development and operation of large-scale information systems. The advantages of implementing this concept are the reduction in the duration of development cycles, the acceleration of development deployment, the flexibility of release planning. To achieve these benefits, DevOps involves the implementation of at least two practices:

• Continuous Integration (CI)
• Continuous Delivery (CD)

These practices also apply to AI/ML platforms to ensure robust and performant builds of productive AI/ML solutions.

AI/ML platforms differ from other information systems in the following aspects:

• Team Competencies: When building an AI/ML solution, the team typically includes data scientists or data scientists who conduct data analysis, model development, and validation. These team members may not be professional developers of productive code.
• Development: AI/ML mechanisms are experimental in nature. In order to solve the problem in the most efficient way, it is required to sort through various combinations of input variables, algorithms, modeling methods and model parameters. The complexity of such a search lies in tracing “what worked / did not work”, ensuring reproducibility of episodes, and generalizing developments for recurring implementations.
• Testing: Testing AI/ML mechanisms requires a larger range of tests than most other developments. In addition to typical unit and integration tests, the validity of the data and the quality of the results of applying the model to training and control samples are tested.
• Deployment: Deployment of AI/ML solutions is not limited to predictive services that apply a once trained model. AI / ML solutions are built around multi-stage pipelines that perform automated training and application of models. Deploying such pipelines involves automating non-trivial steps traditionally performed manually by data scientists in order to be able to train and test models.
• Productive: AI/ML engines can lack performance not only due to inefficient programming, but also due to the ever-changing nature of the input data. In other words, the performance of AI/ML mechanisms can degrade due to a wider range of reasons than the performance of conventional developments. This results in the need to monitor (online) the performance of our AI/ML engines, and send alerts or reject results if performance is not up to expectations.

AI/ML platforms are similar to other information systems in that both require continuous code integration with version control, unit testing, integration testing, continuous development deployment. However, in the case of AI/ML, there are a few important differences:

• CI (Continuous Integration) is no longer limited to testing and validating the code of deployed components – it also includes testing and validating data and AI/ML models.
• CD (Continuous Delivery / Deployment, continuous deployment) is not limited to writing and releasing packages or services, but implies a platform for composing, learning and applying AI / ML solutions.
• CT (Continuous Training, continuous learning) - a new element [approx. by the author of the article: a new element in relation to the traditional concept of DevOps, in which CT is usually Continuous Testing], inherent in AI / ML platforms, responsible for autonomously managing the mechanisms for learning and applying AI / ML models. ... "

We can state that machine learning and artificial intelligence working on real-time data require a wider set of tools and competencies (from code development to orchestration of mathematical modeling environments), closer integration between all functional and subject areas, more efficient organization of human and machine resources.

Real-time Scenario: Recognizing the Development of Defects in Feed Pumps

Continuing to use the field of process control as an example, consider a specific task (already mentioned by us at the very beginning): it is required to provide real-time monitoring of the development of defects in pumps based on the flow of process parameters and reports of maintenance personnel on detected defects.

Figure 2 Formulation of the task of monitoring the development of defects

The peculiarity of the majority of tasks set in this way in practice is that the regularity and efficiency of data receipt (APCS) should be considered against the background of episodic and irregular occurrence (and registration) of various types of defects. In other words: the data from the process control system come once a second correct-accurate, and defects are recorded with an indelible pencil with the date in the general notebook in the workshop (for example: “12.01 - leak into the cover from the side of the 3rd bearing”).

Thus, it is possible to supplement the formulation of the problem with the following important limitation: we have only one "label" of a defect of a particular type (i.e., an example of a defect of a particular type is represented by data from the process control system for a specific date - and we do not have more examples of a defect of this particular type). This restriction immediately takes us beyond the scope of classical machine learning (supervised learning), for which there should be a lot of “labels”.

Figure 3 Refinement of the task of monitoring the development of defects

Can we somehow "multiply" the only "label" at our disposal? Yes we can. The current state of the pump is characterized by the degree of similarity to registered defects. Even without the use of quantitative methods, at the level of visual perception, observing the dynamics of the data values ​​coming from the process control system, you can already learn a lot:

Figure 4 Dynamics of the state of the pump against the background of the “mark” of a defect of a given type

But visual perception (at least for now) is not the most suitable generator of "tags" in our rapidly changing scenario. We will evaluate the similarity of the current state of the pump to the reported defects using a statistical test.

Figure 5 Applying a statistical test to incoming data against the background of a “label” of a defect

The statistical test determines the probability that the records with the values ​​of the technological process parameters in the "flow-packet" received from the process control system are similar to the "label" records of a certain type of defect. The probability value calculated as a result of applying a statistical test (statistical similarity index) is converted to a value of 0 or 1, becoming a “label” for machine learning in each specific record in the similarity package. That is, after processing the newly received package of pump state records with a statistical test, we have the opportunity to (a) add this package to the training sample for training the AI ​​/ ML model and (b) monitor the quality of the current version of the model when it is applied to this package.

Figure 6 Applying a machine learning model to incoming data against the background of a “label” of a defect

In one of our previous webinars we show and explain how the InterSystems IRIS platform allows you to implement any AI / ML mechanism in the form of continuously executing business processes that control the reliability of simulation results and adapt model parameters. When implementing the prototype of our scenario with pumps, we use all the InterSystems IRIS functionality presented during the webinar - implementing in the analyzer process as part of our solution, not classical supervised learning, but rather reinforcement learning, which automatically controls the sample for training models. Records are placed in the training sample on which a “detection consensus” occurs after applying both the statistical test and the current version of the model - i.e. both the statistical test (after the transformation of the similarity index to 0 or 1), and the model produced a result on such records 1. With a new training of the model, during its validation (the newly trained model is applied to its own training sample, with a preliminary application of a statistical test to it), records that “did not hold” the result 1 after processing by the statistical test (due to the constant presence in the training the sample of records from the original "label" of the defect) are removed from the training sample, and the new version of the model learns from the "label" of the defect plus the "held" records from the stream.

Figure 7 Robotization of AI/ML calculations in InterSystems IRIS

If there is a need for a kind of “second opinion” on the quality of detection obtained by local computing in InterSystems IRIS, an advisor process is created to perform training-applying models on a control dataset using cloud services (for example, Microsoft Azure, Amazon Web Services , Google Cloud Platform, etc.):

Figure 8 Second Opinion from Microsoft Azure orchestrated by InterSystems IRIS

The prototype of our scenario in InterSystems IRIS is made in the form of an agent-based system of analytical processes that interact with the equipment object (pump), mathematical modeling environments (Python, R and Julia), and provide self-learning of all involved AI / ML mechanisms - on real-time data flows .

Figure 9 Main functionality of real-time AI/ML solution in InterSystems IRIS

The practical result of our prototype:

• Defect pattern recognized by the model (January 12):

• A developing defect recognized by the model, which was not included in the sample (September 11, the defect itself was ascertained by the repair team only two days later - September 13):

Simulation on real data containing several episodes of the same defect showed that our solution, implemented on the InterSystems IRIS platform, allows us to detect the development of defects of this type several days before they are detected by the repair team.

InterSystems IRIS - universal real-time AI/ML computing platform

The InterSystems IRIS platform simplifies the development, deployment, and operation of real-time data solutions. InterSystems IRIS is able to simultaneously perform transactional and analytical data processing; maintain synchronized views of data in accordance with several models (including relational, hierarchical, object and document); act as an integration platform for a wide range of data sources and individual applications; provide advanced real-time analytics on structured and unstructured data. InterSystems IRIS also provides mechanisms for the use of external analytical tools, allows the flexibility to combine hosting in the cloud and on local servers.

Applications built on the InterSystems IRIS platform have been deployed across industries, helping companies achieve significant economic value from a strategic and operational perspective, enhancing decision making and closing the gaps between event, analysis and action.

Figure 10 InterSystems IRIS architecture in the context of real-time AI/ML

Like the previous diagram, the diagram below combines the new "coordinate system" (CD/CI/CT) with the flow of information between platform work items. Visualization begins with the CD macro-mechanism and continues with the CI and CT macro-mechanisms.

Figure 11 Scheme of information flows between AI/ML elements of the InterSystems IRIS platform

The essence of the CD mechanism in InterSystems IRIS: platform users (AI / ML solution developers) adapt existing and / or create new AI / ML developments using a specialized AI / ML mechanism code editor: Jupyter (full name: Jupyter Notebook; also, for brevity, documents created in this editor are sometimes called). In Jupyter, a developer has the opportunity to write, debug and verify the performance (including using graphics) of a specific AI / ML development before it is hosted (“deployed”) in InterSystems IRIS. It is clear that a new development created in this way will receive only basic debugging (because, in particular, Jupyter does not work with real-time data streams) - this is in the order of things, because the main result of development in Jupyter is confirmation of the fundamental operability of a separate AI / ML-mechanism (“shows the expected result on the data sample”). Similarly, a mechanism already placed in the platform (see the following macro-mechanisms) before debugging in Jupyter may require a "rollback" to the "pre-platform" form (reading data from files, working with data through xDBC instead of tables, direct interaction with globals - multidimensional data arrays InterSystems IRIS - etc.).

An important aspect of CD implementation in InterSystems IRIS is that bidirectional integration is implemented between the platform and Jupyter, which allows you to transfer to the platform (and, further, process in the platform) content in Python, R and Julia (all three are programming languages ​​in the corresponding leading open-source source environments of mathematical modeling). Thus, AI/ML content developers have the ability to “continuously deploy” this content to the platform, working in their familiar Jupyter editor, with familiar libraries available in Python, R, Julia, and performing basic debugging (if necessary) off the platform .

Let's move on to the macro mechanism of CI in InterSystems IRIS. The diagram shows the macro-process of the "real-time robot" (a complex of data structures, business processes and code fragments orchestrated by them in the languages ​​​​of mats and the ObjectScript language - the native development language of InterSystems IRIS). The task of this macro process is to maintain the data queues necessary for the operation of AI / ML mechanisms (based on data flows transmitted to the platform in real time), make decisions about the sequence of application and the “range” of AI / ML mechanisms (they are also “mathematical algorithms”, “ models", etc. - may be called differently depending on the implementation specifics and terminological preferences), keep data structures up to date for analyzing the results of the work of AI / ML mechanisms (cubes, tables, multidimensional data arrays, etc.). etc. - for reports, dashboards, etc.).

An important aspect of the implementation of CI in InterSystems IRIS is that bidirectional integration is implemented between the platform and mathematical modeling environments, which allows executing the content hosted in the platform in Python, R and Julia in their respective environments with the return of execution results. This integration is implemented in both "terminal mode" (i.e. AI/ML content is formulated as ObjectScript code making calls to mats) and "business process mode" (i.e. AI/ML content is formulated as as a business process using a graphic editor, or sometimes using Jupyter, or using an IDE - IRIS Studio, Eclipse, Visual Studio Code). The editability of business processes in Jupyter is reflected by the relationship between IRIS at the CI level and Jupyter at the CD level. A more detailed overview of integration with mathematical modeling environments is given below. At this stage, in our opinion, there is every reason to fix the presence in the platform of all the necessary tools to implement the "continuous integration" of AI / ML developments (coming from "continuous deployment") into real-time AI / ML solutions.

And the main macro mechanism: CT. Without it, there will be no AI / ML platform (although “real time” will be implemented through CD / CI). The essence of CT is the work of the platform with the “artifacts” of machine learning and artificial intelligence directly in the working sessions of mathematical modeling environments: models, distribution tables, matrix vectors, layers of neural networks, etc. This "work", in most cases, consists in creating the mentioned artifacts in the environments (in the case of models, for example, "creation" consists of setting the specification of the model and the subsequent selection of the values ​​of its parameters - the so-called "training" of the model), their application (for models: using them to calculate “model” values ​​of target variables – forecasts, belonging to a category, the probability of an event occurring, etc.) and improving already created and applied artifacts (for example, redefining the set of model input variables based on the results of application – in order to increase prediction accuracy, as an option). The key point in understanding the role of CT is its "abstraction" from the realities of CD and CI: CT will implement all artifacts, focusing on the computational and mathematical specifics of the AI ​​/ ML solution within the capabilities provided by specific environments. Responsibility for "providing input data" and "delivering results" will be the responsibility of CD and CI.

An important aspect of CT implementation in InterSystems IRIS: using the integration with mathematical modeling environments already mentioned above, the platform has the ability to extract the same artifacts from the work sessions running under its control in the environments and (most importantly) turn them into platform data objects. For example, a distribution table that has just been created in a working Python session can be (without stopping the session in Python) transferred to the platform in the form of, for example, a global (multidimensional InterSystems IRIS data array) - and used for calculations in another AI / ML- mechanism (already implemented in the language of another environment - for example, in R) - or a virtual table. Another example: in parallel with the “normal mode” of the model operation (in the Python working session), “auto-ML” is carried out on its input data: automatic selection of optimal input variables and parameter values. And along with the “regular” training, the productive model in real time also receives a “optimization proposal” for its specification - in which the set of input variables changes, the parameter values ​​change (not as a result of training in Python, but as a result of training an “alternative version of itself, for example, in the H2O stack), allowing the overall AI/ML solution to autonomously cope with unforeseen changes in the nature of the input data and the phenomena being modeled.

Let's get acquainted in more detail with the platform AI / ML functionality of InterSystems IRIS, using the example of a real-life prototype.

In the diagram below, on the left side of the slide, there is a part of the business process that implements the processing of scripts in Python and R. In the central part, there are visual logs for the execution of some of these scripts, respectively, in Python and R. Right behind them are examples of content on one and another language, submitted for execution in the appropriate environments. At the end on the right are visualizations based on the results of script execution. The visualizations at the top are made on IRIS Analytics (data are taken from Python to the InterSystems IRIS data platform and displayed on the dashboard using the platform), at the bottom they are made right in the R working session and output from there to graphic files. An important aspect: the presented fragment in the prototype is responsible for training the model (classification of equipment states) on data coming in real time from the process-simulator of the equipment, on command from the process-monitor of the quality of the classification observed during the application of the model. The implementation of an AI/ML solution as a set of interacting processes (“agents”) will be discussed further.

Figure 12 Interaction with Python, R and Julia in InterSystems IRIS

Platform processes (they are also “business processes”, “analytical processes”, “pipelines”, etc. - depending on the context), are primarily edited in a graphical business process editor in the platform itself, and in such a way that both its block diagram and the corresponding AI/ML mechanism (program code) are created simultaneously. Speaking about the fact that “an AI / ML mechanism is obtained”, we initially mean hybridity (within the same process): content in languages ​​of mathematical modeling environments is adjacent to content in SQL (including extensions from IntegratedML), in InterSystems ObjectScript, with other supported languages. Moreover, the platform process provides very wide possibilities for “drawing” in the form of hierarchically nested fragments (as can be seen in the example in the diagram below), which allows you to effectively organize even very complex content without “falling out” of the graphic format anywhere (into “non-graphic » methods/classes/procedures, etc.). That is, if necessary (and it is expected in most projects), absolutely all AI / ML solutions can be implemented in a graphical self-commenting format. Please note that in the central part of the diagram below, which shows a higher “nesting level”, you can see that in addition to the actual work of training the model (using Python and R), an analysis of the so-called ROC curve of the trained model is added, which allows visually (and computationally too) evaluate the quality of training - and this analysis is implemented in the Julia language (it is executed, respectively, in the Julia environment).

Figure 13 Visual environment for composing AI/ML solutions in InterSystems IRIS

As mentioned earlier, the initial development and (in some cases) adaptation of AI / ML mechanisms already implemented in the platform will / can be done outside the platform in the Jupyter editor. In the diagram below, we see an example of adapting an existing platform process (the same as in the diagram above) - this is how the fragment that is responsible for training the model looks in Jupyter. Python content is available for editing, debugging, graphics output directly in Jupyter. Changes (if necessary) can be made with instant synchronization to the platform process, including its production version. Similarly, new content can be transferred to the platform (a new platform process is automatically generated).

Figure 14 Using Jupyter Notebook to Edit AI/ML Engine in InterSystems IRIS Platform

Adaptation of the platform process can be performed not only in a graphical or notebook format, but also in the “total” IDE (Integrated Development Environment) format. These IDEs are IRIS Studio (native IRIS studio), Visual Studio Code (InterSystems IRIS extension for VSCode), and Eclipse (Atelier plugin). In some cases, it is possible for the development team to use all three IDEs at the same time. The diagram below shows an example of editing the same process in IRIS studio, in Visual Studio Code and in Eclipse. Absolutely all content is available for editing: Python / R / Julia / SQL, and ObjectScript, and a business process.

Figure 15 Development of the InterSystems IRIS business process in various IDEs

The InterSystems IRIS business process description and execution tools in the Business Process Language (BPL) deserve special mention. BPL makes it possible to use “ready-made integration components” (activities) in business processes - which, in fact, gives full reason to assert that “continuous integration” is implemented in InterSystems IRIS. Ready-made business process components (activities and links between them) are the most powerful accelerator for assembling an AI/ML solution. And not only assemblies: thanks to the activities and connections between them, over disparate AI / ML developments and mechanisms, an “autonomous management layer” appears that is able to make decisions according to the situation, in real time.

Figure 16 Ready-made business process components for continuous integration (CI) on the InterSystems IRIS platform

The concept of agent systems (they are also “multi-agent systems”) has a strong position in robotics, and the InterSystems IRIS platform organically supports it through the “product-process” construct. In addition to unlimited possibilities for “stuffing” each process with the functionality necessary for the overall solution, endowing the system of platform processes with the property of “agency” allows you to create effective solutions for extremely unstable simulated phenomena (behavior of social/biosystems, partially observable technological processes, etc.).

Figure 16 Operation of AI/ML solution as a business process agent system in InterSystems IRIS

We continue our review of InterSystems IRIS with a story about the application of the platform for solving entire classes of real-time problems (a fairly detailed acquaintance with some of the best practices of platform AI / ML on InterSystems IRIS occurs in one of our previous webinars).

In hot pursuit of the previous diagram, below is a more detailed diagram of the agent system. The diagram shows the same prototype, all four agent processes are visible, the relationships between them are schematically drawn: GENERATOR - handles the creation of data by equipment sensors, BUFFER - manages data queues, ANALYZER - performs machine learning itself, MONITOR - controls the quality of machine learning and feeds a signal that the model needs to be retrained.

Figure 17 Composition of an AI/ML solution as a business process agent system in InterSystems IRIS

The diagram below illustrates the autonomous functioning of another robotic prototype (emotional text recognition) for some time. In the upper part - the evolution of the model learning quality indicator (the quality is growing), in the lower part - the dynamics of the model application quality indicator and the facts of repeated training (red bars). As you can see, the solution has self-learned efficiently and autonomously, and works at a given quality level (the values ​​of the quality indicator do not fall below 80%).

Figure 18 Continuous (self-)learning (CT) on the InterSystems IRIS platform

We also mentioned “auto-ML” earlier, but the diagram below shows the application of this functionality in detail using another prototype as an example. The graphical diagram of the business process fragment shows the activity that starts the simulation in the H2O stack, shows the results of this simulation (obvious dominance of the resulting model over the "man-made" models, according to the comparative diagram of ROC curves, as well as automated identification of the "most influential variables" from those available in original data set). The important point here is the saving of time and expert resources, which is achieved through "auto-ML": what our platform process does in half a minute (finding and training the optimal model), an expert can take from a week to a month.

Figure 19 Auto-ML integration into AI/ML solution based on InterSystems IRIS platform

The diagram below “knocks down the climax” a little, but this is a good way to complete the story about the classes of real-time problems being solved: we remind you that with all the capabilities of the InterSystems IRIS platform, training models under its control is not mandatory. The platform can externally obtain a so-called PMML model specification trained in a tool that is not controlled by the platform - and apply this model in real time from the moment it is imported PMML specifications. At the same time, it is important to take into account that not all AI / ML artifacts can be reduced to a PMML specification, even if most of the most common artifacts allow this. Thus, the InterSystems IRIS platform has an "open loop" and does not mean "platform slavery" for users.

Figure 20 Auto-ML integration into AI/ML solution based on InterSystems IRIS platform

We list the additional platform advantages of InterSystems IRIS (for clarity, in relation to process control), which are of great importance in the automation of artificial intelligence and real-time machine learning:

• Advanced integration tools with any data sources and consumers (PCS/SCADA, equipment, MRO, ERP, etc.)
• Built-in multi-model DBMS for high-performance transactional-analytical processing (Hybrid Transaction/Analytical Processing, HTAP) of any volumes of process data
• Development tools for continuous deployment of real-time AI/ML decision engines based on Python, R, Julia
• Adaptive business processes for continuous integration and (self-)learning mechanisms of real-time AI/ML solutions
• Embedded Business Intelligence tools for visualizing process data and AI/ML solution results
• API Management to deliver the results of the AI ​​/ ML solution to process control systems / SCADA, information and analytical systems, sending alerts, etc.

AI / ML solutions based on the InterSystems IRIS platform easily fit into the existing IT infrastructure. The InterSystems IRIS platform provides highly reliable AI/ML solutions through support for fault-tolerant and disaster-tolerant configurations and flexible deployment in virtual environments, on physical servers, in private and public clouds, Docker containers.

Thus, InterSystems IRIS is a universal real-time AI/ML computing platform. The universality of our platform is confirmed in practice by the absence of de facto restrictions on the complexity of implemented calculations, the ability of InterSystems IRIS to combine (in real time) the processing of scenarios from a wide variety of industries, and the exceptional adaptability of any functions and mechanisms of the platform to specific user needs.

Figure 21 InterSystems IRIS - Universal Real-time AI/ML Computing Platform

For a more substantive interaction with those of our readers who are interested in the material presented here, we recommend that you do not limit yourself to reading it and continue the dialogue “live”. We will be happy to provide support with the formulation of real-time AI / ML scenarios in relation to the specifics of your company, perform joint prototyping on the InterSystems IRIS platform, form and put into practice a roadmap for introducing artificial intelligence and machine learning into your production and management processes. The contact email address of our AI/ML expert group is [email protected].

Source: habr.com