Artificial intelligence (AI) technologies are growing up. Great progress is being made particularly in the field of machine learning (ML) and ML components are increasingly being used in software-intensive systems. This gives the systems the ability to learn from mistakes, to create predictions for future events, and to adapt to changes in their environment. Many applications in logistics, transport, production, or medicine can benefit from this—assuming responsible engineering. In this interview, Klaus Pohl talks about the new possibilities of AI-based software-intensive systems and the challenges these systems pose to software engineering.

paluno: What aspects of AI are of particular interest to you and your research staff?

Pohl: Our focus is on the engineering of software-intensive systems. Nowadays, software is almost always the driver for innovation. In terms of AI, we are particularly interested in how machine learning approaches can be used to improve the development and operation of software-intensive systems and to enable innovation.

For example, we investigated how ML algorithms can be used to improve the development of self-adaptive systems. These systems can be fully automated, to a certain extent, to adapt to changing environmental situations. At the time of development, however, it is impossible for the engineers to predict all future environmental situations. Therefore, we rely on machine learning methods. This enables the systems to learn, from existing environmental situations, which type of adaptation works best in which situation. This means that the adaptive logic of the system continues to evolve during its implementation. In the EU project ENACT, for example, we investigate the use of reinforcement learning algorithms for self-adaptive IoT systems.

Are there any other examples of the use of machine learning methods that you are researching?

AI components are often used to predict future processes or future events. In the EU project DataPorts, together with our partners we are developing a data platform for European seaports. Here, for example, precise, forward-looking recommendations for terminal operators are calculated with the help of suitable methods of machine learning. Based on the experience of the systems, for example, a terminal operator receives precise predictions of whether there are any delays in the logistics process in a deep-sea container and whether it makes sense to intervene in the ongoing process. With the help of the predictions, the operator can react proactively to possible delays and, if necessary, avoid them.

Which ML approach gives you highly accurate predictions?

For this purpose, our team uses Ensemble Deep Learning methods. This ML approach combines predictions from different neural networks. This enables highly accurate predictions, as strengths of the neural networks complement each other and weaknesses of individual networks can be neutralized. The ensembles also provide good estimates of the accuracy of the predictions and thus provide decision support for the terminal operator.

Experts as well as end users must be able to understand how the algorithms have come to their decisions.

Are there aspects of machine learning that you see as critical?

The "black box" character of many methods poses a major problem. Experts as well as end users must be able to understand how the algorithms have come to their decisions. Consider the example of the seaport: the terminal operator must know which factors were decisive for certain predictions before he responds to them. If delivery delays occur because the AI components have provided incorrect predictions, this can have serious consequences for a company—in other words, become very expensive. The traceability of AI results is often not given today. If we simply accept this, we run the risk of relying on unsustainable answers.

What errors could occur due to non-transparent ML methods?

A learning component may draw incorrect conclusions from training data—for example, by taking into account non-relevant properties of training data when classifying objects. This phenomenon has been demonstrated frequently in recent years: for example, supposedly advanced object recognition methods have classified objects in images primarily on the basis of the background. An algorithm that assigns images to the category “ship” because there is a lot of water in the image and classifies images as “train” because there are rails cannot really distinguish ships from trains. The method is merely statistically successful but does not solve the actual classification task. Such mistakes must not happen in situations where they can become dangerous or expensive.

Let us stick to the predicting models. How can the user understand the correlations on which the predicting models are based?

There are basically two possibilities: the first is that the prediction models themselves are designed so transparently that end users can understand how the prediction results come about. These models must remain very simple in order to be understandable and therefore cannot depict any complex correlations in the data. At present, this approach is almost always associated with a lower accuracy of prognosis. For example, a simple model would be a linear regression model that represents the relationship between the input data and the predictions as a linear function. However, this does not reflect any interactions between the input data and thus does not represent more complex relationships. Currently, it is almost always the case that the better the prediction results, the less transparent the predicting models and algorithms are.

The other option is that, in addition to the actual predicting models, simpler models are used to explain the predictions. The actual predicting results are still derived from the more complex and better predicting models but the explanations of the predicting results for the end user are given using the simpler models—for example, by simplifying the results and their main influencing factors and making them understandable.

We need to expand the knowledge gained about the engineering and operation of systems for software-intensive systems with AI components.

What else can software technology contribute to make AI-based software systems more secure and more robust?

We need to make sure that AI-based software-intensive systems—just like all other systems—meet certain quality requirements at all times. To do this, we need to look at the systems as a whole, including the learning processes. This is the only way we can detect unintended effects and ensure that, despite automated learning, requirements remain met at all times. This can be done, for example, by appropriate encapsulation of the learning components, supported by a review of possible adaptation effects during system execution—specifically, before the adaptation is implemented.

At paluno, we have been researching and validating development methods and technologies for complex software-intensive systems and system families for many years. We need to expand the knowledge gained about the engineering and operation of systems for software-intensive systems with AI components. It is only as part of a secure, reliable software system that machine learning can unleash its full potential.