Exploration and Evaluation of Highly Constrained Neural Networks.

[dianasolangel]

Exploration and Evaluation of Highly Constrained Neural Networks.

Context

My internship took place at Michelin’s Research and Development Center, I was a part of the R&D department, specifically the Simulation (SIM) department where I joined the Data Science team. My main objective was to explore and evaluate constrained models applied to various use cases of interest and whose results must meet performance criteria that are physically realistic. The idea was to study how well the integration of constraints helps to improve a model’s performance and its overall results.

Constrained Models

There are three main types of domain knowledge that can directly be translated into alterations of either the input data of a deep network, the loss-function, the model’s structure or its parameters.

We took interest mainly in the following approaches:

MultiplexNet

MultiplexNet ¹ est une approche qui représente les connaissances du domaine sous forme de formule logique en forme normale disjonctive (DNF), ce qui est facile à encoder et à induire par des experts humains. Elle introduit une variable latente catégorielle qui apprend à choisir quels termes de contrainte optimiser directement dans la sortie des algorithmes d'apprentissage existants. Dans cette approche, les contraintes sont directement compilées dans la sortie standard du réseau neuronal, ce qui permet de garantir que toute sortie du réseau MultiplexNet satisfera les contraintes.

Monodense

Dans cette approche ², au lieu de couches pleinement connectées ou denses standards, une couche dense monotone est proposée pour construire des réseaux monotones contraints capables d'approximer n'importe quelle fonction continue partiellement monotone sur un sous-ensemble compact. Une couche entièrement connectée monotone contrainte est introduite, pouvant être utilisée en remplacement direct d'une couche entièrement connectée pour imposer la monotonie.

This is done by construction, by combining the idea of constraining the weights of the fully connected neural network to have only non-negative (for non-decreasing variables) or only non-positive values (for non-ascending) variables. This idea used in conjunction with non-saturated activation functions (the original activation function and two additional monotonic activation functions constructed from it) in a neural network with constrained weights, it can approximate any monotone continuous functions.

Use cases

Generative Model for Tyre’s Components

This application case is part of a project aimed at optimizing tire components through generative design to streamline their development. The goal is to create a generative model that produces new tire architectures, offering designers more diverse options while maintaining 'épure' coherence. At Michelin, 'épure' refers to the tire's interior cross-section, which is critical for analyzing and ensuring the internal geometry meets Michelin's standards. Previous generative models, such as VAEs, have struggled to accurately represent the 'épure' as it would appear in a CAD model, highlighting the need for hard constraints to ensure the generated designs are physically realistic and consistent with real-world 'épure' characteristics.

Figure 1: Illustration of the "épure" component of interest

Prediction of Damage

Predicting tire wear is crucial for ensuring safety and optimizing maintenance schedules. Accurately forecasting the evolution and final extent of tire damage allows for better planning and prevention of potential failures. In this application case, we focused on testing the inclusion of domain and monotonicity constraints into deep learning predictive models while using synthetic data to simulate tire wear progression.

Figure 2: Expected damage evolution over time.

Results and Conclusions

Application Case: Generative model
The approach of including constraints during training is the best one yet,, though ordering products and accurately identifying segments can be challenging. Despite these difficulties, the use of simple constraints has proven effective, with generated data adhering to the specified contact constraints. This highlights the potential of constraints in guiding the generation process. Regarding diversity, while the generated results are generally good, further exploration might be necessary to achieve the expected outcomes.

Figure 2: Example of generation for 3 products composing an "épure".

Application Case: Damage Prediction
Incorporating constraints during training allows the model to focus on essential details, improving overall performance. This approach enables the use of simpler models, reducing computational complexity and enhancing efficiency. However, further exploration of monotonicity constraints is necessary to refine predictions and improve model robustness.

Figure 2: Best Model results LSTM Model.

General Conclusions
Different constraints vary in ease of encoding and implementation, underscoring the importance of selecting the right constraints and models for a given task. While adding hard constraints enhances model reliability and domain adherence, effectively encoding domain knowledge remains a challenge and may require further refinement.

References

Footnotes

1. Hoernle, N., Karampatsis, R. M., Belle, V., & Gal, K. (2022). MultiplexNet: Towards fully satisfied logical constraints in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 5, pp. 5700-5709). Link ↩

2. Runje, D., & Shankaranarayana, S. M. (2023). Constrained Monotonic Neural Networks. In International Conference on Machine Learning (pp. 29338-29353). PMLR. Link ↩