Industrial and Trustworthy AI Challenge: Welding Quality Detection

Join us and engage with a real-world challenge to enhance weld quality inspection in industrial processes.

24 Days

17 Hours

8 Minutes

38 Seconds

Before the Kick-off

# Context

In the highly competitive automotive industry, quality control is essential to ensure the reliability of vehicles and user safety. A failure in quality control can severely jeopardize safety, result in significant financial costs, and cause substantial reputational damage to the company involved.

One of the challenges for Renault is to improve the reliability of quality control for welding seams in automotive body manufacturing. Currently, this inspection is consistently performed by a human operator due to the legal dimension related to user safety. During an industrial process, this task is resource-consuming. The key challenge is to develop an AI-based solution that reduces the number of inspections required by the operator through automated pre-validation.

Within the Confiance.ai (opens new window) Research Program, Renault Group and SystemX worked jointly on the development of trustworthy AI components tackling this problem. Now part of the European Trustworthy Foundation (ETF) (opens new window), we want to ensure that these tools effectively validate the proposed AI models according to the trustworthy criteria defined by the industry (Intended Purpose).

This industrial use case, provided by Renault Group, represents the “Visual Inspection” thematic through a classification problem.

The goal is to be able to assess weld quality from a photo taken by cameras on vehicle production lines.

A weld can have two distinct states:

OK: The welding is normal.
KO: The welding has defects.

The main objective of the challenge is to create an AI component that will assist an operator in performing weld classification while minimizing the need for the operator to inspect the images and double-check the classifications.

For defect identification ("KO"), the system should provide the operator with relevant information on the location of the detected defect in the image, hence reducing the control task duration.

# Expected AI Component

The AI component takes an image as input and optionally some additional metadata. Three possible outputs are possible:

OK: The welding in the image has no defect.
KO: The welding in the image has defects.
UNK: The welding state is UNKNOWN. UNKNOWN is used to indicate that the model is not sure about the predicted class. As we will see later in the evaluation process, an UNKNOWN output can be less penalizing than a False Negative (meaning a true KO predicted as OK), which has a critical cost.

This is illustrated by the figure below:

Optionally, the AI component could additionally return the probability associated with each possible output state. If present, this information will be used by the evaluation process and could improve the final quality score of the solution.

# Operational Domain Design (ODD) Definition

The Operational Domain Design defines the set of input images for which the AI component is expected to return a predicted state.

Here are the conditions and environments in which the AI system is expected to operate effectively and safely:

The luminosity of an image can be between 60 and 140 lumens.
The level of blur (due to vibration of the production line) of an image can be variable.
The orientation of welding seams can vary in the image with a rotation angle between -10° and 10°.
The position of the piece in the image can be translated by 5 millimeters (corresponding to 100 pixels in the image depending on seams and camera position).

For images that are out of the ODD, the AI component shall return “UNKNOWN”, and the image is sent to the operator.

The operational constraints are as follows:

False negative detections (defective welding qualified as OK by AI system) have a safety cost and shall imperatively be minimized. This is a primary objective.
Maximize the accuracy of prediction.
Some welding seams are more critical than others, depending on their position. The level of criticality impacts the cost of false negatives.

# Evaluation Criteria

The submitted AI component will be evaluated according to different quality evaluation metrics, including:

Operational cost metrics: Based on the confusion matrix and a non-symmetrical cost matrix due to operational constraints.
Uncertainty metrics: Measuring the ability of the model to use uncertainty to improve trustworthiness in its output.
Robustness metrics: Measuring the ability of the model to be invariant to empirical perturbations on input images (blur, luminosity, rotation, translation).
Monitoring metrics: Measuring the ability of the model to detect if the given input is within the ODD and adapt its output accordingly.
Explainability metrics: Measuring the ability of the model to provide an explanation for its decision to help the operator save time during inspection.

More details about these different criteria will be added in the coming weeks.

# Dataset

The dataset contains 22,851 images split among three different welding seams. An important property of this dataset is that it is highly unbalanced. There are only 500 KO images in the entire dataset.

Here is below some examples of weldings OK and KO on two different welding seams c10 and c19.

c10 OK c19 OK

c10 KO c19 KO

# Timeline

Competition kick-off: March 17th
Competitors will receive the starter-kit and will have access to the challenge datasets.
Warm-up phase: from March 17th to April 13th.
Participants can get familiar with the competition platform and the provided material. Organizers can use participants’ feedback to adjust the challenge for the next phases.
Development phase: from April 14th to August 17th.
Participants will develop their own solutions, which they can test on a provided dataset. They will be able to see the score of their submitted solution. Any submitted solution can be adjusted until the deadline is met.
Final phase: from August 18th to September.
During this phase, the organizers will review the submitted results, finalize the ranking, and prepare the results.
Results announcement: September 2025
The winners will be announced during the Confiance.ai Community Event 2025.

# Prizes

🥇 1st Place: 4000 €
🥈 2nd Place: 2000 €
🥉 3rd Place: 1000 €
💡 Most Original Solution: 1000 €

Winners will also gain visibility at the Confiance.ai Community Event, the flagship gathering for industrial and responsible AI.

# Protocol

Anyone can take part in this challenge: students, engineers, and everyone else. If you’re up to the challenge, you’re in the right place!

Each participant or organization can submit only one solution. If a participant is part of a team, individual submissions will not be accepted.

The challenge will be hosted on Codabench.

After the kick-off, participants will be invited to create an account and download the starter-kit to prepare their submissions. Once the solutions are ready, participants shall upload them on the Codabench platform.

# nwsltr

# Stay informed about the latest challenge updates

# Contact

Challenge mail: challenge.confiance@irt-systemx.fr

Discord: European Trustworthy AI Foundation #welding-quality-detection-challenge (opens new window)