Highlights
- Developed a retrieval-augmented generation (RAG) prototype.
- RAG prototype uses 7,282 MPA publications on welding and joining.
- MPA engineers write benchmark questions and evaluate answers and sources.
Challenges
The Material Testing Institute (MPA) at the University of Stuttgart combines materials testing, joining research, and industrial transfer. Its knowledge is spread across publications, material tables, formulas, standards, and expert experience. A retrieval-augmented generation (RAG) system — a large language model (LLM) that retrieves and uses information from external knowledge sources — could improve access to this information. However, fluent answers alone do not guarantee engineering correctness. Relevant sources must be retrieved and cited traceably, calculations must be reliable, and uncertainties must remain transparent. Earlier work by the High-Performance Computing Center Stuttgart (HLRS) and the University of Stuttgart’s Institute for Metal Forming Technology (IFU) showed that language models can partially predict temperature-dependent yield curves.
Solution
The collaboration with HammerHAI began with a roadmapping workshop that identified a benchmark test for a RAG system and initiated the development of a RAG prototype. MPA and HLRS ran benchmark tests to see where the RAG prototype helps, where it fails, and how much expert review is required before industrial use could be considered. A living roadmap and weekly meetings now guide prototype reviews and planned tests. The RAG testbed works with 7,282 MPA Stuttgart publications on welding and joining. Its interface allows experiments with source counts, multi-step retrieval, source, page-level references, PDF links, calculations, plots, and a second verification pass. These are prototype options, not yet integrated or validated production features. MPA engineers develop representative questions and evaluate the answers, cited sources, and calculations using the benchmark tests developed by HammerHAI and MPA.
Benefits and Impact
The main result is an evaluation approach and early RGA prototype. The benchmark records assessments conducted by MPA engineers on technical correctness, source traceability, and reproducibility. An exploratory comparison of 11 openly available models illustrates the need for systemic evaluation. This solution has the potential to support future AI-assisted engineering workflows by improving access to specialized engineering knowledge, reducing information retrieval efforts, and supporting the selection of welding parameters to accelerate the calibration of welding machines.
Next Steps
MPA engineers plan to expand the question set and further refine the assessment criteria. HammerHAI intends to test retrieval and generation separately, compare source configurations, and document failure cases. The partners plan to refine RAG functions, calculations, and traceability before considering an operational pilot. In parallel, MPA and HLRS are continuing to prepare for further collaboration in AI-assisted Abaqus simulation
Organizations Involved
Contact
Gerrit Koch, Material Testing Institute (MPA), University of Stuttgart, gerrit.Koch@mpa.uni-stuttgart.de
