HammerHAI Secure AI Model Benchmarking enables independent evaluation of AI models on confidential, real-world datasets without disclosing data to model developers or models to data providers. Acting as a trusted third party, HammerHAI provides secure benchmarking across multiple organizations while protecting intellectual property, data sovereignty, and confidentiality.
Target users
- Data providers: Companies with valuable datasets seeking to evaluate available AI models, compare solutions, and make informed technology or procurement decisions
- AI model developers: Companies and research organizations developing foundation models or other AI models that require independent evaluation on realistic, non-public datasets (ideally in conjunction with a data provider)
- Research and innovation projects: Collaborative initiatives requiring trustworthy and confidential AI model evaluation across organizational boundaries
Description
Evaluation of AI models on representative real-world data is essential for understanding their performance in industrial applications. However, companies are often unable to share sensitive datasets with external model developers. Similarly, AI developers may be unwilling to disclose proprietary models or intellectual property to potential customers. Public benchmarks offer an alternative but may not reflect real-world conditions and can be affected by benchmark contamination or overfitting.
HammerHAI Secure AI Model Benchmarking addresses these challenges by acting as a trusted third party between data providers and model developers. Both parties make their respective assets available to HammerHAI under controlled conditions. The models are evaluated in a secure processing environment without exchanging datasets or models between the participating parties.
The service enables benchmarking across datasets from multiple companies. These datasets are evaluated separately, without mixing the underlying data. Performance results can be aggregated across providers to assess model capabilities under diverse, realistic conditions.
Data access can be integrated into existing Data Spaces, using connectors such as Tractus-X EDC (Eclipse Dataspace Connector). Access and usage are governed by data-sharing agreements that restrict processing to the agreed benchmarking purpose. Data remains within the authorized secure processing environment and is not disclosed to model developers.
The service particularly targets foundation models for tabular, time-series, and graph data, as well as models that can be evaluated through inference or adapted using few-shot learning on relatively small datasets. HammerHAI executes the evaluation and shares agreed performance metrics with both parties, without exposing the underlying data or proprietary models.
Secure AI Model Benchmarking offers the following capabilities:
Confidential benchmarking
- Evaluate proprietary AI models on non-public, real-world datasets.
- Protect datasets and model intellectual property through secure processing and controlled access.
- Prevent direct access to benchmark datasets, reducing the risk of benchmark contamination and overfitting.
Cross-company evaluation
- Benchmark models across datasets contributed by multiple organizations.
- Keep datasets logically separated without combining raw data from different providers.
- Aggregate performance metrics to assess model generalization across different industrial settings.
Data sovereignty and secure processing
- Integrate with Data Spaces and connectors such as Tractus-X EDC.
- Restrict data access and processing to contractually agreed purposes.
- Return agreed evaluation metrics without disclosing protected datasets or models.
AI model evaluation
- Support inference-based benchmarking and few-shot adaptation of AI models.
- Evaluate foundation models for tabular, time-series, and graph data.
- Provide standardized performance metrics and comparable evaluation results.
How it works
HammerHAI coordinates the benchmarking process and provides technical support to both data providers and model developers.
1. Data and model onboarding
- Data providers contribute suitable benchmarking datasets via connectors. HammerHAI can assist in preparing datasets (e.g. anonymizing), defining evaluation tasks, and selecting performance metrics.
- Model developers provide their AI models for confidential evaluation.
- Access permissions, evaluation conditions, and data usage agreements are established.
2. Secure integration
- Datasets are made available through controlled access mechanisms, optionally using Data Space connectors.
- Models and datasets are brought together within a secure HammerHAI processing environment.
- Data and models remain protected from unauthorized access by other participants.
3. Benchmark execution
- HammerHAI performs inference or few-shot learning on the provided datasets.
- Each provider’s dataset is processed separately under agreed upon conditions.
- Standardized performance metrics are calculated and, where appropriate, aggregated across datasets.
4. Results and reporting
- Data providers receive performance metrics to compare AI solutions for their specific applications.
- Model developers receive independent benchmark results reflecting performance on real-world industrial data.
- Only agreed upon evaluation results are shared. Raw datasets, proprietary models, and confidential processing artifacts remain protected.
Applications
The service is particularly valuable for companies and AI developers seeking reliable model evaluations without compromising confidential information. Typical use cases include:
- AI model selection and procurement: Compare available models on company-specific data to support informed purchasing and deployment decisions.
- Independent model benchmarking: Enable AI developers to assess the performance and generalization of their models on realistic, previously unseen datasets.
- Technology and market monitoring: Help data providers monitor advances in AI capabilities and identify solutions relevant to their industrial challenges.
- Cross-industry model evaluation: Assess foundation models across datasets from multiple companies without requiring organizations to share their proprietary data.
- Few-shot learning evaluation: Measure how effectively AI models adapt to new industrial tasks with limited training examples.
How to get access
The service is offered to companies and research projects interested in providing benchmarking datasets or evaluating AI models on a best effort basis.
The offering of the service heavily depends on the availability of fitting data or models. Model providers, please note that benchmarking might currently only be possible based on public data. However, the HammerHAI Data Lab will proactively try to drive matchmaking, particularly if providers are known. Ideally an existing match of a data provider and model provider will request the service.
Via Email: Please contact the HammerHAI team under access@hammerhai.eu with a brief description of your available datasets or AI models and the intended benchmarking application.
Service provided by
Contact
Funding
The service is supported by HammerHAI (European High Performance Computing Joint Undertaking, Grant No. 101234027).
Disclaimer
Benchmarking results depend on the quality, representativeness, and scope of the available datasets and do not guarantee performance in production environments. Data access and processing are subject to individual agreements and applicable usage constraints. The service supports confidential evaluation but does not replace legal or regulatory compliance assessments (typically against the appropriate data space rule books).
