Performance assessment guidebook
Context
In 2025, we launched a series of events under the DRI’s HAI-End project to discuss with digital research professionals how they would address performance analysis challenges given to them as a black-box. Colleagues from industry and tool vendors joined these meetings. The HAI-End consortium collected all outcomes and distilled them into a joint workbook, which we make available through SHAREing.
Above: Members of one of the initial working group meetings.
Versions
This document is still in an early stage of development (arguably a pre-release)
We envisage the performance assessment methodology to be largely made of three stages:
1) Pre-assessment: getting the code ready for assessment
2) High-level assessment: calculating metrics that give an overview of the code’s performance
3) Lower-level assessment: guided by the high-level assessment, we dig down into the performance details
We are happy to announce that an initial version of the Pre-assessment and High-level assessment are included in our guidebook as:
- Chapter 4: Preparation
- Chapter 5: High-level first glance
respectively. These are still in development, but other parts of the assessment methodology are also starting to take shape throughout the guidebook, though are much earlier drafts.
Acknowledgements
This project has received funding through the UKRI Digital Research Infrastructure Programme under grant number UKRI/ST/B000293/1 (HAI-End).
