III.B.4. Evaluation framework and impact assessment

Estimated Execution Time

2-3 weeks

Objective

To define a coherent and structured evaluation framework covering outcomes, outputs, and Key Performance Indicators (KPIs), together with the methodologies for their calculation and assessment. The framework enables both the evaluation of bids; assessing how proposed solutions address the identified unmet needs and the monitoring of implementation, demonstrating impact and added value in terms of performance, cost-effectiveness, usability, and sustainability.

The activity ensures transparency, accountability, and evidence-based decision-making throughout the entire procurement lifecycle, from tender evaluation to solution implementation and impact monitoring.

Who is Involved

Procurers / Contracting Authorities: define evaluation principles, ensure compliance with procurement rules, and validate the framework.
Clinical Experts (from participating hospitals): define outcome-oriented indicators linked to clinical and service impact. This group could be expanded to specialists, nurses, or primary care professionals validating usability and clinical value.
Technical Experts (IT, data, interoperability): define measurable technical and performance indicators.
Operational and Financial Experts: support definition of cost-effectiveness, efficiency, and sustainability indicators.
Project Department / Innovation Unit: coordinate framework development and ensure coherence with project objectives.
Supporting Entities / Knowledge Partners (consultants, impact assessment experts): support KPI design, methodologies, and documentation.

Activities / Tasks

III.B.4.1. Define evaluation principles and scope

Define the scope of the evaluation framework across procurement and implementation phases.
Clarify the distinction and relationship between evaluation criteria for bid assessment and KPIs for implementation monitoring.
Define guiding principles such as transparency, objectivity, proportionality, and outcome orientation.

III.B.4.2. Bid evaluation areas

Define domains for bid evaluation (e.g., excellence, innovativeness, performance, usability, cost-effectiveness) and assign weights.
Define the innovation provider evaluation, team composition, previous experience, consortium capacity.
Include compliance criteria such as financial capacity, ethics declaration, declaration of honor.

III.B.4.3. Monitoring and impact evaluation areas

Establish domains to assess project implementation and long-term impact. Ensure measurability and feasibility of data collection.
Define expected outcomes and outputs linked to the identified unmet needs and innovation challenge.
Ensure outcomes reflect clinical, operational, technical, and organisational dimensions.
Establish clear traceability between unmet needs, functional requirements, and expected results.
Define quantitative and qualitative KPIs for implementation monitoring, assessing performance improvements and functional effectiveness.
Define methodologies for KPI calculation, data sources, baseline assumptions, and measurement frequency. Review and adapt existing frameworks and validated references (e.g., ICHOM, HTA indicators, EU quality standards).
Ensure KPIs are measurable, comparable, and suitable for evidence-based assessment, that reflect success and improvement.
Define usability and user experience indicators reflecting real-world adoption.
Define sustainability indicators covering economic, organisational, and environmental aspects.
Define cost-effectiveness indicators enabling comparison of value delivered versus resources used.
Establish reference values and expected targets, especially when linked to outcome-based or value-based payments.

III.B.4.4. Validation and documentation

Validate the evaluation framework with procurers and key stakeholders, contractors and experts to ensure data collection feasibility and relevance.
Document the framework in a clear and structured manner for inclusion in tender and implementation documentation.

Tips / Common Pitfalls

✅ Clearly separate bid evaluation criteria from implementation monitoring KPIs.

✅ Use measurable, verifiable, and realistic indicators.

✅ Ensure evaluators can clearly understand how decisions will be made.

❌ Avoid overly complex or unmanageable KPI sets.

❌ Avoid ambiguity in scoring, weighting, or calculation methodologies.

❌ Use KPIs not tied to contractor performance for outcome-based payments. Example:

- If procuring an innovative insulin pump, long-term effects (e.g. reduced complications) are not measurable during pilots.
- Instead, assess precision, reliability, usability, and early adherence as short-term indicators of success.

Outcome / Deliverables

Evaluation Framework: Structured document defining relevant bid evaluation areas, associated KPIs, and weightings, ensuring transparent and objective comparison.
Monitoring Framework: Document detailing the areas, KPIs, and weights to monitor during implementation. KPIs remain constant, with phase-specific weights where applicable.
Evaluation and impact assessment framework ready for use throughout the procurement lifecycle.

Resources

Guidelines for Innovation Procurement (PCP/PPI) Example - Tender assessment scorecard (HSMonitor)

Need expert support for this step?

Browse Experts

Browse our directory of experts who can provide professional guidance.

Prepare the Procurement