Set Model Evaluation Criteria Before Comparing Suppliers

Define the decision the model would support

Write a short use statement naming the user, decision frequency, asset coverage and consequence of error. A weekly research prioritization tool needs different evidence from an intraday execution input or a client-facing allocation report. Identify the existing process and the problem the proposed model should solve. Without that baseline, evaluation can drift toward whichever output is easiest to demonstrate rather than the capability the buyer actually needs to improve.

The Federal Reserve's 2026 model-risk guidance emphasizes assessment tailored to model purpose and materiality. Its banking-supervision scope is not a certification standard for every model buyer. Here it supplies context for asking proportionate questions; the procurement framework and numerical examples are illustrative choices. A small research team can document a focused evaluation without claiming that its checklist reproduces a supervisory examination or establishes compliance with banking guidance.

Put mandatory gates outside the score

Identify conditions without which the proposed use cannot proceed: adequate permissions, a known model version, interpretable outputs and access to evidence needed for review. The exact gates depend on the use. Record a gate as met, unmet or unresolved with supporting material. Do not assign a few lost points to an unknown data entitlement when the evaluation requires that data; the missing dependency is not equivalent to a less convenient user interface.

Separate a limitation from a disqualifier. A model that excludes a minor asset class may still fit a narrowly defined task, while a model that cannot identify the information available at decision time may fail a causal historical evaluation. Explain the connection to the intended use. This keeps requirements proportionate and prevents procurement from demanding every conceivable feature while overlooking the single uncertainty most likely to invalidate the buyer's specific workflow.

Assess evidence before interpreting results

Ask how development, validation and later observations were separated, what selection occurred, and which limitations are documented. Request enough detail to connect a summary result to the supplied model identity and evaluation period. The SEC's performance-claims bulletin distinguishes backtested results from actual performance. A buyer should preserve that distinction when recording evidence rather than treat a smooth hypothetical series as operational experience or as proof of attainable future investment returns.

Use appropriate measures for the claimed function. A risk estimate, ranking model and position-sizing tool may need different diagnostics. Describe coverage, stability and known failure conditions alongside any performance statistic. Missing observations should remain visible instead of disappearing from the denominator. When results depend heavily on an execution or cost assumption, record that dependency explicitly and keep the original evidence separate from the buyer's additional sensitivity analysis and assumptions.

Work through a hypothetical weighted assessment

Suppose a hypothetical buyer has already resolved its mandatory gates and predeclares four preference weights: evidence clarity 40%, integration fit 25%, operational support 20% and documentation usability 15%. It uses a zero-to-five rubric with written anchors. An invented assessment of 4, 3, 4 and 2 yields 0.40 x 4 + 0.25 x 3 + 0.20 x 4 + 0.15 x 2 = 3.45 out of five.

The equivalent normalized score is 69 out of 100, but neither number represents a probability of investment success. The team records the evidence behind each rating and tests whether modest weight changes reverse a comparison. If a mandatory permission later becomes unresolved, the score does not rescue the purchase. These weights are buyer-designed teaching assumptions, not an industry standard, a real vendor rating or a substitute for the committee's documented judgment.

Test integration and support as separate claims

Check whether sample files arrive with stable identifiers, documented units and unambiguous timestamps. Ask how revisions, missing inputs and service interruptions are communicated. Demonstrate ingestion with synthetic or properly authorized fixtures before assuming compatibility. A scientifically credible model can still be unsuitable for a workflow whose deadlines or interfaces it cannot meet. Equally, a polished API does not compensate for weak evidence about the calculation it delivers to the buyer.

Translate support claims into observable questions: who handles a disputed output, what information accompanies a correction, and which version remains available for investigation? Record proposed service commitments separately from behavior actually observed during evaluation. One prompt response does not establish a long-term response-time distribution. Budget the buyer's own monitoring and maintenance effort, including the cost of obtaining specialist review when limitations or material changes require more expertise than routine operations provide.

Write a bounded recommendation

A decision memorandum should say whether the model is suitable for the named use, under which limits and with which unresolved conditions. Distinguish declining a purchase for missing evidence from concluding that a strategy lacks merit. An inconclusive trial is not a negative performance result. Record any proposed additional work with a decision purpose and resource limit, so evaluation does not become an open-ended search for a persuasive chart or a convenient passing metric.

Have the responsible reviewers assess legal and regulatory applicability to the actual agreement and jurisdiction; the scorecard does not make those determinations. Preserve the criteria version, evidence references and reasons for exceptions. Reassess when model identity, data dependencies, audience or operational use changes. The goal is a reviewable procurement choice with clearly stated limits, not a generic seal of model quality or an assurance about future returns under conditions the evaluation never observed.

• Use non-compensating gates for critical dependencies.
• Treat weighted scores as decision aids, not probabilities.

Source notes

1. Federal Reserve: SR 26-2, Revised Guidance on Model Risk Management
2. SEC Investor.gov: Investor Bulletin on Performance Claims