NCI staff with responsibility for human specimen resources and the Deputy Director of the National Center for Research Resources, met on October 30, 1997 to consider criteria for assessing the effectiveness of NCI human specimen resources (see attached participant list). Their charge was: To develop a set of criteria that can be used to evaluate a human specimen resource in order to determine whether it is effectively meeting a critical research need. Meeting a critical research need means that the resource facilitates research that contributes to an essential body of knowledge such as discovery of new genes, development and testing of technologies, diagnostic assays, predictors of treatment outcome, etc. The following recommendations developed by the group are consistent with the practices of the National Center for Research Resources and the National Institute of Science and Technology to the extent that those practices are documented in a limited number of publications that were made available to the Committee.
In general, the NCI expects an evaluation to answer three critical questions:
The objective of the evaluation must be clearly stated and well formulated evaluation questions developed, regardless of which evaluation method is chosen. An evaluation can be conducted by NCI staff, by academic or industry scientists, or by existing advisory boards or sub-committees of such boards. The process should be as objective as possible. This argues for including participants who are not directly involved with the resource.
Objective measures generally are used to assess the performance or effectiveness of a resource. While the use of fully objective measures is desirable, appropriate evaluation also requires the use of subjective measures. The subjective measures, while harder to quantitate, are required to measure impact and are essential to the evaluation. Impact measures are required to determine whether a resource is serving a critical need. For human specimen resources in support of cancer research, the primary impact measure is the importance of the research findings resulting from use of the resource. Key research findings can be major contributions to our understanding of cancer biology, development or validation of a critical technology or the development of a useful medical device such as a diagnostic assay or a predictor of therapeutic outcome.
There are a variety of different types of specimen resources, each of which has unique characteristics that must be considered in order to properly evaluate the need for the resource, its impact, and its performance. For instance, a longitudinal study collection will require many years to become a useful resource, and should not be evaluated in its development stage for how it is contributing to critical research needs. In terms of performance, a resource focusing on a single organ system should not be evaluated for the range of tissue types it makes available. Although these are obvious examples, they underscore the central point that judicious selection of appropriate metrics is required in any evaluation. While each resource may require different measures of its effectiveness, the fundamental criterion for the evaluation of any resource remains whether or not it is effectively meeting a critical scientific need.
Performance is by far the easiest item to evaluate. In general the factors listed below are numerical and can be easily tallied if identified in advance. As with the factors discussed below, the choice of specific evaluation factors will be made in the context of what the resource was designed to accomplish and how it was expected to operate.
Useful performance measures include:
The impact of a resource is the most difficult characteristic to measure, although impact becomes easier to measure with time as more researchers are served. Time is required not only for a resource to become established, but also for the critical research findings resulting from use of the resource to be published and appreciated. It is also difficult to find quantitative metrics that clearly aid in the evaluation of impact.
Useful impact measures include:
To determine whether there is a continuing need for an existing resource it is important to consider the nature of the resource, its original objective and how long it has been in existence. Before a resource is created, it is important to clearly state how it is likely to be used, who the users will be and the expected number of users, and other critical expectations.
Questions to determine whether a resource is meeting a continuing research need include:
The NCI always considers the cost of a resource since funds used to support resources might otherwise be available to support investigator initiated research. Comparison of the costs of resources is only useful when comparing two essentially identical resources. In addition, the cost of resources is not a dollar for dollar trade-off against research funding, since the availability of NCI resource support reduces the cost to investigators of obtaining tissue specimens. It is difficult to compare the relative cost of funding acquisition of specimens directly as part of individual research grant support with the cost of providing them through a central resource. While it is relatively easy to assess the cost per specimen of operating a centralized resource, it is difficult to assess the cost per specimen collected as part of a research study, since that includes monetary costs, staff time costs and loss of research productivity costs.
It is even more difficult to try to determine the cost per critical research finding. This is true whether evaluating centralized resources or evaluating the cost of collecting specimens as part of individual research projects. The value of a given research finding is difficult to estimate and the importance of any finding may take many years to become clear. Because of these considerations, we did not believe specific cost metrics can be used for the evaluation of tissue resources.
The approach presented above provides a philosophical framework which if applied thoughtfully should allow one to focus on the critical issues needed to decide whether or not to initiate, continue, or modify a human specimen resource.