Guidance to Select and Use D&I Assessment Instruments
Please review key guidance for selecting and using D&I assessment instruments using this webtool and more broadly.
You can identify relevant assessments instruments four different ways:
(1) If you know the title of the assessment instrument you want to consider you can go directly to the View by D&I Assessment Instruments tab, select the assessment instrument, and review its specifics under the instrument specific page.
(2) If you need to explore instruments based on a number of criteria (e.g., keyword, quantitative/qualitative, type of instrument, construct, etc.), you can go to the View by D&I Assessment Instruments tab and use the right side bar to identify assessment instruments that met the criteria. Then you can review the eligible assessment instruments one by one.
(3) If you are looking for assessment instruments specific to a construct, you can go to the Explore D&I Assessment Instruments tab and select the View by D&I Construct subsection. Here you can identify the relevant constructs and review the associated assessment instruments.
(4) If you have selected a D&I theory/model/framework (TMF) and you are looking for assessment instruments that can be used with the constructs included in the TMF, you can return to the Select section and go to the specific TMF’s individual webpage where all assessment instruments that are linked to the constructs in the TMF will be listed along with the constructs.
We organized constructs by merging similar sub-constructs or elements from specific TMFs. To see what elements are included within each construct, go to the Explore D&I Assessment Instruments tab and select the View by D&I Construct subsection and look at the individual page for constructs that are similar to the construct you are looking for. On this page you will see the specific elements that were combined under this construct. Most of the time you will be able to identify relevant constructs this way.
This webtool is a work in progress and evolving. This is a new section of the website and is being expanded over time. We made two decisions about which instruments to include initially. 1) We followed recommendations of a multidisciplinary advisory panel of experienced D&I researchers as to what were the most important instruments to abstract. 2) We prioritized recommendations for qualitative and quantitative assessments, including more novel methods, such as direct or participant observation, and use of secondary data sources, such as electronic health records (EHRs) or geospatial data, as we think it’s important to support multiple methods and mixed methods research in D&I science. 3) The field is growing rapidly so new instruments are constantly being developed.
Inclusion of an instrument in this database does not indicate that we are endorsing it or that it is the best (e.g., most valid, reliable, pragmatic) instrument to assess a given construct. In conjunction with our advisory panel, we decided not to rate instruments or make recommendations as to which measures are the best. The content area, context, populations, and purpose of the research study in which data are collected are so different that we felt this unwise and have restricted our data fields to objective characteristics.
We have an option for assessment instruments to be rated by users (see Ratings and Comments) to share user experience with the given assessment instrument.
We reviewed the source article and the most recent publications that we could locate for each assessment instrument included. If the instrument you are reviewing does not have a value on a given characteristic, this means that either 1) this characteristic is not relevant for that type of assessment (e.g., norms for qualitative assessment guides) or 2) we were unable to find this information in the published source article or most recent publication we found.
It is important to note that this does not mean that data on a specific characteristic do not exist as we did not conduct a comprehensive literature search on each instrument or contact the developer of each instrument to locate all metadata fields. If this information is important to you, we recommend that you conduct an updated more comprehensive search or contact the developer of the instrument.
Several characteristics are usually relevant for deciding upon which assessment instrument to use. First, the team needs to ensure that the project has a clear research question and stated outcomes, and a TMF (or combination thereof) that supports the question and outcomes. In selecting an appropriate assessment instrument, it is important to find one that fits closely with the key constructs in your selectedTMF. You should also consider the assessment instrument’s appropriateness for a specific context or with the population of interest. What instruments are ‘best’ depends upon the question you are trying to answer and often pragmatic factors such as time to administer, costs, burden, and reliability are important. We recommend that 1) in collaboration with your community or clinical partners you identify which characteristics should be prioritized in making your decision and 2) whenever possible that you include more than one instrument, and potentially more than one type of instrument (e.g., quantitative and qualitative) if appropriate. Ultimately, this selection process includes balancing feasibility and scientific rigor in order to select the best instruments for your project.
There are a variety of places to get information on these tools. Often, the tool has a paper or reference that provides information and the information for the tool development team. Additionally, some of these tools have websites or are located within other toolkits that can be found online.
1) The type of data collection event to be performed (e.g., individual interview, focus group interview, observation).
2) For interviews/focus groups, emphasis should be placed on using easily understood open-ended questions with attention given to their sequencing and timeframe (e.g., present, past, future). Implementation studies typically use semi-structured guides that reflect the topics/subject areas of particular interest and TMF constructs. (Patton, 2014 provides a nice overview of the specific types of questions that one might ask; questions should also reflect the type of evaluation being performed, i.e., formative vs. summative.)
3) For observations, consider protocols/guidelines/templates that specify particular foci and approaches for documenting observations.
4) The extent to which the instrument is tailored to the context in which the work is being carried out (e.g., clinic, school, other community setting) and participant type (e.g., system leader, organizational administrator, direct service provider, staff member, client/patient/service recipient). This is especially important when exploring implementation phenomena at multiple levels. For guidance on this (including information in the appendices) please see Lengnick-Hall et al, 2023 https://link.springer.com/article/10.1186/s13012-023-01302-2.
In addition to this repository, there are other web-based compilation of assessment instruments for D&I research. A few of these include:
- The Society for Implementation Research Collaboration Instrument Repository
https://societyforimplementationresearchcollaboration.org/measures-collection/
- The Implementation Outcomes Repository
https://implementationoutcomerepository.org/
- Washington University in St. Louis Measuring Health Policy Implementation Database
An overarching key concept is that regardless of which instrument is selected, the resulting data is only as good as how the instrument was ultimately applied. This is where the skills of the interviewer, observer, or other type of fieldworker becomes paramount.
A general overview of qualitative methods in implementation science is available from the white paper entitled National Cancer Institute: Qualitative Methods in Implementation Science White paper commissioned by the Implementation Science Division of Population Health at the National Cancer Institute https://cancercontrol.cancer.gov/IS/docs/NCI-DCCPS-ImplementationScience-WhitePaper.pdf
A follow-up webinar discussed advancements in the field since the publication of the white paper:
https://www.youtube.com/watch?v=Mxj7s2Fe89w
Commonly used methods and related publications and resources for qualitative methods in implementation science include:
Rapid Qualitative Approaches:
- Hamilton AB: Rapid Qualitative Analysis: Updates/Developments https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=3846 https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/3846-notes.pdf
- Hamilton, A. B., & Finley, E. P. (2019). Qualitative methods in implementation research: An introduction. Psychiatry research, 280, 112516. https://doi.org/10.1016/j.psychres.2019.112516
- Hamilton, A., Fix, G., Finley, E. (forthcoming). Pragmatic Healthcare Ethnography. Routledge. https://www.routledge.com/Pragmatic-Healthcare-Ethnography-Methods-to-Study-and-Improve-Healthcare/Hamilton-Fix-Finley/p/book/9781032487601?srsltid=AfmBOoo9oHcyiqxkFE3gYrrXQr73-dJorZ16xP6_p36j77TOiiYpV9T4
- St George, S. M., Harkness, A. R., Rodriguez-Diaz, C. E., Weinstein, E. R., Pavia, V., & Hamilton, A. B. (2023). Applying Rapid Qualitative Analysis for Health Equity: Lessons Learned Using “EARS” With Latino Communities. International journal of qualitative methods, 22, https://doi.org/10.1177/16094069231164938. PMID: 38463016 PMCID: PMC10923582.
Ethnographic approaches:
- Gertner, A. K., Franklin, J., Roth, I., Cruden, G. H., Haley, A. D., Finley, E. P., Hamilton, A. B., Palinkas, L. A., & Powell, B. J. (2021). A scoping review of the use of ethnographic approaches in implementation research and recommendations for reporting. Implementation research and practice, 2, 2633489521992743. https://doi.org/10.1177/2633489521992743. PMID: 34056611 PMCID: PMC8153409
- Palinkas, L. A., & Zatzick, D. (2019). Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) in Pragmatic Clinical Trials of Mental Health Services Implementation: Methods and Applied Case Study. Administration and policy in mental health, 46(2), 255–270. https://doi.org/10.1007/s10488-018-0909-3. PMID: 30488143 PMCID: PMC7427407.
- Rabin, B. A., Cain, K. L., Salgin, L., Watson, P. L., Jr, Oswald, W., Kaiser, B. N., Ayers, L., Yi, C., Alegre, A., Ni, J., Reyes, A., Yu, K. E., Broyles, S. L., Tukey, R., Laurent, L. C., & Stadnick, N. A. (2023). Using ethnographic approaches to document, evaluate, and facilitate virtual community-engaged implementation research. BMC public health, 23(1), 409. https://doi.org/10.1186/s12889-023-15299-2. PMID: 36855118 PMCID: PMC9974043.
In addition to this repository, there are other web-based compilation of assessment instruments for D&I research. Most of these are primarily or exclusively focused on quantitative assessment instruments. A few of these include:
- The Society for Implementation Research Collaboration Instrument Repository
https://societyforimplementationresearchcollaboration.org/measures-collection/
- The Implementation Outcomes Repository
https://implementationoutcomerepository.org/
- Washington University in St. Louis Measuring Health Policy Implementation Database
A number of publications are available to provide an overview of key considerations and strategies for selecting and using assessment instruments for implementation science. Here is a list of resources that you might find helpful to explore this topic, without the intent for comprehensiveness:
- Martinez, R. G., Lewis, C. C., & Weiner, B. J. (2014). Instrumentation issues in implementation science. Implementation science : IS, 9, 118. https://doi.org/10.1186/s13012-014-0118-8. PMID: 25185799 PMCID: PMC4164742.
- Stanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., Weiner, B. J., Barwick, M., Wolfenden, L., Damschroder, L. J., & Lewis, C. C. (2021). Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Translational behavioral medicine, 11(1), 11–20. https://doi.org/10.1093/tbm/ibz164. PMID: 31747021 PMCID: PMC7877299.
- Rabin, B. A., Purcell, P., Naveed, S., Moser, R. P., Henton, M. D., Proctor, E. K., Brownson, R. C., & Glasgow, R. E. (2012). Advancing the application, quality and harmonization of implementation science measures. Implementation science : IS, 7, 119. https://doi.org/10.1186/1748-5908-7-119. PMID: 23231885 PMCID: PMC3541131.
- Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013). Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation science : IS, 8, 22. https://doi.org/10.1186/1748-5908-8-22. PMID: 23414420 PMCID: PMC3598720.
- Glasgow, R. E., & Riley, W. T. (2013). Pragmatic measures: what they are and why we need them. American journal of preventive medicine, 45(2), 237–243. https://doi.org/10.1016/j.amepre.2013.03.010. PMID: 23867032.
- Powell, B. J., Stanick, C. F., Halko, H. M., Dorsey, C. N., Weiner, B. J., Barwick, M. A., Damschroder, L. J., Wensing, M., Wolfenden, L., & Lewis, C. C. (2017). Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implementation science : IS, 12(1), 118. https://doi.org/10.1186/s13012-017-0649-x. PMID: 28974248 PMCID: PMC5627503.
- Rabin, B. A., Lewis, C. C., Norton, W. E., Neta, G., Chambers, D., Tobin, J. N., Brownson, R. C., & Glasgow, R. E. (2016). Measurement resources for dissemination and implementation research in health. Implementation science : IS, 11, 42. https://doi.org/10.1186/s13012-016-0401-y. PMID: 27000147 PMCID: PMC4802882.
- National Cancer Institute, Division of Cancer Control and Population Sciences. (2019). Implementation science at a crossroads: Moving towards a comprehensive science of implementation. National Institutes of Health. https://cancercontrol.cancer.gov/sites/default/files/2020-09/nci-dccps-implementationscience-whitepaper.pdf
- Lengnick-Hall, R., Williams, N. J., Ehrhart, M. G., Willging, C. E., Bunger, A. C., Beidas, R. S., & Aarons, G. A. (2023). Eight characteristics of rigorous multilevel implementation research: a step-by-step guide. Implementation science : IS, 18(1), 52. https://doi.org/10.1186/s13012-023-01302-2. PMID: 37872618 PMCID: PMC10594828.