Q00789

SNMMI Guidelines for Evaluation of Artificial Intelligence

Best practices are presented to evaluate AI algorithms developed for different parts of the imaging pipeline ranging from image acquisition to post-processing to clinical decision-making in the context of nuclear medicine.

Course ID: Q00789 Category:
Modalities: , ,

3.0

Satisfaction Guarantee

$34.00

Add to Wishlist

This course has been approved for 3.00 Category A credits.
No discipline-specific Targeted CE credit is currently offered by this course.

Outline

  1. Introduction
  • Components of the Claim
    1. Definition of the Clinical Task
    2. Patient Population for Whom the Task is Defined
    3. Definition of Imaging Process
    4. Process to Extract Task-Specific Information
    5. Figure of Merit (FoM) to Quantify Task Performance
  • Methods for Evaluation
    1. Proof-of-Concept (POC) Evaluation
      1. Objective
      2. Study Design
      3. Output Claim of the POC Study
    2. Technical Task-Specific Evaluation
      1. Objective
      2. Study Design
      3. Output Claim from Evaluation Study
    3. Clinical Evaluation
      1. Objective
      2. Study Design
      3. Output Claim from Clinical Evaluation Study
    4. Postdeployment Evaluation
      1. Objective
      2. Evaluation Strategies
  • Discussion

    • Objectives

    Upon completion of this course, students will:

    1. list the inaccuracies that can result from an AI algorithm
    2. describe the generalizability of AI algorithms to other data sources
    3. list the categories of clinical tasks for AI algorithms
    4. describe examples of classification clinical tasks performed by an AI algorithm
    5. describe examples of quantification tasks performed by an AI algorithm
    6. list the patient population features that should be defined in a task claim
    7. identify the study component needed to obtain a figure of merit (FoM) to quantify task performance
    8. identify the characteristic that provides evidence for AI algorithm generalizability
    9. identify the evaluation class that assesses AI algorithms that improve diagnostic, predictive, prognostic, or therapeutic decisions
    10. list off-label uses of an AI algorithm assessed by postdeployment evaluation
    11. identify the AI algorithm evaluation class that uses task-agnostic FoMs
    12. describe data sources that AI algorithms are evaluated against in POC evaluation
    13. list the evaluation classes that include patient population in the output claim
    14. identify the algorithm evaluation class that does not consider clinical outcomes
    15. describe the technical study evaluation types used to evaluate the accuracy of an AI algorithm
    16. list the types of AI algorithm technical evaluation studies
    17. identify software used to simulate an imaging system for realistic simulation studies
    18. list data types used in realistic simulation studies
    19. describe the study type used to conduct technical evaluation studies
    20. describe the technical task-specific evaluation studies that may need human expert readers
    21. identify the study type that is the most common mechanism for evaluating AI algorithms
    22. list the characteristics of retrospective clinical evaluation studies of AI algorithms
    23. describe the most common and strongest prospective interventional study design
    24. list the characteristics of prospective interventional studies of AI algorithms
    25. identify the study type recommended to evaluate a purely descriptive AI algorithm
    26. identify the study type recommended to evaluate a fully autonomous prescriptive AI algorithm
    27. describe clinical outcomes that are used as reference standards in a clinical evaluation study
    28. list FoMs used to demonstrate the utility of an algorithm in prognostic decision making
    29. describe the postdeployment evaluation technique used to identify root causes for equipment failure
    30. describe the utility of image-quality phantom studies in postdeployment evaluation
    31. identify the RELAINCE guidelines evaluation classes that use external testing cohorts
    32. describe a possible solution to AI algorithm performance problems resulting from data shift
    33. identify the evaluation class in the RELAINCE guidelines that is optional