Getting Started¶
seismometer allows you to evaluate AI
model performance using standardized evaluation criteria that helps you
make decisions based on your own local data. seismometer helps you
validate a model’s initial performance and continue to monitor its
performance over time.
Local validation of an AI model requires cross-referencing data about patients (such as demographics and clinical outcomes) and model performance (including inputs and outputs).
What is in Seismometer¶
Jupyter Notebook examples¶
Open-source Python libraries and Jupyter Notebooks allow data scientists to quickly create a notebook to evaluate model performance.
We expect these examples to continually expand and evolve as new validation and analysis techniques and approaches are created.
Data Schemas¶
Standardized data schemas to incorporate data from single or multiple source systems.
Notebook configuration definitions¶
Within the configuration file, you can control details such as the cohorts to include and the outcomes relevant to the model. You can also provide supplemental model documentation to give data scientists and other report consumers working with the notebook background on the model, definitions of terms and cohorts, and tips for working with data in the notebook.
Install Seismometer¶
From the Python Package Index (PyPI) install the package by running the following at your command line:
pip install seismometer
If you want to utilize fairness audit visualizations, run the following at your command line:
pip install seismometer[audit]
For additional details on installing Python packages, refer to the Python Packaging User Guide.
Gather data¶
Begin by gathering data on model inputs and outputs from production use of a model on your local data. Set up a process for collecting data in the following formats so that you can rerun the notebook for ongoing monitoring of model performance.
Your model developer or health IT vendor might have instructions on how to gather this data. Refer to the Integration Guide for more information.
Format¶
Save data for each of the following tables in the parquet format. This can be done with Pandas to_parquet function.
Predictions
Predictions data includes the model inputs, outputs, patient demographics, and other relevant model features.
It is a wide table where each row represents an individual output on an entity. Each prediction must include:
At least one entity identifier. You can optionally include additional context identifiers.
A primary output and timestamp for when the output was created.
You can optionally, include additional outputs and feature data as needed.
The following is an example of a predictions table.
ID |
|
Predict Time |
Risk Of Scoring |
Has Previous Score |
Categorical Feature1 |
Categorical Feature… |
|---|---|---|---|---|---|---|
123 |
54321 |
1/1/2024 8:00:00 |
0.5 |
0 |
Cat Value1 |
Cat Value1 |
123 |
54321 |
1/1/2024 8:15:00 |
0.99 |
1 |
Cat Value2 |
Cat Value2 |
Events
Events data should be formatted in a long/narrow table design that includes relevant interventions, ground-truth outcomes, and other events.
Lines per entity must include:
At least one identifier that matches the identifier used in the predictions table. Include any additional context identifiers used in the Predictions table.
An event label/type column.
One or two additional nullable columns. The example below shows an event value and timestamp.
The ground truth for a model must be one of the outcome values.
The following is an example of an events table.
ID |
ContactID |
Time |
Type |
Value |
|---|---|---|---|---|
123 |
54321 |
1/1/2024 10:36:25 |
SEP-3 |
true |
123 |
54321 |
1/1/2024 10:45:02 |
CDC |
true |
Provide Configuration Files¶
The configuration file allows you to define relevant cohorts including things like demographics, sensitive groups, and other criteria for evaluating model performance. You can define outcome and intervention events that relate to actions taken based on model output and expected measurable results driven by those actions.
The configuration includes two core elements:
Data definitions to map columns in your data tables to the keys used in the Notebook template. This includes information on how data is used, including associating events to relevant predictions.
Supplemental documentation to give report consumers working in the Notebook background on the model, definitions of terms and cohorts, and tips for working with data in the notebook.
The model developer should provide much of the detail for the configuration file, and you can modify it as needed to fit your population and local workflows. Refer to the Integration Guide for more information.
For details on creating configuration files, refer to the User Guide.