Framework

AVID stores instantiations of AI risks---categorized using the AVID taxonomy---using two base data classes: Vulnerability and Report. A vulnerability (vuln) is a high-level evidence of an AI failure mode, in line with the NIST CVEs. A report is one example of a particular vulnerability occurring, supported by qualitative or quantitative evaluation.

As an example, the vuln AVID-2022-V001 is about gender bias in the large language model bert-base-uncased. This bias is measured through multiple reports, AVID-2022-R0001 and AVID-2022-R0002, which measure gender bias in two separate contexts, using different metrics and datasets, and record salient information and references on those measurements.

The above formulation is similar to how incidents and incident reports are structured in the AI Incident Database. See Figure D.1 for a schematic representation of this structure.

Figure D.1. Schematic of the structure of the AVID taxonomy, vulns, and reports.

To account for diverse levels of details that different groups of AI risk examples can entail, we designate a class for each vulnerability and report. Each such vuln/report class extends the respective base class to a slightly different structure that enables storage of information at different granularities as required. For example, we currently support two vuln/report classes: evaluations of large language models (LLM Evaluation) and incidents from AIID (AIID Incident). Both have the same set of vuln fields but slightly different sets of values to be filled in under references and tags.