Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Connectors

Connectors import or transform data from external sources into AVID-compatible structures. Current first-party connectors in avidtools include:

  • atlas: import/convert MITRE ATLAS case studies

  • cve: import/convert NVD CVEs

  • inspect: convert Inspect AI logs into AVID reports

  • url: scrape a URL and generate an AVID report via LLM-assisted extraction

  • garak: normalize AVID report JSON/JSONL generated from garak-style outputs

List of Connectors

ATLAS Case Study

Adversarial ML case studies hosted on MITRE ATLAS double up as AVID reports or vulnerabilities. Their data are stored as yaml files in the ATLAS GitHub. Given the case study ID, we can import that data as a Report object.

For example, the following code imports the Tay Poisoning case study.

from avidtools.connectors import atlas

CS_ID = 'AML.CS0009'
cs = atlas.import_case_study(CS_ID) # returns a dict in the original schema
report = atlas.convert_case_study(cs) # returns a Report object
NIST CVE

Some AI vulnerabilities related to application security or cybersecurity detection models may be cross-posted as CVEs. This data can be queried from the NIST NVD API. Given a CVE ID, we can import that data into a Vulnerability object.

For example, the following code imports the Proofpoint Evasion vulnerability, the first ever ML CVE reported.

from avidtools.connectors import cve

CVE_ID = 'CVE-2019-20634'
cv = cve.import_cve(CVE_ID) # returns a custom dict
vuln = cve.convert_cve(cv) # returns a Vulnerability object
Inspect AI

The open-source LLM evaluation toolkit Inspect AI enables standardized assessment of LLM behavior across a wide range of capabilities.

The avidtools.connectors.inspect connector converts Inspect evaluation logs (.eval / .json) into one or more AVID Report objects. To do so, first run your evaluation with Inspect AI and save logs to a directory.

For example, the following code runs the BOLD benchmark we contributed to Inspect Evals on gpt-4o-mini.

inspect eval inspect_evals/bold --model openai/gpt-4o-mini --log-dir ./experiment-log

Following this, you can run to convert the Inspect logs to a list of AVID Report objects.

from avidtools.connectors.inspect import convert_eval_log

reports = convert_eval_log("experiment-log/bold_logs.eval")

You can optionally pass normalize=True to apply post-processing that enriches descriptions and formatting:

reports_normalized = convert_eval_log(
    "experiment-log/bold_logs.eval",
    normalize=True,
)

When converting many logs, use the batch helper and write results to JSONL:

from pathlib import Path
from avidtools.connectors.inspect import convert_eval_logs, write_reports_jsonl

log_paths = Path("experiment-log").glob("*.eval")
reports = convert_eval_logs(log_paths, normalize=False)
written = write_reports_jsonl(reports, Path("out/inspect-reports.jsonl"))
print(f"Wrote {written} reports")

Optionally, you can store the original Inspect logs in an S3 bucket and reference that in the report:

reports = convert_eval_log(
    "experiment-log/bold_logs.eval",
    s3_bucket="my-eval-logs",
    s3_key_prefix="inspect",
    s3_region="us-east-1",
)

S3 upload dependency: S3 upload requires boto3, and require configuring AWS secrets using aws configure.

garak

Since the garak CLI can produce AVID reports directly, the garak connector only does optional normalization of the AVID reports that garak produces.

API key optional: garak normalization works without keys, but if OPENAI_API_KEY is available it can use OpenAI-assisted probe-summary enrichment; otherwise it falls back to deterministic descriptions.

URL

Generate an AVID report directly from a URL by scraping page content and extracting report fields.

API key required: URLConnector uses an OpenAI-backed agent and requires OPENAI_API_KEY (either passed as api_key=... or set in environment).

PreviousGiskard
NextIntroduction

Last updated