Skip to content

Add Security and Privacy Terms for WG review - #56

Open
kdruckman wants to merge 1 commit into
aaif:mainfrom
kdruckman:add-security-terms
Open

kdruckman wants to merge 1 commit into
aaif:mainfrom
kdruckman:add-security-terms

Conversation

@kdruckman

Copy link
Copy Markdown
Collaborator

This adds the initial Security & Privacy terms agreed upon in the taxonomy and landscape workstream.

Definitions to be reviewed by the Security and Privacy WG before approving.

In particular, Agent sabotage and Agent mishandling/misuse need refinement.

Add security and privacy wg terms for discussion

@MatthewKhouzam MatthewKhouzam left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are great, could you add some scope, i feel defining examples with sabotage vs derailment would be good to have, also they are related terms. They probably will help accuracy too.

Comment thread taxonomy/taxonomy-data.js
category: '',
aliases: [],
broaderTerm: null,
definition: 'An AI agent that operates outside its authorized boundaries, whether due to compromise, misconfiguration, or emergent behavior.',

@awfrazer awfrazer Sep 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does intention factor in here? Thinking out loud, it could be operating outside of authorized or intended boundaries.

@narko4u

narko4u commented Sep 13, 2026

Copy link
Copy Markdown

Thanks for pulling the Security & Privacy set together, and for flagging the two entries yourself. One observation on why they need work, and three changes that would close it.

The set classifies by attributed cause, and attribution is not observable.

Read the five entries together and they separate on whether an external party caused the behaviour. Agent derailment is explicitly "without malicious external cause". Agent sabotage and Agent mishandling/misuse are "deliberate manipulation ... by an external party". Rogue agent covers "compromise, misconfiguration, or emergent behavior". Three of the five therefore classify the agent by what someone else did to it, or by what went wrong inside it. That is a cause attribution, and a cause attribution is a claim about the world rather than a field in a record.

The two flagged entries are not separable at the point of evidence.

As written, the cause clause in both definitions is identical:

Deliberate manipulation of an AI agent's behavior by an external party, causing it to act against its intended purpose.

Agent sabotage adds "to an elevated degree with severe or criminal consequences". So the only difference between sabotage and mishandling is a judgement about severity, and the taxonomy already has a term for that (Risk classification). A consumer reading an incident record cannot use these two terms to separate the cases.

Nothing in the set states who establishes the attribution.

Both definitions are compatible with the affected party acting and reporting on itself, and a reader has no way to tell. A self-reported attribution and an independently witnessed one are not interchangeable for any downstream purpose: a control, a disclosure, or a risk score.

Concretely, take two incident records identical in every producer-authored field:

  • Record A: the external manipulation is established by material the affected operator did not produce.
  • Record B: the same fields, the attribution asserted only by the affected operator.

Both satisfy the definition of Agent sabotage as currently worded. The taxonomy cannot separate them, so the term carries no decision.

Suggested changes

  1. Differentiate Agent sabotage from Agent mishandling/misuse by consequence and jurisdiction rather than by mechanism, and point both at Risk classification.
  2. Add a scopeNote to both: cause attribution is a claim, and the establishing party is recorded separately from the claim. A self-reported attribution is not interchangeable with an externally witnessed one. The vocabulary for this is already filed as proposals in this workstream at Define: self-reported #52 (self-reported), Define: witness-scope #53 (witness-scope) and Define: three-state reconciliation #54 (three-state reconciliation).
  3. Populate the empty fields: contrastsWith across the three cause-based entries, and relatedTerms only against terms already present in the taxonomy, where Attestation (already an entry, and co-owned by this workstream) and Accountability both apply.
  4. On Privacy-preserving execution, naming "zero-knowledge proofs or trusted execution environments" inside a definition will date it. The durable part is the property, and whether the claim of protection is checkable.
  5. Intent is already an entry. The set uses "intended purpose" for the agent's design intent and "deliberate" for the external party's intent, without saying whose intent is meant.

The distinction in (2) is already carried in the published evidence-grade ladder E0 to E4 (witnessos, Empire Labs), where each rung names what a claim requires and the establishing party is recorded separately from the claim. Offered as a reference for wording, not as something for this workstream to adopt.

Happy to draft the exact entry text as a follow-up PR in the shape of #37, or to supply Records A and B as fixture files if the workstream wants a test case.

@narko4u

narko4u commented Sep 13, 2026

Copy link
Copy Markdown

As offered on this thread: the two scope notes are drafted and open as a PR against this branch.

Each note records, on its own entry, that the attributed cause is a claim rather than a field in a record, and that the establishing party is recorded separately from the claim, so a self-reported attribution is not interchangeable with one established by material the affected operator did not produce. Each note points at the vocabulary already filed in this workstream: #52 (self-reported), #53 (witness-scope) and #54 (three-state reconciliation).

One note per entry rather than one shared note, per CONTRIBUTING section 5.C.2.

Two lines, scopeNote only. Definitions, category, aliases, broaderTerm, relatedTerms, contrastsWith and workgroups are unchanged.

PR: kdruckman#1

If it is easier to apply the two lines directly, take the text and close the PR. If you would rather they land after this merges, say so and I will re-target main.

@julianna-ciq julianna-ciq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants