Feature Request: Team, agent, and memory-level access control

[Sanderhoff-alt]

Summary

I would like to request a discussion around a more detailed permission and sharing model for the open-source Hindsight server.

The current OSS authorization model seems intentionally simple:

• ApiKeyTenantExtension: one shared API key mapped to the configured schema.
• SupabaseTenantExtension: one Supabase user JWT mapped to a schema derived from that user's sub claim.
• Banks are the main application-level resource boundary exposed by the API, but the built-in OSS tenant extensions do not appear to provide organization/team membership, scoped API keys, or memory-level authorization.

This works well for simple self-hosted or per-user-isolated deployments, but it does not fully cover collaborative/team memory use cases.

Use Cases

I would like Hindsight OSS to better support collaborative memory deployments such as:

• A single tenant / organization with multiple users.
• Users with different roles inside the same tenant, such as Owner, Admin, Member, service account, or agent.
• A deployment where each user can have private memory while also participating in shared organization memory.
• Multiple agents working under the same tenant, where each agent may need private working memory and access to selected shared memory.
• Scoped API access for different applications, agents, services, or environments.
• Read-only, write-only, or limited-access integrations.
• Controlled sharing of selected memories between users or agents.
• Optional sharing across tenant or organization boundaries, if the project decides this is an important supported model.

Why This Matters

For collaborative AI agent systems, memory is not always either fully private or fully shared.

Common patterns may include:

• A user has private memories.
• A team has shared memories.
• An agent has its own working memory.
• Some memories should be visible to selected users or agents.
• Some API keys should only access specific banks, agents, sessions, or memory subsets.
• Some users should read shared memories but not modify them.
• Some agents should write memories but not delete or share them.

The current SupabaseTenantExtension provides strong isolation, but each Supabase user effectively becomes their own tenant/schema. That makes organization-level sharing difficult.

Hindsight Cloud appears to support a richer organization model with team roles and bank-scoped API keys. It would be useful to discuss whether some version of that permission model should exist in the OSS server as well, probably as an optional capability rather than a requirement for simple deployments.

PowerMem has a useful reference direction: it wraps the memory core with an agent layer and uses metadata, scopes, and per-memory permissions to support more fine-grained control. I am not saying Hindsight should copy that design directly, and PowerMem's implementation details may not map directly to Hindsight, but it is a useful reference point for wrapper-layer authorization: https://github.com/oceanbase/powermem

Possible Capabilities

This feature could include some or all of the following, depending on what maintainers and users consider in scope:

• Tenant / organization membership.
• User and service-account roles inside a tenant.
• API keys scoped to selected banks.
• API keys scoped to selected operations, such as retain, recall, reflect, update, delete, or share.
• API keys scoped to selected agents, sessions, or metadata filters.
• Private, tenant-shared, and explicitly shared memory visibility modes.
• Agent-level or session-level memory access rules.
• Row-level / memory-level authorization for selected advanced deployments.
• Memory-level ACLs when metadata-based rules are not enough.
• Metadata-based authorization filters.
• Explicit memory sharing between users or agents.
• Optional cross-tenant / cross-organization sharing, if maintainers consider it in scope.
• Extension hooks for custom authorization policies.
• Query-time authorization filtering during recall/search.

I do not think all of these necessarily need to be implemented, and some may be too complex or outside the project's intended scope. The main goal is to identify which permission layers are genuinely needed and where they should live.

Design Considerations

This probably needs careful design. There is a real tradeoff between under-design and over-design.

Under-design risks:

• Teams cannot safely share memory.
• Users need to share one API key or one Supabase user identity as a workaround.
• Applications needing private + shared memory must fork Hindsight or build a large wrapper.

Over-design risks:

• Permission logic becomes hard to understand.
• Recall/search performance may suffer.
• Simple self-hosted deployments become harder to configure.
• Core Hindsight becomes too opinionated.

Ideally, the design would keep simple deployments simple, while allowing advanced deployments to opt into richer authorization.

Some capabilities may already be expressible with existing tags, metadata, or bank-level conventions. Those should probably remain lightweight and avoid new core concepts. Other capabilities, such as durable ACLs, scoped API keys, revocation, or enforced query-time authorization, may require new primitives. It would be important to separate these layers clearly so that Hindsight does not overfit one permission model while still providing enough structure for robust access control.

Suggested Direction

My preferred direction would be to explore this primarily as a wrapper layer, plugin, or extension-based authorization layer, with minimal intrusion into the Hindsight core.

At a high level, the core memory engine should ideally remain focused on memory operations, while the authorization layer is responsible for resolving the caller, enforcing access rules, and limiting which memories are visible for each operation. The core could expose the necessary hooks and request context, but avoid hard-coding one opinionated permission model.

A useful design distinction may be:

Simple/default model:
  caller -> allowed banks
  access control remains bank-scoped

Fine-grained model:
  caller -> richer authorization context
  memory rows -> ownership / visibility / metadata / ACL attributes
  retrieval and mutation operations -> enforce those rules before returning or modifying memories

In other words, bank-scoped access can remain the simple default. More granular access, including row-level memory access control, would require the authorization layer to resolve additional caller context and compare it against memory attributes, metadata, or dedicated ACL state. Some policies may be expressible with existing tags or metadata; stricter policies may need dedicated authorization tables or persisted permission records.

This may fit naturally with existing extension points, or it may require a new dedicated authorization extension. Either way, the goal would be to keep the core stable, make fine-grained permissions opt-in, and leave room for the community to decide which permission layers are actually worth supporting.

Perhaps this should start in a dedicated experimental branch, plugin, wrapper package, or design document, then collect feedback from Hindsight users before merging anything into main.

Open Questions

• Should Hindsight OSS support organization/team-style access control?
• Should bank-level permissions be enough, or is memory-level access control needed?
• Should memory sharing be modeled explicitly?
• Should authorization be implemented as a wrapper/plugin/extension layer rather than directly in core?
• What minimal hooks or request context would core need to expose for this to work cleanly?
• Which policies can be handled with existing tags/metadata, and which require new persisted authorization state?
• Should authorization be schema-based, row-level, metadata-based, ACL-table-based, or extension-driven?
• Should cross-tenant sharing be supported at all?
• What permission model do current Hindsight users actually need?

Community Feedback

Since I am not a repository maintainer, this would need maintainer input before it can move beyond discussion. I would especially like to hear @nicoloboschi's thoughts on whether this direction fits the project and whether the right boundary should be an extension, a wrapper layer, or some combination of both.

I would also like to hear from other Hindsight users about their real authorization and sharing needs. Different deployments may need very different tradeoffs, so it would be useful to collect examples before choosing a design.**

Examples that would be helpful:

• How you currently model tenants, users, agents, sessions, and API keys.
• Whether bank-level access control is enough for your use case.
• Whether you need private + shared memory in the same deployment.
• Whether metadata/tags would be sufficient for your permission model.
• Whether you need durable ACLs or stronger enforced authorization.
• What performance, security, or operational constraints matter most.

If there is enough interest, perhaps this could become a community-driven effort: start with a shared design, prototype it in an experimental branch, plugin, or wrapper, and then merge the parts that prove useful and maintainable.

[nicoloboschi]

this request makes sense in general.

our take is that each company/enterprise likes auth/authz in different ways and, yes, Hindsight is intentionally simple in this area.

this provides users and adopters to easily build a middleware layer on top of it with custom acls, syncs on orgs auth (e.g. google ws, okta..)

we want to keep it simple. there's already the extension mechanism for users to fork it.

on the isolation:

• banks can be fully isolated to teams
• inside a team, you should use tags. if a person shouldn't see others data, it should be enforced to use tags. an admin tho should be able to see all users data

all the primitives are already there

[Sanderhoff-alt]

I opened a draft PR as a first implementation slice for this discussion: #2262.

The PR is intentionally closer to the current Hindsight Cloud model than to the full set of fine-grained permissions discussed here. It adds an opt-in Supabase-backed organization profile with organizations, members, owner/admin/member roles, manual invites, organization switching, and Hindsight scoped API keys that can be limited by Bank and operation.

It should be treated as a discussion draft rather than a complete answer to this issue. The current scope is still Bank-level access control. It does not implement memory-level ACLs, row-level memory authorization, agent/session-level isolation, cross-tenant sharing, billing/usage enforcement, SSO, automatic invite email, password recovery UI, or full organization deletion.

My hope is that this gives the community and maintainers a concrete baseline to review: whether the organization/team/API-key shape is the right foundation, whether this should live as extensions plus a control-plane wrapper, and which finer-grained authorization primitives should come next without over-designing the core.

Feedback is very welcome, especially from users who already need multi-user teams, shared Banks, agent-scoped access, or memory-level sharing/isolation.