Fence user.yaml synchronization can delete dynamically managed Arborist users

[flashguerdon]

Fence user.yaml synchronization can delete dynamically managed Arborist users

Executive summary

On 31 July 2026, a Fence fence-create sync job removed multiple active Auth0 identities from Arborist. Authentication in Fence continued to succeed, but subsequent login-time group assignment failed because the corresponding Arborist user records no longer existed.

The deletion was not caused by an Arborist database migration. It was performed intentionally by Fence's user synchronization code when remove_users_with_no_policies=True.

The failure requires the following interaction:

1. A dynamic user has temporary project access in Arborist.
2. Arborist's delete_expired_access job removes an expired usr_grp or usr_policy row.
3. The user retains only generic policies inherited from anonymous or logged-in groups.
4. A later fence-create sync --yaml ... run excludes those generic policies from consideration.
5. Fence concludes that the user has no meaningful policies and deletes the Arborist identity.

The direct defect is not expiry cleanup. Expired authorization should be removed. The unsafe behavior is treating the absence of non-generic authorization as a reason to delete the user identity.

Impact

After the Arborist user is deleted:

• Auth0 login to Fence can still succeed.
• Fence retains its own database user record.
• Fence attempts to add the identity to groups during login.
• Arborist rejects those requests with user does not exist.
• The user receives incomplete or denied access until the Arborist identity is recreated.

Users with an unexpired non-generic project policy may survive the synchronization. This makes the behavior appear inconsistent even though the deletion rule is deterministic.

Components involved

Arborist expired-access maintenance

The Arborist maintenance job executes:

DELETE FROM usr_grp WHERE expires_at <= now();
DELETE FROM usr_policy WHERE expires_at <= now();

This removes expired group memberships and direct policy grants. It does not delete users from usr.

Fence user.yaml synchronization

The Helm job executes:

fence-create sync \
  --arborist http://arborist-service \
  --yaml /var/www/fence/user.yaml

This command synchronizes more than resources, roles, policies, and groups. It also reconciles existing Arborist users.

Deletion logic

Fence retrieves the user's effective Arborist policies, then removes policies configured as anonymous_policies and all_users_policies from consideration.

The user is deleted when:

remove_users_with_no_policies \
and not incoming_policies \
and not user_existing_policies

The full-sync user.yaml path currently hard-codes:

remove_users_with_no_policies=True

The runtime/dynamic path uses False, but that does not protect users when the deployment-time user.yaml job runs.

Reproduction

1. Create an Arborist user.
2. Grant a project group membership with an expiry.
3. Allow the membership to expire.
4. Run Arborist delete_expired_access.
5. Confirm the user remains but has only generic logged-in policies.
6. Run fence-create sync --arborist ... --yaml ....
7. Observe Deleting user ... from Arborist (since they have no policies).
8. Attempt login-time group synchronization.
9. Observe Arborist reject group assignment because the user no longer exists.

A simpler reproduction is possible by creating a user with only generic policies and running the user.yaml sync.

Root cause

Fence conflates two different states:

• the user currently has no non-generic authorization;
• the user identity is obsolete and should be deleted.

That assumption is unsafe for mixed authorization models where identities and memberships are managed dynamically through Auth0, REMS, GA4GH visas, or other external systems.

A user absent from user.yaml may still be a valid active identity. Expiry of authorization must not imply deletion of identity.

Proposed fix

Add explicit CLI control to fence-create sync:

--prune-users
--no-prune-users

Behavior:

• --prune-users: preserve current behavior for backward compatibility.
• --no-prune-users: preserve Arborist identities that have no remaining non-generic policies.

The default remains pruning enabled in the initial PR to avoid changing existing deployments silently. Dynamic deployments must opt out explicitly.

Deployment usage

For BioCommons deployments using Auth0 and REMS:

fence-create sync \
  --arborist http://arborist-service \
  --yaml /var/www/fence/user.yaml \
  --no-prune-users

Suggested Helm value:

usersync:
  pruneUsers: false

Suggested template fragment:

fence-create sync \
  --arborist http://arborist-service \
  --yaml /var/www/fence/user.yaml \
  {{- if .Values.usersync.pruneUsers }}
  --prune-users
  {{- else }}
  --no-prune-users
  {{- end }}

Backward compatibility

The proposed CLI default is --prune-users, matching current behavior. Existing invocations without either flag behave as before.

Security and authorization considerations

Disabling identity pruning does not preserve expired access:

• delete_expired_access still removes expired group memberships and policies.
• user.yaml synchronization can still revoke obsolete direct policies.
• the change only prevents deletion of the Arborist user row solely because no non-generic policy remains.

This separates identity lifecycle from authorization lifecycle.

Operational workaround before rollout

Until the patched Fence image is deployed:

1. Snapshot active Fence usernames before every user.yaml sync.
2. Run the sync.
3. Compare Fence users with Arborist users.
4. Recreate missing Arborist identities through the API.
5. Verify dynamic memberships on next login or replay the relevant entitlement event.

Validation plan

Automated tests

• Existing behavior deletes a policy-less user when pruning is enabled.
• The same user is preserved when pruning is disabled.
• The CLI flag propagates through sync_users, UserSyncer.sync, _sync, and _update_authz_in_arborist.
• Dynamic/single-user synchronization remains non-pruning.

Test environment

1. Record users and groups before synchronization.
2. Run fence-create sync ... --no-prune-users.
3. Confirm resources, roles, policies, and groups update normally.
4. Confirm no Deleting user messages appear.
5. Confirm all pre-existing Arborist users remain.
6. Remove an expired membership and rerun sync.
7. Confirm access is removed but the user identity remains.

Monitoring recommendations

Alert on:

Deleting user .* from Arborist \(since they have no policies\)

After every authorization deployment, compare active Fence users with Arborist users and fail the release if dynamically managed identities are missing.