feat(policy): DSPX-898 NDR database schema #2055

ryanulit · 2025-04-08T15:01:51Z

Proposed Changes

Important

For historical purposes: Non Data Resource was the original name of this object, but per discussion below we landed on Registered Resource

Create new database migrations for NDR schema

Checklist

I have added or updated unit tests
I have added or updated integration tests (if appropriate)
I have added or updated documentation

Testing Instructions

service/policy/db/migrations/20250407000000_add_non_data_resources.sql

github-actions · 2025-04-10T17:58:21Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.067 ms	0.040 ms	0.112 ms
rewrap-nanotdf	4887	409.485 ms	45.547 ms	783.115 ms
rewrap-tdf3	5001	448.332 ms	61.713 ms	960.994 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	376.719981ms
Throughput	265.45 requests/second

github-actions · 2025-04-10T18:04:25Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.081 ms	0.049 ms	0.138 ms
rewrap-nanotdf	4600	445.169 ms	25.043 ms	933.536 ms
rewrap-tdf3	5001	458.440 ms	42.081 ms	1099.012 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	356.584642ms
Throughput	280.44 requests/second

github-actions · 2025-04-10T18:15:31Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.070 ms	0.031 ms	0.133 ms
rewrap-nanotdf	4788	431.527 ms	54.188 ms	1021.112 ms
rewrap-tdf3	5001	451.186 ms	58.251 ms	993.586 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	352.461013ms
Throughput	283.72 requests/second

service/policy/db/migrations/20250407000000_add_registered_resources.md

github-actions · 2025-04-11T19:04:02Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.071 ms	0.045 ms	0.111 ms
rewrap-nanotdf	4645	439.712 ms	49.354 ms	1054.901 ms
rewrap-tdf3	5001	448.409 ms	44.065 ms	1106.873 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	348.314481ms
Throughput	287.10 requests/second

github-actions · 2025-04-11T19:28:35Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.070 ms	0.036 ms	0.109 ms
rewrap-nanotdf	4962	441.450 ms	76.182 ms	1076.160 ms
rewrap-tdf3	5001	478.847 ms	68.519 ms	1075.427 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.875355ms
Throughput	274.82 requests/second

github-actions · 2025-04-11T19:29:11Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.069 ms	0.042 ms	0.101 ms
rewrap-nanotdf	4703	425.664 ms	31.790 ms	971.556 ms
rewrap-tdf3	5001	454.674 ms	41.235 ms	1232.370 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	352.09304ms
Throughput	284.02 requests/second

jakedoublev

🎉

service/policy/db/schema_erd.md

github-actions · 2025-04-11T20:03:47Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.072 ms	0.036 ms	0.126 ms
rewrap-nanotdf	4987	415.415 ms	57.621 ms	844.881 ms
rewrap-tdf3	5001	449.663 ms	56.857 ms	1063.923 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	359.953412ms
Throughput	277.81 requests/second

🤖 I have created a release *beep* *boop* --- ## [0.5.3](service/v0.5.2...service/v0.5.3) (2025-05-22) ### Features * **authz:** authz v2 versioning implementation ([#2173](#2173)) ([557fc21](557fc21)) * **authz:** authz v2, ers v2 protos and gencode for ABAC with actions & registered resource ([#2124](#2124)) ([ea7992a](ea7992a)) * **authz:** export entity id prefix constant from entity instead of authorization service v1 ([#2261](#2261)) ([94079a9](94079a9)) * **authz:** subject mapping plugin support for ABAC with actions ([#2223](#2223)) ([d08b939](d08b939)) * bulk keycloak provisioning ([#2205](#2205)) ([59e4485](59e4485)) * **core:** add otel to opentdf services ([#1858](#1858)) ([53a7aa0](53a7aa0)) * **core:** Adds EC withSalt options ([#2126](#2126)) ([67b6fb8](67b6fb8)) * **core:** enhance db configuration options ([#2285](#2285)) ([ed9ff59](ed9ff59)) * **core:** New Key Index and Manager Plugin SPI ([#2095](#2095)) ([eb446fc](eb446fc)) * **core:** support onConfigUpdate hook when registering services ([#1992](#1992)) ([366d4dc](366d4dc)) * **core:** v2 ERS with proto updates ([#2210](#2210)) ([a161ef8](a161ef8)) * **policy:** actions crud service endpoints and proto validation ([#2037](#2037)) ([e933fa9](e933fa9)) * **policy:** actions service RPCs should actually hit storage layer CRUD ([#2063](#2063)) ([da4faf5](da4faf5)) * **policy:** add enhanced standard/custom actions protos ([#2020](#2020)) ([bbac53f](bbac53f)) * **policy:** Add platform key indexer. ([#2189](#2189)) ([861ef8d](861ef8d)) * **policy:** consume lib/identifier parse function ([#2181](#2181)) ([1cef22b](1cef22b)) * **policy:** DSPX-1018 NDR retrieval by FQN support ([#2131](#2131)) ([0001041](0001041)) * **policy:** DSPX-1057 registered resource action attribute values (DB + Service implementation) ([#2191](#2191)) ([6bf1b2e](6bf1b2e)) * **policy:** DSPX-1057 registered resource action attribute values (protos only) ([#2217](#2217)) ([6375596](6375596)) * **policy:** DSPX-893 NDR define crud protos ([#2056](#2056)) ([55a5c27](55a5c27)) * **policy:** DSPX-898 NDR database schema ([#2055](#2055)) ([2a10a6a](2a10a6a)) * **policy:** DSPX-901 NDR database crud ([#2071](#2071)) ([20e0a5f](20e0a5f)) * **policy:** DSPX-902 NDR service crud implementation (2/2) ([#2066](#2066)) ([030ad33](030ad33)) * **policy:** DSPX-902 NDR service crud protos only (1/2) ([#2092](#2092)) ([24b6cb5](24b6cb5)) * **policy:** Finish resource mapping groups ([#2224](#2224)) ([5ff754e](5ff754e)) * **policy:** GetMatchedSubjectMappings should provide value FQN ([#2151](#2151)) ([ad80044](ad80044)) * **policy:** key management crud ([#2110](#2110)) ([4c3d53d](4c3d53d)) * **policy:** Key management proto ([#2115](#2115)) ([561f853](561f853)) * **policy:** Modify get request to search for keys by kasid with keyid. ([#2147](#2147)) ([780d2e4](780d2e4)) * **policy:** Restrict KAS deletion when tied to Key ([#2144](#2144)) ([4c4ab13](4c4ab13)) * **policy:** Return KAS Key structure ([#2172](#2172)) ([7f97b99](7f97b99)) * **policy:** rotate keys rpc ([#2180](#2180)) ([0d00743](0d00743)) * **policy:** stored enhanced actions database migration, CRUD queries, SM updates ([#2040](#2040)) ([e6b7c79](e6b7c79)) * **sdk:** Add a KAS allowlist ([#2085](#2085)) ([d7cfdf3](d7cfdf3)) * **sdk:** add nanotdf plaintext policy ([#2182](#2182)) ([e5c56db](e5c56db)) * **sdk:** Use ConnectRPC in the go client ([#2200](#2200)) ([fc34ee6](fc34ee6)) ### Bug Fixes * **core:** access pdp cleanup before actions in ABAC decisioning ([#2123](#2123)) ([9b38a3c](9b38a3c)) * **core:** Autobump service ([#2080](#2080)) ([006c724](006c724)) * **core:** Autobump service ([#2104](#2104)) ([1f72cc7](1f72cc7)) * **core:** Autobump service ([#2108](#2108)) ([be5b7d7](be5b7d7)) * **core:** bump to go 1.24 and bump service proto module dependencies ([#2064](#2064)) ([94891a0](94891a0)) * **core:** Fix DPoP with grpc-gateway ([#2044](#2044)) ([4483ef2](4483ef2)) * **core:** fix service go.mod ([#2141](#2141)) ([3b98f6d](3b98f6d)) * **core:** Improves errors when under heavy load ([#2132](#2132)) ([4490a14](4490a14)) * **core:** Let legacy KAOs use new trust plugins ([#2218](#2218)) ([5aa6916](5aa6916)) * **core:** migrate from mitchellh/mapstructure to go-viper/mapstructure ([#2087](#2087)) ([0a3a82e](0a3a82e)) * **core:** update viper to 1.20.1 ([#2088](#2088)) ([09099e9](09099e9)) * **core:** Updates vulnerable dep go/x/net ([#2072](#2072)) ([11c02cd](11c02cd)) * **deps:** bump github.com/creasty/defaults from 1.7.0 to 1.8.0 in /service ([#2242](#2242)) ([86a9b46](86a9b46)) * **deps:** bump github.com/jackc/pgx/v5 from 5.5.5 to 5.7.5 in /service ([#2249](#2249)) ([d8f3b67](d8f3b67)) * **deps:** bump the internal group across 1 directory with 2 updates ([#2296](#2296)) ([7f92c70](7f92c70)) * **deps:** bump toolchain in /lib/fixtures and /examples to resolve CVE GO-2025-3563 ([#2061](#2061)) ([9c16843](9c16843)) * handle empty private and public key ctx structs ([#2272](#2272)) ([f3fc647](f3fc647)) * **policy:** remove predefined rules in actions protos ([#2069](#2069)) ([060f059](060f059)) * **policy:** return kas uri on keys for definition, namespace and values ([#2186](#2186)) ([6c55fb8](6c55fb8)) * update key_mode to provide more context ([#2226](#2226)) ([44d0805](44d0805)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>

ryanulit added 3 commits April 7, 2025 14:41

initial schema and mermaid diagram creation

1b95322

Merge branch 'main' into DSPX-898-ndr-database-schema

4750e7c

update comments

8615804

jakedoublev reviewed Apr 8, 2025

View reviewed changes

add metadata column and FK cascade delete

8ebf2ba

jakedoublev reviewed Apr 10, 2025

View reviewed changes

service/policy/db/migrations/20250407000000_add_non_data_resources.sql Outdated Show resolved Hide resolved

ryanulit added 2 commits April 10, 2025 11:21

Merge branch 'main' into DSPX-898-ndr-database-schema

8c03140

fix bad sql in migration

9294e6d

add unique constraint per PR comments

3d8c1a2

update mermaid diagram

380b3a0

rename to registered resources

f6ac917

github-actions bot reviewed Apr 11, 2025

View reviewed changes

ryanulit added 2 commits April 11, 2025 13:24

update mermaid diagram title

888a53e

Merge branch 'main' into DSPX-898-ndr-database-schema

57bd476

run make policy-erd-gen

c6ab94a

jakedoublev approved these changes Apr 11, 2025

View reviewed changes

github-actions bot reviewed Apr 11, 2025

View reviewed changes

ryanulit marked this pull request as ready for review April 11, 2025 20:01

ryanulit requested review from a team as code owners April 11, 2025 20:01

ryanulit enabled auto-merge April 11, 2025 20:02

ryanulit added this pull request to the merge queue Apr 11, 2025

Merged via the queue into main with commit 2a10a6a Apr 11, 2025
25 checks passed

ryanulit deleted the DSPX-898-ndr-database-schema branch April 11, 2025 20:21

opentdf-automation bot mentioned this pull request Apr 11, 2025

chore(main): release service 0.5.3 #2046

Merged

feat(policy): DSPX-898 NDR database schema #2055

feat(policy): DSPX-898 NDR database schema #2055

Uh oh!

Conversation

ryanulit commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Checklist

Testing Instructions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 10, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 10, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 10, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 11, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 11, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 11, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

jakedoublev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 11, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

Uh oh!

Uh oh!

ryanulit commented Apr 8, 2025 •

edited

Loading