feat(eth2): add PostgreSQL bulk key loading with AWS KMS envelope encryption - #1214
Open
usmansaleem wants to merge 10 commits into
Open
feat(eth2): add PostgreSQL bulk key loading with AWS KMS envelope encryption#1214usmansaleem wants to merge 10 commits into
usmansaleem wants to merge 10 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| @Override | ||
| public long getDbHealthCheckTimeoutMilliseconds() { | ||
| return dbHealthCheckTimeoutMilliseconds; | ||
| } |
There was a problem hiding this comment.
Unused health-check timeout option
Medium Severity
--postgres-keystore-db-health-check-timeout-milliseconds and getDbHealthCheckTimeoutMilliseconds() are exposed and documented, but nothing reads the value. Unlike slashing-protection, no periodic postgres keystore DB health check is scheduled, so configuring the option has no effect.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ce7db26. Configure here.
…ryption Loading 20,000 BLS keys directly from Azure Key Vault/AWS Secrets Manager takes ~8 minutes due to one HTTP call per key. This adds an alternative eth2 key source: keys are pre-encrypted (AES-256-GCM, envelope encryption) and stored in PostgreSQL by an external provisioning process, and Web3Signer bulk-streams and decrypts them locally, making exactly one vault call per tenant (not per key) to unwrap that tenant's DEK. This first slice wires up the AWS KMS-backed KEK resolver end-to-end: new `keys-postgres` module (schema, streaming loader, AES-GCM/AAD crypto, reference-counted DEK cache with safe wipe-on-evict), `--postgres-keystore-*` CLI options on the `eth2` subcommand, and Eth2Runner wiring with health check + metrics. Azure and HashiCorp KEK resolvers follow in subsequent PRs. See designs/postgres-bulk-key-loading.md for the full design and the provisioning-side SQL/AAD contract.
usmansaleem
force-pushed
the
feature/postgres-keystore-aws-kms
branch
from
July 28, 2026 02:34
960b4de to
2b0d21d
Compare
| dekForRow, | ||
| cipherThreadLocal, | ||
| results, | ||
| errorCount))); |
There was a problem hiding this comment.
DEK wipe races active decrypts
Medium Severity
loadAll resolves a tenant DEK, then queues decrypt work without holding a lease. If the Caffeine TTL eviction runs in that window, pendingWipe flips and later acquireForRead calls fail, so in-flight keys for that tenant are skipped as errors—especially near TTL expiry on /reload.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 2b0d21d. Configure here.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| } | ||
|
|
||
| public Lease acquireForRead() { | ||
| lock.readLock().lock(); |
|
There was a problem hiding this comment.
Pull request overview
Adds a new ETH2/BLS key-loading backend that bulk-streams envelope-encrypted validator keys from PostgreSQL and locally decrypts them using a tenant-scoped DEK, unwrapped via AWS KMS (one KMS call per tenant). This introduces a new keys-postgres module, integrates it into the existing signer-loading pipeline (startup + /reload), and documents the provisioning-side cryptographic/schema contract.
Changes:
- Introduces
keys-postgres: schema + version check, streaming JDBC bulk loader, AES-256-GCM + mandatory AAD, and a TTL DEK cache with wipe-safety. - Adds AWS KMS KEK resolver (per-region KMS clients) and wires Postgres bulk loading into
Eth2Runnerwith health check + metrics. - Adds new
eth2CLI options for Postgres keystore + AWS KMS KEK credentials, plus tests and distribution packaging for migrations.
Reviewed changes
Copilot reviewed 36 out of 37 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| signing/src/main/java/tech/pegasys/web3signer/signing/config/PostgresKeystoreParameters.java | Defines Postgres keystore config surface (enablement, DB params, cache TTL, parallelism, timeout). |
| signing/src/main/java/tech/pegasys/web3signer/signing/config/PostgresAwsKmsKekParameters.java | Defines AWS KMS KEK-unwrap credential parameters for Postgres keystore. |
| signing/src/main/java/tech/pegasys/web3signer/signing/config/metadata/SignerOrigin.java | Adds POSTGRES origin to distinguish Postgres-loaded keys. |
| signing/src/main/java/tech/pegasys/web3signer/signing/bulkloading/BlsPostgresBulkLoader.java | Bridges keys-postgres decrypted key DTOs into BlsArtifactSigners; long-lived loader to preserve DEK cache across reloads. |
| signing/build.gradle | Adds dependency on new :keys-postgres module. |
| settings.gradle | Includes new keys-postgres Gradle module. |
| keys-postgres/src/testFixtures/java/tech/pegasys/web3signer/keystorage/postgres/PostgresKeystoreTestUtil.java | Embedded Postgres + Flyway migration helper for tests. |
| keys-postgres/src/test/java/tech/pegasys/web3signer/keystorage/postgres/PostgresBulkKeyLoaderTest.java | Integration tests for bulk load behavior (per-tenant vault calls, cache hits, failure isolation). |
| keys-postgres/src/test/java/tech/pegasys/web3signer/keystorage/postgres/crypto/TenantDekTest.java | Tests wipe/lease behavior for DEK byte lifecycle safety. |
| keys-postgres/src/test/java/tech/pegasys/web3signer/keystorage/postgres/crypto/TenantDekCacheTest.java | Tests DEK caching semantics (tenant/version keying, wipe on close). |
| keys-postgres/src/test/java/tech/pegasys/web3signer/keystorage/postgres/crypto/AesGcmKeyCipherTest.java | Tests AES-GCM encrypt/decrypt and rejection of tampering/wrong AAD/key. |
| keys-postgres/src/test/java/tech/pegasys/web3signer/keystorage/postgres/crypto/AadCodecTest.java | Tests deterministic/uncolliding AAD encoding and version differentiation. |
| keys-postgres/src/main/resources/migrations/keystore-postgresql/V00001__initial.sql | Initial schema for tenants/keys + database_version. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/TenantRecord.java | Tenant DTO used by KEK resolvers and AAD binding. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/PostgresKeystoreVersionChecker.java | Startup schema version verification (no auto-migrations). |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/PostgresConnectionFactory.java | Builds read-only Hikari DataSource for Postgres keystore access. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/PostgresBulkKeyLoader.java | Streaming query + per-tenant DEK resolution + parallel row decrypt into decrypted key DTOs. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/kek/KekResolver.java | Abstraction for vault-backed KEK unwrapping. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/kek/KekResolutionException.java | Error type for DEK unwrap failures with logging constraints. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/kek/awskms/PostgresAwsKmsKekResolver.java | AWS KMS implementation of KEK resolver; caches clients per region. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/kek/awskms/AwsKmsKekCredentials.java | Credential interface used by the AWS KMS KEK resolver. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/DecryptedBlsKey.java | Module-local DTO for decrypted key material (signing-layer maps to Teku types). |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/crypto/TenantDekCache.java | Caffeine-based DEK cache with wipe on eviction/close. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/crypto/TenantDek.java | Lease-based DEK byte holder to avoid wipe races during concurrent decrypt. |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/crypto/AesGcmKeyCipher.java | AES-256-GCM codec for `IV |
| keys-postgres/src/main/java/tech/pegasys/web3signer/keystorage/postgres/crypto/AadCodec.java | Length-prefixed AAD encoding used to bind ciphertexts to (tenant, key, version). |
| keys-postgres/build.gradle | Build config and dependencies for new module, including AWS SDK + JDBC + cache libs. |
| gradle/spotless.java.license | Updates current license header template. |
| gradle/spotless.java.former.license | Adds legacy license header template for older files. |
| designs/postgres-bulk-key-loading.md | Full design + schema + cryptographic/provisioning contract documentation. |
| core/src/main/java/tech/pegasys/web3signer/core/Eth2Runner.java | Wires Postgres bulk loader into eth2 signer load pipeline, adds metrics + health check, ensures loader is long-lived. |
| core/src/main/java/tech/pegasys/web3signer/core/config/HealthCheckNames.java | Adds health-check name for Postgres bulk loading. |
| commandline/src/test/java/tech/pegasys/web3signer/commandline/CommandlineParserTest.java | Adds CLI parsing/validation tests for Postgres keystore options. |
| commandline/src/main/java/tech/pegasys/web3signer/commandline/subcommands/Eth2SubCommand.java | Adds CLI mixins + validation for Postgres keystore and AWS KMS KEK parameters. |
| commandline/src/main/java/tech/pegasys/web3signer/commandline/PicoCliPostgresKeystoreParameters.java | Implements PostgresKeystoreParameters via PicoCLI options. |
| commandline/src/main/java/tech/pegasys/web3signer/commandline/PicoCliPostgresAwsKmsKekParameters.java | Implements PostgresAwsKmsKekParameters via PicoCLI options. |
| build.gradle | Spotless dual-license header handling + includes new migrations in distribution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+81
to
+83
| this.decryptionParallelism = | ||
| Math.clamp(decryptionParallelism, 1, Runtime.getRuntime().availableProcessors()); | ||
| } |
Comment on lines
+95
to
+102
| private static String extractRegion(final String kmsKeyArn) { | ||
| final List<String> parts = Splitter.on(':').splitToList(kmsKeyArn); | ||
| if (parts.size() < 4 || !"arn".equals(parts.getFirst())) { | ||
| throw new KekResolutionException( | ||
| "tenants.kek_key_id is not a valid KMS key ARN: " + kmsKeyArn); | ||
| } | ||
| return parts.get(3); | ||
| } |
Comment on lines
+79
to
+80
| final byte[] iv = new byte[IV_LENGTH_BYTES]; | ||
| SecureRandom.getInstanceStrong().nextBytes(iv); |
Comment on lines
+63
to
+66
| @Override | ||
| public void close() throws IOException { | ||
| db.close(); | ||
| } |
Comment on lines
+146
to
+148
| // postgresBulkKeyLoader also closes the datasource, since it created it via | ||
| // PostgresConnectionFactory and owns its lifecycle. | ||
| postgresBulkKeyLoader.close(); |
Comment on lines
+37
to
+39
| default long getDbHealthCheckTimeoutMilliseconds() { | ||
| return 3000; | ||
| } |
joshuafernandes
force-pushed
the
master
branch
from
September 13, 2026 22:01
32bea26 to
e58a979
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Loading BLS validator keys directly from Azure Key Vault / AWS Secrets Manager issues one HTTP call per key, which takes ~8 minutes for 20,000 keys at startup and on every
/reload. This PR adds an alternative eth2 key source: keys are pre-encrypted (AES-256-GCM envelope encryption) and stored in PostgreSQL by an external/offline provisioning process, and Web3Signer bulk-streams and decrypts them locally — making exactly one vault call per tenant (not per key) to unwrap that tenant's Data Encryption Key (DEK).This PR wires up the AWS KMS-backed KEK resolver end-to-end as the first of three planned backends (Azure and HashiCorp to follow in subsequent PRs, tracked in
designs/postgres-bulk-key-loading.md).keys-postgresmodule: schema + version check (migrations are packaged but never auto-run, matchingslashing-protection's convention), a streaming bulk loader (fetchSize=1000, forward-only), AES-256-GCM decrypt with mandatory AAD binding each ciphertext to(tenant, key, dek_version), and a reference-counted DEK cache (Caffeine, 15 min TTL) that only wipes key bytes once no in-flight decrypt is using them.signing:BlsPostgresBulkLoader— long-lived by design (constructed once, unlike the otherBls*BulkLoaders) so the DEK cache actually persists across/reloadcycles.commandline: new--postgres-keystore-*and--postgres-keystore-aws-kms-*options on theeth2subcommand, with validation following the existing hand-written style (no@ArgGroup, per this repo's convention).core: wired intoEth2Runner.bulkLoadSigners()with a health check and newpostgres_bulk_load_time/postgres_kek_vault_calls_totalmetrics.designs/postgres-bulk-key-loading.md: full design write-up plus the provisioning-side contract (schema, exact ciphertext/AAD byte layout, per-vault KEK-wrapping notes, exampleINSERTs, and worked encrypt examples in Java/Python/Node.js) for whoever builds the external provisioning process.Test plan
./gradlew :keys-postgres:test— cipher round-trip + tampered-tag/AAD-mismatch rejection, DEK cache concurrency/wipe-safety, and embedded-postgres integration tests proving exactly-N-vault-calls-per-tenant and DEK-cache-hit behavior across reloads./gradlew :signing:test :commandline:test :core:test— CLI parsing/validation tests for the new option group./gradlew compileTestJava compileIntegrationTestJavarepo-wide — no regressions in existing modules./gradlew spotlessApplyNote
High Risk
Introduces a new path that decrypts validator private keys and depends on AWS KMS, database content, and cryptographic contracts; misconfiguration or provisioning mistakes could prevent loading or affect which keys sign.
Overview
Adds an eth2 bulk key source that reads envelope-encrypted BLS keys from PostgreSQL instead of one HTTP call per key from Azure/AWS/GCP secrets. Keys are decrypted locally after one vault unwrap per tenant (DEK via KEK); a 15-minute DEK cache is kept on a long-lived loader so
/reloadcan avoid repeat KMS calls.Introduces the
keys-postgresmodule (streaming JDBC load, AES-256-GCM + mandatory AAD, schema version check, packaged migrations not auto-applied) andBlsPostgresBulkLoaderinsigning, wired throughEth2Runnerwithkeys-check/postgres-bulk-loadingandpostgres_bulk_load_time/postgres_kek_vault_calls_totalmetrics. New--postgres-keystore-*and--postgres-keystore-aws-kms-*CLI options (separate from existing AWS Secrets Manager credentials);SignerOrigin.POSTGRES. AWS KMS is the only KEK backend implemented in this PR. Includesdesigns/postgres-bulk-key-loading.mdand minor build/distribution tweaks (Spotless dual license headers, keystore migrations in the dist).Reviewed by Cursor Bugbot for commit 2d68bd4. Bugbot is set up for automated code reviews on this repo. Configure here.