[RFC] Cryptographic design decisions for integration of Parquet Modular Encryption

[nibix]

Introduction

This RFC describes necessary design decisions for integrating Parquet Modular Encryption (PME) into the OpenSearch plugins parquet-data-format and analytics-backend-datafusion.

For obtaining deeper insights into the design decisions, a prototype has been developed. It is available at #22067

Please review this document regarding cryptographic and architectural considerations.

Key management and derivation

The following diagram shows the retrieval/derivation flow for the encryption keys used for the Parquet files. It has been designed following the OpenSearch lucene encryption plugin.

%%{init: {'themeVariables': {'fontSize': '17px'}}}%%

flowchart TD
    KMS["KMS wrapping key<br/>KMS/provider<br/>wraps encryptedKey[N] and unwraps it to dataKey[32]"]

    subgraph NodeA["OpenSearch node A"]
        direction TB

        KeyfileA["<strong>encrypted DEK</strong><br/>encryptedKey[N]<br/>persisted on node A disk as index keyfile<br/>nodeA/.../indices/{indexUUID}/keyfile<br/>meaning: encrypted form of dataKey[32]"]

        DataKeyA["<strong>decrypted DEK</strong><br/>dataKey[32]<br/>Lifecycle tied to shard<br/>cached in node A JVM heap<br/>loaded at shard open or cache miss, zeroed on shard close<br/>meaning: decrypted node-local index data key"]

        Metadata["key_metadata JSON<br/>persisted plaintext in each Parquet file footer<br/>written per file, parsed per file read<br/>contains: version=1, data_key_id=default, message_id"]

        MessageId["messageId[16]<br/>generated on each file write,<br/>recovered from metadata on each file read<br/>public per-file derivation input<br/>not secret key material"]

        Context["FOOTER_KEY_CONTEXT<br/>constant<br/>KDF domain-separation input<br/><small>opensearch/parquet-pme/footer-key/v1</small>"]

        Extract["PRK[48]<br/>derived on the fly per file write/read operation<br/><small>HMAC-SHA384(key=dataKey[32], data=messageId[16])</small>"]

        Expand["T1[48]<br/>derived on the fly per file write/read operation<br/>HMAC-SHA384(key=PRK[48], data=context || 0x01)"]

        FooterKey["<strong>Derived Key</strong><br>footerKey[16]<br/>derived on the fly per file write/read operation<br/>passed to parquet-rs for AES-GCM footer/page crypto<br/>meaning: first 16 bytes of T1"]

        AadPrefix["aadPrefix<br/>derived on the fly per file write/read operation<br/>passed to parquet-rs as GCM AAD prefix<br/>domain + version + data_key_id + messageId[16]"]

        ParquetFile["encrypted Parquet ciphertext<br/>persisted on node A disk per Parquet file<br/>encrypted with footerKey[16]<br/>key_metadata: plaintext bootstrap data"]
    end

    subgraph NodeB["OpenSearch node B"]
        direction TB
        KeyfileB["<strong>encrypted DEK</strong><br/>encryptedKey[N]<br/>persisted on node B disk<br/><small>nodeB/.../indices/{indexUUID}/keyfile</small>"]
        DataKeyB["dataKey[32]"]
        FooterKeyB["footerKey[16]"]
    end

    KMS -->|"wraps or unwraps"| KeyfileA
    KeyfileA -->|"decryptKey(encryptedKey)"| DataKeyA
    DataKeyA -->|"dataKey[32]"| Extract

    Metadata -->|"decode"| MessageId
    MessageId --> Extract
    MessageId --> AadPrefix
    Context --> Expand

    Extract -->|"PRK[48]"| Expand
    Expand -->|"T1[0..15]"| FooterKey

    Metadata --> ParquetFile
    FooterKey --> ParquetFile
    AadPrefix --> ParquetFile

    KMS -->|"separate provider calls"| KeyfileB
    KeyfileB --> DataKeyB
    DataKeyB --> FooterKeyB

Loading

Node-local keyfiles

One property should be specially highlighted: There is one persisted DEK for the combination (index,node). That means that on each node, a different DEK is used for a particular index. This follows the design of the OpenSearch Lucene encryption plugin.

Encryption algorithm

The AES-GCM algorithm is used for encryption; parquet-rs only supports AES-GCM for encryption, it does not support AES-GCM-CTR: apache/arrow-rs#7258. The keys shall be 256 bit keys; the prototype currently deviates from this and uses 128 bit keys, as the used version of parquet-rs does not support 256 bit keys.
This might be fixed with parquet-rs version 59: apache/arrow-rs#9203

Parquet key_metadata JSON

OpenSearch stores per-file PME bootstrap metadata as compact UTF-8 JSON in the
Parquet encryption key_metadata field. This data is used to re-hydrate the per-file PME key.

The v1 JSON shape is:

{"version":1,"data_key_id":"default","message_id":"<base64url-message-id>"}

The fields are:

Field	Meaning
version	Metadata schema version. The current Java code accepts only 1.
data_key_id	Logical identifier of the index data key. In v1 this must be default.
message_id	Base64url-without-padding encoding of the 16-byte random per-file messageId.

Properties of the current Java contract:

• key_metadata contains no key material. It stores only public bootstrap data
  needed to re-derive the per-file Parquet key.
• message_id is generated once per Parquet file write using SecureRandom,
  encoded with base64url without padding, and must decode to exactly 16 bytes.
• data_key_id is fixed to default in v1. It resolves to the node-local,
  index-level encrypted DEK stored in .../indices/{indexUUID}/keyfile.
• The JSON writer emits compact JSON with fields in this order:
  version, data_key_id, message_id.
• The Java parser is fail-closed: it rejects empty metadata, malformed JSON,
  unknown fields, missing fields, unsupported versions, any data_key_id other
  than default, and any message_id that is not valid base64url or does not
  decode to 16 bytes.

AAD prefix

PME files allow the inclusion of "additional authenticated data" (AAD) in the
encryption. The AAD is a prefix that is authenticated along with the ciphertext
but not encrypted. The AAD is used to identify the file and the module that
produced it. It enables readers to detect tampering, module swaps, and full-file
replacement.

That AAD prefix must be built from identity known before the Parquet file is opened,
not from data that is only carried inside the file being verified.

For the OpenSearch PME integration, the AAD prefix shall bind the file to its logical shard
file identity. It shall contain the following fields encoded in a binary format:

Field	Source	Purpose
domain	Constant opensearch/parquet-pme/aad/v1	Domain separation and AAD-prefix format versioning.
index_uuid	Shard/index context	Binds the file to the OpenSearch index.
shard_id	Shard context	Prevents replacement with a file from another shard of the same index.
parquet_file_name	Writer/catalog file identity	Binds the ciphertext to the expected Parquet file; current file names include the writer generation.

Note: this is not yet implemented this way in the prototype.