Implement OpaPolarisAuthorizer #2680

sungwy · 2025-09-25T03:03:16Z

Resolves #138

RFC: https://docs.google.com/document/d/1HadMFygjbuZathZZPanO6cFVorx0Ju0FopkICxX1tCE/edit?tab=t.0
Mailing list discussion: https://lists.apache.org/thread/54qdbsxs3j7wwhv3tsccqj6qng5lqgmz

Some followup items highlighted as a part of this PR review:

introduce OpaHttpClientFactory that utilizes PoolingHttpClientConnectionManager to get opa-http-client to be more production ready
publish user-facing docs on OPA authorization
publish json schema for docs and opa server checks
introduce per-realm configuration support

adutra

Thanks @sungwy for this draft PR, I couldn't resist and took a look 😄

This is really interesting imho and a very nice addition to Polaris. We need to start thinking about ways to make Polaris RBAC fully pluggable.

...sions/auth/opa/src/main/java/org/apache/polaris/extension/auth/opa/OpaPolarisAuthorizer.java

runtime/defaults/src/main/resources/application.properties

runtime/service/src/main/java/org/apache/polaris/service/config/AuthorizationConfiguration.java

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

runtime/service/build.gradle.kts

dimas-b

Very interesting idea! Just some preliminary comments below :)

polaris-core/build.gradle.kts

runtime/defaults/src/main/resources/application.properties

flyrain · 2025-09-25T22:32:34Z

Thanks for working on it @sungwy ! Looking forward to the RFC/design doc!

flyrain · 2025-10-08T18:05:30Z

Will post my complete review soon. Adding one thing first, currently all grant and revoke privilege operation (e.g. grantPrivilegeOnCatalogToRole) won’t work as expected, since these mapping has to happen in the OPA side. How do we position these APIs? If we still want to support them within Polaris, we need new interfaces between the OPA service and Polaris. I think the least thing we could do now is blocking these operation for OPA authorizer. Given the current PR is large, I’m OK to leave it in another PR. Thanks @singhpk234 for bring it up for discussion.

flyrain

Hi @sungwy, thanks a lot for working on it. I believe this is the right direction. It opens a door for a lot of possibilities. Left some comments. Feel free to reach out for questions.

polaris-core/src/main/java/org/apache/polaris/core/auth/TokenProvider.java

polaris-core/src/main/java/org/apache/polaris/core/auth/JwtDecoder.java

runtime/service/src/main/java/org/apache/polaris/service/auth/OpaPolarisAuthorizerFactory.java

...e/service/src/main/java/org/apache/polaris/service/auth/DefaultPolarisAuthorizerFactory.java

...sions/auth/opa/src/main/java/org/apache/polaris/extension/auth/opa/OpaPolarisAuthorizer.java

polaris-core/src/main/java/org/apache/polaris/core/auth/OpaPolarisAuthorizer.java

polaris-core/src/main/java/org/apache/polaris/core/auth/FileTokenProvider.java

runtime/service/src/main/java/org/apache/polaris/service/auth/OpaPolarisAuthorizerFactory.java

runtime/service/src/main/java/org/apache/polaris/service/config/AuthorizationConfiguration.java

sungwy · 2025-10-09T14:54:44Z

Thank you all for the reviews. I think this PR is ready for another round of feedback.

I've made note of the next TODO items we could start working on soon after this initial PR is merged, and listed them in the PR description

flyrain

@sungwy , thanks for the quick response. The changes look great overall. I think we are pretty close. Could you also share some thoughts on this comment, #2680 (comment)?

flyrain · 2025-10-09T16:24:42Z

polaris-core/src/main/java/org/apache/polaris/core/auth/OpaPolarisAuthorizer.java

+      String trustStorePassword,
+      Object client) {
+
+    if (Strings.isNullOrEmpty(opaServerUrl)) {


Nit: -> Preconditions.checkArgument(...)

flyrain · 2025-10-09T18:18:37Z

...sions/auth/opa/src/main/java/org/apache/polaris/extension/auth/opa/OpaPolarisAuthorizer.java

+    input.set("actor", buildActorNode(principal));
+    input.put("action", op.name());
+    input.set("resource", buildResourceNode(targets, secondaries));
+    input.set("context", buildContextNode());
+    ObjectNode root = objectMapper.createObjectNode();
+    root.set("input", input);


A followup PR sounds good to me.

runtime/service/src/main/java/org/apache/polaris/service/auth/OpaPolarisAuthorizerFactory.java

flyrain · 2025-10-09T18:50:23Z

polaris-core/src/main/java/org/apache/polaris/core/auth/OpaPolarisAuthorizer.java

+      } else {
+        if (sslSocketFactory != null) {
+          httpClient =
+              HttpClients.custom()
+                  .setDefaultRequestConfig(requestConfig)
+                  .setConnectionManager(
+                      PoolingHttpClientConnectionManagerBuilder.create()
+                          .setSSLSocketFactory(sslSocketFactory)
+                          .build())
+                  .build();
+        } else {
+          httpClient = HttpClients.custom().setDefaultRequestConfig(requestConfig).build();
+        }


Given the http client is passed from the parameter, do we still need line 129 to line 145? Should createSslSocketFactory also be part of ServiceProducer::opaHttpClient()? Otherwise, the ssl config may not take any effect.

dimas-b

Hi @sungwy , sorry for challenging the whole source layout, but it's the first time I looked at your PR for real 😅

I believe it would be valuable to isolate OPA from core and from runtime/service and only include it into runtime/server. So, I stopped commenting when I realized I was getting into the dependencies too much.

Here's a sketch of a possible source layout:
extensions/opa

authorizer - OpaPolarisAuthorizer and related stuff
runtime - CDI stuff (if necessary... it may be ok to keep producers inside the authorizer module)
test - OpaTestResource and stuff - only for tests that require a full server

test should be able to include :polaris-runtime-service as a test dep. to get a running Quarkus server

WDYT?

dimas-b · 2025-10-09T20:28:44Z

polaris-core/build.gradle.kts


 dependencies {
  implementation(project(":polaris-api-management-model"))
+  implementation(libs.apache.httpclient5)


I think it might be preferable to move OPA stuff out of "core". I propose extensions/opa (include into runtime/server).

Rationale:

the new authorizer is pluggable, but not all downstream projects may want to have it by default.

reducing core dependencies (IIRC, Quarkus runtime already has httpclient5, but core does not have to depend on it).

I think that makes a lot of sense as well. I was curious to hear opinions on the new dependencies and OPA plugin being introduced as a part of polaris-core.

Let me take a stab at separating out these changes as a separate build in extensions/auth/opa

I believe it would be valuable to isolate OPA from core and from runtime/service and only include it into runtime/server.
the new authorizer is pluggable, but not all downstream projects may want to have it by default.

I'm OK to move OPA related classes from core to runtime/service, but I don't think we should put it in runtime/server. Here are the reason:

I think it's perfect fine to have another implementation of authorizer, we did that in multiple places.

The native RBAC is still the default.

There is no single extra lib dependencies introduced when it landed in the service module. http and jwt libs are there already.

Do we still need this dep. in core?

@flyrain : if OPA is not added to runtime/server users of the OSS distributions will not be able to use it. I personally think it's a valuable feature for OSS users... Let's sync up on this aspect on the dev ML thread for visibility.

Let me review this dependency again.

I've moved most of the OPA factory and configuration classes into extensions/auth/opa as suggested, but now I'm seeing a test failure at PolarisEventListenerTest that's related to the Jandex index. I'll look into this a bit more and get this resolved.

PolarisEventListenerTest > testAllEventTypesHandled() FAILED

Do we still need this dep. in core?

No. I think it’s perfectly fine to keep the OPA authorizer within the polaris-runtime-service module rather than moving it to a separate extension module.
The rationale for separating components like hive or hadoop federation is that they introduce heavy and intrusive Hadoop dependencies that we don’t want to pull into the service module.
The OPA authorizer, on the other hand, adds no additional dependencies, it integrates cleanly within the service module, so there’s no downside to keeping it there.

Moreover, OpaFileTokenIntegrationTest and OpaIntegrationTest are still be part of the polaris-runtime-service module. What's the point of having the tested classes(like OpaPolarisAuthorizer) in an extension module?

The pluggability aspect is not about "heavy" or "light" depenpencies per se, but about giving downstream projects control over what to enable/bundle or not. As this appears to be a point where a bit of misunderstanding developed, let's move the discussion about this particular aspect to the dev ML for visibility and reaching consensus within the broader community (not everyone watches PRs).

dimas-b · 2025-10-09T20:36:44Z

polaris-core/src/main/java/org/apache/polaris/core/auth/OpaPolarisAuthorizer.java

+   * @return SSLConnectionSocketFactory for HTTPS connections, or null for HTTP
+   * @throws Exception if SSL configuration fails
+   */
+  private static SSLConnectionSocketFactory createSslSocketFactory(


SSLConnectionSocketFactory appears to be deprecated 🤔 Should we use DefaultClientTlsStrategy as suggested by its javadoc?

dimas-b · 2025-10-09T20:39:06Z

polaris-core/src/main/java/org/apache/polaris/core/auth/OpaPolarisAuthorizer.java

+
+      // Configure SSL for HTTPS connections
+      SSLConnectionSocketFactory sslSocketFactory =
+          createSslSocketFactory(opaServerUrl, verifySsl, trustStorePath, trustStorePassword);


This will be called for every request (REST API). I'm not sure how heavy is the socket factory, but would it make sense to keep it in OpaPolarisAuthorizerFactory (to be shared across OpaPolarisAuthorizer instances)?

Yes, I'm already refactoring this as per @flyrain 's comment. It looks like I'll need to introduce an OpaHttpClientFactory anyways within this PR to clean this up.

dimas-b · 2025-10-09T20:42:08Z

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

+  @Produces
+  @Singleton
+  @Identifier("opa-bearer-token-provider")
+  public BearerTokenProvider opaBearerTokenProvider(


I wonder it this producer could be moved into OpaPolarisAuthorizerFactory. It should only be required when that factory is active.

Side note: if we take OPA out or core, it would be nice to avoid direct references to it here as well and rely only on CDI to find the right classes via annotations.

I took a stab at this. Let me know if this aligns with what you were suggesting

dimas-b · 2025-10-09T20:46:36Z

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

+  @Produces
+  @Singleton
+  @Identifier("opa-http-client")
+  public CloseableHttpClient opaHttpClient() {


@Singleton might be an overkill. It is only needed when OPA is active. It should be a dependent bean, I think.

Also, OpaPolarisAuthorizerFactory might be able to produce / dispose of this bean. Beans can produce other beans, AFAIK.

If dependent beans prove to be too hard to manage, perhaps we can keep the client as a field in the factory and dispose of it together with the factory.

dimas-b · 2025-10-09T20:49:10Z

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

+      java.time.Duration refreshInterval =
+          java.time.Duration.ofSeconds(bearerToken.refreshInterval());
+      boolean jwtExpirationRefresh = bearerToken.jwtExpirationRefresh();
+      java.time.Duration jwtExpirationBuffer =


nit: import?

dimas-b · 2025-10-09T20:54:32Z

runtime/service/src/test/java/org/apache/polaris/service/auth/OpaFileTokenIntegrationTest.java

+      try {
+        tokenFile = Files.createTempFile("opa-test-token", ".txt");
+        Files.writeString(tokenFile, "test-opa-bearer-token-from-file-67890");
+        tokenFile.toFile().deleteOnExit();


nit: this is not guaranteed to delete the file, AFAIK. Plus the "JVM" may be long running, e.g. a Gradle process.

It might be best to manage this as a QuarkusTestResourceLifecycleManager, which can produce extra server config entries.

dimas-b · 2025-10-09T21:02:51Z

runtime/service/src/main/java/org/apache/polaris/service/config/AuthorizationConfiguration.java

+  @WithDefault("default")
+  String type();
+
+  OpaConfig opa();


If we move OPA out of core, let's create a separate config interface for it (with prefix polaris.authorization.opa). This class will only declare type() (which is common for all authorizers).

sungwy · 2025-10-10T20:58:18Z

Will post my complete review soon. Adding one thing first, currently all grant and revoke privilege operation (e.g. grantPrivilegeOnCatalogToRole) won’t work as expected, since these mapping has to happen in the OPA side. How do we position these APIs? If we still want to support them within Polaris, we need new interfaces between the OPA service and Polaris. I think the least thing we could do now is blocking these operation for OPA authorizer. Given the current PR is large, I’m OK to leave it in another PR. Thanks @singhpk234 for bring it up for discussion.

Hi @flyrain - I agree! In our planned architecture, we are thinking of returning false on the Management APIs through the OPA Server rego. But I agree that we should review whether it would make sense to just systematically fail on these actions within the OpaPolarisAuthorizer itself before sending the request to OPA .

dimas-b · 2025-10-10T22:41:37Z

gradle/projects.main.properties

 polaris-misc-types=tools/misc-types
 polaris-extensions-federation-hadoop=extensions/federation/hadoop
 polaris-extensions-federation-hive=extensions/federation/hive
+polaris-extensions-auth-opa=extensions/auth/opa


nit: I'd personally be fine with polaris-auth-opa (for brevity). The module name does not have to repeat all sub-dir names of its location. I think the polaris prefix helps in case modules are imported into another project, but otherwise shorter name are easier to use, IMHO.

I do like the consistency across the extension modules. We already have polaris-extensions-federation-hadoop and polaris-extensions-federation-hive above

dimas-b · 2025-10-10T22:47:07Z

polaris-core/build.gradle.kts


 dependencies {
  implementation(project(":polaris-api-management-model"))
+  implementation(libs.apache.httpclient5)


Do we still need this dep. in core?

dimas-b · 2025-10-10T22:49:34Z

extensions/auth/opa/build.gradle.kts

+
+dependencies {
+  implementation(project(":polaris-core"))
+  implementation(project(":polaris-runtime-service"))


why does OPA need polaris-runtime-service?

Updated the dependencies 👍

dimas-b · 2025-10-10T22:52:07Z

...ervice/src/intTest/java/org/apache/polaris/service/auth/opa/OpaFileTokenIntegrationTest.java

+
+@QuarkusTest
+@TestProfile(OpaFileTokenIntegrationTest.FileTokenOpaProfile.class)
+public class OpaFileTokenIntegrationTest {


Would it be possible to keep this test under extensions/auth/opa/... somewhere? In light of making OPA a proper plugin, it would be nice to avoid hard dependencies from the main service module.

Yeah I did try this out earlier and I found it quite challenging.

These are @QuarkusTest integration tests that need to start the full Polaris runtime service application and configure the complete CDI container with all service dependencies to run an end to end test of the authorization workflow. The OPA extension module is a pure library without the Quarkus application context needed for @QuarkusTest, and I think there's value in leaving it at just that.

IMHO I don't see too much of a problem in adding the extensions into runtime/service as a test dependency.

dimas-b · 2025-10-10T22:53:56Z

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

+      RealmConfig realmConfig,
+      @Any Instance<PolarisAuthorizerFactory> authorizerFactories) {
+    PolarisAuthorizerFactory factory =
+        authorizerFactories.select(Identifier.Literal.of(authorizationConfig.type())).get();


Could you move the factory into a separate @ApplicationScoped producer to avoid .select() calls on every request?

Good point @dimas-b - I took care of this, and also made some adjustments into OpaPolarisAuthorizorFactory to validate the configurations and initialize the HttpClient at the application scope.

...sions/auth/opa/src/main/java/org/apache/polaris/extension/auth/opa/OpaHttpClientFactory.java

jbonofre · 2025-10-11T15:11:18Z

...sions/auth/opa/src/main/java/org/apache/polaris/extension/auth/opa/OpaPolarisAuthorizer.java

+    input.set("actor", buildActorNode(principal));
+    input.put("action", op.name());
+    input.set("resource", buildResourceNode(targets, secondaries));
+    input.set("context", buildContextNode());
+    ObjectNode root = objectMapper.createObjectNode();
+    root.set("input", input);


Yes, I agree that it would be "simpler/cleaner" to create beans (eventually usign Jackson annotations) to generate the JSON input, generating the associate schema and validating it. It's pretty straight forward as we already "bundle" Jackson.

sungwy added 9 commits September 20, 2025 17:53

OpaPolarisAuthorizer

be48131

add CDI AuthorizerProducer

317cdc4

inject polarisAuthorizer in ServiceProducers CDI

0739855

add integration test

7be0482

Merge branch 'main' into opa-authorizer

c18d4d2

license

c7701cb

add integration tests

ec3c142

Merge branch 'main' into opa-authorizer

eec60c2

minor fixes

ed6f265

github-project-automation bot added this to Basic Kanban Board Sep 25, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Sep 25, 2025

adutra reviewed Sep 25, 2025

View reviewed changes

dimas-b reviewed Sep 25, 2025

View reviewed changes

polaris-core/build.gradle.kts Outdated Show resolved Hide resolved

runtime/defaults/src/main/resources/application.properties Show resolved Hide resolved

sungwy added 2 commits September 26, 2025 18:01

adopt review feedback

5caf1f4

remove comment

3935c6a

HonahX mentioned this pull request Sep 29, 2025

Implement Finer Grained Operations and Privileges For Update Table #2697

Merged

sungwy added 7 commits September 30, 2025 02:34

support https and bearer token authz

5ad1030

file token provider and token refresh

0785bdb

Merge branch 'main' into opa-authorizer

4b950e3

fix

85baedc

refactoring

6421275

refactor tests, disable ssl verification in integration tests

36f687c

use http in integration tests

c1ae608

HonahX mentioned this pull request Oct 7, 2025

Add new PolaisAuthorizer.authorizeOrThrow for Fine Grained AuthorizableOperations #2767

Open

sungwy added 2 commits October 8, 2025 01:55

remove properties from initial implementation

6516726

remove unused ssl dependencies

edfe61a

sungwy marked this pull request as ready for review October 8, 2025 02:05

sungwy requested review from adutra and dimas-b October 8, 2025 02:14

sungwy requested a review from collado-mike October 8, 2025 12:34

flyrain reviewed Oct 8, 2025

View reviewed changes

sungwy added 6 commits October 9, 2025 02:57

adopt review feedback

723dec1

Notes about Beta

479ac60

Merge branch 'main' into opa-authorizer

c946d0d

adopt more feedback

f46f97b

remove JwtDecoder in favor of auth0 java-jwt

81de61e

use httpclient 5

d014a97

sungwy requested a review from flyrain October 9, 2025 14:43

flyrain reviewed Oct 9, 2025

View reviewed changes

dimas-b reviewed Oct 9, 2025

View reviewed changes

sungwy added 2 commits October 9, 2025 22:09

opa http client factory refactoring

944d005

extensions/auth/opa refactoring

477839a

sungwy added 2 commits October 10, 2025 21:17

fix opa tests

4252a44

lint

c0053f9

dimas-b reviewed Oct 10, 2025

View reviewed changes

refactoring and cleaning up dependencies

0eb0a97

jbonofre reviewed Oct 11, 2025

View reviewed changes

remove old integration test files

7b61eee

Implement OpaPolarisAuthorizer #2680

Are you sure you want to change the base?

Implement OpaPolarisAuthorizer #2680

Uh oh!

Conversation

sungwy commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adutra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

flyrain commented Sep 25, 2025

Uh oh!

flyrain commented Oct 8, 2025

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sungwy commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sungwy commented Sep 25, 2025 •

edited

Loading

sungwy commented Oct 9, 2025 •

edited

Loading