SOLR-16903: Migrate all tests using File to NIO Path #3263

mlbiscoc · 2025-03-13T21:05:15Z

https://issues.apache.org/jira/browse/SOLR-16903

Description

WIP - Leaving in draft state for now as there is still more work from refactor needed but most tests pass so can be semi-reviewed.

Migrate all imports using File to Path. There should be no more imports using File in the project. We are close to making File a forbidden API after this but still more work needed.

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

solr/core/src/java/org/apache/solr/blockcache/BlockDirectory.java

epugh

Wow, so much work! In general I love the removing of the indirection that using Path is showing....

solr/core/src/test/org/apache/solr/SolrInfoBeanTest.java

solr/core/src/test/org/apache/solr/cli/SolrProcessManagerTest.java

solr/core/src/test/org/apache/solr/cli/StreamToolTest.java

solr/core/src/test/org/apache/solr/cli/TestSolrCLIRunExample.java

mlbiscoc · 2025-03-19T14:33:50Z

Tests seem to fail on TestCoreDiscovery.java due to

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")

Seems Files.getPosixFilePermissions isn't allowed on Crave running the tests. I could try to get around it without the getPosixFilePermissions to make files readable or unreadable but I suspect we will still run into the same issue of Files.setAttribute after...

dsmiley · 2025-03-19T20:20:11Z

A quick google suggests we should add this to our tests policy for the security manager:

`permission java.lang.RuntimePermission "accessUserInformation";`

But it wasn't needed before... nonetheless if this fixes the matter then it's fine. Eventually we'll be disabling the security manager and perhaps implementing other measures to stop tests from writing where they shouldn't.

mlbiscoc · 2025-03-19T20:50:41Z

A quick google suggests we should add this to our tests policy for the security manager:
`permission java.lang.RuntimePermission "accessUserInformation";`
But it wasn't needed before... nonetheless if this fixes the matter then it's fine. Eventually we'll be disabling the security manager and perhaps implementing other measures to stop tests from writing where they shouldn't.

I think that worked. I removed permissions from default.policy on my local machine to reproduce and added in the accessUserInformation in 6b37693 which looks to pass now.

dsmiley

Just a few more observations but I won't be looking further than this. I expect a contributor on Windows that I've been chatting with will validate this as well.

solr/core/src/test/org/apache/solr/core/TestFileSystemConfigSetService.java

solr/core/src/test/org/apache/solr/core/TestLazyCores.java

solr/core/src/test/org/apache/solr/handler/component/FacetPivot2CollectionsTest.java

mlbiscoc

Got a VM of windows on my laptop to run the tests. The only test that failed was the file permissions tests in TestCoreDiscovery and fixed that accordingly. Still think would be good if someone else ran the test locally (windows and unix). I think this is in a state I'm happy with.

solr/core/src/test/org/apache/solr/core/TestFileSystemConfigSetService.java

solr/core/src/test/org/apache/solr/handler/component/FacetPivot2CollectionsTest.java

mlbiscoc · 2025-03-21T18:35:24Z

solr/core/src/test/org/apache/solr/core/TestCoreDiscovery.java

+    try {
+      Set<PosixFilePermission> perms = Files.getPosixFilePermissions(homeDir);
+      perms.remove(PosixFilePermission.OWNER_READ);
+      perms.remove(PosixFilePermission.GROUP_READ);
+      perms.remove(PosixFilePermission.OTHERS_READ);
+      Files.setAttribute(homeDir, "posix:permissions", perms);
+    } catch (IOException e) {
+      throw new AssumptionViolatedException(
+          "Cannot make " + homeDir + " non-readable. Test aborted.", e);
+    }


Looks like on windows this wasn't possible which is why there is a assumeTrue call. This will replicate that and just ignore the test on windows.

dsmiley · 2025-03-24T18:20:30Z

I ran the tests last Friday, especially the Windows sensitive one you found and they all passed. I don't recall if there's any outstanding comment of mine that is unfulfilled. I know there's a contributor that may share his results when running Windows.

mlbiscoc · 2025-03-24T20:22:49Z

I ran the tests last Friday, especially the Windows sensitive one you found and they all passed. I don't recall if there's any outstanding comment of mine that is unfulfilled. I know there's a contributor that may share his results when running Windows.

Thanks for helping run the tests. Trying to be careful for something this large of a change. Did a few passthroughs of the PR since opening this. At this point I should have went through all the comments in this PR and resolved them but happy to make more changes if anyone catches something.

dsmiley · 2025-03-28T05:31:59Z

I plan to merge this Friday afternoon. Looks like the tests passed on Windows for another contributor.

dsmiley

Glad to have merged this finally! Thanks again @mlbiscoc !

dsmiley · 2025-03-28T21:33:22Z

In IntelliJ, I did a find-usages of File (methods, fields, etc.) across project files. Excluding gradle (uses File a lot), there are still a number of usages. For example TestRestore. On that spot in particular, wow, the few lines there are so back & forth on Path -> File -> String -> Path ... makes my head spin

HoustonPutman · 2025-03-31T17:59:54Z

@mlbiscoc I think we only have 1 failing test from this, so good job!

CustomTLogDirTest.testExternal is failing often enough on the MacOS and Linux builds. One example:
https://jenkins.thetaphi.de/view/Solr/job/Solr-main-macOS/5363

mlbiscoc · 2025-03-31T18:05:25Z

@mlbiscoc I think we only have 1 failing test from this, so good job!

CustomTLogDirTest.testExternal is failing often enough on the MacOS and Linux builds. One example: https://jenkins.thetaphi.de/view/Solr/job/Solr-main-macOS/5363

Ah so close.... Thanks @HoustonPutman Will try and fix this today.

HoustonPutman · 2025-03-31T19:52:14Z

Oh, well it looks like the nightly tests are less happy... https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main

mlbiscoc · 2025-03-31T20:47:55Z

Thats sad. I ran these tests many times locally. Does nightly not clean up directories? Many of those tests seem to fail where a file/directory already exists. ./gradlew test doesn't seem to run into this issue. Hoping the fix should be small.

CustomTLogDirTest.testExternal is an interesting one. That seed that was run (or I guess depending on the seed it generates), LuceneTestCase.createTempDir() is returning a windowsPath object instead of it's normal filterPath and Path doesn't play well with it. I ran into an issue with LuceneTestCase.createTempDir() earlier in this PR and fixed it with FilterPath.unwrap and looks like unwraping here bandaids the issue.

Will try to fix these errors today.

mlbiscoc · 2025-04-01T01:21:32Z

PR to fix nightly tests and CustomTLogDirTest.testExternal

* Remove unnecessary toAbsolutePath() calls * Remove some callers of FileSystems.getDefault().getSeparator() * Remove Path.toString in where it could easily be avoided. * Use Path.toFile().setReadable in TestCoreDiscovery * Throw AssumptionViolatedException for file permissions on windows Co-authored-by: David Smiley <[email protected]>

mlbiscoc added 7 commits March 6, 2025 09:36

Part 1 of refactor

3860bc3

Fix TODOs so it's easier to find

be9d7a0

Part 2 of refactor

3ef37ee

Part 3 of refactor

36489ea

Build issues from import

9cd71c9

Fix tests to be runnable

37a4911

Fix additional failing tests

2c36bee

epugh reviewed Mar 13, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/blockcache/BlockDirectory.java Show resolved Hide resolved

epugh reviewed Mar 13, 2025

View reviewed changes

Merge branch 'apache:main' into SOLR-16903-move-to-path-tests

542e5cf

mlbiscoc added 2 commits March 19, 2025 16:28

Remove some callers of FileSystems.getDefault().getSeparator()

23d12c1

Add accessUserInformation for making files readable

6b37693

github-actions bot added the tool:build label Mar 19, 2025

Remove toString in ZkConfigFilesTest

fe31ab2

dsmiley reviewed Mar 20, 2025

View reviewed changes

mlbiscoc added 4 commits March 21, 2025 13:57

Use toFile().setReadable in TestCoreDiscovery

2befa83

Changes from review

4b969a5

Throw AssumptionViolatedException for file permissions on windows

63a8bf2

Catch UnsupportedOperationException

e24d0f5

mlbiscoc commented Mar 21, 2025

View reviewed changes

mlbiscoc marked this pull request as ready for review March 21, 2025 19:10

Dupe assert

f36df49

dsmiley mentioned this pull request Mar 27, 2025

Add Paths#get to forbidden APIs #3289

Merged

7 tasks

Merge branch 'main' into SOLR-16903-move-to-path-tests

c47a8b8

dsmiley merged commit 86f2afd into apache:main Mar 28, 2025
1 of 2 checks passed

dsmiley reviewed Mar 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOLR-16903: Migrate all tests using File to NIO Path #3263

SOLR-16903: Migrate all tests using File to NIO Path #3263

mlbiscoc commented Mar 13, 2025

epugh left a comment

mlbiscoc commented Mar 19, 2025

dsmiley commented Mar 19, 2025

mlbiscoc commented Mar 19, 2025

dsmiley left a comment

mlbiscoc left a comment

mlbiscoc Mar 21, 2025

dsmiley commented Mar 24, 2025

mlbiscoc commented Mar 24, 2025

dsmiley commented Mar 28, 2025

dsmiley left a comment

dsmiley commented Mar 28, 2025

HoustonPutman commented Mar 31, 2025

mlbiscoc commented Mar 31, 2025

HoustonPutman commented Mar 31, 2025

mlbiscoc commented Mar 31, 2025

mlbiscoc commented Apr 1, 2025

SOLR-16903: Migrate all tests using File to NIO Path #3263

SOLR-16903: Migrate all tests using File to NIO Path #3263

Conversation

mlbiscoc commented Mar 13, 2025

Description

Checklist

epugh left a comment

Choose a reason for hiding this comment

mlbiscoc commented Mar 19, 2025

dsmiley commented Mar 19, 2025

mlbiscoc commented Mar 19, 2025

dsmiley left a comment

Choose a reason for hiding this comment

mlbiscoc left a comment

Choose a reason for hiding this comment

mlbiscoc Mar 21, 2025

Choose a reason for hiding this comment

dsmiley commented Mar 24, 2025

mlbiscoc commented Mar 24, 2025

dsmiley commented Mar 28, 2025

dsmiley left a comment

Choose a reason for hiding this comment

dsmiley commented Mar 28, 2025

HoustonPutman commented Mar 31, 2025

mlbiscoc commented Mar 31, 2025

HoustonPutman commented Mar 31, 2025

mlbiscoc commented Mar 31, 2025

mlbiscoc commented Apr 1, 2025