Refactor date parsing #1372

Isira-Seneviratne · 2025-09-12T02:54:27Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

Update the DateWrapper class to store an Instant, and remove the deprecated methods.
[YouTube] Improve the upload date parsing so that non-approximate dates can be returned.
[SoundCloud] Return the original instant string in the stream extractor.

…actor

TobiGr

thanks

TobiGr · 2025-09-12T17:04:09Z

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

+        final String dateStr = playerMicroFormatRenderer.getString("uploadDate",
+                playerMicroFormatRenderer.getString("publishDate", ""));
+        if (!dateStr.isEmpty()) {
+            return dateStr;
        }

-        final JsonObject liveDetails = playerMicroFormatRenderer.getObject(
-                "liveBroadcastDetails");
-        if (!liveDetails.getString("endTimestamp", "").isEmpty()) {
-            // an ended live stream
-            return liveDetails.getString("endTimestamp");
-        } else if (!liveDetails.getString("startTimestamp", "").isEmpty()) {
-            // a running live stream
-            return liveDetails.getString("startTimestamp");
+        final var liveDetails = playerMicroFormatRenderer.getObject("liveBroadcastDetails");
+        final String timestamp = liveDetails.getString("endTimestamp", // an ended live stream
+                liveDetails.getString("startTimestamp", "")); // a running live stream
+
+        if (!timestamp.isEmpty()) {
+            return timestamp;


this makes it harder to understand what's going on. I'd prefer to keep it readable and revert this change.

I feel like the code below line 180 as a whole could be removed. I checked how the date is being extracted, and it's always taken from uploadDate or publishDate.

YouTube might return different layouts in different cases, so it's better to keep all cases

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

TobiGr · 2025-09-17T06:30:37Z

Please list all the breaking changes in the PR description so they can be copied to the release notes.

TobiGr · 2025-09-17T06:42:41Z

The tests are now dependent on the runner's time zone, which is not good. While the automated tests pass (server is set to UTC), tests on my computer fail:

results


YoutubeStreamExtractorControversialTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2010-09-09T15:40:44> but was: <2010-09-09T17:40:44>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$DescriptionTestPewdiepie > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2019-08-24T15:39:57> but was: <2019-08-24T17:39:57>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$DescriptionTestUnboxing > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2018-06-19T19:41:34> but was: <2018-06-19T21:41:34>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$NoVisualMetadataVideoTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2017-05-16T14:50:53> but was: <2017-05-16T16:50:53>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$PublicBroadcasterTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2023-07-07T15:30:08> but was: <2023-07-07T17:30:08>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$RatingsDisabledTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2023-01-13T21:53:57> but was: <2023-01-13T22:53:57>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestMaiLab > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2020-11-19T05:30:01> but was: <2020-11-19T06:30:01>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestMaiLab > testName() SKIPPED

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestMaiLab > testTags() SKIPPED

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestTagesschau > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2021-03-17T19:56:59> but was: <2021-03-17T20:56:59>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorLivestreamTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2022-07-12T12:12:29> but was: <2022-07-12T14:12:29>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorRelatedMixTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2015-07-09T16:34:35> but was: <2015-07-09T18:34:35>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorUnlistedTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2017-09-22T12:15:21> but was: <2017-09-22T14:15:21>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
        at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

Stypox · 2025-09-17T08:38:52Z

Mmmh, what's the advantage of using Instant instead of OffsetDateTime? I see this PR does many changes but I don't see clear advantages. I mean, sure, some code is shorter, but the longer code we had before worked just fine, or was there some limitation I am not getting?

Isira-Seneviratne · 2025-09-18T00:27:22Z

Mmmh, what's the advantage of using Instant instead of OffsetDateTime? I see this PR does many changes but I don't see clear advantages. I mean, sure, some code is shorter, but the longer code we had before worked just fine, or was there some limitation I am not getting?

Instant represents a single point of time in the UTC time zone, while OffsetDateTime stores a zone offset. As such, I felt that storing an Instant would be better as it avoids having to convert between different time zones in order to obtain it, and a local date/time value can be obtained via their factory methods.

Stypox

Thank you! Generally this looks good to me, except for a few nitpicks

extractor/src/main/java/org/schabi/newpipe/extractor/localization/TimeAgoParser.java

extractor/src/main/java/org/schabi/newpipe/extractor/utils/ExtractorHelper.java

...ctor/src/main/java/org/schabi/newpipe/extractor/services/peertube/PeertubeParsingHelper.java

...a/org/schabi/newpipe/extractor/services/soundcloud/extractors/SoundcloudStreamExtractor.java

.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java

...abi/newpipe/extractor/services/youtube/extractors/kiosk/YoutubeChartsBaseKioskExtractor.java

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

...org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamInfoItemExtractor.java

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

# Conflicts: # extractor/src/main/java/org/schabi/newpipe/extractor/localization/DateWrapper.java

Co-authored-by: Stypox <[email protected]>

TobiGr · 2025-10-10T07:33:22Z

Please add this PR to the release notes and list the breaking changes in the "Breaking" section.

Isira-Seneviratne added 5 commits September 12, 2025 08:16

Update DateWrapper to store an Instant

953189f

Allow retrieval of a non-approximate DateWrapper in YoutubeStreamExtr…

758c31b

…actor

Improve tests

2dc17d4

Fix checkstyle issues, added more refactorings

6bc9d32

Refactor PeerTube date parsing method

380c303

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from cadda5e to 380c303 Compare September 12, 2025 16:46

Rm substring call

e9dd84f

TobiGr reviewed Sep 13, 2025

View reviewed changes

Apply code review suggestions

c79afc0

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from ddaef9a to c79afc0 Compare September 13, 2025 01:19

Isira-Seneviratne mentioned this pull request Sep 13, 2025

Refactor codebase to use Instant instead of OffsetDateTime TeamNewPipe/NewPipe#12633

Open

5 tasks

Isira-Seneviratne added 3 commits September 16, 2025 06:40

Do additional refactoring

f8d6f94

Use LocalDateTime instead of LocalDate

8727659

Fix some issues

f9f71cb

Fix tests

59e78c2

Avoid potential NPEs in Soundcloud extractors

f38b72c

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from 0411b8a to f38b72c Compare September 18, 2025 00:54

Fix nested tests

1a5ac2f

Stypox approved these changes Oct 5, 2025

View reviewed changes

Address code review comments

8997187

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from 1ff529a to afe9454 Compare October 8, 2025 01:38

Merge branch 'dev' into Refactor-date-parsing

f4084ed

# Conflicts: # extractor/src/main/java/org/schabi/newpipe/extractor/localization/DateWrapper.java

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from afe9454 to f4084ed Compare October 8, 2025 01:56

Isira-Seneviratne and others added 2 commits October 8, 2025 07:27

Apply suggestion from @Stypox

a84d831

Co-authored-by: Stypox <[email protected]>

Reduce some lines of code

9355798

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from 2b784dd to 9355798 Compare October 8, 2025 02:53

Isira-Seneviratne added 3 commits October 8, 2025 08:24

Fix nullable annotations

2d794e7

Update YoutubeStreamInfoItemLockupExtractor

fa4b717

Added documentation for parsing methods

e26cc84

Isira-Seneviratne force-pushed the Refactor-date-parsing branch 2 times, most recently from 34c495a to b3fd6b3 Compare October 10, 2025 03:05

Remove unthrown exception

2b306a3

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from b3fd6b3 to 2b306a3 Compare October 10, 2025 03:12

Isira-Seneviratne merged commit 39b9482 into TeamNewPipe:dev Oct 10, 2025
4 checks passed

Uh oh!

Refactor date parsing #1372

Refactor date parsing #1372

Uh oh!

Conversation

Isira-Seneviratne commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiGr left a comment

Choose a reason for hiding this comment

Uh oh!

TobiGr Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

Isira-Seneviratne Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

Stypox Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TobiGr commented Sep 17, 2025

Uh oh!

TobiGr commented Sep 17, 2025

Uh oh!

Stypox commented Sep 17, 2025

Uh oh!

Isira-Seneviratne commented Sep 18, 2025

Uh oh!

Stypox left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TobiGr commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Isira-Seneviratne commented Sep 12, 2025 •

edited

Loading