Merge pull request #5 from NonoElRobot/dev

[YT] Fix 403 again merge latest extractor fixes to zen

Author: whistlingwoods

.github/CONTRIBUTING.md

@@ -1,3 +1,5 @@
+### Please do **not** open pull requests for *new features* now, as we are planning to rewrite large chunks of the code. Only bugfix PRs will be accepted. More details will be announced soon!
+
 NewPipe contribution guidelines
 ===============================
 
@@ -40,10 +42,6 @@ You'll see *exactly* what is sent, be able to add **your comments**, and then se
 * Create PRs that cover only **one specific issue/solution/bug**. Do not create PRs that are huge monoliths and could have been split into multiple independent contributions.
 * NewPipe uses [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor) to fetch data from services. If you need to change something there, you must test your changes in NewPipe. Telling NewPipe to use your extractor version can be accomplished by editing the `app/build.gradle` file: the comments under the "NewPipe libraries" section of `dependencies` will help you out.
 
-### Kotlin in NewPipe
-* NewPipe will remain mostly Java for time being
-* Contributions containing a simple conversion from Java to Kotlin should be avoided. Conversions to Kotlin should only be done if Kotlin actually brings improvements like bug fixes or better performance which are not, or only with much more effort, implementable in Java. The core team sees Java as an easier to learn and generally well adopted programming language.
-
 ### Creating a Pull Request (PR)
 
 * Make changes on a **separate branch** with a meaningful name, not on the _master_ branch or the _dev_ branch. This is commonly known as *feature branch workflow*. You may then send your changes as a pull request (PR) on GitHub.

.github/ISSUE_TEMPLATE/question.yml .github/DISCUSSION_TEMPLATE/questions.yml

@@ -1,11 +1,8 @@
-name: Question
-description: Ask about anything NewPipe-related
-labels: [question, needs triage]
 body:
   - type: markdown
     attributes:
       value: |
-        Thanks for taking the time to fill out this issue! :hugs:
+        Thanks for taking the time to fill out this form! :hugs:
 
         Note that you can also ask questions on our [IRC channel](https://web.libera.chat/#newpipe).
 
@@ -14,7 +11,7 @@ body:
     attributes:
       label: "Checklist"
       options:
-        - label: "I made sure that there are *no existing issues* - [open](https://github.com/TeamNewPipe/NewPipe/issues) or [closed](https://github.com/TeamNewPipe/NewPipe/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
+        - label: "I made sure that there are *no existing issues or discussions* - [open](https://github.com/TeamNewPipe/NewPipe/issues) or [closed](https://github.com/TeamNewPipe/NewPipe/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
           required: true
         - label: "I have read the [FAQ](https://newpipe.net/FAQ/) and my question isn't listed."
           required: true

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -14,7 +14,7 @@ body:
     attributes:
       label: "Checklist"
       options:
-        - label: "I am able to reproduce the bug with the [latest version](https://github.com/TeamNewPipe/NewPipe/releases/latest)."
+        - label: "I am able to reproduce the bug with the latest version given here: [CLICK THIS LINK](https://github.com/TeamNewPipe/NewPipe/releases/latest)."
           required: true
         - label: "I made sure that there are *no existing issues* - [open](https://github.com/TeamNewPipe/NewPipe/issues) or [closed](https://github.com/TeamNewPipe/NewPipe/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
           required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +1,8 @@
 blank_issues_enabled: false
 contact_links:
+  - name: ❓ Question
+    url: https://github.com/TeamNewPipe/NewPipe/discussions/new?category=questions
+    about: Ask about anything NewPipe-related
   - name: 💬 IRC
     url: https://web.libera.chat/#newpipe
     about: Chat with us via IRC for quick Q/A

.github/PULL_REQUEST_TEMPLATE.md

@@ -28,7 +28,7 @@
 #### APK testing
 <!-- Use a new, meaningfully named branch. The name is used as a suffix for the app ID to allow installing and testing multiple versions of NewPipe, e.g. "commentfix", if your PR implements a bugfix for comments. (No names like "patch-0" and "feature-1".)  -->
 <!-- Remove the following line if you directly link the APK created by the CI pipeline. Directly linking is preferred if you need to let users test.-->
-The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR.
+The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR. You can find more info and a video demonstration [on this wiki page](https://github.com/TeamNewPipe/NewPipe/wiki/Download-APK-for-PR).
 
 #### Due diligence
 - [ ] I read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md).

.github/changed-lines-count-labeler.yml

@@ -0,0 +1,17 @@
+# Add 'size/small' label to any changes with less than 50 lines
+size/small:
+  max: 49
+
+# Add 'size/medium' label to any changes between 50 and 249 lines
+size/medium:
+  min: 50
+  max: 249
+
+# Add 'size/large' label to any changes between 250 and 749 lines
+size/large:
+  min: 250
+  max: 749
+
+# Add 'size/giant' label to any changes for more than 749 lines
+size/giant:
+  min: 750

.github/workflows/ci.yml

@@ -6,6 +6,7 @@ on:
     branches:
       - dev
       - master
+      - refactor
       - release**
       - zen
     paths-ignore:
@@ -39,62 +40,74 @@ jobs:
       contents: read
 
     steps:
-      - uses: actions/checkout@v3
-      - uses: gradle/wrapper-validation-action@v1
+      - uses: actions/checkout@v4
+      - uses: gradle/wrapper-validation-action@v2
 
       - name: create and checkout branch
         # push events already checked out the branch
         if: github.event_name == 'pull_request'
-        run: git checkout -B ${{ github.head_ref }}
+        env:
+          BRANCH: ${{ github.head_ref }}
+        run: git checkout -B "$BRANCH"
 
-      - name: set up JDK 11
-        uses: actions/setup-java@v3
+      - name: set up JDK 17
+        uses: actions/setup-java@v4
         with:
-          java-version: 11
+          java-version: 17
           distribution: "temurin"
           cache: 'gradle'
 
       - name: Build debug APK and run jvm tests
         run: ./gradlew assembleDebug lintDebug testDebugUnitTest --stacktrace -DskipFormatKtlint
 
       - name: Upload APK
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: app
           path: app/build/outputs/apk/debug/*.apk
 
   test-android:
-    # macos has hardware acceleration. See android-emulator-runner action
-    runs-on: macos-latest
+    runs-on: ubuntu-latest
     timeout-minutes: 20
     strategy:
       matrix:
-        # api-level 19 is min sdk, but throws errors related to desugaring
-        api-level: [ 21, 29 ]
+        include:
+          - api-level: 21
+            target: default
+            arch: x86
+          - api-level: 33
+            target: google_apis # emulator API 33 only exists with Google APIs
+            arch: x86_64
 
     permissions:
       contents: read
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+
+      - name: Enable KVM
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
 
-      - name: set up JDK 11
-        uses: actions/setup-java@v3
+      - name: set up JDK 17
+        uses: actions/setup-java@v4
         with:
-          java-version: 11
+          java-version: 17
           distribution: "temurin"
           cache: 'gradle'
 
       - name: Run android tests
         uses: reactivecircus/android-emulator-runner@v2
         with:
           api-level: ${{ matrix.api-level }}
-          # workaround to emulator bug: https://github.com/ReactiveCircus/android-emulator-runner/issues/160
-          emulator-build: 7425822
+          target: ${{ matrix.target }}
+          arch: ${{ matrix.arch }}
           script: ./gradlew connectedCheck --stacktrace
 
       - name: Upload test report when tests fail # because the printed out stacktrace (console) is too short, see also #7553
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         if: failure()
         with:
           name: android-test-report-api${{ matrix.api-level }}
@@ -107,19 +120,19 @@ jobs:
       contents: read
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
 
-      - name: Set up JDK 11
-        uses: actions/setup-java@v3
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
         with:
-          java-version: 11 # Sonar requires JDK 11
+          java-version: 17
           distribution: "temurin"
           cache: 'gradle'
 
       - name: Cache SonarCloud packages
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ~/.sonar/cache
           key: ${{ runner.os }}-sonar

.github/workflows/image-minimizer.js

@@ -17,6 +17,8 @@ module.exports = async ({github, context}) => {
         initialBody = context.payload.comment.body;
     } else if (context.eventName == 'issues') {
         initialBody = context.payload.issue.body;
+    } else if (context.eventName == 'pull_request') {
+        initialBody = context.payload.pull_request.body;
     } else {
         console.log('Aborting: No body found');
         return;
@@ -30,10 +32,12 @@ module.exports = async ({github, context}) => {
     }
 
     // Regex for finding images (simple variant) ![ALT_TEXT](https://*.githubusercontent.com/<number>/<variousHexStringsAnd->.<fileExtension>)
-    const REGEX_IMAGE_LOOKUP = /\!\[(.*)\]\((https:\/\/[-a-z0-9]+\.githubusercontent\.com\/\d+\/[-0-9a-f]{32,512}\.(jpg|gif|png))\)/gm;
+    const REGEX_USER_CONTENT_IMAGE_LOOKUP = /\!\[(.*)\]\((https:\/\/[-a-z0-9]+\.githubusercontent\.com\/\d+\/[-0-9a-f]{32,512}\.(jpg|gif|png))\)/gm;
+    const REGEX_ASSETS_IMAGE_LOCKUP = /\!\[(.*)\]\((https:\/\/github\.com\/[-\w\d]+\/[-\w\d]+\/assets\/\d+\/[\-0-9a-f]{32,512})\)/gm;
 
     // Check if we found something
-    let foundSimpleImages = REGEX_IMAGE_LOOKUP.test(initialBody);
+    let foundSimpleImages = REGEX_USER_CONTENT_IMAGE_LOOKUP.test(initialBody)
+        || REGEX_ASSETS_IMAGE_LOCKUP.test(initialBody);
     if (!foundSimpleImages) {
         console.log('Found no simple images to process');
         return;
@@ -47,53 +51,8 @@ module.exports = async ({github, context}) => {
     var wasMatchModified = false;
 
     // Try to find and replace the images with minimized ones
-    let newBody = await replaceAsync(initialBody, REGEX_IMAGE_LOOKUP, async (match, g1, g2) => {
-        console.log(`Found match '${match}'`);
-        
-        if (g1.endsWith(IGNORE_ALT_NAME_END)) {
-            console.log(`Ignoring match '${match}': IGNORE_ALT_NAME_END`);
-            return match;
-        }
-        
-        let probeAspectRatio = 0;
-        let shouldModify = false;
-        try {
-            console.log(`Probing ${g2}`);
-            let probeResult = await probe(g2);
-            if (probeResult == null) {
-                throw 'No probeResult';
-            }
-            if (probeResult.hUnits != 'px') {
-                throw `Unexpected probeResult.hUnits (expected px but got ${probeResult.hUnits})`;
-            }
-            if (probeResult.height <= 0) {
-                throw `Unexpected probeResult.height (height is invalid: ${probeResult.height})`;
-            }
-            if (probeResult.wUnits != 'px') {
-                throw `Unexpected probeResult.wUnits (expected px but got ${probeResult.wUnits})`;
-            }
-            if (probeResult.width <= 0) {
-                throw `Unexpected probeResult.width (width is invalid: ${probeResult.width})`;
-            }
-            console.log(`Probing resulted in ${probeResult.width}x${probeResult.height}px`);
-            
-            probeAspectRatio = probeResult.width / probeResult.height;
-            shouldModify = probeResult.height > IMG_MAX_HEIGHT_PX && probeAspectRatio < MIN_ASPECT_RATIO;
-        } catch(e) {
-            console.log('Probing failed:', e);
-            // Immediately abort
-            return match;
-        }
-        
-        if (shouldModify) {
-            wasMatchModified = true;
-            console.log(`Modifying match '${match}'`);
-            return `<img alt="${g1}" src="${g2}" width=${Math.min(600, (IMG_MAX_HEIGHT_PX * probeAspectRatio).toFixed(0))} />`;
-        }
-        
-        console.log(`Match '${match}' is ok/will not be modified`);
-        return match;
-    });
+    let newBody = await replaceAsync(initialBody, REGEX_USER_CONTENT_IMAGE_LOOKUP, minimizeAsync);
+    newBody = await replaceAsync(newBody, REGEX_ASSETS_IMAGE_LOCKUP, minimizeAsync);
 
     if (!wasMatchModified) {
         console.log('Nothing was modified. Skipping update');
@@ -117,9 +76,17 @@ module.exports = async ({github, context}) => {
             repo: context.repo.repo,
             body: newBody
         });
+    } else if (context.eventName == 'pull_request') {
+        console.log('Updating pull request', context.payload.pull_request.number);
+        await github.rest.pulls.update({
+            pull_number: context.payload.pull_request.number,
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            body: newBody
+        });
     }
 
-    // Asnyc replace function from https://stackoverflow.com/a/48032528
+    // Async replace function from https://stackoverflow.com/a/48032528
     async function replaceAsync(str, regex, asyncFn) {
         const promises = [];
         str.replace(regex, (match, ...args) => {
@@ -129,4 +96,52 @@ module.exports = async ({github, context}) => {
         const data = await Promise.all(promises);
         return str.replace(regex, () => data.shift());
     }
+
+    async function minimizeAsync(match, g1, g2) {
+            console.log(`Found match '${match}'`);
+
+            if (g1.endsWith(IGNORE_ALT_NAME_END)) {
+                console.log(`Ignoring match '${match}': IGNORE_ALT_NAME_END`);
+                return match;
+            }
+
+            let probeAspectRatio = 0;
+            let shouldModify = false;
+            try {
+                console.log(`Probing ${g2}`);
+                let probeResult = await probe(g2);
+                if (probeResult == null) {
+                    throw 'No probeResult';
+                }
+                if (probeResult.hUnits != 'px') {
+                    throw `Unexpected probeResult.hUnits (expected px but got ${probeResult.hUnits})`;
+                }
+                if (probeResult.height <= 0) {
+                    throw `Unexpected probeResult.height (height is invalid: ${probeResult.height})`;
+                }
+                if (probeResult.wUnits != 'px') {
+                    throw `Unexpected probeResult.wUnits (expected px but got ${probeResult.wUnits})`;
+                }
+                if (probeResult.width <= 0) {
+                    throw `Unexpected probeResult.width (width is invalid: ${probeResult.width})`;
+                }
+                console.log(`Probing resulted in ${probeResult.width}x${probeResult.height}px`);
+
+                probeAspectRatio = probeResult.width / probeResult.height;
+                shouldModify = probeResult.height > IMG_MAX_HEIGHT_PX && probeAspectRatio < MIN_ASPECT_RATIO;
+            } catch(e) {
+                console.log('Probing failed:', e);
+                // Immediately abort
+                return match;
+            }
+
+            if (shouldModify) {
+                wasMatchModified = true;
+                console.log(`Modifying match '${match}'`);
+                return `<img alt="${g1}" src="${g2}" width=${Math.min(600, Math.floor(IMG_MAX_HEIGHT_PX * probeAspectRatio))} />`;
+            }
+
+            console.log(`Match '${match}' is ok/will not be modified`);
+            return match;
+        }
 }

.github/workflows/image-minimizer.yml

@@ -5,6 +5,8 @@ on:
     types: [created, edited]
   issues:
     types: [opened, edited]
+  pull_request:
+    types: [opened, edited]
 
 permissions:
   issues: write
@@ -15,17 +17,17 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/setup-node@v3
+      - uses: actions/setup-node@v4
         with:
           node-version: 16
 
       - name: Install probe-image-size
         run: npm i [email protected] --ignore-scripts
 
       - name: Minimize simple images
-        uses: actions/github-script@v6
+        uses: actions/github-script@v7
         timeout-minutes: 3
         with:
           script: |