feat: Improve onyxsdk-pen integration #1452

zinstack625 · 2025-03-19T23:06:37Z

Updated onyxsdk dependencies

Makes stuff work on some newer devices. Don't really know why, but before that, stylus input resulted in full refreshes, as if there were no support for that altogether. This was observed both on Play Store and on local builds. May, but should not break other devices support

Reworked OnyxsdkPenArea.kt

Removed explicit drawing
Much of the stuff done in drawPreview had absolutely no effects. Whole magic seems to happen in TouchHelper. Removing that manual drawing, again, may break stuff on other devices, but most of the logic remained nonetheless. In case of breakages, should be trivial to bring that stuff back, or, if my intuition is correct, further simplify this class. Would really like to hear some feedback on that removed code if possible
Established message passing between OnyxsdkPenArea and Dart
In order to modify state of the PlatformView, message passing was required. It is used to update stroke parameters from Flutter UI toolbar to the PlatformView. I see the use as "refreshing" the creationParams. Might be useful in the future, both in of itself and as a precedent for other usecases
Mapped used tools to those in onyxsdk-pen
This in effect resulted in directly-refreshed preview looking very similar to the underlying image. Most of the tools are practically indistinguishable and are only insignificantly changed in shape (underlying image is less detailed) and, in some cases significantly in color during full refresh (seen on attached video). Exception being the pencil (seen on attached video), but it's still pretty close. Don't think it can get closer without some legally questionable voodoo magic with onyxsdk

Dart side

Most are due to preparation of tool parameters to Kotlin side, some are due to canvas being unaware of the tool being used. I tried to make significant changes as close to the actual use as possible, but may have introduced a couple of unnecessary abstractions and translations along the way. Could possibly be done better, but I'm inexperienced in Flutter and Dart. Hopefully the changes are not disruptive

Seems to be doing good on Note Air 4C, but would be good if someone could test on other devices. This does not seem to introduce any slowdowns, does not touch any significant logic and does not introduce any UI changes

20250320.webm

Great project BTW

codecov · 2025-03-19T23:45:38Z

Codecov Report

Attention: Patch coverage is 96.42857% with 1 line in your changes missing coverage. Please review.

Project coverage is 47.29%. Comparing base (7d60459) to head (a9c8748).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/components/canvas/canvas.dart	95.45%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1452      +/-   ##
==========================================
+ Coverage   47.15%   47.29%   +0.13%     
==========================================
  Files         114      114              
  Lines        9002     9027      +25     
==========================================
+ Hits         4245     4269      +24     
- Misses       4757     4758       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fe-Ti · 2025-03-20T20:32:10Z

Hi,

On my NoteAir 2 there is a huge lag (approx. 30s) after I lift my pen and before the actual stroke is rendered from scribble.

I tried to fix this issue and came up with this solution (same for eraser):

        override fun onBeginRawDrawing(b: Boolean, touchPoint: TouchPoint) {
            // begin of stylus data
            touchHelper.setRawDrawingRenderEnabled(true)
            reset()
        }

        override fun onEndRawDrawing(b: Boolean, touchPoint: TouchPoint) {
            // end of stylus data
            touchHelper.setRawDrawingRenderEnabled(false)
            reset()
            scheduleRefresh()
        }

With such toggling the lag disappeared and expirience became closer to proprietary note app.

Also there is an issue with releasing resources of touchHelper. For some reason the dispose() function is not called if I exit an app via navigation buttons/gestures (e.g. going to browser or launcher). So the pen continues to scribble all over the screen :)

zinstack625 · 2025-03-21T14:08:24Z

On my NoteAir 2 there is a huge lag (approx. 30s) after I lift my pen and before the actual stroke is rendered from scribble

Regarding this lag, this should be handled in scheduleRefresh(). Setting it too low may result in display refreshing too often, which in itself may be undesirable(?). 1 sec should be reasonable, but 30 secs is definitely unintended behavior. Perhaps forcing a refresh with EpdController is better than hoping it would on itself...

Also there is an issue with releasing resources of touchHelper. For some reason the dispose() function is not called if I exit an app via navigation buttons/gestures (e.g. going to browser or launcher). So the pen continues to scribble all over the screen :)

Can reproduce, dispose() is indeed not called on app minimization, and is even skipped on app close (however view is still destroyed and this sideeffect is gone). No method from PlatformView is called on this action. I guess the best option is to figure whether canvas is shown from Dart side (is there such a callback somewhere?) and pass it to Kotlin...

override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: Result) {
    if (call.method == "updateStroke") {
        val params = call.arguments<Map<String, Any>?>()
        updateStroke(params)
        result.success(null)
    } else if (call.method == "disableDraw") {
        touchHelper.setRawDrawingEnabled(false)
    } else if (call.method == "enableDraw") {
        touchHelper.setRawDrawingEnabled(true)
    } else {
        result.notImplemented()
    }
}

Fe-Ti · 2025-03-21T15:11:13Z

I guess the best option is to figure whether canvas is shown from Dart side (is there such a callback somewhere?) and pass it to Kotlin...

In such way overriding dispose() in _OnyxSdkPenAreaState may help

  @override
  void dispose() {
    channel.invokeMethod('disableDraw', creationParams).catchError((e) {});
  }

I'm not sure if it doesn't break anything in onyxsdk package interface.

Iey4iej3 · 2025-04-05T08:09:29Z

I am unable to compile and test, but did you encounter #1459 before the fix?

zinstack625 · 2025-04-06T15:28:51Z

I am unable to compile and test, but did you encounter #1459 before the fix?

Kinda, in some circumstances, not really sure, might just be my writing. If it is there, it will probably still be there, I did not mess with the code that gets the stylus data that's used after the refresh. I however can not test anything before 800e450, stuff just straight up does not work for me until that (I have a note air 4c, which is probably too new for version used before that)

Iey4iej3 · 2025-04-07T06:38:11Z

Kinda, in some circumstances, not really sure, might just be my writing.

The most obvious strange thing is that, the top dot in "i" is thickened into a bullet. It does not look like this in the preview, but becomes this after a refresh.

Iey4iej3 · 2025-04-21T10:05:49Z

I am unable to compile and test, but did you encounter #1459 before the fix?

Kinda, in some circumstances, not really sure, might just be my writing. If it is there, it will probably still be there, I did not mess with the code that gets the stylus data that's used after the refresh. I however can not test anything before 800e450, stuff just straight up does not work for me until that (I have a note air 4c, which is probably too new for version used before that)

Actually, your patch seems to fix #1459. At least, it is much better. To clarify, it is not fixed by only upgrading onyxsdk as in 800e450. It is unclear why this patch fixes the issue. Thank you very much!

adil192

Really impressive, thank you! Just requesting some tiny changes

adil192 · 2025-05-12T08:18:03Z

packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt

+    enum class StrokeStyle(val Id: Int) {
+        FountainPen(0),
+        Pen(1),
+        Brush(2),
+        Pencil(3),
+        Marker(4)
+    }


We should define this enum in the dart package too

adil192 · 2025-05-12T08:18:41Z

lib/components/canvas/canvas.dart

+  int toolToOnyx(Tool currentTool) {
+    if (placeholder) return 5;
+    if (currentTool is Pencil) {
+      return 3;
+    } else if (currentTool is Highlighter) {
+      return 4;
+    } else if (currentTool is Eraser) {
+      return 1;
+    } else if (currentTool is Select) {
+      return 1;
+    } else if (currentTool is LaserPointer) {
+      return 1;
+    } else if (currentTool is Pen) {
+      if (currentTool.isPressureEnabled()) {
+        return 2;
+      } else {
+        return 1;
+      }
+    } else {
+      return 5;
+    }
+  }
+


This can then use the enum's values instead of ints

adil192 · 2025-05-12T08:19:37Z

lib/components/canvas/canvas.dart

+  int getColor() {
+      if (currentTool is Pen) {
+        return (currentTool as Pen).color.toARGB32();
+      } else {
+        return Colors.black.toARGB32();
+      }
+  }
+  double getWidth() {
+      if (currentTool is Pen) {
+        double baseSize = (currentTool as Pen).getSize() * currentScale;
+        if ((currentTool as Pen).isPressureEnabled()) {
+          return baseSize;
+        } else {
+          return baseSize * 2;
+        }
+      } else {
+        return 3;
+      }
+  }


I think these methods would be best named getOnyxTool, getOnyxColor, getOnyxWidth so their purpose is more clear

adil192 · 2025-05-12T08:20:31Z

lib/data/tools/pen.dart

+  bool isPressureEnabled() {
+    return pressureEnabled;
+  }
+  double getSize() {
+    return options.size;
+  }
+


These seem like redundant getters, we can just use the underlying fields themselves

adil192 · 2025-05-12T08:22:40Z

packages/onyxsdk_pen/android/src/main/kotlin/com/example/onyxsdk_pen/OnyxsdkPenArea.kt

+        /*
+         Flutter ints are variable width
+         Personally, I think it is utterly dumb. I hope there is a way to fix
+         int size in flutter (why would you need 64 bits to store srgb??),
+         but unless or until there isn't, this stupidity should be performed
+         */
+        val pureColor = (paramsRef?.get("strokeColor") as? Long)?.toInt()
+                              ?: paramsRef?.get("strokeColor") as? Int
+                              ?: Color.BLACK


We have the fixnum package for fixed size ints, though I think I prefer this hack over polluting the dart code

adil192 · 2025-05-12T08:27:54Z

packages/onyxsdk_pen/lib/onyxsdk_pen_area.dart

@@ -21,6 +24,9 @@ class OnyxSdkPenArea extends StatefulWidget {
  /// is still writing, which will make the screen get stuck in a half-drawn
  /// state.
  final Duration refreshDelay;
+  final int strokeStyle;


This should be an enum arg too like in kotlin: e.g. strokeStyle: StrokeStyle.fountainPen

zinstack625 added 6 commits March 20, 2025 01:07

Chore: Update onyxsdk

800e450

Feat: Implement sdk strokes in onyx preview

4437187

Feat: pass selected tool to preview [1/3]

3c77dd9

Feat: pass selected tool to preview [2/3]

bb7589d

Feat: pass selected tool to preview [3/3]

1314e6c

fix: fixup tests

ae94dc6

fix: make linter happy

ac04b9b

github-actions bot mentioned this pull request Mar 21, 2025

Top Issues Dashboard #1196

Open

zinstack625 added 2 commits March 21, 2025 18:11

fix: force display refresh on stylus lift

711908a

fix: disable drawing on app hide

a9c8748

zinstack625 force-pushed the main branch from 556ccfe to a9c8748 Compare March 21, 2025 15:33

This comment was marked as off-topic.

Sign in to view

Iey4iej3 mentioned this pull request Apr 21, 2025

Onyx Boox: strokes become strange after refresh #1459

Open

2 tasks

Iey4iej3 mentioned this pull request May 9, 2025

Onyx Device: Strokes go over Toolbar and icons #1465

Open

2 tasks

adil192 requested changes May 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Improve onyxsdk-pen integration #1452

feat: Improve onyxsdk-pen integration #1452

zinstack625 commented Mar 19, 2025

codecov bot commented Mar 19, 2025 •

edited

Loading

Fe-Ti commented Mar 20, 2025 •

edited

Loading

zinstack625 commented Mar 21, 2025

Fe-Ti commented Mar 21, 2025 •

edited

Loading

Iey4iej3 commented Apr 5, 2025

zinstack625 commented Apr 6, 2025

Iey4iej3 commented Apr 7, 2025

This comment was marked as off-topic.

Iey4iej3 commented Apr 21, 2025 •

edited

Loading

adil192 left a comment

adil192 May 12, 2025

adil192 May 12, 2025 •

edited

Loading

adil192 May 12, 2025

adil192 May 12, 2025

adil192 May 12, 2025

adil192 May 12, 2025

feat: Improve onyxsdk-pen integration #1452

Are you sure you want to change the base?

feat: Improve onyxsdk-pen integration #1452

Conversation

zinstack625 commented Mar 19, 2025

Updated onyxsdk dependencies

Reworked OnyxsdkPenArea.kt

Dart side

codecov bot commented Mar 19, 2025 • edited Loading

Codecov Report

Fe-Ti commented Mar 20, 2025 • edited Loading

zinstack625 commented Mar 21, 2025

Fe-Ti commented Mar 21, 2025 • edited Loading

Iey4iej3 commented Apr 5, 2025

zinstack625 commented Apr 6, 2025

Iey4iej3 commented Apr 7, 2025

This comment was marked as off-topic.

Iey4iej3 commented Apr 21, 2025 • edited Loading

adil192 left a comment

Choose a reason for hiding this comment

adil192 May 12, 2025

Choose a reason for hiding this comment

adil192 May 12, 2025 • edited Loading

Choose a reason for hiding this comment

adil192 May 12, 2025

Choose a reason for hiding this comment

adil192 May 12, 2025

Choose a reason for hiding this comment

adil192 May 12, 2025

Choose a reason for hiding this comment

adil192 May 12, 2025

Choose a reason for hiding this comment

codecov bot commented Mar 19, 2025 •

edited

Loading

Fe-Ti commented Mar 20, 2025 •

edited

Loading

Fe-Ti commented Mar 21, 2025 •

edited

Loading

Iey4iej3 commented Apr 21, 2025 •

edited

Loading

adil192 May 12, 2025 •

edited

Loading