Log linearized curve overlays #455
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Weird, CI breaking on Not sure when that started? (fixed with #457) |
goodboy
force-pushed
the
overlays_interaction_latency_tuning
branch
from
e280e48 to
07d941d
Compare
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
94e212d to
3c4031e
Compare
goodboy
force-pushed
the
overlays_interaction_latency_tuning
branch
from
07d941d to
ea3ea8d
Compare
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
d4262c9 to
4f204dc
Compare
goodboy
force-pushed
the
overlays_interaction_latency_tuning
branch
from
ea3ea8d to
02b92e8
Compare
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
4f204dc to
38bb493
Compare
goodboy
force-pushed
the
overlays_interaction_latency_tuning
branch
from
02b92e8 to
fefb0de
Compare
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
38bb493 to
f697354
Compare
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
f697354 to
1d6270f
Compare
77 tasks
goodboy
commented
Feb 14, 2023
piker/ui/_interaction.py
Outdated
| @@ -340,6 +360,49 @@ async def handle_viewmode_mouse( | |||
| view.order_mode.submit_order() | |||
|
|
|||
|
|
|||
| class OverlayT(Struct): | |||
There was a problem hiding this comment.
Note this code isn't used at all yet, it's just a draft for a more encapsulated API for the main logic inside .interact_graphics_cycle() below.
Base automatically changed from
overlays_interaction_latency_tuning
to
master
February 14, 2023 18:48
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
1d6270f to
3d091f2
Compare
goodboy
added a commit
that referenced
this pull request
Feb 16, 2023
As part of solving a final bullet-issue in #455, which is specifically a case: - with N > 2 curves, one of which is the "major" dispersion curve" and the others are "minors", - we can run into a scenario where some minor curve which gets pinned to the major (due to the original "pinning technique" -> "align to major") at some `P(t)` which is *not* the major's minimum / maximum due to the minor having a smaller/shorter support and thus, - requires that in order to show then max/min on the minor curve we have to expand the range of the major curve as well but, - that also means any previously scaled (to the major) minor curves need to be adjusted as well or they'll not be pinned to the major the same way! I originally was trying to avoid doing the recursive iteration back through all previously scaled minor curves and instead decided to try implementing the "per side" curve dispersion detection (as was originally attempted when first starting this work). The idea is to decide which curve's up or down "swing in % returns" would determine the global y-range *on that side*. Turns out I stumbled on the "align to first" technique in the process: "for each overlay curve we align its earliest sample (in time) to the same level of the earliest such sample for whatever is deemed the major (directionally disperse) curve in view". I decided (with help) that this "pin to first" approach/style is equally as useful and maybe often more so when wanting to view support-disjoint time series: - instead of compressing the y-range on "longer series which have lesser sigma" to make whatever "shorter but larger-sigma series" pin to it at an intersect time step, this instead will expand the price ranges based on the earliest time step in each series. - the output global-returns-overlay-range for any N-set of series is equal to the same in the previous "pin to intersect time" technique. - the only time this technique seems less useful is for overlaying market feeds which have the same destination asset but different source assets (eg. btceur and btcusd on the same chart since if one of the series is shorter it will always be aligned to the earliest datum on the longer instead of more naturally to the intersect sample level as was in the previous approach). As such I'm going to keep this technique as discovered and will later add back optional support for the "align to intersect" approach from previous (which will again require detecting the highest dispersion curve direction-agnostic) and pin all minors to the price level at which they start on the major. Further details of the implementation rework in `.interact_graphics_cycle()` include: - add `intersect_from_longer()` to detect and deliver a common datum from 2 series which are different in length: the first time-index sample in the longer. - Rewrite the drafted `OverlayT` to only compute (inversed log-returns) transforms for a single direction and use 2 instances, one for each direction inside the `Viz`-overlay iteration loop. - do all dispersion-per-side major curve detection in the first pass of all `Viz`s on a plot, instead updating the `OverlayT` instances for each side and compensating for any length mismatch and rescale-to-minor cases in each loop cycle.
goodboy
added a commit
that referenced
this pull request
Feb 21, 2023
As part of solving a final bullet-issue in #455, which is specifically a case: - with N > 2 curves, one of which is the "major" dispersion curve" and the others are "minors", - we can run into a scenario where some minor curve which gets pinned to the major (due to the original "pinning technique" -> "align to major") at some `P(t)` which is *not* the major's minimum / maximum due to the minor having a smaller/shorter support and thus, - requires that in order to show then max/min on the minor curve we have to expand the range of the major curve as well but, - that also means any previously scaled (to the major) minor curves need to be adjusted as well or they'll not be pinned to the major the same way! I originally was trying to avoid doing the recursive iteration back through all previously scaled minor curves and instead decided to try implementing the "per side" curve dispersion detection (as was originally attempted when first starting this work). The idea is to decide which curve's up or down "swing in % returns" would determine the global y-range *on that side*. Turns out I stumbled on the "align to first" technique in the process: "for each overlay curve we align its earliest sample (in time) to the same level of the earliest such sample for whatever is deemed the major (directionally disperse) curve in view". I decided (with help) that this "pin to first" approach/style is equally as useful and maybe often more so when wanting to view support-disjoint time series: - instead of compressing the y-range on "longer series which have lesser sigma" to make whatever "shorter but larger-sigma series" pin to it at an intersect time step, this instead will expand the price ranges based on the earliest time step in each series. - the output global-returns-overlay-range for any N-set of series is equal to the same in the previous "pin to intersect time" technique. - the only time this technique seems less useful is for overlaying market feeds which have the same destination asset but different source assets (eg. btceur and btcusd on the same chart since if one of the series is shorter it will always be aligned to the earliest datum on the longer instead of more naturally to the intersect sample level as was in the previous approach). As such I'm going to keep this technique as discovered and will later add back optional support for the "align to intersect" approach from previous (which will again require detecting the highest dispersion curve direction-agnostic) and pin all minors to the price level at which they start on the major. Further details of the implementation rework in `.interact_graphics_cycle()` include: - add `intersect_from_longer()` to detect and deliver a common datum from 2 series which are different in length: the first time-index sample in the longer. - Rewrite the drafted `OverlayT` to only compute (inversed log-returns) transforms for a single direction and use 2 instances, one for each direction inside the `Viz`-overlay iteration loop. - do all dispersion-per-side major curve detection in the first pass of all `Viz`s on a plot, instead updating the `OverlayT` instances for each side and compensating for any length mismatch and rescale-to-minor cases in each loop cycle.
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
5223eb1 to
020e1ae
Compare
goodboy
added a commit
that referenced
this pull request
Feb 28, 2023
As part of solving a final bullet-issue in #455, which is specifically a case: - with N > 2 curves, one of which is the "major" dispersion curve" and the others are "minors", - we can run into a scenario where some minor curve which gets pinned to the major (due to the original "pinning technique" -> "align to major") at some `P(t)` which is *not* the major's minimum / maximum due to the minor having a smaller/shorter support and thus, - requires that in order to show then max/min on the minor curve we have to expand the range of the major curve as well but, - that also means any previously scaled (to the major) minor curves need to be adjusted as well or they'll not be pinned to the major the same way! I originally was trying to avoid doing the recursive iteration back through all previously scaled minor curves and instead decided to try implementing the "per side" curve dispersion detection (as was originally attempted when first starting this work). The idea is to decide which curve's up or down "swing in % returns" would determine the global y-range *on that side*. Turns out I stumbled on the "align to first" technique in the process: "for each overlay curve we align its earliest sample (in time) to the same level of the earliest such sample for whatever is deemed the major (directionally disperse) curve in view". I decided (with help) that this "pin to first" approach/style is equally as useful and maybe often more so when wanting to view support-disjoint time series: - instead of compressing the y-range on "longer series which have lesser sigma" to make whatever "shorter but larger-sigma series" pin to it at an intersect time step, this instead will expand the price ranges based on the earliest time step in each series. - the output global-returns-overlay-range for any N-set of series is equal to the same in the previous "pin to intersect time" technique. - the only time this technique seems less useful is for overlaying market feeds which have the same destination asset but different source assets (eg. btceur and btcusd on the same chart since if one of the series is shorter it will always be aligned to the earliest datum on the longer instead of more naturally to the intersect sample level as was in the previous approach). As such I'm going to keep this technique as discovered and will later add back optional support for the "align to intersect" approach from previous (which will again require detecting the highest dispersion curve direction-agnostic) and pin all minors to the price level at which they start on the major. Further details of the implementation rework in `.interact_graphics_cycle()` include: - add `intersect_from_longer()` to detect and deliver a common datum from 2 series which are different in length: the first time-index sample in the longer. - Rewrite the drafted `OverlayT` to only compute (inversed log-returns) transforms for a single direction and use 2 instances, one for each direction inside the `Viz`-overlay iteration loop. - do all dispersion-per-side major curve detection in the first pass of all `Viz`s on a plot, instead updating the `OverlayT` instances for each side and compensating for any length mismatch and rescale-to-minor cases in each loop cycle.
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
1b83f4b to
de2f77b
Compare
|
tweaked base branch to that from #471 for now just until we either drop those commits from this history or land that branch (which is slightly more critical 😂 ) |
goodboy
added a commit
that referenced
this pull request
Feb 28, 2023
As part of solving a final bullet-issue in #455, which is specifically a case: - with N > 2 curves, one of which is the "major" dispersion curve" and the others are "minors", - we can run into a scenario where some minor curve which gets pinned to the major (due to the original "pinning technique" -> "align to major") at some `P(t)` which is *not* the major's minimum / maximum due to the minor having a smaller/shorter support and thus, - requires that in order to show then max/min on the minor curve we have to expand the range of the major curve as well but, - that also means any previously scaled (to the major) minor curves need to be adjusted as well or they'll not be pinned to the major the same way! I originally was trying to avoid doing the recursive iteration back through all previously scaled minor curves and instead decided to try implementing the "per side" curve dispersion detection (as was originally attempted when first starting this work). The idea is to decide which curve's up or down "swing in % returns" would determine the global y-range *on that side*. Turns out I stumbled on the "align to first" technique in the process: "for each overlay curve we align its earliest sample (in time) to the same level of the earliest such sample for whatever is deemed the major (directionally disperse) curve in view". I decided (with help) that this "pin to first" approach/style is equally as useful and maybe often more so when wanting to view support-disjoint time series: - instead of compressing the y-range on "longer series which have lesser sigma" to make whatever "shorter but larger-sigma series" pin to it at an intersect time step, this instead will expand the price ranges based on the earliest time step in each series. - the output global-returns-overlay-range for any N-set of series is equal to the same in the previous "pin to intersect time" technique. - the only time this technique seems less useful is for overlaying market feeds which have the same destination asset but different source assets (eg. btceur and btcusd on the same chart since if one of the series is shorter it will always be aligned to the earliest datum on the longer instead of more naturally to the intersect sample level as was in the previous approach). As such I'm going to keep this technique as discovered and will later add back optional support for the "align to intersect" approach from previous (which will again require detecting the highest dispersion curve direction-agnostic) and pin all minors to the price level at which they start on the major. Further details of the implementation rework in `.interact_graphics_cycle()` include: - add `intersect_from_longer()` to detect and deliver a common datum from 2 series which are different in length: the first time-index sample in the longer. - Rewrite the drafted `OverlayT` to only compute (inversed log-returns) transforms for a single direction and use 2 instances, one for each direction inside the `Viz`-overlay iteration loop. - do all dispersion-per-side major curve detection in the first pass of all `Viz`s on a plot, instead updating the `OverlayT` instances for each side and compensating for any length mismatch and rescale-to-minor cases in each loop cycle.
Instead delegate directly to `Viz.default_view()` throughout charting startup and interaction handlers. Also add a `ChartPlotWidget.reset_graphics_caches()` context mngr which resets all managed graphics object's cacheing modes on enter and restores them on exit for simplified use in interaction handling code.
Previously when very zoomed out and using the `'r'` hotkey the interaction handler loop wouldn't trigger a re-(up)sampling to get a more detailed curve graphic and instead the previous downsampled (under-detailed) graphic would show. Fix that by ensuring we yield back to the Qt event loop and do at least a couple render cycles with paired `.interact_graphics_cycle()` calls. Further this flips the `.start/signal_ic()` methods to use the new `.reset_graphics_caches()` ctr-mngr method.
When the target pinning curve (by default, the dispersion major) is shorter then the pinned curve, we need to make sure we find still find the x-intersect for computing returns scalars! Use `Viz.i_from_t()` to accomplish this as well and, augment that method with a `return_y: bool` to allow the caller to also retrieve the equivalent y-value at the requested input time `t: float` for convenience. Also tweak a few more internals around the 'loglin_ref_to_curve' method: - only solve / adjust for the above case when the major's xref is detected as being "earlier" in time the current minor's. - pop the major viz entry from the overlay table ahead of time to avoid a needless iteration and simplify the transform calc phase loop to avoid handling that needless cycle B) - add much better "organized" debug printing with more clear headers around which "phase"/loop the message pertains and well as more explicit details in terms of x and y-range values on each cycle of each loop.
There's been way too many issues when trying to calculate this dynamically from the input array, so just expect the caller to know what it's doing and don't bother with ever hitting the error case of calculating and incorrect value internally.
As per the change to `slice_from_time()` this ensures this `Viz` always passes its self-calculated time indexing step size to the time slicing routine(s). Further this contains a slight impl tweak to `.scalars_from_index()` to slice the actual view range from `xref` to `Viz.ViewState.xrange[1]` and then reading the corresponding `yref` from the first entry in that array; this should be no slower in theory and makes way for further caching of x-read-range to `ViewState` opportunities later.
Again, as per the signature change, never expect implicit time step calcs from overlay processing/machinery code. Also, extend the debug printing (yet again) to include better details around "rescale-due-to-minor-range-out-of-view" cases and a detailed msg for the transform/scaling calculation (inputs/outputs), particularly for the cases when one of the curves has a lesser support.
For the purposes of eventually trying to resolve last-step indexing synchronization (an intermittent but still existing) issue(s) that can happen due to races during history frame query and shm writing during startup. In fact, here we drop all `hist_viz` info queries from the main display loop for now anticipating that this code will either be removed or improved later.
Not sure how i missed this (and left in handling of `list.remove()` and it ever worked for that?) after the `samplerd` impl in 5ec1a72 but, this adjusts the remove-broken-subscriber loop to catch the correct `set.remove()` exception type on a missing (likely already removed) subscription entry.
This is particularly more "good looking" when we boot with a pair that doesn't have historical 1s OHLC and thus the fast chart is empty from outset. In this case it's a lot nicer to be already zoomed to a comfortable preset number of "datums in view" even when the history isn't yet filled in. Adjusts the chart display `Viz.default_view()` startup to explicitly ensure this happens via the `do_min_bars=True` flag B)
Solve this by always scaling the y-range for the major/target curve *before* the final overlay scaling loop; this implicitly always solve the case where the major series is the only one in view. Tidy up debug print formatting and add some loop-end demarcation comment lines.
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
ca71e32 to
1aab9f1
Compare
|
Outstanding bullets added to #461. Think this is ready to land 🏄🏼 |
goodboy
force-pushed
the
log_linearized_curve_overlays
branch
from
5a07542 to
889e920
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The cross-symbol y-alignment you always wanted..
Yah it's for #420 🏄🏼 now swapped with #461.
wUt dIs?!
CurveperPlotItemy-range sortingChartView.interact_graphics_cycle()which does the more efficient pipelining ofVizrelated graphics update calls per mouse event._displayloop as well.Qt.Signalstuff for interaction which results in more reliable, easier to understand, and generally faster interaction handling code 😂Solved ToDo bugs:
we need to add back y-ranging on the vlm chart to make sure the RHS label is always in view (done in 0d45495)
stupid gap on LHS of vlm fsp chart which still seems to be due to some kinda
PlotItem.layoutrelated setting?LUL, fixed in d4262c9
get the immediate y-range incremental update on L1 working again?
rewrite commit 325fe3c which is duplicate of 325fe3c from Overlays interaction latency tuning #453 after cherry picking it there.
Final refinements
OverlayTtype and API and use lru caching on range (inverse) transform outputsFINAL FINAL bug fixes and tweaks:
fix upsampling (re-)triggering when using the
'r'to reset chart hotkey in 1d649e5there's still an issue with de-synced 1m last bars?
lol, the alignment to "sigma major" default can definitely still cause a terrible config as per this example of bitcoin overlayed with a deribit call:
align_to_majordisp_still_borked_Peek.2023-03-06.10-53.mp4