Proposal: API for Reading MIDI/Audio Data from DAW in Advance

[GasimoCodes]

I may have missed something, but I haven’t found an API in the documentation that allows plugins to actively read MIDI/audio data from the DAW. Implementing such functionality could represent a generational leap in the quality and flexibility of virtual instruments and effects. If Musicians can read all of the notes from a note sheet before playing, so should CLAP.

Example Benefits of This Feature:

• Effect plugins could analyze audio without requiring playback of the whole track. They could also grab audio of other tracks to gather more context.
• Instruments could understand and leverage note contexts and choose articulations or samples accordingly, reducing the need for extensive keyswitching or automation. (Some libraries already attempt this with workarounds, e.g., requiring users to shift MIDI tracks by several bars.)
• Instruments could precompute and cache audio, reducing the amount of what needs to be computed (again) during a playback. This is extra useable on slower machines when working with larger orchestras or (in the future) ai generated midi->audio tools

Proposal

Plugins would query the DAW for MIDI/audio events for their own or neighboring tracks.

Example pseudocode:

MidiEvent[] GetTrackEvents(int trackIndex, long start, long end);

Such a functionality would also require more code such as getting TrackModified events from DAW etc. Such needed additions can be discussed in this issue.

There is already API to get which track CLAP is on, there also exists API to control DAW transport. This proposed feature seems like a natural addition to the API. If such functionality already exists and I overlooked it, please let me know! (And if someone knows why it isnt in VST3/4 spec already please also let me know!)

[baconpaul]

Wouldn’t ara do this?

[GasimoCodes]

Wouldn’t ara do this?

Thanks for the quick reply @baconpaul . ARA2 seems to be able to do this and even provides a CLAP sample on their github (while being under the Apache License). It flew completely under my radar.
Considering ARA2 was out since 2018, its strange that it appears to be rarely used by plugins. The list on Avid's website (and Wikipedia) lists a total of 13 plugins which use it. Is there any technical reason for this or does it go down to ARA having a bad reputation or is integrating it into existing plugins difficult?

Having the ability to read midi data ahead of time sounds like a big thing all the realistic sample players would rush to implement asap. Why hasn't it happened and we see companies forcing users to shift midi beats or adding in fake latency just to read ahead?

[baconpaul]

I think quite a few more plugins use ARA than listed on the website. Many things which do a lookahead do I think, as do a lot of scoring plugins.

As to why sampler authors haven't used it, I'm not sure! @defiantnerd worked with the team on ARA in clap so maybe he would know? It's also not supported in every daw (but is in most and is in the big pro daws).

My point was more "I think what you asked for exists and works in clap today using ARA in the DAWs which support it".