proposal - integration documentation API/input in some form

[KTrain5169]

as far as I know, neuro is entirely in charge of what she will remember (with some exceptions like action force state which can partially be controlled by the integration toggling ephemeral_context). this works great ~a fair chunk of the time at least, but this is still imo a bit of a problem because game instructions may be disregarded or forgotten by neuro, causing her to make potentially ill-informed plays since the game instructions was discarded. while we all love neuro throwing on-stream sometimes, there's gotta be a limit on that so that she isn't perpetually bad at video games.

what I propose is either or both of the following:

• an additional API packet (or additional data to startup packets, though I think that's less flexible) that has some instructions on how to play the game that can be kept in a read-only (to neuro) context. the reason why I prefer an additional API packet is that it could allow the game to progressively update her "instruction book", so that not only is she not overloaded with context from documentation, but it also allows the integration to only reveal information as it's progressively discovered, possibly preventing spoilers (at least in a vacuum - google-sama can come out afterall) and also not distracting neuro from what she can do at that point in time.
  • an alternative to this could be potentially adding a new query type of packet that allows neuro to query the integration for information on-demand, similar to making it like an actual action that would take a single input for category. though she can forget to use that tool whenever she needs information, and this may not play nice with action forces.
• A method of delivering static Markdown files with frontmatter metadata. This would mean that it's easier to setup (in theory) and it's in a much more readable format which helps integration devs, and the files can be pre-downloaded and manually put in if needed. Then, when the integration starts, neuro can load that into her if she chooses (or load on a page-by-page/category basis, this would be where frontmatter metadata is important). partially inspired by KTANE's manual being "loaded" into neuro's context.
  • if pre-downloading the files is too much effort ("tool" integrations, as I've gotten to calling them, such as the vscode extension, may require the documentation to be updated somewhat regularly), the integration can simply send a bunch of markdown to the server on startup, potentially being able to combine with the other proposal, but this doesn't sound super practical and may actually cause more strain on processing on the server end (at least, I think, I'm not sure how computationally expensive this would be). also, this might be complicated for integration devs to set up.

honestly just wanted to throw my thoughts into here and see if this would be useful as an addition to v1/v2 neuro api, since the initial context containing all instructions can be buried pretty quickly and I think it's quite overdue for something like this to exist

[Govorunb]

A lot of the time when an LLM 'forgets' something it's just focusing on other things - the instructions are still in context, it's just that something else takes the attention. But, in general, to mitigate forgetfulness right now, you can:

1. Explicitly (or implicitly) ask Neuro to remember/recall it

("implicitly" meaning something like "This is important." over "YOU MUST REMEMBER THIS.")

I don't know if it's "nice" to do this - so, maybe look at the other methods first. I'm making some observational assumptions here (from a position no different to yours) but I don't think Neuro is incapable of choosing to remember important things, as long as it's reasonably clear what those important things are and when they are important. A high signal-to-noise ratio is always valuable. Which brings me to:

2. Improve instruction prompts themselves

Compact/summarize for information density, trim out things an LLM is reasonably likely to 'know' in its weights, provide examples to steer the model towards success, etc.

There are quite a few so-called 'techniques' (prompting is far from an exact science). I'd wager a guess that if you package the information well on your side, Neuro can function quite well (Pokemon was pretty good imo). That's not to say "if it doesn't work it's a skill issue on your part" - but LLMs are pretty good at picking things up these days.

3. Remind Neuro automatically (periodically/on logical breakpoints) OR let her remind herself

To be clear - slapping 5k tokens of context on a hardcoded timer is probably the worst way to go about doing this. Sending a short rules blurb on e.g. a new round in-game may help to "reset" a bit, but it's neither here nor there.

As for letting Neuro remind herself, I don't think you need a new message type - an action that gives her instructions fits the description of "allows neuro to query [...] on-demand" pretty well. If there's literal pages of instructions (a sign you may need to trim them down), you can add "navigation" through args as you suggested. If you're worried she'll stim-spam it and turtle-in-a-six-pack-ring her own context - since you control when she can do it, you can put a cooldown on it. And so on. With a system like this on your side rather than the server's, you have much more freedom.

I think if "I have too much documentation" is a recurring problem, it means there's opportunity space for a clientside library that encapsulates the "doc browsing" actions. (Especially considering we're coming up on a year since the v2 proposal, I'd advise looking for clientside workarounds before suggesting a spec change - sincerely hoping this sentence will age poorly.)

---

Longer-term, I think the issue will "solve itself" as models get better (e.g. RLMs are pretty interesting). Philosophically, it also seems (to me) like the longer-term goal is to have fewer and fewer "supporting" systems that do something for (or instead of) Neuro and more levers she can pull to do something with. Like, a prosthetic arm over a fully autonomous ArmBot-3000 - the more Neuro can express herself, the better. So, I'd try to frame things in these terms instead.

(obviously this is just me yapping, take it with a salt mine)

[KTrain5169]

A lot of the time when an LLM 'forgets' something it's just focusing on other things - the instructions are still in context, it's just
that something else takes the attention.

The problem is that eventually they start focusing on that specific thing without the rules. It can work, depending on what it is they are focusing on, I just don't think it's necessarily enough because it gets buried under more and more context.

1. Explicitly (or implicitly) ask Neuro to remember/recall it

("implicitly" meaning something like "This is important." over "YOU MUST REMEMBER THIS.")

I don't know if it's "nice" to do this - so, maybe look at the other methods first. I'm making some observational assumptions here (from a position no different to yours) but I don't think Neuro is incapable of choosing to remember important things, as long as it's reasonably clear what those important things are and when they are important. A high signal-to-noise ratio is always valuable.

I would say this is the less "intrusive" method besides checkpoint-based reminders, but this would be a problem if, in the course of a game, there are some things that are also "important to remember", which then means neuro has to start weighing what is more important in arbitrary values she may not fully understand.

Which brings me to:

2. Improve instruction prompts themselves

Compact/summarize for information density, trim out things an LLM is reasonably likely to 'know' in its weights, provide examples to steer the model towards success, etc.

There are quite a few so-called 'techniques' (prompting is far from an exact science). I'd wager a guess that if you package the information well on your side, Neuro can function quite well (Pokemon was pretty good imo). That's not to say "if it doesn't work it's a skill issue on your part" - but LLMs are pretty good at picking things up these days.

I agree - the problem is that in longer games or over the course of a game round I'm not sure if this would hold up.

3. Remind Neuro automatically (periodically/on logical breakpoints) OR let her remind herself

To be clear - slapping 5k tokens of context on a hardcoded timer is probably the worst way to go about doing this. Sending a short rules blurb on e.g. a new round in-game may help to "reset" a bit, but it's neither here nor there.

For most games I'd imagine this is good enough, however for more open-ended games there aren't as likely anything to use as checkpoints. ditto for tool integrations but I'm trying to come at a game perspective here.

As for letting Neuro remind herself, I don't think you need a new message type - an action that gives her instructions fits the description of "allows neuro to query [...] on-demand" pretty well. If there's literal pages of instructions (a sign you may need to trim them down), you can add "navigation" through args as you suggested. If you're worried she'll stim-spam it and turtle-in-a-six-pack-ring her own context - since you control when she can do it, you can put a cooldown on it. And so on. With a system like this on your side rather than the server's, you have much more freedom.

the reason why I wanted a new message type is because making it as an action is a workaround I don't think really plays super nice with low prio action forces, where neuro probably could look at rules before deciding to make a move. Also, "If there's literal pages of instructions (a sign you may need to trim them down)" -> some game mechanics may require a bit more explanation, and while part of it would be a communication skill issue there can be legitimate mechanics that takes a fair amount to instruct. it would be more convenient if it was stored in a "book" somewhere.

I think if "I have too much documentation" is a recurring problem, it means there's opportunity space for a clientside library that encapsulates the "doc browsing" actions.

I do agree that there probably is an opportunity for a library of some kind to encapsulate doc browsing actions. I was going to see if I could make a prototype of what it would look like before dumping this idea here but decided against it as when I wrote this it was already late at night.

(Especially considering we're coming up on a year since the v2 proposal, I'd advise looking for clientside workarounds before suggesting a spec change - sincerely hoping this sentence will age poorly.)

I also (respectfully) hope the sentence ages poorly.

Longer-term, I think the issue will "solve itself" as models get better (e.g. RLMs are pretty interesting). Philosophically, it also seems (to me) like the longer-term goal is to have fewer and fewer "supporting" systems that do something for (or instead of) Neuro and more levers she can pull to do something with. Like, a prosthetic arm over a fully autonomous ArmBot-3000 - the more Neuro can express herself, the better. So, I'd try to frame things in these terms instead.

If we take humans as a baseline example, they still need to be able to read instruction manuals to do things, so I'd say it would still make sense for Neuro to have this sort of feature. Maybe not long-term if she's able to absorb that knowledge, but I definitely still think it would make sense within those goals.

(obviously this is just me yapping, take it with a salt mine)

fair enough

---

also, I tried my best to come from a game perspective even though the api is more genericised now, mainly because the first line still says it's a protocol for letting neuro play games, but I also fully understand this wouldn't be used too much by integrations that aren't complex or that expect vedal to play alongside (probably). but I still think being able to have a documentation book for neuro somewhere would be useful to flesh out said complex integrations and also I suspect that sometimes she wouldn't have intended to choose the "worse" option had she known it would be worse (can't always tell though due to ragebait existing)

[Govorunb]

Right, yeah - I think I may have ended up unnecessarily downplaying things - it's a valid issue. Prompting/contexting is a mitigation and it can only go so far (though pretty far imo). Just saying that your current options for a solution (as I see them) are to wait 3 months to a year (or more) for model intelligence, wait (gestures vaguely) for this spec change (plus potentially v2), or to try and get incremental improvements out "today" (comparatively). In my eyes, a clientside solution is the best because you can take it into your own hands to ensure success. Best of luck though, whichever way you decide to pursue.

[KTrain5169]

Right, yeah - I think I may have ended up unnecessarily downplaying things - it's a valid issue. Prompting/contexting is a mitigation and it can only go so far (though pretty far imo). Just saying that your current options for a solution (as I see them) are to wait 3 months to a year (or more) for model intelligence, wait (gestures vaguely) for this spec change (plus potentially v2), or to try and get incremental improvements out "today" (comparatively). In my eyes, a clientside solution is the best because you can take it into your own hands to ensure success. Best of luck though, whichever way you decide to pursue.

I'm currently looking into what the "best" option is in place of this proposal but this currently requires analysing many lines of code and a few too many hours of vods than I can handle.

waiting 3mo+ for model intelligence upgrades at least has a higher chance of happening before spec change/v2 (aware). in the meantime I probably will see if I have the time to write something like a data store module and make it available for everyone to use

[CoolCat467]

I would also be quite interested in working on a data store/book module, that sounds interesting to me, so if it were for Python I would like to collaborate.

[Ubuntufanboy]

I'm inclined to agree that client-side is the way to go. As nice as it would be to have any solution implemented in the actual API, we'd be waiting for ages. Although, I still suggest building the tools around the assumption that Neuro's intelligence will be improved.

Although, to improve current performance, it should be the standard to required (or strongly recommend) to have all the instructions use few-shot prompting. Ideally, all instructions when registered should have some "examples" field to put in context. Even less intelligent models are still fully capable of performing calls given examples. The extra 100-200 tokens is well worth it for more clear instructions. Plus, since Neuro would be managing her own context (probably still with some form of a cooldown), this would still be more efficient than instruction injecting.

We just need to build a client-side tool that can perform instruction RAG decently well. I could/would do this but I am still learning the exact nuances of the API. I'm nowhere near as experienced as you all.

[KTrain5169]

@Ubuntufanboy

Although, to improve current performance, it should be the standard to required (or strongly recommend) to have all the instructions use few-shot prompting. Ideally, all instructions when registered should have some "examples" field to put in context. Even less intelligent models are still fully capable of performing calls given examples.

I agree too, the problem is that examples is marked as a "maybe it works" for action schemas, which means to actually have examples we would need to load it into description (not very fun)
This is in addition to examples of game mechanics taking up probably more tokens than necessary (though I don't actually know what the balance is)

[Ubuntufanboy]

I think the best thing to do is for somebody to make a backwards compatible fork with experimental community updates so developers can try it out better implementations of current less reliable features. If Vedal/Alex were to see a direct comparison between LLM action accuracy in completing actions with certain features, it would be much more compelling to them to implement. The token count I'm not too worried about because this would be done during registry. Even if the original instructions go out of context, it's okay because her own usages of the tools should surely be enough. She isn't going to suddenly "forget" how to use a tool she has been using before.

[CoolCat467]

I don't see them ever open-sourcing the server side unless it were a separate module the backend plugs into, but a sort of wrapper sort of deal could certainly work. Ie wrapper acts as primary neuro sdk client that neuro actually talks too, and also acts as a server itself that passes through everything to primary client but with alterations?

[KTrain5169]

I don't see them ever open-sourcing the server side unless it were a separate module the backend plugs into, but a sort of wrapper sort of deal could certainly work. Ie wrapper acts as primary neuro sdk client that neuro actually talks too, and also acts as a server itself that passes through everything to primary client but with alterations?

ehhh that doesn't sound like it would help here (also too much effort)