Symptom
On POST /v1/chat/completions and POST /v1/completions with "stream": true, a stop
sequence only ends the answer when it happens to fall inside a single token. A sequence that
spans a token boundary is never seen, and the generation runs on to its budget.
When it does match, the token carrying the match has already been sent. The answer therefore
ends one token late: the stop sequence itself, and whatever the model put after it inside that
same token, reach the client. OpenAI cuts exactly at the sequence and never emits it.
Cause
The check looks at each token on its own, and nothing trims what was already emitted:
if any(s in token for s in stop_sequences):
break
No text is accumulated across tokens, so a sequence can only ever match within one.
Repro
curl -sN localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<model>","stream":true,
"messages":[{"role":"user","content":"Repeat exactly, nothing else: alpha beta gamma delta"}],
"stop":["beta gamma"]}'
The two-word sequence is tokenized apart, so no single chunk contains it and the stream runs to
the end. With a sequence that is one token — "stop":["gamma"] — the stream does end, but the
chunk containing gamma has already been delivered.
Symptom
On
POST /v1/chat/completionsandPOST /v1/completionswith"stream": true, astopsequence only ends the answer when it happens to fall inside a single token. A sequence that
spans a token boundary is never seen, and the generation runs on to its budget.
When it does match, the token carrying the match has already been sent. The answer therefore
ends one token late: the stop sequence itself, and whatever the model put after it inside that
same token, reach the client. OpenAI cuts exactly at the sequence and never emits it.
Cause
The check looks at each token on its own, and nothing trims what was already emitted:
No text is accumulated across tokens, so a sequence can only ever match within one.
Repro
The two-word sequence is tokenized apart, so no single chunk contains it and the stream runs to
the end. With a sequence that is one token —
"stop":["gamma"]— the stream does end, but thechunk containing
gammahas already been delivered.