feat(api): oasdiff OpenAI openAPI spec against ours #3529
Conversation
meta-cla
bot
added
the
CLA Signed
cdoern
force-pushed
the
oasdiff-for-openai
branch
10 times, most recently
from
67b7c6e to
b509e85
Compare
cdoern
requested review from
ashwinb,
bbrowning,
ehhuang,
hardikjshah,
leseb,
mattf,
raghotham,
reluctantfuturist,
slekkala1,
terrytangyuan and
yanxi0830
as code owners
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
b509e85 to
8a2bb5f
Compare
.github/workflows/conformance.yml
Outdated
| "$OPENAI_SPEC" \ | ||
| "$LOCAL_SPEC" \ | ||
| --strip-prefix-revision "/v1/openai/v1" \ | ||
| --match-path "(^/v1/openai/v1)?/(chat/completions|completions|embeddings|files|models|moderations|vector_stores|responses)(/.*)?$" |
There was a problem hiding this comment.
we should probably have this list in code or as annotations to the specific apis in the code instead of listing the paths here: (^/v1/openai/v1)?/(chat/completions|completions|embeddings|files|models|moderations|vector_stores|responses)
| if: steps.cache-openapi.outputs.cache-hit != 'true' | ||
| run: | | ||
| mkdir -p ~/openai-openapi | ||
| curl -L https://app.stainless.com/api/spec/documented/openai/openapi.documented.yml -o ~/openai-openapi/openai-openapi.yml |
There was a problem hiding this comment.
do we not want to pin the version of the openai api?
There was a problem hiding this comment.
sure, I can pin it. Do we have a version in mind we want to use?
There was a problem hiding this comment.
I think the pinning is important, what's the latest version so we can get started with it?
| id: cache-openapi | ||
| uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 | ||
| with: | ||
| path: ~/openai-openapi |
There was a problem hiding this comment.
I believe the cache action by default has access only to a runners home dir? I could be wrong but that is where I have been storing and writing the cached binaries+files to/from
| key: openai-openapi.yml | ||
|
|
||
| # Cache oasdiff to avoid checksum failures and speed up builds | ||
| - name: Cache oasdiff |
There was a problem hiding this comment.
we should probably also support a force no-cache conformance check as well - we should probably run it at the time of release.
There was a problem hiding this comment.
I agree, but should I make this a separate action so it can be triggered by maintainers manually?
cdoern
force-pushed
the
oasdiff-for-openai
branch
4 times, most recently
from
da078af to
c5bd55e
Compare
cdoern
force-pushed
the
oasdiff-for-openai
branch
3 times, most recently
from
d7e8b77 to
5e68691
Compare
|
this now checks all of our APIs against all of the openAI apis. The removal of |
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
5e68691 to
f423104
Compare
diff the `/v1/` routes that are OpenAI compatible against the OpenAI openAPI spec. This will of course only trigger on PRs where the spec is changed. This will catch errors with new handwritten additions to our openAI compat routes. Signed-off-by: Charlie Doern <[email protected]>
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
f423104 to
d9abb2e
Compare
3 participants
What does this PR do?
diff the
/v1/openai/v1routes against the OpenAI openAPI spec. This will of course only trigger on PRs where the spec is changed.This will catch errors with new handwritten additions to our openAI compat routes.
I am marking this test with
continue-on-error: true, until we get rid of all of the errors. Nevertheless, this is a nice utility to have so folks know if their spec changes introduce more breaking changes or fix breakages when comparing to the OpenAI openapi spec.Test Plan
test should pass.