feat: vector buckets #774
feat: vector buckets #774
Conversation
⛔ Snyk checks have failed. 7 issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
![snyk-io[bot]](https://avatars.githubusercontent.com/in/372950?s=60&v=4){kind=small}
fenos
force-pushed
the
feat/vector-buckets
branch
5 times, most recently
from
1cf95bd to
01797cc
Compare
fenos
force-pushed
the
feat/vector-buckets
branch
from
01797cc to
a2715e7
Compare
Pull Request Test Coverage Report for Build 18404797921Details
💛 - Coveralls |
| app.register(routes.cdn, { prefix: 'cdn' }) | ||
| app.register(routes.healthcheck, { prefix: 'health' }) | ||
| app.register(routes.iceberg, { prefix: 'iceberg/v1' }) | ||
| app.register(routes.vectors, { prefix: 'vectors' }) |
There was a problem hiding this comment.
should this be singular ("vector")? to match: bucket, object, ...
| : 400 | ||
|
|
||
| if (statusCode === 500) { | ||
| console.log('error') |
| fastify.addHook('preParsing', async (request: AWSRequest, reply, bodyPayload) => { | ||
| if ( | ||
| opts.skipIfJwtToken && | ||
| request.headers.authorization?.replace('Bearer ', '')?.match(JWT_SHAPE) |
There was a problem hiding this comment.
Maybe this could be moved into a helper function in @internal/auth/jwt.ts
| ? byteHasherStream!.toReadable({ autoCleanup: true }) | ||
| : (body as Readable) | ||
|
|
||
| const { secret: jwtSecret, jwks } = await getJwtSecret(request.tenantId) |
There was a problem hiding this comment.
If this should use the same signing secret as URL signing (not sure if that's the intention or not) you should use urlSigningKey instead of secret which uses the JWK signing key if set otherwise falls back to jwtSecret inside getJwtSecret().
| ChunkSignatureV4Parser, | ||
| V4StreamingAlgorithm, | ||
| } from '@storage/protocols/s3/signature-v4-stream' | ||
| // @ts-expect-error - no types for compose |
There was a problem hiding this comment.
Probably better to manually declare typedef for compose and remove this bypass.
Adding this down by the fastify declare (line 38) seems to resolve it and maintain stricter typing
declare module 'stream' {
export function compose<A extends Stream, B extends Stream>(s1: A, s2: B): B & A;
}|
|
||
| const indexResult = await request.s3Vector.listVectors(request.body) | ||
|
|
||
| return response.send(indexResult) |
There was a problem hiding this comment.
Might be nice to align this with the existing listObjectsV2 naming for consistency.
- include
hasNextboolean in response - rename to
cursor(request) andnextCursor(response)
| } | ||
|
|
||
| export default async function routes(fastify: FastifyInstance) { | ||
| fastify.post<listIndexRequest>( |
There was a problem hiding this comment.
same as listVectors. Align pagination types with existing list V2
| } | ||
| } | ||
|
|
||
| throw ERRORS.TransactionError('Transaction failed after maximum retries', lastError) |
There was a problem hiding this comment.
Might be nice to make "after maximum retries" conditional on if there were retires, and/or maybe include the retry count in the error message to avoid confusion if we need to debug this in the future.
| return { | ||
| vectorBucket: { | ||
| vectorBucketName: vectorBucket.id, | ||
| creationTime: Math.floor(vectorBucket.created_at?.getTime() / 1000), |
There was a problem hiding this comment.
check vectorBucket.created_at is typeof Date to avoid error/NaN
| return { | ||
| vectorBuckets: bucketResult.vectorBuckets.map((bucket) => ({ | ||
| vectorBucketName: bucket.id, | ||
| creationTime: Math.floor(bucket.created_at?.getTime() / 1000), |
There was a problem hiding this comment.
check vectorBucket.created_at is typeof Date to avoid error/NaN
3 participants
What kind of change does this PR introduce?
Feature
What is the new behaviour?
Implement Vector Bucket data source
Supported Operations:
Authentication mechanisms: