[Bug] Issue with data prep leads to incorrectly sized vectors

[zeke-emerson]

Is this a new bug?

• I believe this is a new bug
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Presently, when I run npm run index to try to create and populate the index, I receive the following error on every attempted upsert (up until I force quit):

Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 61440 does not match the dimension of the index 384]

I've attached a sample output error file as well: errors.txt

Expected Behavior

The repository should run as is and properly create the index. If I am improperly running the project, then the other open issue should be addressed to provide clearer usage instructions.

Steps To Reproduce

1. Clone the repository.
2. Create your .env file and run npm install.
3. Run npm run index to initiate creating the index.
4. Observe the error messages.

Relevant log output

Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 61440 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 99072 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 56448 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 52992 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 50304 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 100224 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 48384 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 40704 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 78336 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 96000 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 76800 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 40320 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 95616 does not match the dimension of the index 384]
Error upserting chunk [PineconeError: PineconeClient: Error calling upsert: PineconeError: Vector dimension 47616 does not match the dimension of the index 384]

Environment

No response

Additional Context

No response

[junefish]

Having the same issue in my project.