firestore: bulkwriter loses documents

[alsofr]

Client

firestore

Environment

go 1.23.4

Code and Dependencies

Here is a simplified example of the code generating the faulty behaviour.
Call the function with a larger slice of objects to persist (several thousands).

func (c *Client) BulkCreate(ctx context.Context, collID string, src any) error {
	if reflect.TypeOf(src).Kind() != reflect.Slice {
		return errors.New("bulk create: src must be a slice")
	}

	w := c.delegate.BulkWriter(ctx)
	defer w.End() // This will Flush all the remaining documents at the end

	docs := reflect.ValueOf(src)
	collRef := c.delegate.Collection(collID)

	for i := 0; i < docs.Len(); i++ {
		bwj, err := w.Create(collRef.NewDoc(), docs.Index(i).Interface())
		if err != nil {
			return fmt.Errorf("bulk create: %w", err)
		}
	}
	return nil
}

go.mod

go 1.23.4

require (
   cloud.google.com/go/firestore v1.17.0
)

Expected behavior

All documents passed to a BulkWriter are written to the database. If some documents fail, an error is thrown.

Actual behavior

When adding documents quickly to a bulkWriter, some documents are skipped silently. The bulk operation succeeds but there is no indication that any data is missing.

Possible explanation/work-around

After reading the source code, the following work-around was created to make sure that no documents are skipped.

		// The maxBatchSize for the bulkwriter is 20 after which the bulkwriter tries to write to the database
		// Wait for the write results before continuing. This prevents silent loss of data.
		if (i+1)%maxBatchSize == 0 {
			if _, err := bwj.Results(); err != nil {
				return fmt.Errorf("bulk create document: %w", err)
			}
		}

The faulty code seems to be the silencing of errors coming from the bundler.Add function:

It's possible that the bundler is throwing an ErrOverflow error when the document addition rate to the queue is faster than the Firestore persistence rate.