Bigquery Output Plugin HTTP buffer size is not configurable. #9061
Description
Is your feature request related to a problem? Please describe.
BigQuery output plugin's buffer size is 4096 which is way too small. I'm using the stackdriver output plugin, and it's working with no issues. Google Cloud has a default dumb limit of 60 read requests per minute, which doesn't allow me to pull from the logs.
In the meantime I wanna upload things to bigquery, and I wanted to use the bigquery output plugin for that.
I can see in the bigquery.c file that the HTTP buffer size is fixed to 4096 (Bytes I'm assuming), running that with my pod I'm getting this:
[2024/07/09 00:18:21] [ warn] [http_client] cannot increase buffer: current=4192 requested=36960 max=4192
[2024/07/09 00:18:21] [ warn] [http_client] cannot increase buffer: current=4192 requested=36960 max=4192
Describe the solution you'd like
I can see that other plugins allow for the user to set the HTTP buffer (kubernetes filter, for example.) Would be great to have that here as well.
Describe alternatives you've considered
Ditch fluentbit & write my own service.
Here's my cconfig:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Buffer_Chunk_Size 64KB
Buffer_Max_Size 128KB
[OUTPUT]
Name stackdriver
Match kube.*
resource k8s_container
k8s_cluster_name CLUSTER_NAME
k8s_cluster_location CLUSTER_LOCATION
google_service_credentials /fluent-bit/service-account.json
[OUTPUT]
Name bigquery
Match kube.*
google_service_credentials /fluent-bit/service-account.json
project_id YOUR_PROJECT_ID
dataset_id YOUR_DATASET_ID
table_id YOUR_TABLE_ID
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L