out_stackdriver: Modify tag value of log entry based on the value stream #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Example configuration file for the changefluent-bit.conf [SERVICE]
Flush 1
Daemon Off
Log_Level debug
Parsers_File myparser.conf
[INPUT]
Name tail
Tag k8s_container.test
Parser JSON
Path /home/jeffluoo/fluentbit/testContainer.json
[OUTPUT]
Name stackdriver
Match k8s_container.*
resource k8s_container
k8s_cluster_name k8s_cluster_name_test
k8s_cluster_location k8s_cluster_location_test
[FILTER]
Name record_modifier
Match k8s_container.*
Record logging.googleapis.com/local_resource_id k8s_container.namespace_name_test.podt_name_test.container_name_testmyparser.conf [PARSER]
Name JSON
Format json/home/jeffluoo/fluentbit/testContainer.json {"stream":"stderr","time":"2019-03-09T13:52:36.806411254Z","log":"test Log Message","kubernetes":{"labels":{"controller-revision-hash":"foorbarr","pod-template-generation":"1","app":"logging-availability-test"},"namespace_name":"test-components","docker_id":"testDockerId","container_name":"logging-test","pod_name":"logging-availability-test-v8gwl","annotations":{"prometheus.io\/path":"\/","prometheus.io\/scrape":"true"},"host":"gke-test-host","pod_id":"test-pod-id"}} |
Debug log output from testing the change |
Attached Valgrind output that shows no leaks or memory corruption was found |
Signed-off-by: Jeff Luo <[email protected]>
erain
reviewed
Jun 19, 2020
JeffLuoo
commented
Jun 19, 2020
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
Signed-off-by: Jeff Luo <[email protected]>
JeffLuoo
force-pushed
the
feature/modify_tag_value
branch
from
c31f2bb to
9b24ad2
Compare
JeffLuoo
force-pushed
the
feature/k8s_container
branch
5 times, most recently
from
0680c24 to
7f663f2
Compare
JeffLuoo
force-pushed
the
feature/k8s_container
branch
from
7f663f2 to
d88101d
Compare
JeffLuoo
force-pushed
the
feature/k8s_container
branch
from
6419928 to
a111c08
Compare
JeffLuoo
force-pushed
the
feature/k8s_container
branch
from
a111c08 to
2e451c8
Compare
JeffLuoo
force-pushed
the
feature/k8s_container
branch
from
b9cac4e to
94dae4e
Compare
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
JeffLuoo
pushed a commit
that referenced
this pull request
Jun 16, 2021
output_thread patch #1 : pipe creation is platform dependent, in unixes it's achieved through the pipe syscall but in windows it's a socket pair created through libevent which means instead of read/write we need to use recv/send which was already abstracted through flb_pipe_(r/w). output_thread patch #2 : in windows the SOCKET data type is defined as an UINT_PTR which means in 64 bit operating systems it's an 8 byte number instead of 4. libevent abstracts this through the evutil_socket_t data type which in turn is abstracted using the flb_pipefd_t data type in fluent bit. The problem comes to play when calling mk_event_channel_create which receives 2 int pointers to return both pipe endpoints. This means there are 2 possible bugs (one of which was happening and the other is not really a concern I think) : Since mk_event_channel_create was only modifying the low part of the elements in the ch_parent_events array and the structure was not zeroed when allocated, the high parts of those 64 bit members contained garbage which caused any winsock calls to return error 10038 (not a socket). That's the reason why I added a memset call in line 421. There is still a possible issue with mk_event_channel_create which we should fix by moving our platform dependent data type to mk_lib and defining our flb local data types to those if need be (for consistency). Signed-off-by: Leonardo Alminana <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One feature of stackdriver output plugin in fluentd is that we will provide more information on log name for the stream of the log. This feature is missing in fluent bit and this patch will add this feature.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Documentation
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.