Skip to content

Commit b8dca44

Browse files
jiridanekjiridanek
committed
todo: add wip logging notes and configs for vector and fluentd
1 parent f5078c5 commit b8dca44

File tree

6 files changed

+197
-0
lines changed

6 files changed

+197
-0
lines changed

‎docs/notes/logging/fluentd.conf‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
2+
3+
# <source>
4+
# @type forward
5+
# port 24224
6+
# </source>
7+
8+
# https://docs.fluentd.org/configuration/config-file
9+
10+
<source>
11+
# https://docs.fluentd.org/input/tail
12+
@type tail
13+
path sample.log
14+
read_from_head true
15+
#pos_file sample.log.pos
16+
tag skrouterd
17+
# https://docs.fluentd.org/configuration/parse-section
18+
<parse>
19+
# https://docs.fluentd.org/parser/regexp
20+
# https://docs.fluentd.org/parser/regexp#how-to-debug-my-regexp-pattern
21+
@type regexp
22+
# https://ruby-doc.org/core-2.4.1/Regexp.html
23+
#expression /^(?<time>[^ ]+ [^ ]+ [^ ]+) (?<component>[^ ]+) \((?<severity>[^ ]+)\) (?<message>.*)$/
24+
# notice that .*? in message field is lazy capture, but .* would also work here (but not in vector)
25+
expression /^(?<time>[^ ]+ [^ ]+ [^ ]+) (?<component>[^ ]+) \((?<severity>[^ ]+)\) (?<message>.*?)(?: \((?<source>.*:\d+)\))?$/
26+
time_key time
27+
# https://docs.fluentd.org/configuration/parse-section#time-parameters
28+
time_format %Y-%m-%d %H:%M:%S.%N %z
29+
types component:string,level:string,message:string,source:string
30+
</parse>
31+
</source>
32+
33+
<source>
34+
# https://docs.fluentd.org/input/tail
35+
@type tail
36+
path sample_locations.log
37+
read_from_head true
38+
#pos_file sample_locations.log.pos
39+
tag skrouterd
40+
# https://docs.fluentd.org/configuration/parse-section
41+
<parse>
42+
# https://docs.fluentd.org/parser/regexp
43+
# https://docs.fluentd.org/parser/regexp#how-to-debug-my-regexp-pattern
44+
@type regexp
45+
expression /^(?<time>[^ ]+ [^ ]+ [^ ]+) (?<component>[^ ]+) \((?<severity>[^ ]+)\) (?<message>.*) \((?<source>.*:\d+)\)$/
46+
time_key time
47+
# https://docs.fluentd.org/configuration/format-section#time-parameters
48+
time_format %Y-%m-%d %H:%M:%S.%N %z
49+
types user_id:integer,paid:bool,paid_usd_amount:float
50+
</parse>
51+
</source>
52+
53+
54+
<match skrouterd>
55+
@type file
56+
path fluent.out
57+
</match>
58+
59+
# https://docs.fluentd.org/configuration/config-file#embedded-ruby-code

‎docs/notes/logging/readme.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Logging
2+
3+
Skupper Router produces two kinds of logs. First, the operation logs and then flow logs.
4+
5+
This document deals with the operation logs only.
6+
7+
## OpenShift
8+
9+
[OpenShift 4.12 by default uses Fluentd, Elasticsearch, and Kibana](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.12/html/logging/cluster-logging) for its logging subsystem.
10+
[Fluentd collects the logs](https://docs.openshift.com/container-platform/4.12/logging/config/cluster-logging-collector.html), then
11+
[Elasticsearch is used to store the collected data](https://docs.openshift.com/container-platform/4.12/logging/config/cluster-logging-log-store.html), and finally there is
12+
[Kibana to visualize the collected data](https://docs.openshift.com/container-platform/4.12/logging/cluster-logging-visualizer.html).
13+
14+
In the above, Fluentd can be [replaced by Vector](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.12/html/logging/cluster-logging#cluster-logging-about-vector_cluster-logging), and
15+
[an external log store can substitute for Logstash](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.12/html/logging/cluster-logging#cluster-logging-forwarding-about_cluster-logging).
16+
17+
### Log parsing
18+
19+
At some point throughout the log processing pipeline, the logs, which skrouterd produces in a plain text format, need to be parsed.
20+
Every component of the pipeline is capable of performing this step
21+
Fluentd [<parse> directive](https://docs.fluentd.org/configuration/parse-section)
22+
Vector [remap with the VRL language](https://vector.dev/docs/reference/vrl/) or transform with an [arbitrary lua program](https://vector.dev/docs/reference/configuration/transforms/lua/)
23+
Elasticsearch [Grok filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)
24+
25+
#### Date and time
26+
27+
ruby -e 'require "time"; puts Time.strptime("2023-03-12 11:54:24.084418 +0100", "%Y-%m-%d %H:%M:%S.%N %z")'
28+
29+
In general, the log processing pipeline may consists from gathering the log messages,
30+
31+
### Log collection
32+
33+
#### Fluentd
34+
35+
```shell
36+
sudo dnf install -y ruby-devel
37+
gem install fluentd --no-doc
38+
```
39+
40+
Check the config syntax with `fluentd --dry-run -c fluent.conf`
41+
42+
Test with
43+
44+
fluentd -c fluentd.conf
45+
46+
#### Vektor
47+
48+
```shell
49+
sudo dnf install -y https://packages.timber.io/vector/0.28.1/vector-0.28.1-1.$(arch).rpm
50+
```
51+
52+
Test with
53+
54+
vector -c vector.conf < sample.log
55+
56+
### Visualization
57+
58+
## Kibana
59+
60+
### Putting it all together
61+
62+
63+
64+
65+
66+
[]{}

‎docs/notes/logging/sample.log‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
2023-03-12 11:54:24.077669 +0100 SERVER (info) Container Name: Standalone_ZIsKPjNEXUzhcok
2+
2023-03-12 11:54:24.078884 +0100 ROUTER (info) Router started in Standalone mode
3+
2023-03-12 11:54:24.078984 +0100 ROUTER (info) Version: 2.1.0-156-g2a85d022
4+
2023-03-12 11:54:24.080946 +0100 ROUTER_CORE (info) Streaming link scrubber: Scan interval: 30 seconds, max free pool: 128 links
5+
2023-03-12 11:54:24.081033 +0100 ROUTER_CORE (info) Core module enabled: streaming_link_scrubber
6+
2023-03-12 11:54:24.081103 +0100 ROUTER_CORE (info) Core module present but disabled: mobile_sync
7+
2023-03-12 11:54:24.081208 +0100 ROUTER_CORE (info) Stuck delivery detection: Scan interval: 30 seconds, Delivery age threshold: 10 seconds
8+
2023-03-12 11:54:24.081326 +0100 ROUTER_CORE (info) Core module enabled: stuck_delivery_detection
9+
2023-03-12 11:54:24.082128 +0100 ROUTER_CORE (info) Core module enabled: heartbeat_server
10+
2023-03-12 11:54:24.082205 +0100 ROUTER_CORE (info) Core module present but disabled: heartbeat_edge
11+
2023-03-12 11:54:24.082301 +0100 ROUTER_CORE (info) Core module enabled: address_lookup_client
12+
2023-03-12 11:54:24.082352 +0100 ROUTER_CORE (info) Core module present but disabled: mesh_discovery_interior
13+
2023-03-12 11:54:24.082411 +0100 ROUTER_CORE (info) Core module present but disabled: mesh_discovery_edge
14+
2023-03-12 11:54:24.082473 +0100 ROUTER_CORE (info) Core module present but disabled: edge_addr_tracking
15+
2023-03-12 11:54:24.082528 +0100 ROUTER_CORE (info) Core module present but disabled: core_test_hooks
16+
2023-03-12 11:54:24.082578 +0100 ROUTER_CORE (info) Core module present but disabled: edge_router
17+
2023-03-12 11:54:24.083734 +0100 FLOW_LOG (info) Protocol logging started
18+
2023-03-12 11:54:24.084008 +0100 ROUTER_CORE (info) Protocol adaptor registered: tcp
19+
2023-03-12 11:54:24.084180 +0100 ROUTER_CORE (info) Protocol adaptor registered: http2
20+
2023-03-12 11:54:24.084418 +0100 ROUTER_CORE (info) Protocol adaptor registered: http/1.x
21+
2023-03-12 11:54:24.085748 +0100 ROUTER_CORE (info) Router Core thread running. 0/Standalone_ZIsKPjNEXUzhcok
22+
2023-03-12 11:54:24.085979 +0100 ROUTER_CORE (info) Protocol adaptor registered: amqp
23+
2023-03-12 11:54:24.086084 +0100 ROUTER_CORE (info) In-process subscription M/$management
24+
2023-03-12 11:54:24.105875 +0100 AGENT (info) Activating management agent on $_management_internal
25+
2023-03-12 11:54:24.105934 +0100 ROUTER_CORE (info) In-process subscription L/$management
26+
2023-03-12 11:54:24.106039 +0100 ROUTER_CORE (info) In-process subscription M/sfe.VbCkR:0
27+
2023-03-12 11:54:24.106079 +0100 ROUTER_CORE (info) In-process subscription L/$_management_internal
28+
2023-03-12 11:54:24.107232 +0100 POLICY (info) Policy configured maxConnections: 65535, policyDir: '',access rules enabled: 'false', use hostname patterns: 'false'
29+
2023-03-12 11:54:24.108581 +0100 POLICY (info) Policy fallback defaultVhost is defined: '$default'
30+
2023-03-12 11:54:24.112510 +0100 CONN_MGR (info) Configured Listener: 0.0.0.0:amqp proto=any, role=normal
31+
2023-03-12 11:54:24.113131 +0100 SERVER (notice) Operational, 4 Threads Running (process ID 2151406)
32+
2023-03-12 11:54:24.113479 +0100 SERVER (notice) Process VmSize 20.00 TiB (31.11 GiB available memory)
33+
2023-03-12 11:54:24.113508 +0100 SERVER (info) Running in DEBUG Mode
34+
2023-03-12 11:54:24.114054 +0100 SERVER (notice) Listening on 0.0.0.0:amqp

‎docs/notes/logging/sample_locations.log‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2023-03-11 13:56:17.661912 +0000 HTTP_ADAPTOR (info) Listener httpListener/0.0.0.0:24162: stopped listening for client connections on 0.0.0.0:24162 (/build/src/adaptors/adaptor_listener.c:168)

‎docs/notes/logging/skrouterd_vector.vrl‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# https://vector.dev/docs/reference/vrl
2+
# https://vector.dev/docs/reference/vrl/#parse-custom-logs
3+
4+
# language documentation at https://vrl.dev
5+
# https://vector.dev/docs/reference/vrl/expressions/
6+
# try your code in the VRL REPL, learn more at https://vrl.dev/examples
7+
8+
# https://vector.dev/docs/reference/vrl/functions/#parse_regex
9+
# https://docs.rs/regex/latest/regex/#syntax
10+
# see also https://vector.dev/docs/reference/vrl/functions/#parse_grok
11+
12+
#. = parse_regex!(.message, r'^(?P<timestamp>[^ ]+ [^ ]+ [^ ]+) (?P<component>[^ ]+) \((?P<severity>[^ ]+)\) (?P<message>.*)$')
13+
#. = parse_regex!(.message, r'^(?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+\.\d+ \+\d+) (?P<component>[^ ]+) \((?P<severity>[^ ]+)\) (?P<message>.*)$')
14+
15+
# uses lazy capture in message field
16+
. = parse_regex!(.message, r'^(?P<timestamp>[^ ]+ [^ ]+ [^ ]+) (?P<component>[^ ]+) \((?P<severity>[^ ]+)\) (?P<message>.*?)(?: \((?P<source>.*:\d+)\))?$')
17+
#. = parse_regex!(.message, r'^(?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+\.\d+ \+\d+) (?P<component>[^ ]+) \((?P<severity>[^ ]+)\) (?P<message>.*?)(?: \((?P<source>.*:\d+)\))?$')
18+
19+
# Coerce parsed fields
20+
.timestamp = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S.%N %z") ?? now()

‎docs/notes/logging/vector.conf‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# https://vector.dev/docs/reference/configuration/
2+
3+
# https://vector.dev/docs/reference/configuration/sources/
4+
[sources.skrouterd_logs]
5+
type = "stdin"
6+
7+
# https://vector.dev/docs/reference/configuration/transforms/
8+
[transforms.skrouterd_parser]
9+
inputs = ["skrouterd_logs"]
10+
type = "remap"
11+
file = "skrouterd_vector.vrl"
12+
13+
# https://vector.dev/docs/reference/configuration/sinks/
14+
[sinks.console]
15+
inputs = ["skrouterd_parser"]
16+
type = "console"
17+
encoding.codec = "json"

0 commit comments

Comments
 (0)