google-fluent silently ignores structured log entries with improper time field

[jacek-jablonski]

Hi,
I am using google-fluentd version 1.11.2. I've noticed that google-fluentd silently ignores log entries with improper time field. Here is configuration:

<source>
  @type tail
  format json
  path /home/jenkins/workspace/*/storage/logs/*.log
  pos_file /var/lib/google-fluentd/pos/jenkins.pos
  read_from_head true

  tag jobs-logs
</source>

<filter jobs-logs*>
  @type record_transformer
  <record>
    severity ${record["severity"]}
    service "jobs"
  </record>
</filter>

Sample log entry looks like that:

{
  "log_name": "big_query",
  "message": "new request",
  "message_template": "new request",
  "time": "2020-11-23T18:57:08.694567+00:00",
  "severity": "INFO",
  "stack": {
    "thread": 140232798734080,
    "process": 6,
    "exc_info": null,
    "exc_text": null,
    "stack_info": null
  },
  "extra": {
    "req": {
      "query": "...",
      "project": "xxx",
      "params": "xxx"
    },
    "res": {
      "state": "DONE",
      "duration": 1.209,
      "latency": 1.552014112,
      "tb_billed": 1.0486e-05,
      "cost_uds": 5,
      "cache_hit": false,
      "errors": null
    },
    "extra_data": {
      "job_name": "xxx"
    }
  }
}

The problem here is probably with the time field. By default google-fluentd expects a float. However, these log entries are not visible in LogsViewer and there is no information in logs or in a verbose mode that such log entries are dropped.

When I add to the configuration that time_type string and also some (invalid for test case) time_format %Y-%-m..., then google-fluentd correctly reports that time_format is invalid.

I think google-fluentd should also report about the first situation. Without any information, it made debugging missing logs very difficult.

[scr4bble]

I have the same problem though in my case the time field was always present, just not recognized by fluentd collector.

"time":"2021-01-21T21:19:27.935+00:00"

I managed to fix it by adding time_format %iso8601 into the configuration.

<source>
  @type tail
  <parse>
    @type json
    time_format %iso8601
  </parse>
  path xxx.log
  pos_file /var/lib/google-fluentd/pos/xxx.pos
  read_from_head true
  tag xxx
</source>

Please produce some error in the google-fluentd when the failed parsing results into ignored log lines!
(so it is visible in systemctl status google-fluentd.service and one doesn't have to spend hours debugging what is wrong).