gNMI - TLS handshake failure on Ciena devices

[whizkidTRW]

Relevant telegraf.conf

[[inputs.gnmi]]
  interval = "5m"
  alias = "ciena-gnmi"
  addresses = [ "XX.XX.XX.XX:6702" ]
      
  username = "XXXXXXXXXXXXXX"
  password = "XXXXXXXXXXXXXX"
      
  encoding = "proto"
  redial = "10s"
  tls_enable = true
  insecure_skip_verify = true
  tls_ca = "/etc/telegraf/ciena-ca.cert.pem"
  tls_cert = "/etc/telegraf/ciena-client.cert.pem"
  tls_key = "/etc/telegraf/ciena-client.key.pem"
  name_override = "saos10xgnmi"
  updates_only = true

  fieldpass = ["path","source","name", "in_crc_error_pkts", "in_discards", "in_errors", "in_octets", "out_errors", "out_octets"]
  tagexclude = ["path","name"]

  [[inputs.gnmi.subscription]]
     name = "ifcounters"
     origin = "Ciena"
     path = "/oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters"
     subscription_mode = "sample"
     sample_interval = "30s"

Logs from Telegraf

Broken starting with the 1.29.2 release (through latest):
---------------------------------------------------------
telegraf  | 2025-02-04T14:40:45Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf  | 2025-02-04T14:40:45Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf  | 2025-02-04T14:40:45Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 2.0.0: use 'fieldinclude' instead
telegraf  | 2025-02-04T14:40:45Z I! Starting Telegraf 1.29.2 brought to you by InfluxData the makers of InfluxDB
telegraf  | 2025-02-04T14:40:45Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
telegraf  | 2025-02-04T14:40:45Z I! Loaded inputs: gnmi
telegraf  | 2025-02-04T14:40:45Z I! Loaded aggregators: 
telegraf  | 2025-02-04T14:40:45Z I! Loaded processors: converter rename strings
telegraf  | 2025-02-04T14:40:45Z I! Loaded secretstores: 
telegraf  | 2025-02-04T14:40:45Z W! Outputs are not used in testing mode!
telegraf  | 2025-02-04T14:40:45Z I! Tags enabled: host=10.5.200.224
telegraf  | 2025-02-04T14:40:45Z D! [agent] Initializing plugins
telegraf  | 2025-02-04T14:40:45Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/oc-if:interface/oc-if:state/oc-if:counters:ifcounters]
telegraf  | 2025-02-04T14:40:45Z D! [agent] Starting service inputs
telegraf  | 2025-02-04T14:40:45Z E! [inputs.gnmi::ciena-gnmi] Error in plugin: failed to setup subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
telegraf  | 2025-02-04T14:40:55Z E! [inputs.gnmi::ciena-gnmi] Error in plugin: failed to setup subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/0
 ✔ Container telegraf  Stopped                                                                                                                                                                                                



Exact same config works fine up to 1.29.1 release:
--------------------------------------------------
telegraf  | 2025-02-04T14:41:15Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf  | 2025-02-04T14:41:15Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf  | 2025-02-04T14:41:15Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 2.0.0: use 'fieldinclude' instead
telegraf  | 2025-02-04T14:41:15Z I! Starting Telegraf 1.29.1 brought to you by InfluxData the makers of InfluxDB
telegraf  | 2025-02-04T14:41:15Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
telegraf  | 2025-02-04T14:41:15Z I! Loaded inputs: gnmi
telegraf  | 2025-02-04T14:41:15Z I! Loaded aggregators: 
telegraf  | 2025-02-04T14:41:15Z I! Loaded processors: converter rename strings
telegraf  | 2025-02-04T14:41:15Z I! Loaded secretstores: 
telegraf  | 2025-02-04T14:41:15Z W! Outputs are not used in testing mode!
telegraf  | 2025-02-04T14:41:15Z I! Tags enabled: host=10.5.200.224
telegraf  | 2025-02-04T14:41:15Z D! [agent] Initializing plugins
telegraf  | 2025-02-04T14:41:15Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/oc-if:interface/oc-if:state/oc-if:counters:ifcounters]
telegraf  | 2025-02-04T14:41:15Z D! [agent] Starting service inputs
telegraf  | 2025-02-04T14:41:15Z D! [inputs.gnmi::ciena-gnmi] Connection to gNMI device 10.255.32.14:6702 established
telegraf  | > interface,agent_host=10.255.32.14,host=10.5.200.224,ifIndex=1 ifHCInOctets=94533575503546i,ifHCOutOctets=873939187818389i,ifInCrcErrors=0i,ifInDiscards=236i,ifInErrors=0i,ifOutErrors=0i 1738680087844000000
telegraf  | > interface,agent_host=10.255.32.14,host=10.5.200.224,ifIndex=2 ifHCInOctets=94569510275635i,ifHCOutOctets=874902982673682i,ifInCrcErrors=0i,ifInDiscards=0i,ifInErrors=0i,ifOutErrors=0i 1738680087844000000

. . . (output trimmed fo clarity)

Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/0
 ✔ Container telegraf  Stopped

System info

Telegraf 1.29.2+, Docker 4.37.2, MacOS 15.2 (m4-Max)

Docker

services:
  telegraf:
    image: telegraf:1.29.2-alpine
    container_name: telegraf
    restart: no
    command: telegraf --debug --test-wait 45
    volumes:
      - /etc/snmp:/etc/snmp:ro
      - ./mibs:/usr/share/snmp/mibs:rw
      - ./telegraf/etc:/etc/telegraf:rw
    ports: 
      - '8125:8125'
    logging:
      options:
        max-size: "1m"
        max-file: "5"

Steps to reproduce

1. Start with Telegraf 1.29.2
2. Subscribe to a Ciena device
3. TLS Authentication fails (certs are valid / vendor supplied, don't expire until 2050)
4. Revert to Telegraf 1.29.1, connection works fine

Expected behavior

TLS handshake is expected to still work with known good config from 1.29.1 to subsequent versions

Actual behavior

TLS handshake breaks starting in 1.29.2

Additional info

Confirmed to be working with my Cisco IOS-XR devices, so this problem is unique to Ciena.