Memcached protocol doesn't work

[yizhengx]

Hi, we implemented memcached protocol in aeraki and used DeathStarBenchmark/hotel-reservation to test.

Meta-protocol-proxy implementation: https://github.com/yizhengx/meta-protocol-proxy/tree/local-queue/src/application_protocols/memcached

We built it using bazel following the instructions in the document.

Example yamls for memcached services

• deployment

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      kompose.cmd: kompose convert
      kompose.version: 1.22.0 (955b78124)
    creationTimestamp: null
    labels:
      io.kompose.service: memcached-rate
    name: memcached-rate
  spec:
    replicas: 1
    selector:
      matchLabels:
        io.kompose.service: memcached-rate
    strategy: {}
    template:
      metadata:
        annotations:
          kompose.cmd: kompose convert
          kompose.version: 1.22.0 (955b78124)
          sidecar.istio.io/statsInclusionPrefixes: cluster.outbound,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grp,listener,connection_manager
          sidecar.istio.io/statsInclusionRegexps: http.*
          sidecar.istio.io/bootstrapOverride: aeraki-bootstrap-config
          sidecar.istio.io/proxyImage: yizhengx/aeraki-proxy:rate-limit-cas # custom image we built
        creationTimestamp: null
        labels:
          io.kompose.service: memcached-rate
      spec:
        containers:
          - env:
              - name: MEMCACHED_CACHE_SIZE
                value: "128"
              - name: MEMCACHED_THREADS
                value: "2"
            image: memcached
            name: hotel-reserv-rate-mmc
            ports:
              - containerPort: 11211
            resources:
              requests:
                cpu: 100m
              limits:
                cpu: 1000m
        restartPolicy: Always
  status: {}
  

• service

  apiVersion: v1
  kind: Service
  metadata:
   annotations:
     kompose.cmd: kompose convert
     kompose.version: 1.22.0 (955b78124)
   creationTimestamp: null
   labels:
     io.kompose.service: memcached-rate
   name: memcached-rate
  spec:
   ports:
     - name: tcp-metaprotocol-memcached-memcached-rate
       protocol: TCP
       port: 11211
       targetPort: 11211
   selector:
     io.kompose.service: memcached-rate
  status:
   loadBalancer: {}
  

I really appreciate any advice on this including what we missed and how we can find the root cause. Thanks!

[zhaohuabing]

@yizhengx Can you paste the Envoy config dump?

[yizhengx]

@yizhengx Can you paste the Envoy config dump?

Hi! I'm not sure what exactly you're looking for, but I ran istioctl proxy-status for one of the memcached proxy:

yizhengx@node0:~/aeraki-benchmark/hotel-res$ istioctl proxy-status memcached-profile-6d4b6b647f-nw6rl.hotel-res
Clusters Match
--- Istiod Listeners
+++ Envoy Listeners
@@ -3233,51 +3233,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-rate.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-rate.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "memcached-rate.hotel-res.svc.cluster.local_11211"
-                                                        },
-                                                "statPrefix": "outbound|11211||memcached-rate.hotel-res.svc.cluster.local"
+                                                            "route_config_name": "memcached-rate.hotel-res.svc.cluster.local_11211"
+                                                        },
+                                                "stat_prefix": "outbound|11211||memcached-rate.hotel-res.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -4991,51 +4991,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-reserve.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-reserve.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "memcached-reserve.hotel-res.svc.cluster.local_11211"
-                                                        },
-                                                "statPrefix": "outbound|11211||memcached-reserve.hotel-res.svc.cluster.local"
+                                                            "route_config_name": "memcached-reserve.hotel-res.svc.cluster.local_11211"
+                                                        },
+                                                "stat_prefix": "outbound|11211||memcached-reserve.hotel-res.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -5889,51 +5889,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "memcached-profile.hotel-res.svc.cluster.local_11211"
-                                                        },
-                                                "statPrefix": "outbound|11211||memcached-profile.hotel-res.svc.cluster.local"
+                                                            "route_config_name": "memcached-profile.hotel-res.svc.cluster.local_11211"
+                                                        },
+                                                "stat_prefix": "outbound|11211||memcached-profile.hotel-res.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -6214,51 +6214,51 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "thrift",
+                                                "application_protocol": "thrift",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.thrift"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "thrift-sample-server.meta-thrift.svc.cluster.local"
+                                                                                    "destination_service": "thrift-sample-server.meta-thrift.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
-                                                        },
-                                                "statPrefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local"
+                                                            "route_config_name": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
+                                                        },
+                                                "stat_prefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local"
                                             }
                                     }
                                 }
                             ]
                         }
                     ],
                     "trafficDirection": "OUTBOUND",
@@ -7170,44 +7170,44 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
-                                                "routeConfig": {
+                                                "route_config": {
                                                             "name": "inbound|11211||",
                                                             "routes": [
                                                                         {
                                                                                     "route": {
                                                                                                 "cluster": "inbound|11211||"
                                                                                             }
                                                                                 }
                                                                     ]
                                                         },
-                                                "statPrefix": "inbound|11211||"
+                                                "stat_prefix": "inbound|11211||"
                                             }
                                     }
                                 }
                             ],
                             "transportSocket": {
                                 "name": "envoy.transport_sockets.tls",
                                 "typedConfig": {
@@ -7314,44 +7314,44 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
-                                                "routeConfig": {
+                                                "route_config": {
                                                             "name": "inbound|11211||",
                                                             "routes": [
                                                                         {
                                                                                     "route": {
                                                                                                 "cluster": "inbound|11211||"
                                                                                             }
                                                                                 }
                                                                     ]
                                                         },
-                                                "statPrefix": "inbound|11211||"
+                                                "stat_prefix": "inbound|11211||"
                                             }
                                     }
                                 }
                             ],
                             "name": "0.0.0.0_11211"
                         },
                         {
@@ -7383,44 +7383,44 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "applicationProtocol": "memcached",
+                                                "application_protocol": "memcached",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.memcached"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "memcached-profile.hotel-res.svc.cluster.local"
+                                                                                    "destination_service": "memcached-profile.hotel-res.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
-                                                "routeConfig": {
+                                                "route_config": {
                                                             "name": "inbound|11211||",
                                                             "routes": [
                                                                         {
                                                                                     "route": {
                                                                                                 "cluster": "inbound|11211||"
                                                                                             }
                                                                                 }
                                                                     ]
                                                         },
-                                                "statPrefix": "inbound|11211||"
+                                                "stat_prefix": "inbound|11211||"
                                             }
                                     }
                                 }
                             ],
                             "name": "0.0.0.0_11211"
                         }
                     ],

Routes Match (RDS last loaded at Tue, 30 Apr 2024 22:41:35 CDT)

Output from running istioctl proxy-config for a specific pod

yizhengx@node0:~/aeraki-benchmark/hotel-res$ istioctl proxy-config all -n hotel-res memcached-profile-6d4b6b647f-nw6rl
Error: error unmarshalling config dump response from Envoy: proto: (line 21366:13): unable to resolve "type.googleapis.com/aeraki.meta_protocol_proxy.admin.v1alpha.RoutesConfigDump": "not found"

[zhaohuabing]

@yizhengx The config dump looks good to me since the original TCP_Filter has been replaced with Meta_protocol_filter and the codec is "name": "aeraki.meta_protocol.codec.memcached".

For the next step, I suggest you turn on trace and look into the log to see how a memchace request is processed and what could be potentially wrong.