The Zeta server cannot find other members via Kubernetes since after bump a version from 2.3.5 to 2.3.7 #7750

loustler · 2024-09-26T04:53:27Z

loustler
Sep 26, 2024

Hi folks

A Zeta server couldn't find other members via Kubernetes API since after bump a version from 2.3.5 to 2.3.7
I've glanced over a PR #6947 I think related with this issue, but I couldn't find any problems.
Below a configuration is still valid since after 2.3.6?
It's working with a same configuration in 2.3.5.

Starting a cluster like this

/opt/seatunnel/bin/seatunnel-cluster.sh

hazelcast.yaml

hazelcast:
      cluster-name: seatunnel
      network:
        rest-api:
          enabled: true
          endpoint-groups:
            CLUSTER_WRITE:
              enabled: true
            DATA:
              enabled: true
        join:
          multicast:
            enabled: false
          kubernetes:
            enabled: true
            service-port: 5801
            namespace: MY_NAMESPACE
            service-name: seatunnel
        port:
          auto-increment: false
          port: 5801
      properties:
        hazelcast.logging.type: log4j2
        hazelcast.invocation.max.retry.count: 100
        hazelcast.invocation.retry.pause.millis: 1000
        hazelcast.tcp.join.port.try.count: 30
        hazelcast.slow.operation.detector.stacktrace.logging.enabled: true
        hazelcast.operation.generic.thread.count: 50
        hazelcast.client.invocation.timeout.seconds: 3600

Thanks for all your supports

Answered by loustler

Sep 27, 2024

I created #7757

View full answer

loustler · 2024-09-26T05:02:57Z

loustler
Sep 26, 2024
Author

Do you mind if I ask this issue? @EricJoy2048

0 replies

Hisoka-X · 2024-09-27T03:46:50Z

Hisoka-X
Sep 27, 2024
Collaborator

have you tried with this way https://seatunnel.apache.org/docs/start-v2/kubernetes/ ?

17 replies

loustler Sep 27, 2024
Author

A both versions are work with a Discovery SPI, I think it's not the root cause

https://github.com/hazelcast/hazelcast/blob/966c57c236a7de898824ea9f5a38bfdd5f79995d/hazelcast/src/main/java/com/hazelcast/instance/impl/Node.java#L841-L857

Below log from 2.3.5

INFO: Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2024-09-27 14:34:43,531 INFO  [o.a.s.e.s.SeaTunnelServer     ] [main] - SeaTunnel server start...
2024-09-27 14:34:43,533 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2024-09-27 14:34:43,533 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Cluster name: seatunnel
2024-09-27 14:34:43,533 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1]

 _____               _____                             _
/  ___|             |_   _|                           | |
\ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
 `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
/\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
\____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|


2024-09-27 14:34:43,533 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2024-09-27 14:34:43,533 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-09-27 14:34:43,537 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-09-27 14:34:44,099 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: seatunnel, service-port: 5801, service-label: null, service-label-value: true, namespace: default, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, expose-externally-mode: AUTO, use-node-name-as-external-address: false, service-per-pod-label: null, service-per-pod-label-value: null, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2024-09-27 14:34:44,481 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Kubernetes Discovery activated with mode: KUBERNETES_API
2024-09-27 14:34:44,483 INFO  [c.h.s.security                ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2024-09-27 14:34:44,578 INFO  [c.h.i.i.Node                  ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Using Discovery SPI
2024-09-27 14:34:44,583 WARN  [c.h.c.CPSubsystem             ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2024-09-27 14:34:44,905 INFO  [.c.c.DefaultClassLoaderService] [main] - start classloader service with cache mode
Sep 27, 2024 2:34:44 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
Sep 27, 2024 2:34:44 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/seatunnel/config/seatunnel.yaml
Sep 27, 2024 2:34:44 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
Sep 27, 2024 2:34:44 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2024-09-27 14:34:44,930 WARN  [o.a.s.e.s.TaskExecutionService] [pool-2-thread-1] - [THIS_POD]:5801 [seatunnel] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2024-09-27 14:34:44,935 INFO  [o.a.s.e.s.TaskExecutionService] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Created new BusWork : 1891974590
2024-09-27 14:34:44,949 WARN  [a.s.e.s.s.s.DefaultSlotService] [hz.main.seaTunnel.slotService.thread] - failed send heartbeat to resource manager, will retry later. this address: [THIS_POD]:5801
2024-09-27 14:34:44,954 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-6-thread-1] - [THIS_POD]:5801 [seatunnel] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2024-09-27 14:34:45,048 INFO  [c.h.i.d.Diagnostics           ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-09-27 14:34:45,057 INFO  [c.h.c.LifecycleService        ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] [THIS_POD]:5801 is STARTING
2024-09-27 14:34:45,104 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [THIS_POD]:5801 [seatunnel] [5.1] Initialized new cluster connection between /THIS_POD:5801 and /POD_B:37243
2024-09-27 14:34:45,106 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-0] - [THIS_POD]:5801 [seatunnel] [5.1] Initialized new cluster connection between /THIS_POD:5801 and /POD_C:46971
2024-09-27 14:34:45,172 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Cannot fetch the current zone, ZONE_AWARE feature is disabled
2024-09-27 14:34:45,198 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Kubernetes plugin discovered node name: NODE_NAME
2024-09-27 14:34:45,260 WARN  [c.h.k.KubernetesClient        ] [main] - Cannot fetch public IPs of Hazelcast Member PODs, you won't be able to use Hazelcast Smart Client from outside of the Kubernetes network

Below log from 2.3.7

2024-09-27 14:36:13,295 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2024-09-27 14:36:13,295 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2024-09-27 14:36:13,298 INFO  [o.a.s.e.c.c.SeaTunnelConfig   ] [main] - seatunnel.home is /opt/seatunnel
2024-09-27 14:36:13,412 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2024-09-27 14:36:13,412 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2024-09-27 14:36:13,882 INFO  [o.a.s.e.s.SeaTunnelServer     ] [main] - SeaTunnel server start...
2024-09-27 14:36:13,884 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2024-09-27 14:36:13,884 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Cluster name: seatunnel
2024-09-27 14:36:13,884 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1]

 _____               _____                             _
/  ___|             |_   _|                           | |
\ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
 `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
/\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
\____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|


2024-09-27 14:36:13,884 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2024-09-27 14:36:13,884 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-09-27 14:36:13,886 INFO  [c.h.system                    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-09-27 14:36:14,365 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: seatunnel, service-port: 5801, service-label: null, service-label-value: true, namespace: default, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, expose-externally-mode: AUTO, use-node-name-as-external-address: false, service-per-pod-label: null, service-per-pod-label-value: null, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2024-09-27 14:36:14,643 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Kubernetes Discovery activated with mode: KUBERNETES_API
2024-09-27 14:36:14,644 INFO  [c.h.s.security                ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2024-09-27 14:36:14,727 INFO  [c.h.i.i.Node                  ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Using Discovery SPI
2024-09-27 14:36:14,731 WARN  [c.h.c.CPSubsystem             ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2024-09-27 14:36:14,971 INFO  [.c.c.DefaultClassLoaderService] [main] - start classloader service with cache mode
2024-09-27 14:36:14,976 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2024-09-27 14:36:14,976 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2024-09-27 14:36:14,982 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2024-09-27 14:36:14,982 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2024-09-27 14:36:14,998 WARN  [o.a.s.e.s.TaskExecutionService] [pool-2-thread-1] - [THIS_POD]:5801 [seatunnel] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2024-09-27 14:36:15,000 INFO  [o.a.s.e.s.TaskExecutionService] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Created new BusWork : 1758876146
2024-09-27 14:36:15,009 WARN  [a.s.e.s.s.s.DefaultSlotService] [hz.main.seaTunnel.slotService.thread] - failed send heartbeat to resource manager, will retry later. this address: [THIS_POD]:5801
2024-09-27 14:36:15,012 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-6-thread-1] - [THIS_POD]:5801 [seatunnel] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2024-09-27 14:36:15,061 INFO  [c.h.i.d.Diagnostics           ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-09-27 14:36:15,067 INFO  [c.h.c.LifecycleService        ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] [THIS_POD]:5801 is STARTING
2024-09-27 14:36:15,171 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Cannot fetch the current zone, ZONE_AWARE feature is disabled
2024-09-27 14:36:15,198 INFO  [c.h.s.d.i.DiscoveryService    ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Kubernetes plugin discovered node name: NODE_NAME
2024-09-27 14:36:15,198 WARN  [c.h.i.i.Node                  ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] No join method is enabled! Starting standalone.
2024-09-27 14:36:15,210 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [THIS_POD]:5801 [seatunnel] [5.1] This node become a new active master node, begin init coordinator service

I couldn't find sort of this log from 2.3.7

2024-09-27 14:34:45,104 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [THIS_POD]:5801 [seatunnel] [5.1] Initialized new cluster connection between /THIS_POD:5801 and /POD_B:37243

And I found this log from 2.3.7

2024-09-27 14:36:15,198 WARN  [c.h.i.i.Node                  ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] No join method is enabled! Starting standalone.

It means a joiner is not created.

https://github.com/hazelcast/hazelcast/blob/966c57c236a7de898824ea9f5a38bfdd5f79995d/hazelcast/src/main/java/com/hazelcast/instance/impl/Node.java#L817-L822

liunaijie Sep 27, 2024
Collaborator

In 2.3.7, i see the log

2024-09-27 14:36:14,727 INFO  [c.h.i.i.Node                  ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] Using Discovery SPI

When has this log, it will create DiscoveryJoiner.
https://github.com/hazelcast/hazelcast/blob/966c57c236a7de898824ea9f5a38bfdd5f79995d/hazelcast/src/main/java/com/hazelcast/instance/impl/Node.java#L853

But this log means the joiner is null.

2024-09-27 14:36:15,198 WARN  [c.h.i.i.Node                  ] [main] - [THIS_POD]:5801 [seatunnel] [5.1] No join method is enabled! Starting standalone.

It's so wired.

loustler Sep 27, 2024
Author

I resolved this, I'll create a PR

loustler Sep 27, 2024
Author

I created #7757

Answer selected by loustler

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The Zeta server cannot find other members via Kubernetes since after bump a version from 2.3.5 to 2.3.7 #7750

{{title}}

Replies: 2 comments 17 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

The Zeta server cannot find other members via Kubernetes since after bump a version from 2.3.5 to 2.3.7 #7750

loustler Sep 26, 2024

Replies: 2 comments · 17 replies

loustler Sep 26, 2024 Author

Hisoka-X Sep 27, 2024 Collaborator

loustler Sep 27, 2024 Author

liunaijie Sep 27, 2024 Collaborator

loustler Sep 27, 2024 Author

loustler Sep 27, 2024 Author

loustler
Sep 26, 2024

Replies: 2 comments 17 replies

loustler
Sep 26, 2024
Author

Hisoka-X
Sep 27, 2024
Collaborator

loustler Sep 27, 2024
Author

liunaijie Sep 27, 2024
Collaborator

loustler Sep 27, 2024
Author

loustler Sep 27, 2024
Author