feat: Add Kubernetes Service Discovery Readiness Check

[ChuanFF]

Description

Currently, APISIX may serve traffic before completing the initial sync from Kubernetes service discovery, potentially causing routing errors.

I'd like to add a readiness check to ensure APISIX only reports as ready after successfully pulling K8s data. Would extending apisix.status_ready() to include this check be appropriate?

I'm ready to implement this via PR.

[Baoyuantop]

Hi @ChuanFF, thanks for your report. Could you please first provide your detailed technical proposal?

[ChuanFF]

I propose setting a flag in the shared_dict to indicate that APISIX has completed data ingestion in apisix.discovery.kubernetes.post_list(). The post_list function is called after APISIX pulls Kubernetes endpoints data. We can then check whether this flag has been written into every relevant shared_dict. The readiness check code is as follows:

local function check_ready(dict_name)
    local endpoint_dict = ngx.shared[dict_name]
    if not endpoint_dict then
        core.log.error("failed to get lua_shared_dict:", dict_name, ", please check your APISIX version")
        return false, "failed to get lua_shared_dict: ", dict_name, ", please check your APISIX version"
    end
    -- check flag
    local ready = endpoint_dict:get("discovery_ready")
    if not ready then
        return false, "discovery not ready"
    end
    return true
end

local function single_mode_check_discovery_ready()
    local _, err = check_ready(dict_base_name)
    if err then
        core.log.warn(err)
        return 500, err
    end
    return 200, "discovery ready"
end

local function multiple_mode_check_discovery_ready(confs)
    for _, conf in ipairs(confs) do
        local id = conf.id
        local dict_name = dict_base_name .. "-" .. id
        local _, err = check_ready(dict_name)
        if err then
            return 500, err
        end
    end
    return 200, "discovery ready"
end

local function check_discovery_ready()
    local discovery_conf = local_conf.discovery and local_conf.discovery.kubernetes
    if not discovery_conf then
        return 200, "discovery disable"
    end
    if #discovery_conf == 0 then
        return single_mode_check_discovery_ready()
    else
        return multiple_mode_check_discovery_ready(discovery_conf)
    end
end

[Baoyuantop]

It appears we must first complete #12288, otherwise post_list will not be called correctly.

[Baoyuantop]

The informer_factory maintains fetch_state during the List/Watch process: "connecting" → "listing" → "list finished" → "watching".
Can we start a lightweight timer in init.lua to periodically read handle.fetch_state? When list finished is first observed, write discovery_ready=true to the corresponding shared dictionary and mark it as set (to avoid duplicates). This way, we wouldn't rely on fixing the pre_list function. What do you think?

[ChuanFF]

Using a timer approach introduces some latency (the interval for apisix.timers.register_timer is set to 1s), and the additional timer makes the solution less elegant. Perhaps we can delay the Kubernetes service discovery readiness feature until the post_list issue is resolved?

[Baoyuantop]

I agree with that.