tezos: HTTP status 401

[copolycube]

Hello,

the tezos-node exporter is currently failing on both testnets

mainnet : healthy but 401 errors

sudo docker-compose  ps
             Name                            Command                  State                                             Ports
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
mainnet_indexer-db-schema_1       ./db.bash postgresql://ind ...   Exit 0
mainnet_indexer-mempool_1         /home/tezos/utils/run-tezo ...   Up
mainnet_indexer-token-support_1   /home/tezos/utils/run-tezo ...   Up
mainnet_mezos_1                   ./run-mezos.bash run --cha ...   Up (healthy)   0.0.0.0:29090->29090/tcp,:::29090->29090/tcp
mainnet_node_1                    /usr/local/bin/entrypoint. ...   Up (healthy)   0.0.0.0:8733->8733/tcp,:::8733->8733/tcp, 0.0.0.0:9733->9733/tcp,:::9733->9733/tcp
mainnet_postgres-exporter_1       postgres_exporter                Up             0.0.0.0:9187->9187/tcp,:::9187->9187/tcp
mainnet_postgres_1                docker-entrypoint.sh postgres    Up (healthy)   5432/tcp
mainnet_tezos-node-exporter_1     /usr/bin/tezos_exporter -- ...   Up (healthy)   0.0.0.0:9489->9489/tcp,:::9489->9489/tcp

with the following logs

tezos-node-exporter_1    | 2021-11-20T21:34:45.085230043Z time="2021-11-20T21:34:45Z" level=error msg="error getting point stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:34:51.975682978Z time="2021-11-20T21:34:51Z" level=error msg="error getting connections stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:34:51.975840759Z time="2021-11-20T21:34:51Z" level=error msg="error getting peer stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:34:51.975978206Z time="2021-11-20T21:34:51Z" level=error msg="error getting point stats" error="tezos: HTTP status 401"

Granadanet KO

granadanet_tezos-node-exporter_1     /usr/bin/tezos_exporter -- ...   Up (unhealthy)   0.0.0.0:9490->9489/tcp,:::9490->9489/tcp`
 in 

with the following logs

sudo docker-compose  ps
               Name                             Command                   State                                                  Ports
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
granadanet_indexer-db-schema_1       ./db.bash postgresql://ind ...   Exit 0
granadanet_indexer-mempool_1         /home/tezos/utils/run-tezo ...   Up
granadanet_indexer-token-support_1   /home/tezos/utils/run-tezo ...   Up
granadanet_mezos_1                   ./run-mezos.bash run --cha ...   Up (healthy)     0.0.0.0:19090->19090/tcp,:::19090->19090/tcp
granadanet_node_1                    /usr/local/bin/entrypoint. ...   Up (healthy)     0.0.0.0:18732->18732/tcp,:::18732->18732/tcp, 0.0.0.0:19732->19732/tcp,:::19732->19732/tcp
granadanet_postgres-exporter_1       postgres_exporter                Up               0.0.0.0:9188->9187/tcp,:::9188->9187/tcp
granadanet_postgres_1                docker-entrypoint.sh postgres    Up (healthy)     5432/tcp
granadanet_tezos-node-exporter_1     /usr/bin/tezos_exporter -- ...   Up (unhealthy)   0.0.0.0:9490->9489/tcp,:::9490->9489/tcp

tezos-node-exporter_1    | 2021-11-20T21:20:09.977702506Z time="2021-11-20T21:20:09Z" level=error msg="error getting connections stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:20:09.977871569Z time="2021-11-20T21:20:09Z" level=error msg="error getting peer stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:20:09.978039546Z time="2021-11-20T21:20:09Z" level=error msg="error getting point stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:20:26.094371165Z time="2021-11-20T21:20:26Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused

Hangzhounet KO

hangzhounet_tezos-node-exporter_1 /usr/bin/tezos_exporter -- ... Up (unhealthy) 0.0.0.0:9491->9489/tcp,:::9491->9489/tcp

in

sudo docker-compose  ps
               Name                              Command                   State                                                  Ports
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hangzhounet_indexer-db-schema_1       ./db.bash postgresql://ind ...   Exit 0
hangzhounet_indexer-mempool_1         /home/tezos/utils/run-tezo ...   Up
hangzhounet_indexer-token-support_1   /home/tezos/utils/run-tezo ...   Up
hangzhounet_mezos_1                   ./run-mezos.bash run --cha ...   Up (healthy)     0.0.0.0:39090->39090/tcp,:::39090->39090/tcp
hangzhounet_node_1                    /usr/local/bin/entrypoint. ...   Up (healthy)     0.0.0.0:38732->38732/tcp,:::38732->38732/tcp, 0.0.0.0:39732->39732/tcp,:::39732->39732/tcp
hangzhounet_postgres-exporter_1       postgres_exporter                Up               0.0.0.0:9189->9187/tcp,:::9189->9187/tcp
hangzhounet_postgres_1                docker-entrypoint.sh postgres    Up (healthy)     5432/tcp
hangzhounet_tezos-node-exporter_1     /usr/bin/tezos_exporter -- ...   Up (unhealthy)   0.0.0.0:9491->9489/tcp,:::9491->9489/tcp

with the following logs

tezos-node-exporter_1    | 2021-11-20T21:26:21.017308792Z time="2021-11-20T21:26:21Z" level=error msg="error getting connections stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:26:21.017443250Z time="2021-11-20T21:26:21Z" level=error msg="error getting peer stats" error="tezos: HTTP status 401"
tezos-node-exporter_1    | 2021-11-20T21:26:21.017565468Z time="2021-11-20T21:26:21Z" level=error msg="error getting point stats" error="tezos: HTTP status 401"

config used

docker image image: ecadlabs/tezos_exporter:v0.0.1-beta.14
as defined in umami-stack-orchestration : https://gitlab.com/nomadic-labs/umami-wallet/umami-stack-orchestration/-/blob/main/umami-stack-compose/docker-compose.yml

# https://github.com/ecadlabs/tezos_exporter
  tezos-node-exporter:
          #image: ecadlabs/tezos_exporter
    image: ecadlabs/tezos_exporter:v0.0.1-beta.14
    command: --tezos-node-url http://node:${node_port_rpc}
    ports:
      - ${prom_tezos_exporter}:9489
    networks:
      - node
    restart: unless-stopped
    healthcheck:
       test: ["CMD", "wget", "--tries=1", "--spider", "http://localhost:${prom_tezos_exporter}/metrics"]
       interval: 30s
       timeout: 10s

  postgres-exporter:
    image: bitnami/postgres-exporter:latest
    networks:
      - database
    ports:
      - ${prom_postgres_exporter}:9187
    restart: always

If you have any recommendations as to what modify in the docker-compose orchestration to make sure it works correctly, feel free to suggest them !

Thanks in advance for your help !

[danielelisi]

This is likely caused by the ACLs introduced in one of the latest version of Octez.

You need to add the requested endpoints to an allow-list or you can add the --allow-all-rpc parameter to tezos-node run command.

You can find more info in the official docs here https://tezos.gitlab.io/user/node-configuration.html?highlight=allow%20all%20rpc#access-control-lists

[copolycube]

Hello, thanks @danielelisi for your answer.
if that was the case then the error would be the same on mainnet, no ?

as you can see here :
https://gitlab.com/nomadic-labs/umami-wallet/umami-stack-orchestration/-/blob/main/umami-stack-compose/docker-compose.yml#L19 the octez tezos node is already launched with the --allow-all-rpc option

    command: tezos-node --net-addr :${node_port_p2p} --rpc-addr :${node_port_rpc} --network ${tezos_net} --history-mode=archive --allow-all-rpc :${node_port_rpc}

and both node and tezos-node-exporter are on the same docker network (i.e. node).

I might be missing something obvious, but I'm not sure what :-/

[danielelisi]

if that was the case then the error would be the same on mainnet, no ?

Which Octez version are you running on mainnet?

Can you try specifying the bind address as 0.0.0.0:${node_port_rpc} for both your --rpc-addr and -allow-all-rpc

[copolycube]

all my octez nodes (mainnet, granadanet, hangzhounet) are running with node_docker_image=tezos/tezos:v11.0

[copolycube]

adding the 0.0.0.0: to rpc-addr and allow-all-rpc, with the following command in docker-compose :

  node:
    image: ${node_docker_image}
    hostname: node
    command: tezos-node --net-addr :${node_port_p2p} --rpc-addr 0.0.0.0:${node_port_rpc} --network ${tezos_net} --history-mode=rolling --allow-all-rpc 0.0.0.0:${node_port_rpc}

makes the following logs in tezos-node-exporter

tezos-node-exporter_1    | 2021-11-24T16:53:35.339430882Z time="2021-11-24T16:53:35Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
tezos-node-exporter_1    | 2021-11-24T16:53:35.342599780Z time="2021-11-24T16:53:35Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
tezos-node-exporter_1    | 2021-11-24T16:53:35.342933518Z time="2021-11-24T16:53:35Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
tezos-node-exporter_1    | 2021-11-24T16:53:35.342958097Z time="2021-11-24T16:53:35Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
tezos-node-exporter_1    | 2021-11-24T17:02:27.548435032Z time="2021-11-24T17:02:27Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
tezos-node-exporter_1    | 2021-11-24T17:02:27.550979823Z time="2021-11-24T17:02:27Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied
tezos-node-exporter_1    | 2021-11-24T17:02:27.551284432Z time="2021-11-24T17:02:27Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
tezos-node-exporter_1    | 2021-11-24T17:02:27.552041323Z time="2021-11-24T17:02:27Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
tezos-node-exporter_1    | 2021-11-24T17:06:55.195410030Z time="2021-11-24T17:06:55Z" level=error msg="error monitoring mempool operations" error="Get \"http://node:18732/chains/main/mempool/monitor_operations?branch_refused\": dial tcp 172.28.0.2:18732: connect: connection refused" pool=branch_refused
tezos-node-exporter_1    | 2021-11-24T17:06:55.195427394Z time="2021-11-24T17:06:55Z" level=error msg="error monitoring mempool operations" error="Get \"http://node:18732/chains/main/mempool/monitor_operations?branch_delayed\": dial tcp 172.28.0.2:18732: connect: connection refused" pool=branch_delayed
tezos-node-exporter_1    | 2021-11-24T17:06:55.512968089Z time="2021-11-24T17:06:55Z" level=error msg="error monitoring mempool operations" error="Get \"http://node:18732/chains/main/mempool/monitor_operations?refused\": dial tcp 172.28.0.2:18732: connect: connection refused" pool=refused
tezos-node-exporter_1    | 2021-11-24T17:06:55.657465093Z time="2021-11-24T17:06:55Z" level=error msg="error monitoring mempool operations" error="Get \"http://node:18732/chains/main/mempool/monitor_operations?applied\": dial tcp 172.28.0.2:18732: connect: connection refused" pool=applied
tezos-node-exporter_1    | 2021-11-24T17:07:02.181104185Z time="2021-11-24T17:07:02Z" level=error msg="error getting bootstrap status" error="Get \"http://node:18732/monitor/bootstrapped\": dial tcp 172.28.0.2:18732: connect: connection refused"
tezos-node-exporter_1    | 2021-11-24T17:07:02.183299887Z time="2021-11-24T17:07:02Z" level=error msg="error getting bootstrap status" error="Get \"http://node:18732/chains/main/is_bootstrapped\": dial tcp 172.28.0.2:18732: connect: connection refused"
tezos-node-exporter_1    | 2021-11-24T17:09:25.388884463Z time="2021-11-24T17:09:25Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_delayed
tezos-node-exporter_1    | 2021-11-24T17:09:25.388905646Z time="2021-11-24T17:09:25Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=branch_refused
tezos-node-exporter_1    | 2021-11-24T17:09:25.389254355Z time="2021-11-24T17:09:25Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=refused
tezos-node-exporter_1    | 2021-11-24T17:09:25.389276503Z time="2021-11-24T17:09:25Z" level=error msg="error monitoring mempool operations" error="json: cannot unmarshal array into Go struct field Operation.contents of type map[string]interface {}" pool=applied

[copolycube]

Hello,

could it be linked to https://gitlab.com/tezos/tezos/-/issues/2229 ?