[Bug]: Error creating WACZ - ETIMEDOUT

[tkrn]

Browsertrix Version

v1.12.0-6032e28

What did you expect to happen? What happened instead?

When I have an external defined minio instance and when the crawl job moves to the upload/creating warz phase, it fails with the following error: "Error creating WACZ" with a more verbose error of "Error: connect ETIMEDOUT 10.215.7.8:9000". With that being said, the credentials are correct and I am able to enumerate the S3 bucket through the minio client (mc) with success on the same box with the same host and access/secret key.

Reproduction instructions

1. Create a custom S3 storage target as follows:

minio_local: false
storages:
- name: "default"
  type: "s3"
  region: "docker"
  access_key: "<access>"
  secret_key: "<secret>"
  bucket_name: "btrix"
  endpoint_url: "http://<ip>:9000/"

2. Create a new crawl job. Wait for error.

Screenshots / Video

Error Message:

local-config.yaml for Browertrix:

Minio Client Connection Parameters:

Minio Client Successful Connection:

Environment

No response

Additional details

Both the MongoDB instance and Minio instance are running on the same physical server but through docker compose. The ports are listening to 0.0.0.0:9000 allowing any connections including hairpin and localhost connections. This configuration is working successfully for the MongoDB instance which is outside of the Browsertrix deployment, although physically on the same server but logically outside of the microk8s cluster. The be stated, the MongoDB instance is working as expected with this type of deployment.

[tkrn]

After a bit more research, I'm having no luck with working with external storage that is non-AWS. I've deployed various instances of MinIO (bare metal, docker compose and Kubernetes variants). All are successfully deployed, accessible, and routable. I've deployed non-TLS and TLS configurations. It appears that the external S3 storage components simply are not working.

Is anyone else having this problem? Can someone else replicate it? Thanks!

[ikreymer]

Sorry you're still having issues - we use minio locally as part of the default deployment, though haven't tested with a standalone minio. We've tested with non-AWS cloud providers in production, including DigitalOcean, Wasabi, etc...
Based on the message, it seems like its a routing issue as your minio deployment is on an internal network (10.215.x.x) and isn't being reached from the k8s cluster... How are you deploying minio?

[tkrn]

I don't mean to be a pain but I think there is something here. I've posted my Kubernete's deployment of MinIO here - https://github.com/tkrn/kubernetes-minio-deployment

Additionally, it's worth mentioning that I've deployed a test pod where I deployed the mc client and referanced the MinIo instance by Kubernete's Service Name with success. I've attached screenshots.

Lastly, I've noticed I haven't has this type of issue with my external MongoDB instance which is setup outside of the Browsertrix deployment. This rules out the possibility of a routing issue. I do understand tests are fine externally with major providers.

My certificate is self-signed, and also was not working when there was no certificate at all when I was addressing with the http protocol and not https.

As you can see, I'm even in the same broadcast domain from a connectivity perspective within the Kubernetes cluster.

Minio in a seperate namespace:

My default namespace:

Tester Ubuntu workload, connecting to the MinIO instance in the deployment/pod/namespace above.

Lastly, the config.json for the mc client:

Same error, but now with an instance hosted within the same Kubernetes cluster:

The only other thing I could think of which is different in a self-hosted environment is the use of non-standard ports which has the potential to have a timeout on a port that is not listening if the port number settings are not getting parsed/applied.

Thank you for your support and please let me know if there is anything else you may need.

[tkrn]

UPDATE: After much troubleshooting, trial and error I was able to find a work around for this. Ultimately, I exposed my MinIO instance publically behind a Traefik reverse proxy and Let's Encrypt TLS resolver. After a few firewall rules and after a successful SSL certificate was issued, I was able to use my public FQDN which his the router and loops back into the Traefik reverse proxy then down to MinIO. I don't love this idea but there appears to be issues around non-SSL, non-standard ports with the service.

[roastedGhostt]

@tkrn Hi, I have found a resolution to the issue that does not involve opening a MinIO instance to the public web.
First off I really need to thank you for all your diligent troubleshooting and documentation, without it I would have gotten stuck repeating all the same things. I would also like to caution that I am a novice with Kubernetes so I don't know if my terminology is correct or my solution is the best mechanism to use/most durable.

I've found that there is a network policy for the kubernetes cluster which blocks most outgoing connections on your local network which can be seen here.

egress:
    - to:
      - ipBlock:
          cidr: 0.0.0.0/0
          except: # Exclude traffic to Kubernetes service IPs and pods
          - 10.0.0.0/8
          - 172.16.0.0/12
          - 192.168.0.0/16

In my case, my minIO instance is on a separate machine on my local network located at 192.168.1.200 and uses port 9000 for api

Working from that I found that this network policy can be modified using the command:
kubectl edit netpol crawler-limit-egress -n crawlers

Add the following block to networkpolicies.yaml or using above command:

  - ports:
    - port: 9000
      protocol: TCP
    to:
    - ipBlock:
        cidr: 192.168.1.200/24

Unfortunately any time you upgrade your cluster from the source (e.g. helm3 upgrade --install btrix browsertrix/browsertrix) this will overwrite the networkpolicy so you'll have to remember to renew your changes.

After that making that change I reran my crawl and it was successfully able to store the WACZ file in my MinIO instance. Hope this helps.

[ikreymer]

@roastedGhostt I suspect this is indeed the issue that @tkrn was happeing, thank you for troubleshooting this!

We added the default network policy to make sure the crawler doesn't start crawling internal resources which probably makes sense, but of course need to be balance that with ability to have custom configurations. In the default policy, we indeed allow access to the local minio: https://github.com/webrecorder/browsertrix/blob/main/chart/templates/networkpolicies.yaml#L68

The policy is overridable with a custom Helm chart values file, which you'd probably want to use anyway to change default passwords, etc...

If you have an overrides file local.yaml, you can specify a custom policy to allow your IP or just disable it altogether with:

...
crawler_enable_network_policy: false

For just that setting, you can run it on the command line helm3 upgrade --install btrix browsertrix/browsertrix --set crawler_enable_network_policy=false and that should disable it altogether.

Or, you can optionally provide a custom policy with

crawler_enable_network_policy:
   - to:
      - ipBlock:
          cidr: 0.0.0.0/0
          except:
          - <block IPs>

though you'd need to make sure to enable some of the default services as well.

We'll add something about this to the docs to make this easier to find for whomever else needs this!

[ikreymer]

We can probably make this simpler by providing a way to specify additional IPs that should be allowed, with the default network policy. It probably doesn't make sense to override the whole thing, but perhaps adding additional IPs to allow and block could be useful? @vnznznz what do you think?

[vnznznz]

What about something like this, where one can add proper NetworkPolicyEgress rules to add to the existing list? It's not overly verbose for a single IP but adds more options for more complicated use-cases.

crawler_network_policy_additional_egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.1/32

Edit: Opened a PR that adds this setting.

[ikreymer]

This works, though I wonder if we should instead be blocking the browser rather than the whole crawler. We can do this with one of several ways:

• In the crawler with: blockRules
• In the browser with: URLBlockList enterprise policy
• In the browser, with PAC policy: Support host-specific proxies with proxy config YAML browsertrix-crawler#837 (setting PROXY to unresolvable address for blocked resources)

This has the advantage of just applying to the browser, and can block access to services the the crawler still needs..
Would require injecting a new volume into the crawler, which we'll need to do for PAC policy anyway, so that might be one way to go..