How do you run pub/sub client behind a proxy?

[boranby]

Hi,

In our setup the servers don't have any outside internet access. However, we run squid outside, connecting to the VPN and creating a proxy over ssh. Basically, we can download updates etc. without any issues. When I try to use Pub/Sub client behind the proxy, as I understand it has issues because it's a gRPC client (please correct me if I'm wrong on this).

Is it possible to find a way to have the Pub/Sub client connection behind the proxy? It stuck at the last line of the logs.

The code and logs:

let config = ClientConfig::default().with_auth().await.unwrap();
let client = Client::new(config).await.unwrap();

2024-10-15T06:37:41.363527Z DEBUG reqwest::connect: starting new connection: https://oauth2.googleapis.com/
2024-10-15T06:37:41.363615Z DEBUG reqwest::connect: proxy(http://127.0.0.1:3129) intercepts 'https://oauth2.googleapis.com/'
2024-10-15T06:37:41.363667Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:3129
2024-10-15T06:37:41.363814Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:3129
2024-10-15T06:37:41.649662Z DEBUG hyper_util::client::legacy::pool: pooling idle connection for ("https", oauth2.googleapis.com)
2024-10-15T06:37:41.649826Z DEBUG publisher: ClientConfig: ClientConfig { pool_size: Some(4), project_id: Some("orders-435328"), environment: GoogleCloud(), endpoint: "pubsub.googleapis.com", connection_option: ConnectionOptions { timeout: None, connect_timeout: None } }
2024-10-15T06:37:41.650167Z DEBUG resolve{host=pubsub.googleapis.com}: hyper_util::client::legacy::connect::dns: resolving host="pubsub.googleapis.com"

Connection:

ssh -R 3129:localhost:3128 [email protected]

Environment:

export http_proxy=http://127.0.0.1:3129
export https_proxy=http://127.0.0.1:3129

[yoshidan]

Yes, it is.

While reqwest automatically retrieves the proxy from the environment variable, tonic does not seem to have such a function.

It seems to work if we modify the following sections.

google-cloud-rust/foundation/gax/src/conn.rs

Line 180 in 497f2d2

async fn connect(endpoint: Endpoint) -> Result<TonicChannel, tonic::transport::Error> {

Get the environment variables and if proxy is set,
Use connect_with_connector etc. etc.

[boranby]

Thanks for your quick response. I checked the connect_with_connector and https://github.com/hyperium/tonic/blob/master/examples/src/uds/client.rs#L21-L28 . I tried to connect via UnixStream as in the example, I tried TcpStrem and I tried to create a new connection through new endpoint let endpoint = TonicChannel::from_shared... and then let channel = endpoint.connect().await? . None of them worked. Do you have any suggestion which structs/functions I should use?

[yoshidan]

How about using hyper_proxy2 ?

let raw_connector = ...
let proxy = Proxy::new(Intercept::All, "http://proxy.server:8080".parse()?);
let proxy_connector = ProxyConnector::from_proxy(raw_connector, proxy)?;
let channel = endpoint.connect(proxy_connector).await?;

[boranby]

My fork: 10c7dab

if let Ok(proxy) = std::env::var("HTTPS_PROXY") {
    let proxy = {
        let proxy_uri = proxy.parse().unwrap();
        let proxy = Proxy::new(Intercept::All, proxy_uri);
        let connector = HttpConnector::new();
        let proxy_connector = ProxyConnector::from_proxy(connector, proxy).unwrap();
        proxy_connector
    };
    let channel = endpoint.connect_with_connector(proxy).await?;
    Ok(channel)
} else {
     let channel = endpoint.connect().await?;
     Ok(channel)
}

Logs (it stuck):

2024-10-17T12:59:27.759035Z DEBUG reqwest::connect: starting new connection: https://oauth2.googleapis.com/
2024-10-17T12:59:27.759076Z DEBUG reqwest::connect: proxy(http://127.0.0.1:3129) intercepts 'https://oauth2.googleapis.com/'
2024-10-17T12:59:27.759127Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:3129
2024-10-17T12:59:27.759234Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:3129

Edit: I verified that it goes into the proxy branch and uses connect_with_connector.