GCP ValueError: Service account info was not in the expected format, missing fields token_uri

[roarjn]

Trying to create a GCPCluster after specifying my GCP projectid in ~/.config/dask/cloudprovider.yaml I get below error related to ValueError: Service account info was not in the expected format, missing fields token_uri.

from dask_cloudprovider.gcp import GCPCluster
from dask.distributed import wait

cluster = GCPCluster(
    zone="us-central1-a",
    machine_type="n1-standard-8",
    n_workers=1,
    docker_image="rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8",
    worker_class="dask_cuda.CUDAWorker",
    env_vars={"EXTRA_PIP_PACKAGES": "gcsfs"}
)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/auth/_default.py in load_credentials_from_file(filename, scopes, quota_project_id)
    134             credentials = service_account.Credentials.from_service_account_info(
--> 135                 info, scopes=scopes
    136             )

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/oauth2/service_account.py in from_service_account_info(cls, info, **kwargs)
    210         signer = _service_account_info.from_dict(
--> 211             info, require=["client_email", "token_uri"]
    212         )

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/auth/_service_account_info.py in from_dict(data, require)
     50             "Service account info was not in the expected format, missing "
---> 51             "fields {}.".format(", ".join(missing))
     52         )

ValueError: Service account info was not in the expected format, missing fields token_uri.

The above exception was the direct cause of the following exception:

DefaultCredentialsError                   Traceback (most recent call last)
<ipython-input-23-066423d9340a> in <module>
     17     docker_image="rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8",
     18     worker_class="dask_cuda.CUDAWorker",
---> 19     env_vars={"EXTRA_PIP_PACKAGES": "gcsfs"}
     20 )

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/dask_cloudprovider/gcp/instances.py in __init__(self, projectid, zone, machine_type, source_image, docker_image, ngpus, gpu_type, filesystem_size, auto_shutdown, bootstrap, **kwargs)
    512     ):
    513 
--> 514         self.compute = GCPCompute()
    515 
    516         self.config = dask.config.get("cloudprovider.gcp", {})

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/dask_cloudprovider/gcp/instances.py in __init__(self)
    551 
    552     def __init__(self):
--> 553         self._compute = self.refresh_client()
    554 
    555     def refresh_client(self):

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/dask_cloudprovider/gcp/instances.py in refresh_client(self)
    570                     # take first row
    571                     f_.write(creds_rows[0][1])
--> 572                 creds, _ = google.auth.load_credentials_from_file(filename=f)
    573             return googleapiclient.discovery.build("compute", "v1", credentials=creds)
    574 

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/auth/_default.py in load_credentials_from_file(filename, scopes, quota_project_id)
    138             msg = "Failed to load service account credentials from {}".format(filename)
    139             new_exc = exceptions.DefaultCredentialsError(msg, caught_exc)
--> 140             six.raise_from(new_exc, caught_exc)
    141         if quota_project_id:
    142             credentials = credentials.with_quota_project(quota_project_id)

~/.local/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

DefaultCredentialsError: ('Failed to load service account credentials from /tmp/tmp79wsxl8a.', ValueError('Service account info was not in the expected format, missing fields token_uri.'))

Anything else we need to know?:

Environment:

• Dask version: 2.30.0
• Python version: 3.7.9
• Operating System: Ubuntu 18.04.4 LTS
• Install method (conda, pip, source): conda

[roarjn]

Bit more context: I have set my GCP credentials using gcloud auth login in the terminal. The GCP documentation seems to suggest that you can either use gcloud auth login or use service account and pass credentials export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json.

But even after using gcloud auth login I seem to be getting error specific to service account credentials. I am passing service account credentials but perhaps making a mistake with it.

[quasiben]

Yes, you should be able to use either gcloud auth or the env var. When authenticating with gcloud auth, it should drop a sqlite db here: ~/.config/gcloud/credentials.db . What version of gcloud are you using ?

[roarjn]

thanks @quasiben. I see the the file ls -lah ~/.config/gcloud/credentials.db. Where does this file get used?

$ gcloud --version
Google Cloud SDK 303.0.0
alpha 2020.07.24
beta 2020.07.24
bq 2.0.58
core 2020.07.24
gsutil 4.52
kubectl 1.15.11

[quasiben]

I am using Google Cloud SDK 315.0.0 -- at some point google made the switch from pickled json objects to sqlite though I am not sure when that happened.

This is all used in the auth section of cloudprovider:

dask-cloudprovider/dask_cloudprovider/gcp/instances.py

Lines 549 to 573 in 53d3c92

	class GCPCompute:
	"""Wrapper for the ``googleapiclient`` compute object."""
	def __init__(self):
	self._compute = self.refresh_client()
	def refresh_client(self):
	if os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", False):
	return googleapiclient.discovery.build("compute", "v1")
	else:
	import google.auth.credentials # google-auth
	path = os.path.join(
	os.path.expanduser("~"), ".config/gcloud/credentials.db"
	)
	if not os.path.exists(path):
	raise GCPCredentialsError()
	conn = sqlite3.connect(path)
	creds_rows = conn.execute("select * from credentials").fetchall()
	with tmpfile() as f:
	with open(f, "w") as f_:
	# take first row
	f_.write(creds_rows[0][1])
	creds, _ = google.auth.load_credentials_from_file(filename=f)
	return googleapiclient.discovery.build("compute", "v1", credentials=creds)

If you are using a conda env you could upgrade easily with https://anaconda.org/conda-forge/google-cloud-sdk ?

[roarjn]

updated to Google Cloud SDK 319.0.0 but still replicating this issue. ValueError: Service account info was not in the expected format, missing fields token_uri

[quasiben]

Hmm, can you try re-authenticating with the new version ?

[roarjn]

yes, re-authenticated but same issue.

[quasiben]

Thanks for your patience here @roarjn -- maybe we need to activate a service account associated with your login?

`gcloud auth activate-service-account -- from https://cloud.google.com/iam/docs/creating-managing-service-account-keys

@bradmiro do you happen to know this off hand ? If not, no worries

[bradmiro]

Hey @roarjn, sorry you're having issues! I have two preliminary suggestions:

1. Can you try manually deleting the ~/.config/gcloud/credentials.db file and try re-authenticating?
2. If that didn't work, can you try this with a service account (following the link @quasiben provided) and setting GOOGLE_APPLICATION_CREDENTIALS in your environment, pointing to your new key?

[quasiben]

Thanks for chiming in @bradmiro !

[roarjn]

Thank you so much @bradmiro and @quasiben. Brad's suggestion #1 worked after I deleted the ~/.config/gcloud/credentials.db and re-authenticated:

rm ~/.config/gcloud/credentials.db

gcloud auth login

[bradmiro]

Glad it's working for you now @roarjn !

[quasiben]

Huzzah!!!

Closing now.

[roarjn]

Couple more issues i am running into. Appreciate any suggestion.

1. trying to read_csv from GCS with below steps seeing this error. Corresponding read from S3 works in my setup.
   $ `data = dask_cudf.read_csv( "gs://airlines-2019.csv")
   

gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.

2. Also, I am looking to specify GPU machine_type. For A100 instance I can see the machine type can be machine_type="a2-highgpu-2g". I am trying to find the machine_type for a T4 GCP instance. Is there a way to specify accelerator_type in GCPCluster?
   Tried below:

cluster = GCPCluster(
    zone="us-central1-a",
    machine_type="n1-standard-8",
#     machine_type="a2-highgpu-1g", # launches an A100 GPU instance
    gpu_type="nvidia-tesla-t4",  # Test for a T4 instance
    n_workers=1,
    docker_image="rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8",
    worker_class="dask_cuda.CUDAWorker",
    env_vars={"EXTRA_PIP_PACKAGES": "gcsfs"}
)

which resulted in:
TypeError: Parameter "project" value "" does not match the pattern "(?:(?:[-a-z0-9]{1,63}\.)*(?:[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?):)?(?:[0-9]{1,19}|(?:[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?))"