Diverse updates and fixes

- Update cloud_storage and local to abstract out common code and
  make the systems more usable
- Fix Firestore implementation
- Add the key_uploader
- Improve READEME.md

Author: davidharcombe

README.md

@@ -36,7 +36,7 @@ to enable the API you need.
 ### Secret Manager
 
 Two secrets will need to be manually added to Secret Manager before the library
-can be used. These are the client id and client secret. The easiest way to do
+can be used. These are the client id and client secret. One way to do
 this is using a small shell script like this:
 
 ```
@@ -78,11 +78,18 @@ echo "{ \"client_id\": \"${CLIENT_SECRET}\" }" | gcloud --project ${PROJECT} sec
 The library will create any further secrets and versions automatically. It will
 also remove all but the latest secret each time an update occurs. This reduces
 the usage cost of Secret Manager substantially as projects are charged based
-partially on number of _active_ (ie not destroyed) secret versions.
+partially on number of _active_ and _disabled_ (ie not destroyed) secret
+versions.
+
+You can also use [`KeyUploader`](#keyuploader).
 
 ### Firestore
 
-Firestore requires no additional configuration.
+Firestore requires no additional configuration, although you will have to
+seed the Firestore database with the client secret data, much like with Secret
+Manager above. Unfortunately there is no CLI access through `gcloud` to
+Firestore so we can't write a simple shell script. In this case, please see
+below in the section on [`KeyUploader`](#keyuploader).
 
 ### Google Cloud Storage
 
@@ -91,6 +98,7 @@ token files and project secrets are to be stored and to which the app's service
 account has read/write access. This should then be locked down so that no other
 non-administrators have access.
 
+
 ### Local files
 
 No special configuration is required. This implementation is HIGHLY insecure,
@@ -157,3 +165,43 @@ manager = SecretManager(project='<gcp project name>')
 
 manager.list_documents()
 ```
+
+## [`KeyUploader`](#keyuploader)
+
+The Key Uploader (`key_upload.py`) is a way of inserting `json` files into your
+datastore from the command line. It will work for any key, and any of the
+supplied datastores. This will allow you to pre-load `Firestore` with the
+`client_secret` keys necessary, but can also be used for `CloudStorage`,
+`LocalFile` and `SecretManager` implementations simply by changing a switch.
+
+The file to be uploaded can be stored either locally or on
+`Google Cloud Storage`.
+
+For example: to run the `KeyUploader` and install the client secrets file into
+Firestore, you would do the following:
+
+```
+python auth.cli.key_upload.py         \
+  --key=client_secret                 \
+  --firestore                         \
+  --email=YOUR_EMAIL                  \
+  --file=PATH/TO/client_secrets.json
+  ```
+
+Your available command-line switches are:
+
+| Switch             |                | Description                                                                                  |
+| ------------------ | -------------- | -------------------------------------------------------------------------------------------- |
+| `--project`        | _optional_     | The Google Cloud project to use (if it is not your default)                                  |
+| `--email`          | _optional_     | Your email address. This will be placed in the document.                                     |
+| `--file`           | **_required_** | The file containing the json to be uploaded.                                                 |
+| `--key`            | **_required_** | The name of the key to install. This must be valid for your storage method.                  |
+| `--encode_key`     | _optional_     | Base64 encode the key. Do this for any user tokens, but **NOT** for the `client_secret` key. |
+| `--local`          | _optional_     | Create/write to a local file.                                                                |
+| `--firestore`      | _optional_     | Use Firestore.                                                                               |
+| `--secret_manager` | _optional_     | Use Secret Manager.                                                                          |
+| `--cloud_storage`  | _optional_     | Use Google Cloud Storage.                                                                    |
+| `--bucket`         | _optional_     | The bucket in GCS to store the data in.                                                      |
+
+**NOTE** _One and only one_ of `--local`, `--firestore`, `--cloud_storage`
+and `--secret_manager` must be specified

auth/abstract_datastore.py

@@ -14,6 +14,7 @@
 from __future__ import annotations
 
 from typing import Any, Dict, List, Optional
+from auth import decorators
 
 
 class AbstractDatastore(object):
@@ -29,7 +30,23 @@ class AbstractDatastore(object):
   All unimplemented functions raise a NotImplementedError() rather than
   simply 'pass'.
   """
-  def get_document(self, key: Optional[str]=None) -> Dict[str, Any]:
+  @decorators.lazy_property
+  def project(self) -> str:
+    return self._project
+
+  @project.setter
+  def project(self, project: str) -> None:
+    self._project = project
+
+  @decorators.lazy_property
+  def email(self) -> str:
+    return self._email
+
+  @email.setter
+  def email(self, email: str) -> None:
+    self._email = email
+
+  def get_document(self, key: Optional[str] = None) -> Dict[str, Any]:
     """Fetches a document (could be anything, 'type' identifies the root.)
 
     Fetch a document
@@ -48,9 +65,7 @@ def store_document(self, id: str,
                      document: Dict[str, Any]) -> None:
     """Stores a document.
 
-    Store a document in Firestore. They're all stored by Type
-    (DCM/DBM/SA360/ADH) and each one within the type is keyed by the
-    appropriate report id.
+    Store a document in the datastore.
 
     Arguments:
         id (str): report id
@@ -61,7 +76,7 @@ def store_document(self, id: str,
   def update_document(self, id: str, new_data: Dict[str, Any]) -> None:
     """Updates a document.
 
-    Update a document in Firestore. If the document is not already there, it
+    Update a document in the datastore. If the document is not already there, it
     will be created as a net-new document. If it is, it will be updated.
 
     Args:
@@ -70,10 +85,10 @@ def update_document(self, id: str, new_data: Dict[str, Any]) -> None:
     """
     raise NotImplementedError('Must be implemented by child class.')
 
-  def delete_document(self, id: str, key: Optional[str]=None) -> None:
+  def delete_document(self, id: str, key: Optional[str] = None) -> None:
     """Deletes a document.
 
-    This removes a document or partial document from the Firestore. If a key is
+    This removes a document or partial document from the datastore. If a key is
     supplied, then just that key is removed from the document. If no key is
     given, the entire document will be removed from the collection.
 
@@ -83,15 +98,10 @@ def delete_document(self, id: str, key: Optional[str]=None) -> None:
     """
     raise NotImplementedError('Must be implemented by child class.')
 
-  def list_documents(self, key: str=None) -> List[str]:
+  def list_documents(self, key: str = None) -> List[str]:
     """Lists documents in a collection.
 
-    List all the documents in the collection 'type'. If a key is give, list
-    all the sub-documents of that key. For example:
-
-    list_documents(Type.SA360_RPT) will show { '_reports', report_1, ... }
-    list_documents(Type.SA360_RPT, '_reports') will return
-      { 'holiday_2020', 'sa360_hourly_depleted', ...}
+    List all the documents.
 
     Args:
         key (str, optional): the sub-key. Defaults to None.
@@ -104,7 +114,7 @@ def list_documents(self, key: str=None) -> List[str]:
   def get_all_documents(self) -> List[Dict[str, Any]]:
     """Fetches all documents
 
-    Fetches all documents of a given Type.
+    Fetches all documents.
 
     Returns:
         runners (List[Dict[str, Any]]): contents of all documents

auth/cli/key_upload.py

@@ -0,0 +1,99 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import json
+import logging
+import os
+from contextlib import suppress
+from datetime import datetime
+
+import gcsfs
+from absl import app, flags
+from auth.credentials_helpers import encode_key
+
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string('project', None, 'GCP Project.')
+flags.DEFINE_string('email', None, 'Report owner/user email.')
+flags.DEFINE_string('key', None, 'Key to create/update')
+flags.DEFINE_string('file', None, 'File containing json data')
+flags.DEFINE_string('bucket', None, 'GCS Bucket for the datastore.')
+flags.DEFINE_bool('encode_key', False, 'Encode the key (for tokens).')
+flags.DEFINE_bool('local', False, 'Local storage.')
+flags.DEFINE_bool('firestore', False, 'Send to Firestore.')
+flags.DEFINE_bool('secret_manager', False, 'Send to Secret Manager.')
+flags.DEFINE_bool('cloud_storage', False, 'Send to GCS.')
+flags.mark_flags_as_required(['file', 'key'])
+flags.mark_bool_flags_as_mutual_exclusive(
+    ['local', 'firestore', 'secret_manager', 'cloud_storage'], required=True)
+
+
+def upload(**args) -> None:
+  """Uploads data to firestore.
+
+  Args:
+      key (str): the data key.
+      file (str): the file containing the data.
+      encode_key (bool): should the key be encoded (eg is it an email).
+      local_store (bool): local storage (True) or Firestore (False).
+  """
+  _project = args.get('project')
+  _key = args.get('key')
+
+  if file := args.get('file'):
+    if file.startswith('gs://'):
+      with gcsfs.GCSFileSystem(project=_project).open(file, 'r') as data_file:
+        src_data = json.loads(data_file.read())
+    else:
+      # Assume locally stored token file
+      with open(file, 'r') as data_file:
+        src_data = json.loads(data_file.read())
+
+  if args.get('encode_key'):
+    key = encode_key(_key)
+
+  else:
+    key = _key
+
+  src_data['email'] = _key
+
+  if args.get('local_store'):
+    from auth.datastore.local_datastore import LocalDatastore
+    f = LocalDatastore()
+
+  if args.get('firestore'):
+    from auth.datastore.firestore import Firestore
+    f = Firestore()
+
+  if args.get('secret_manager'):
+    from auth.datastore.secret_manager import SecretManager
+    f = SecretManager(project=_project, email=args.get('email'))
+
+  if args.get('cloud_storage'):
+    from auth.datastore.cloud_storage import CloudStorage
+    f = CloudStorage(project=_project,
+                     email=args.get('email'),
+                     bucket=args.get('bucket'))
+
+  f.update_document(id=key, new_data=src_data)
+
+
+def main(unused_argv):
+  upload(**FLAGS.flag_values_dict())
+
+
+if __name__ == '__main__':
+  with suppress(SystemExit):
+    app.run(main)

auth/cli/key_upload_test.py

@@ -0,0 +1,73 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import unittest
+from unittest import mock
+
+from auth import local_datastore
+from classes.report_type import Type
+from cli import key_upload
+
+from copy import deepcopy
+from typing import Any, Callable, Dict, Mapping
+
+CLASS_UNDER_TEST = 'cli.key_upload'
+
+
+class KeyUploadTest(unittest.TestCase):
+  def setUp(self) -> None:
+    self.valid_source = {'test_root': {'a': 'A', 'b': 'B'}, 'email': 'key'}
+    self.open = mock.mock_open(read_data=json.dumps(self.valid_source))
+    self.mock_datastore = mock.MagicMock()
+    self.mock_datastore.update_document.return_value = None
+
+  def test_good_unencoded(self):
+    with mock.patch(f'{CLASS_UNDER_TEST}.open', self.open):
+      with mock.patch.object(local_datastore.LocalDatastore,
+                             'update_document',
+                             return_value=None) as mock_method:
+        event = {
+          'key': 'key',
+          'file': 'test.json',
+          'encode_key': False,
+          'local_store': True,
+        }
+        _ = key_upload.upload(**event)
+        self.open.assert_called_with('test.json', 'r')
+        self.open().read.assert_called()
+        mock_method.assert_called()
+        mock_method.assert_called_with(type=Type._ADMIN, id='key',
+                                       new_data=self.valid_source)
+
+  def test_good_encoded(self):
+    with mock.patch(f'{CLASS_UNDER_TEST}.open', self.open):
+      with mock.patch.object(local_datastore.LocalDatastore,
+                             'update_document',
+                             return_value=None) as mock_method:
+        event = {
+          'key': 'key',
+          'file': 'test.json',
+          'encode_key': True,
+          'local_store': True,
+        }
+        _ = key_upload.upload(**event)
+        self.open.assert_called_with('test.json', 'r')
+        self.open().read.assert_called()
+        mock_method.assert_called()
+        expected = deepcopy(self.valid_source)
+
+        mock_method.assert_called_with(type=Type._ADMIN, id='a2V5',
+                                       new_data=expected)
+