RFC: Support OpenAPI generation for micro functions via CLI/script

[ran-isenberg]

Is this related to an existing feature request or issue?

No response

Which Powertools for AWS Lambda (Python) utility does this relate to?

Event Handler - REST API

Summary

Hi,
I'd like to use event handler with event validation turned on for my micro functions service. At the moment, the /swagger generation is done via one endpoint and unless that handler defines all routes in the app, there's no way to generate all the routes for the OpenAPI.

Use case

I have a service with multiple micro function, I'm not a fan of mono lambda for large service.
In addition, I'm more interested in generating the OpenAPI doc offline and saving it to my service repository where it can be later published to various location, and not necessarily served via the rest endpoint.

Proposal

Create a CLI that I can run locally on my service in the shell that generates the OpenAPI doc. It will need to init the event handler and it's all uses in the different lambda handlers and then generate the document and save it to a destination folder in a JSON format.
I have created a CLI that works with the current implementation - it accesses the deployed API GW at the /swagger?format=json and saves the output. Now, this is obviously not the solution, but it's an example of a possible interface (replace stack output input with the lambda handlers folder patch and it might be enough.

Here's the current CLI:
https://github.com/ran-isenberg/aws-lambda-handler-cookbook/blob/main/generate_openapi.py

Out of scope

I have added this script as a makefile command of my AWS Lambda handler cookbook and now it's part of my CI/CD pipeline where I can fail PRs if developers added/changed the REST API but forgot to update the documentation. Expect a blog post about it very soon :)

Potential challenges

Thanks @leandrodamascena for this section:

1. • OS compatibility: It's a CLI, we'll make sure it works on at least Windows, Linux and Mac.
2. • Security: what are the possible problems when we provide a CLI (even if it is pure python code) to install on someone' computer?
     3 - Performance: do we need to worry about any performance issues? - i dont think so, it runs locally,. can be several seconds at max.

Please note that it can even be just a script that i can run from powertools library.

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[leandrodamascena]

Hey @ran-isenberg! Thanks for kicking off this RFC – it's a great starting point to outline the requirements for this CLI.

After reading the code, I saw you've built a sort of "downloader" for an existing OpenAPI schema deployed in AWS via Cloudformation. While I see potential use cases for this implementation, I think it's important to consider some drawbacks that must be reviewed.

• One of the key features of this CLI could be the ability to generate a unique OpenAPI schema from one or multiple directories containing Lambda functions. How can we cover this?

• Customers may not use Cloudformation to deploy their resources in AWS and it's a potential limitation.

• Customers may prefer not to deploy the generated OpenAPI file directly to AWS. Instead, they may wish to create a local OpenAPI file from multiple Lambda functions and then upload it to their organization's dev portal, for example. This approach provides flexibility for teams to collaborate, document and establish contracts between applications across the organization.

• Customers might run into issues deploying resources on AWS, like making endpoints public (or private) without authentication. Do we need to think about adding authentication when this client downloads an OpenAPI spec from a given URL? And what kind of authentication?

Taking these points into account, it might be a good idea to reconsider our approach to building this CLI. Perhaps we should place more emphasis on local generation, rather than assuming that the file has already been deployed to AWS.

Please let me know what do you think.

BTW, I'll be adding the help-wanted and need-feedback tags to gather additional ideas from others as well.

Thank you!

[ran-isenberg]

@leandrodamascena I think i didnt explain myself well.
I totally agree with the local approach. I meant that the CLI that i built works with current implementation of the event handler.
Going forward an offline variation is preferred like you wrote.

[leandrodamascena]

@leandrodamascena I think i didnt explain myself well. I totally agree with the local approach. I meant that the CLI that i built works with current implementation of the event handler. Going forward an offline variation is preferred like you wrote.

I think this CLI should have the ability to download OpenAPI, as you wrote, but my opinion is that the core features should be something more offline to allow for more functionality with fewer restrictions.
What do you think about writing down some ideas in this direction and after that thinking about a PoC to validate?

[ran-isenberg]

We have implemented an offline OpenAPI generator CLI internally so i know it can be done.

What kind of ideas? i thought i already did that :)
"Create a CLI that I can run locally on my service in the shell that generates the OpenAPI doc. It will need to init the event handler and it's all uses in the different lambda handlers and then generate the document and save it to a destination folder in a JSON format."

what did you have in mind?

[rberger]

If effort was put into this, it would be nice to not require that CDK/Cloudformation be the basis of generating the OpenAPI info.

[ran-isenberg]

@anafalcao what are we discussing? :) let's build it

[leandrodamascena]

Hi @ran-isenberg! I have a branch I'm working on for this issue! I hope to have it ready in the near future.

The reason we converted to RFC is to give you more control over the things we're discussing as RFCs.

[ran-isenberg]

amazing!

[leandrodamascena]

Hello everyone! Just a heads up, as part of the effort to create a new class to generate OpenAPI Schema from multiple Lambda files, I created this issue: #6122.

I also have a PoC running and be able to create a single OpenAPI schema from multiple files. This is not the final experience that I want for customers - this is only an experiment - but it's working something like this:

folder_path = './my_files'
resolver_files = find_resolver_files(folder_path)
print("\nResolver files found:")
for file_path, instances in resolver_files:
    print(f"File: {file_path}")
    for instance in instances:
        print(f"  - {instance}")

merged_resolver = merge_resolvers(resolver_files)
print(merge_openapi_specs(json.loads(merged_resolver[0]),json.loads(merged_resolver[1])))

Resolver files found:

File: ./my_files/func1.py
  - app
File: ./my_files/func2.py
  - app
Starting to merge resolvers
[('./my_files/func1.py', ['app']), ('./my_files/func2.py', ['app'])]
Processing file: ./my_files/func1.py
  Merging instance: app
Processing file: ./my_files/func2.py
  Merging instance: app
Merging completed

In the my_files folder I have the following files:

func1.py

from typing import Optional

from pydantic import BaseModel, Field

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver, Response, content_types
from aws_lambda_powertools.event_handler.openapi.exceptions import RequestValidationError
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver(enable_validation=True)


class Todo(BaseModel):
    userId: int
    id_: Optional[int] = Field(alias="id", default=None)
    title: str
    completed: bool


@app.post("/todos")
def create_todo(todo: Todo) -> int:

    return 2


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

print(app.__dict__)

func2.py

from typing import Optional
from pydantic import BaseModel, Field
import requests
from requests import Response

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver(enable_validation=True)

class MyClassItems(BaseModel):
    name: str

class MyClass(BaseModel):
    name: str
    items: list[MyClassItems]

@app.get("/todos")
@tracer.capture_method
def get_todos() -> MyClass:
    todos: Response = requests.get("https://jsonplaceholder.typicode.com/todos")
    todos.raise_for_status()

    # for brevity, we'll limit to the first 10 only
    return MyClass(name="a",items=[MyClassItems(name="LEO")])


# You can continue to use other utilities just as before
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

OpenAPI Schema output

{
   "openapi":"3.1.0",
   "info":{
      "title":"Powertools API",
      "version":"1.0.0"
   },
   "servers":[
      {
         "url":"/"
      }
   ],
   "paths":{
      "/todos":{
         "post":{
            "summary":"POST /todos",
            "operationId":"create_todo_todos_post",
            "requestBody":{
               "content":{
                  "application/json":{
                     "schema":{
                        "$ref":"#/components/schemas/Todo"
                     }
                  }
               },
               "required":true
            },
            "responses":{
               "422":{
                  "description":"Validation Error",
                  "content":{
                     "application/json":{
                        "schema":{
                           "$ref":"#/components/schemas/HTTPValidationError"
                        }
                     }
                  }
               },
               "200":{
                  "description":"Successful Response",
                  "content":{
                     "application/json":{
                        "schema":{
                           "type":"integer",
                           "title":"Return"
                        }
                     }
                  }
               }
            }
         },
         "get":{
            "summary":"GET /todos",
            "operationId":"get_todos_todos_get",
            "responses":{
               "422":{
                  "description":"Validation Error",
                  "content":{
                     "application/json":{
                        "schema":{
                           "$ref":"#/components/schemas/HTTPValidationError"
                        }
                     }
                  }
               },
               "200":{
                  "description":"Successful Response",
                  "content":{
                     "application/json":{
                        "schema":{
                           "$ref":"#/components/schemas/MyClass"
                        }
                     }
                  }
               }
            }
         }
      }
   },
   "components":{
      "schemas":{
         "HTTPValidationError":{
            "properties":{
               "detail":{
                  "items":{
                     "$ref":"#/components/schemas/ValidationError"
                  },
                  "type":"array",
                  "title":"Detail"
               }
            },
            "type":"object",
            "title":"HTTPValidationError"
         },
         "Todo":{
            "properties":{
               "userId":{
                  "type":"integer",
                  "title":"Userid"
               },
               "id":{
                  "anyOf":[
                     {
                        "type":"integer"
                     },
                     {
                        "type":"null"
                     }
                  ],
                  "title":"Id"
               },
               "title":{
                  "type":"string",
                  "title":"Title"
               },
               "completed":{
                  "type":"boolean",
                  "title":"Completed"
               }
            },
            "type":"object",
            "required":[
               "userId",
               "title",
               "completed"
            ],
            "title":"Todo"
         },
         "ValidationError":{
            "properties":{
               "loc":{
                  "items":{
                     "anyOf":[
                        {
                           "type":"string"
                        },
                        {
                           "type":"integer"
                        }
                     ]
                  },
                  "type":"array",
                  "title":"Location"
               },
               "type":{
                  "type":"string",
                  "title":"Error Type"
               }
            },
            "type":"object",
            "required":[
               "loc",
               "msg",
               "type"
            ],
            "title":"ValidationError"
         },
         "MyClass":{
            "properties":{
               "name":{
                  "type":"string",
                  "title":"Name"
               },
               "items":{
                  "items":{
                     "$ref":"#/components/schemas/MyClassItems"
                  },
                  "type":"array",
                  "title":"Items"
               }
            },
            "type":"object",
            "required":[
               "name",
               "items"
            ],
            "title":"MyClass"
         },
         "MyClassItems":{
            "properties":{
               "name":{
                  "type":"string",
                  "title":"Name"
               }
            },
            "type":"object",
            "required":[
               "name"
            ],
            "title":"MyClassItems"
         }
      }
   }
}

I'm working on getting this PoC as close to the final experience and will update this ticket to hear your feedback.

@ran-isenberg I'll ping you to hear your opinions before publishing here.

Thanks everyone.

[ran-isenberg]

@leandrodamascena thank you for pushing this!!! i'm really excited for this. I'll feature it in a blog post once you iron it out.
amazing progress, thank you!!!

[sthulb]

From my point of view, I'm not super frilled by us creating a CLI tool for the project. I can see the requirement for this, but the tool is a slippery slope to us creating more CLI tools in the future.