Skip to content

Commit

Permalink
Merge pull request #118 from dlt-hub/fix/remove_print_statements_upda…
Browse files Browse the repository at this point in the history
…te_readme

remove print statements and update readme
  • Loading branch information
sh-rp authored May 27, 2024
2 parents b0f6f59 + de7525a commit af79f60
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 13 deletions.
30 changes: 18 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ You will need Python 3.9 or higher installed, as well as pip. You can run `pip i

We will create a simple example pipeline from a [PokeAPI spec](https://pokeapi.co/) in our repo. You can point to any other OpenAPI Spec instead if you prefer.

```console
```sh
# 1.a. Run the generator with a URL:
$ dlt-init-openapi pokemon --url https://raw.githubusercontent.com/dlt-hub/dlt-init-openapi/devel/tests/cases/e2e_specs/pokeapi.yml --global-limit 2

Expand All @@ -35,9 +35,12 @@ $ dlt-init-openapi pokemon --path ./my_specs/pokeapi.yml

# 2. You can now pick both of the endpoints from the popup.

# 3. After selecting your Pokemon endpoints and hitting Enter, your pipeline will be rendered.
# 3. After selecting your Pokemon endpoints and hitting Enter,
# your pipeline will be rendered.

# 4. If you have any kind of authentication on your pipeline (this example does not), open the `.dlt/secrets.toml` and provide the credentials. You can find further settings in the `.dlt/config.toml`.
# 4. If you have any kind of authentication on your pipeline (this example does not),
# open the `.dlt/secrets.toml` and provide the credentials. You can find further
# settings in the `.dlt/config.toml`.

# 5. Go to the created pipeline folder and run your pipeline.
$ cd pokemon-pipeline
Expand All @@ -46,22 +49,25 @@ $ PROGRESS=enlighten python pipeline.py # we use enlighten for a nice progress b
# 6. Print the pipeline info to the console to see what got loaded.
$ dlt pipeline pokemon_pipeline info

# 7. You can now also install Streamlit to see a preview of the data; you should have loaded 40 Pokemons and their details.
# 7. You can now also install Streamlit to see a preview of the data; you should
# have loaded 40 Pokemons and their details.
$ pip install pandas streamlit
$ dlt pipeline pokemon_pipeline show

# 8. You can go to our docs at https://dlthub.com/docs to learn how to modify the generated pipeline to load to many destinations, place schema contracts on your pipeline, and many other things.
# 8. You can go to our docs at https://dlthub.com/docs to learn how to modify
# the generated pipeline to load to many destinations, place schema contracts
# on your pipeline, and many other things.

# NOTE: We used the `--global-limit 2` CLI flag to limit the requests to the PokeAPI for this example. This way, the Pokemon collection endpoint only gets queried twice, resulting in 2 x 20 Pokemon

details being rendered.
# NOTE: We used the `--global-limit 2` CLI flag to limit the requests to the PokeAPI
# for this example. This way, the Pokemon collection endpoint only gets queried
# twice, resulting in 2 x 20 Pokemon details being rendered.
```

## What will be created?

When you run the `init` command above, the following files will be generated:

```
```text
pokemon_pipeline/
├── .dlt/
│ ├── config.toml # dlt config, learn more at dlthub.com/docs
Expand All @@ -84,7 +90,7 @@ This file contains the configuration dictionary for the [dlt rest_api](https://d

The generated dictionary will look something like this:

```python
```py
{
"client": {
"base_url": base_url,
Expand Down Expand Up @@ -134,7 +140,7 @@ The generated dictionary will look something like this:

## CLI command

```console
```sh
$ dlt-init-openapi <source_name> [OPTIONS]
# example:
$ dlt-init-openapi pokemon --path ./path/to/my_spec.yml --no-interactive --output-path ./my_pipeline
Expand Down Expand Up @@ -170,7 +176,7 @@ package_name: "other_package_name"
And use it with the config argument:
```console
```sh
$ dlt-init-openapi pokemon --url ... --config config.yml
```

Expand Down
1 change: 0 additions & 1 deletion dlt_init_openapi/parser/endpoints.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,6 @@ def unresolvable_path_param_names(self) -> List[str]:
@property
def unresolvable_query_params(self) -> List[Parameter]:
"""returns a list of required query param names with params that are used by the paginator excluded"""
print("HERE")
paginator_params = self.detected_pagination.param_names if self.detected_pagination else []
query_params: List[Parameter] = []
for param in self.list_all_parameters:
Expand Down

0 comments on commit af79f60

Please sign in to comment.