-
Notifications
You must be signed in to change notification settings - Fork 72
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Source name
Generic REST API #313
Make sure that
- I checked the README in the source folder
- If I have problems with the credentials I checked the credentials docs and troubleshooting
Describe the bug
The resource returned by rest_api.rest_api_source()
is not callable
To Reproduce
@dlt.resource(name='table_name', write_disposition='replace')
def generate_rows():
for i in range(10):
yield {'id':i, 'example_string':'abc'}
@dlt.source
def source_name():
return generate_rows
pokemon_source = rest_api_source({
"client": {
"base_url": "https://pokeapi.co/api/v2/",
"default_paginator": "json_links",
},
"endpoints": {
"pokemon": {
"params": {
"limit": 100, # Default page size is 20
},
"resource": {
"primary_key": "id",
}
},
},
})
print(list(source_name().with_resources('table_name').add_limit(1))) # works
print(list(pokemon_source.with_resources('pokemon').add_limit(1))) # returns 1 page of data
print(list(pokemon_source.pokemon.add_limit(1))) # returns 1 page of data
print(list(pokemon_source().with_resources('pokemon').add_limit(1))) # crashes: TypeError: 'DltSource' object is not callable
Expected behavior
I'm not sure if it should return one page or data or 1 record. Since it calls the paginator I think 1 page is fine.
print(list(pokemon_source().with_resources('pokemon').add_limit(1))) # should it return only 1 record instead of 1 page?
print(list(pokemon_source().with_resources('pokemon').add_limit(1))) # should not crash?
Running environment
- OS: MacOs
dlt
Version 0.4.4- verified-sources 4428038
Additional context
I tried to implement a function that checks if the source can connect to the API endpoint by fetching only the first page of a resource.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
Status
Ready for Deployment