Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scrapy raises exception on error status code #308

Merged
merged 8 commits into from
Jan 1, 2021
Merged

scrapy raises exception on error status code #308

merged 8 commits into from
Jan 1, 2021

Conversation

ebleb
Copy link
Contributor

@ebleb ebleb commented Dec 20, 2020

scrapy now raises an exception when the request returns an error status code as requested in issue #188

@ubergesundheit
Copy link
Member

Looks nice! Would it be possible to define the custom_settings and handle_error outside of the scraper files to reuse them?

@ubergesundheit
Copy link
Member

Thanks for quickly reacting!

@bCyberBasti lets think about implications of running these changes on the server. As I understand, right now, scrapers don't fail on incorrect http status errors. Will this tear down all scraping once an error is encountered or will it just skip to the next scraper?

@ebleb
Copy link
Contributor Author

ebleb commented Dec 22, 2020

I'm not sure if that answers your question, but the changes made lead to scrapy only closing the spider that encountered the error.

@ubergesundheit
Copy link
Member

Just a quick heads up: I'm planning to merge #268 before this, If you don't mind to be patient, propbably your PR needs some changes then.

@ubergesundheit ubergesundheit added the component/backend Issues related to the backend/database label Dec 31, 2020
@ubergesundheit
Copy link
Member

@ebleb would you be so kind to adapt your PR for the changed backend?

@ebleb
Copy link
Contributor Author

ebleb commented Jan 1, 2021

The scrapers should now work with the new backend. The build fails, probably because I don't have access to the docker login secrets as I'm not a collaborator.

Anyone with collaborator access to [a] repository can use [...] secrets for Actions.

@ubergesundheit ubergesundheit merged commit 0d0efb7 into codeformuenster:master Jan 1, 2021
@ubergesundheit
Copy link
Member

Thanks :)

@ebleb ebleb deleted the scrapy-error-invalid-code branch January 1, 2021 18:02
@ebleb ebleb mentioned this pull request Jan 2, 2021
Closed
ubergesundheit pushed a commit that referenced this pull request Jan 12, 2021
@ebleb @ubergesundheit
scrapy raises exception on error status code (#308)
eb9c61b 
* scrapy now raises an exception when the request returns an error status code

* fixes performance issue

* fixed styling

* refactor

* fixes performance issue
ubergesundheit added a commit that referenced this pull request Jan 19, 2021
@ubergesundheit @dependabot
@ebleb @jdemaeyer
Calendar button (#274)
1269841 
* Remove DateArrow

* add calendar popup

* super little style tweaks

* Bump @types/react from 16.9.49 to 17.0.0 in /frontend (#291)

* Bump typescript from 4.0.3 to 4.1.2 in /frontend (#290)

* Bump eslint-plugin-react from 7.21.3 to 7.21.5 in /frontend (#289)

* deprecate set-env (#292)

* Bump eslint-plugin-jsx-a11y from 6.4.0 to 6.4.1 in /frontend (#295)

* Bump @popperjs/core from 2.5.3 to 2.5.4 in /frontend (#294)

* Bump @testing-library/jest-dom from 5.11.4 to 5.11.6 in /frontend (#293)

* Bump @testing-library/react from 11.0.4 to 11.2.2 in /frontend (#296)

* Bump @testing-library/user-event from 12.2.2 to 12.5.0 in /frontend (#298)

* Bump @types/jest from 26.0.15 to 26.0.16 in /frontend (#297)

* Bump @types/node from 14.14.6 to 14.14.13 in /frontend (#302)

* Bump actions/setup-node from v2.1.2 to v2.1.3 (#299)

* Bump @types/react-dom from 16.9.8 to 17.0.0 in /frontend (#301)

* Bump actions/setup-node from v2.1.3 to v2.1.4 (#305)

Bumps [actions/setup-node](https://github.com/actions/setup-node) from v2.1.3 to v2.1.4.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](actions/setup-node@v2.1.3...c46424e)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix react-router scrolling bug (#306)

* fix react-router scrolling bug

* fixed styling

* fixed styling

* Rewrite backend with Django (#268)

* Rewrite backend with Django

* Black-ify Django backend

* Fix missing backend dependency

* Remove trailing slahes from backend

* Add date, location, and organizer filters to events API

* Update backend README

* Black-ify all the things, again...

* Add Django admin for events

* Update backend flake8 config for new Django setup

* Fix help text for formatted_description field

* Update frontend API client

* Fix backend Docker container binding

* Use black to make files less readable

* Preserve event edits across crawls

* add neccessary env stuff for django to infrastructure

* prepare backend for deployment & add nginx static container

* try fix ci

* maybe - instead of _

* try out build-push-action@v2 for local caching

* use dest instead of src for cache-to

* try to address container source tag with digest

* try out local registry

* initialize buildx with network=host

* fix the yaml

* use correct build-arg

* Mark all readOnly properties as required in OpenAPI Spec

* Add hardcoded 400 and 404 response schemas to OpenAPI spec

* add updated generated-client

* change useGetEvents hook to accomodate changed backend

* attempt to fix client check in ci

* debug client check in ci

* disable client check for now

* add ssh-key gerald

* shave a step from backend dockerfile

* change migration command & rename static service in deployment compose file

* add some deployment docs

* change staging commands to use django

* add ansible.cfg

Co-authored-by: Gerald Pape <[email protected]>
Co-authored-by: ubergesundheit <[email protected]>
Co-authored-by: Gerald Pape <[email protected]>

* Django staging fixes (#312)

* fix env for api container

* enable cors for django

* try reenabling api client validation

* Django staging fixes 2 (#313)

* rename traefik router of api-static

* document admin user creation

* Django staging fixes 3 (#314)

* fix DJANG_DEBUG env default

* change production deployment command in ci

* set SESSION_COOKIE_SECURE and CSRF_COOKIE_SECURE to True

* Upgrade deps (2020-12-31) (#315)

* upgrade deps

* revert dev stuff

* changes how date paths are generated (#311)

* changes how date paths are generated

* fixes import statement

Co-authored-by: Gerald Pape <[email protected]>

* scrapy raises exception on error status code (#308)

* scrapy now raises an exception when the request returns an error status code

* fixes performance issue

* fixed styling

* refactor

* fixes performance issue

* frontend deps upgrade 2020-01-04 (#322)

* Bump ky from 0.24.0 to 0.25.1 in /frontend (#317)

Bumps [ky](https://github.com/sindresorhus/ky) from 0.24.0 to 0.25.1.
- [Release notes](https://github.com/sindresorhus/ky/releases)
- [Commits](sindresorhus/ky@v0.24.0...v0.25.1)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump csp-html-webpack-plugin from 4.0.0 to 5.0.1 in /frontend (#319)

Bumps [csp-html-webpack-plugin](https://github.com/slackhq/csp-html-webpack-plugin) from 4.0.0 to 5.0.1.
- [Release notes](https://github.com/slackhq/csp-html-webpack-plugin/releases)
- [Commits](slackhq/csp-html-webpack-plugin@v4.0.0...5.0.1)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* remove staticmethod (#323)

* Bump lxml from 4.5.2 to 4.6.2 in /backend (#324)

Bumps [lxml](https://github.com/lxml/lxml) from 4.5.2 to 4.6.2.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](lxml/lxml@lxml-4.5.2...lxml-4.6.2)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @types/jest from 26.0.19 to 26.0.20 in /frontend (#328)

* Bump eslint-plugin-prettier from 3.3.0 to 3.3.1 in /frontend (#327)

* Bump ky from 0.25.1 to 0.26.0 in /frontend (#325)

* Bump @types/node from 14.14.19 to 14.14.20 in /frontend (#326)

* Bump @testing-library/react from 11.2.2 to 11.2.3 in /frontend (#329)

* make calendar only full width for xs screens

* add close button to calendar

* align calendarbutton

* fiddle with spacings a little

* change size of date button in calendar again to keep it circular

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Benedikt Eble <[email protected]>
Co-authored-by: Jakob de Maeyer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
component/backend Issues related to the backend/database
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ebleb @ubergesundheit