Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle existing URLs in citations #91

Open
wants to merge 67 commits into
base: master
Choose a base branch
from
Open

Handle existing URLs in citations #91

wants to merge 67 commits into from

Conversation

nemobis
Copy link
Member

@nemobis nemobis commented Dec 6, 2023

Bug: T196255

nemobis added 30 commits December 6, 2023 13:53
@nemobis
Update copyright guidance link
5623edf 
The Meta-Wiki page is out of step with recent legislation.
LIBER has helpfully provided new informational pages.
@nemobis
Revert "Remove multiprocessing"
997191d 
This reverts commit 1b5a04f.
@nemobis
Make refresh faster
83529b5 
* Introduce a timeout for the first Unpaywall phase.
* Throw away old suggestions to avoid wasting resources on stale data.
* Run the Dissemin prefill with its own Python script as the import
  takes most of the time when issued once per title.
@nemobis
Only add blank URL parameters where already present
72bdbef
@nemobis
Set header to python3, that's what we use already
49812ac
@nemobis
Better handle existing links
7d32576 
* Keep URL if it already offers a PDF. Useful for example when another
  repository link is present, or a resolver which goes to the same place.
* Add url-access for existing links which we can't override because there
  is no open access copy.
@nemobis
Don't fetch non-existing URL
338018e
@nemobis
Check for URL presence
20c9caf
@nemobis
Time out after 5 seconds on PDF test fetch
dd4f6fe 
Otherwise we may be stuck forever.
@nemobis
Use HTTPS
1421a33
@nemobis
Default for undefined oa_status in resp
412939b
@nemobis
Randomise titles to prefill
548a67b
@nemobis
Shuffle properly
63c764e
@nemobis
Use session, handle exception for missing resp
79b13c9
@nemobis
Handle requests exception
e5fe463
@nemobis
Also consider unknown OA status when no Unpaywall data
5c06e03
@nemobis
Handle Unpaywall connection errors
45f3431
@nemobis
Fix typo in template
963ef8d
@nemobis
Preserve Unpaywall oa_status, add tests, reduce url-access=limited
16e7389
@nemobis
Avoid spaces in comment edit
d066248
@nemobis
Actually return default oa_status
0ec5f89
@nemobis
Define oa_status default earlier
f8ffb30
@nemobis
Attempt to add preview in logged out main page
323acaa
@nemobis
Undefined page_name
7f55818
@nemobis
Undefined orig_hash
826c8e4
@nemobis
Simplify condition
c372e43 
Avoids

  TypeError: argument of type 'NoneType' is not iterable
@nemobis
Check for existence
0d25ec4
@nemobis
Check for length
00c422e
@nemobis
Don't check for http link
b4f32fa
@nemobis
Change classification for replaced URLs
5c24f47
nemobis added 25 commits December 11, 2023 01:00
@nemobis
Follow redirects when checking URLs
c92ed6e 
For example HAL is an HTTP 301:
https://hal.archives-ouvertes.fr/jpa-00241038/document
@nemobis
Use correct url variable
c7db11e
@nemobis
Check for variable existence in template
c0e5b35
@nemobis
Use correct name of variable in template
1deb1a9
@nemobis
Revert "Check for variable existence in template"
a6bf16d 
This reverts commit c0e5b35.
@nemobis
Also consider URL archival aliases
61f9c3a
@nemobis
Update default edit summary as links aren't always added
d092a88
@nemobis
Show proposed edit in frontpage (also when no link is available)
04baf29
@nemobis
Avoid relative import
e426f0a 
ImportError: attempted relative import with no known parent package
@nemobis
Add server_default sequence
44a752a 
https://docs.sqlalchemy.org/en/20/core/defaults.html#associating-a-sequence-as-the-server-side-default

Trying to avoid, during testing,

sqlalchemy.exc.StatementError: (sqlalchemy.exc.OperationalError) (pymysql.err.OperationalError) (4091, "Unknown SEQUENCE: 'userstats_id_seq'")
@nemobis
Use autoincrement in userstats table
98318a2 
Bug: T354144
@nemobis
Update UserStats comment to new database schema, OAuth ID
324745c
@nemobis
Add query to update the leaderboard
4917289 
Bug: T354144
@nemobis
Restore SS directory, move url-access edits too
b79a076
@nemobis
Clarify scope of Sherpa/Romeo advice: publisher information
79643eb
@nemobis
Check existing URL also when not adding a new one
1cd6152
@nemobis
Add test case for existing URL on closed DOI
76e5bb8
@nemobis
Continue looking for OA URLs and IDs if DOI is gold/hybrid/bronze
5b2012c
@nemobis
Remove extra printing
c784a67
@nemobis
Don't use requests session with multithreading
8b0776f
@nemobis
Remove special-casing for hdl, citeseerx
ddab25a 
We want to convert all hdl URLs and to rank all hdl, citeseerx URLs even
if they are not in the best_oa_location object.
@nemobis
Avoid undefined variable in frontpage
21985ab
@nemobis
Handle filenames beginning by hyphen, avoid error: "invalid option"
ce32e5f
@nemobis
Revert "Don't use requests session with multithreading"
8676a52 
Didn't help with avoiding hanging, probably caused by insufficient
CPU resources.

This reverts commit 8b0776f.
@nemobis
Restore citeseerx special casing
8895319 
Partially reverts ddab25a

Bug: T354471
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@nemobis