Doesn't work since 2/24 8:00 AM UTC

[lota09]

from autoscraper import AutoScraper

url = 'https://stackoverflow.com/questions/2081586/web-scraping-with-python'

# We can add one or multiple candidates here.
# You can also put urls here to retrieve urls.
wanted_list = ["What are metaclasses in Python?"]

scraper = AutoScraper()
result = scraper.build(url, wanted_list)
print(result)

Above is example code written on the README.

Even there's phrase "What are metaclasses in Python?" in that page, empty array returned. ( [] )
since 2/24 8:00 AM UTC

[lota09]

Is there anyone has this working correctly?

[amine-bs]

I had the same problem. It looks like a problem with beautifulsoup4 version 4.13.x. I tested it with beatufilsoup 4.12.3 and it worked well

[lota09]

I had the same problem. It looks like a problem with beautifulsoup4 version 4.13.x. I tested it with beatufilsoup 4.12.3 and it worked well

Thanks for sharing.

For those who struggle with this problem :

1. If you install beautifulsoup4-4.12.3 like below this problem may be fixed.
   pip install beautifulsoup4==4.12.3

2. Or creating virtual environment is great solution for this.
   Here's requirements.txt for virtual environment.

autoscraper==1.1.14
beautifulsoup4==4.12.3
bs4==0.0.2
certifi==2025.1.31
charset-normalizer==3.4.1
idna==3.10
lxml==5.3.1
requests==2.32.3
soupsieve==2.6
typing_extensions==4.12.2
urllib3==2.3.0

place it on your project and type pip -m venv .venv then type below to activate venv.

✅ Windows (PowerShell)
.venv\Scripts\Activate
✅ Windows (CMD)
.venv\Scripts\activate.bat
✅ Linux/macOS (bash/zsh)
source .venv/bin/activate

Then type pip install -r requirements.txt then intact version of beautifulsoup and autoscraper will be installed.

Now I understand why my professors told me to use virtual environment...

[lota09]

Maybe we want to ask @alirezamika to update autoscraper to be compatible with latest version of bs4,
or designate beautifulsoup 4.12.3 as compatible dependency at setup.py
For those who encounter this issue in the future..