Skip to content

Bug — Webscraper not working for the upcoming semesters #905

@Tevetron

Description

@Tevetron

Describe the bug
Our web scraper works for all past and current semesters, however it has begun to crash when scraping upcoming semesters like Fall 2025.

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'cron-job' branch (if not merged yet)
  2. run 'python3 cron/no_login.py'
  3. wait and see if it crashes while scraping a course
  4. if this works, we proceed to checking the docker container instead of the script
  5. run 'docker compose -f docker-compose.yml -f docker-compose.development.yml up'
  6. view the 'yacs_cron' container by method of your choosing
  7. go to your 'command line' of choice in however you are viewing the docker container (this would be the 'exec' tab docker desktop
  8. see steps 2 and 3

Expected behavior

  1. The python file should run for a few minutes
  2. it will produce output about which types of courses it is currently scraping e.g. CSCI
  3. It should produce a csv file named after the semester after running

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions