edx-dl not able to download videos from edx platform

[MATRIX30]

🚨Please review the Troubleshooting section
before reporting any issue. Don't forget also to check the current issues to
avoid duplicates.

Subject of the issue

edx-dl fails to extract and download videos for "https://courses.edx.org/courses/course-v1:EdinburghX+PA1.1x+3T2019/course/" on www.edx.org
it seems the videos for this course are sourced from "https://media.ed.ac.uk/" and not youtube
Need help on resolving this issue

Your environment

• Operating System (name/version):windows 10 Professional
• Python version: 3.7.0
• youtube-dl version: 2019.09.28
• edx-dl version: 0.1.10

Steps to reproduce

--- create an account on Edx

--- enroll for the course "https://courses.edx.org/courses/course-v1:EdinburghX+PA1.1x+3T2019/course/"

---- type the following into CMD
edx-dl -u username -p password -o path --ignore-errors --cache https://courses.edx.org/courses/course-v1:EdinburghX+PA1.1x+3T2019/course/

Expected behaviour

download to start normally

Actual behaviour

edx_dl version 0.1.10
Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Downloading Introduction to Predictive Analytics [course-v1:EdinburghX+PA1.1x+3T2019/co]
Downloading 0 section(s)
loading 2329 urls from cache [edx-dl.cache]
Extracting all units information in parallel.
No downloadable video found.

[YukunXia]

Having the same issue :(

[mor3dr3ad]

Confirmed with different url:
https://courses.edx.org/courses/course-v1:MITx+14.750x+3T2019/course/

Output of --debug:

root[main] edx_dl version 0.1.10
root[parse_file_formats] file_formats: ['e?ps', 'pdf', 'txt', 'doc', 'xls', 'ppt', 'docx', 'xlsx', 'pptx', 'odt', 'ods', 'odp', 'odg', 'zip', 'rar', 'gz', 'mp3', 'R', 'Rmd', 'ipynb', 'py']
root[edx_get_headers] Building initial headers for future requests.
root[_get_initial_token] Getting initial CSRF token.
root[_get_initial_token] Found CSRF token.
root[edx_get_headers] Headers built: {'User-Agent': 'edX-downloader/0.01', 'Accept': 'application/json, text/javascript, /; q=0.01', 'Content-Type': 'application/x-www-form-urlencoded;charset=utf-8', 'Referer': 'https://courses.edx.org/login_ajax', 'X-Requested-With': 'XMLHttpRequest', 'X-CSRFToken': 'PUsSLjqYvxBtMFO07I7RfYRpxPPZdHE0zWBVoJk4aqqo8AOSciOeEoSTr49FvNeH'}
root[edx_login] Logging into Open edX site: https://courses.edx.org/login_ajax
root[get_courses_info] Extracting course information from dashboard.
root[get_courses_info] Data extracted: ["lotsofcourseswhichidontwanttoshare"]
root[get_available_sections] Extracting sections for :https://courses.edx.org/courses/course-v1:MITx+14.750x+3T2019/course/
root[get_available_sections] Extracted sections: []
root[_display_selections] Downloading Political Economy and Economic Development [course-v1:MITx+14.750x+3T2019/co]
root[_display_sections] Downloading 0 section(s)
root[extract_all_units_in_sequence] Extracting all units information in sequentially.
root[extract_all_units_in_sequence] urls: []
root[parse_units] No downloadable video found.

[adizukerman]

Same issue with https://courses.edx.org/courses/course-v1:MITx+2.830.2x+3T2019/course/

[ozhaggis]

Same issue with multiple courses.

edx_dl version 0.1.10
Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Downloading Data Science: Machine Learning [course-v1:HarvardX+PH125.8x+2T2019/co]
Downloading 0 section(s)
Extracting all units information in parallel.
No downloadable video found.

[EugeneLoy]

Same issue. Course: https://courses.edx.org/courses/course-v1:MITx+18.6501x+3T2019/course/

It looks like edx-dl is missing most of the sections of the course. In my example, it sees only 1 section, while edx site displays more than 5 (at the moment):

> edx-dl.py -u <username> --list-sections https://courses.edx.org/courses/course-v1:MITx+18.6501x+3T2019/course/
edx_dl version 0.1.10
Password:
Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Fundamentals of Statistics [course-v1:MITx+18.6501x+3T2019/co] has 1 sections so far
 1 - Download Entrance Survey videos

[not-lucky]

Here's mine...

Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Downloading Calculus Applied! [course-v1:HarvardX+CalcAPL1x+2T2019/co]
Downloading 3 section(s)
Section 1: Optional Sections (CHOOSE 1 of 3)
Optional Sections
Section 2: Section 12: Course Wrap Up
End of Course Survey
Course Feedback Forum (Optional)
Section 3: Acknowledgements
Course Team and Special Thanks
Section 1: What Makes a Good Test Question? Mathematical Models to Measure Knowledge and Improve Learning
Section 2: Economic Applications of Calculus: Elasticity and A Tale of Two Cities
Section 3: From X-rays to CT scans: Mathematics and Medical Imaging
Section 4: What is Middle Income? Thinking about Income Distributions with Statistics and Calculus
Section 5: Population Dynamics Part I: the Evolution of Population Models and Section 6: Population Dynamics II: A Biological Puzzle OR How Fishing Affects a Predator-Prey System
Section 7: Extinction, Chaos and other Bifurcation Behavior, Section 8: Bifurcation Part II: Outbreak! Budworm Populations and Bifurcations, Section 9: Bifurcation Part III: Species in Competition: Coexistence or Exclusion
Section 10: E = mc²: Taylor Approximation and the Energy Equation
Final Assessments
Extracting all units information in parallel.
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@944fb6867b354e2cafb41415aae41415'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@2101c542ac614691acc54224d3c314a8'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@5864500159ef40f9839d66d2492fea58'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@13aed97186fd4c7588a5ea1399e096df'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@a53371a01e9c4fd28dcb1a1609614da7'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@ebf2c858d37e418583f839965631108f'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@d4e29c075ff14ad583a3750767faf698'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@6cc97f049d444c4f8470b88ad3fdbc52'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@0f7edf523c55490e8380b6e9a809df33'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@fb7c4d1c1a2649b29e472b2ef86a36ce'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@edb436fadf2c4b74b175b9b5b6334b48'
Processing 'https://courses.edx.org/courses/course-v1:HarvardX+CalcAPL1x+2T2019/jump_to/block-v1:HarvardX+CalcAPL1x+2T2019+type@vertical+block@1ccb65aca6b34beda14dedfa6bffafbc'
Removed 0 duplicated urls from 0 in total
Output directory: Downloaded

[abeckman]

Same issue with multiple courses.

[lubaroli]

Same issue here, edx-dl only sees the first section.

Heres the log:
root[main] edx_dl version 0.1.10 root[parse_file_formats] file_formats: ['e?ps', 'pdf', 'txt', 'doc', 'xls', 'ppt', 'docx', 'xlsx', 'pptx', 'odt', 'ods', 'odp', 'odg', 'zip', 'rar', 'gz', 'mp3', 'R', 'Rmd', 'ipynb', 'py'] Password: root[edx_get_headers] Building initial headers for future requests. root[_get_initial_token] Getting initial CSRF token. root[_get_initial_token] Found CSRF token. root[edx_get_headers] Headers built: {'User-Agent': 'edX-downloader/0.01', 'Accept': 'application/json, text/javascript, */*; q=0.01', 'Content-Type': 'application/x-www-form-urlencoded;charset=utf-8', 'Referer': 'https://courses.edx.org/login_ajax', 'X-Requested-With': 'XMLHttpRequest', 'X-CSRFToken': 'wWr0eKCgnA1uusK8rQvzPJHFK8bXmxn4i1pxyGtnuxsy0MRE8LXYh87mk8DN1eST'} root[edx_login] Logging into Open edX site: https://courses.edx.org/login_ajax root[get_courses_info] Extracting course information from dashboard. root[get_courses_info] Data extracted: [Fundamentals of Statistics: https://courses.edx.org/courses/course-v1:MITx+18.6501x+3T2019/course/, TOEFL Test Preparation: The Insider’s Guide: https://courses.edx.org/courses/course-v1:ETSx+TOEFLx+3T2017/course/, Minds and Machines: https://courses.edx.org/courses/course-v1:MITx+24.09x+3T2015/course/, Practical Learning Analytics: https://courses.edx.org/courses/course-v1:MichiganX+PLAx+2T2016/course/, Embedded Systems - Shape the World: https://courses.edx.org/courses/course-v1:UTAustinX+UT.6.03x+1T2016/course/, The Science of Everyday Thinking: https://courses.edx.org/courses/course-v1:UQx+Think101x+2T2015/course/, Electronic Interfaces: https://courses.edx.org/courses/course-v1:BerkeleyX+EE40LX+2T2015/course/, Autonomous Navigation for Flying Robots: https://courses.edx.org/courses/TUMx/AUTONAVx/2T2014/course/, Next Generation Infrastructures - Part 2: https://courses.edx.org/courses/DelftX/NGI102x/3T2014/course/, Solar Energy: https://courses.edx.org/courses/DelftX/ET.3034TU/3T2014/course/, Circuits and Electronics: https://courses.edx.org/courses/MITx/6.002_4x/3T2014/course/] root[get_available_sections] Extracting sections for :https://courses.edx.org/courses/course-v1:MITx+18.6501x+3T2019/course/ root[get_available_sections] Extracted sections: [<edx_dl.common.Section object at 0x1042f6110>] root[_display_selections] Downloading Fundamentals of Statistics [course-v1:MITx+18.6501x+3T2019/co] root[_display_sections] Downloading 1 section(s) root[_display_sections] Section 1: Entrance Survey root[_display_sections] 1. Entrance Survey root[extract_all_units_in_parallel] Extracting all units information in parallel. root[extract_all_units_in_parallel] urls: ['https://courses.edx.org/courses/course-v1:MITx+18.6501x+3T2019/jump_to/block-v1:MITx+18.6501x+3T2019+type@vertical+block@entrancesurvey-tab1'] root[extract_units] Processing 'https://courses.edx.org/courses/course-v1:MITx+18.6501x+3T2019/jump_to/block-v1:MITx+18.6501x+3T2019+type@vertical+block@entrancesurvey-tab1' root[main] Removed 0 duplicated urls from 0 in total root[download] Output directory: Downloaded

[wzhuwz]

Looks like edx-dl is missing most of the sections of the course. My case https://courses.edx.org/courses/course-v1:GTx+ISYE6669+2T2018/course/.

Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Downloading FA18: Deterministic Optimization [course-v1:GTx+ISYE6669+2T2018/co]
Downloading 5 section(s)
Section 1: Getting Started
Welcome Message
Syllabus
Getting Help
Getting to Know Each Other
Section 2: Discussions and Q&A
Discussions and Q&A Forums
Section 3: Proctoring Information - Verified Learners
Section 4: Midterm Exam - Verified Learners
Section 5: Final Exam - Verified Learners
Extracting all units information in parallel.
Processing 'https://courses.edx.org/courses/course-v1:GTx+ISYE6669+2T2018/jump_to/block-v1:GTx+ISYE6669+2T2018+type@vertical+block@b4e0e428596e4a438b61d9c44a66ff45'
Processing 'https://courses.edx.org/courses/course-v1:GTx+ISYE6669+2T2018/jump_to/block-v1:GTx+ISYE6669+2T2018+type@vertical+block@6e0eef9f7a9b4eed99ea9c1ad8e37b16'
Processing 'https://courses.edx.org/courses/course-v1:GTx+ISYE6669+2T2018/jump_to/block-v1:GTx+ISYE6669+2T2018+type@vertical+block@d827bed0374e46b5a0abe62978b7cca8'
Processing 'https://courses.edx.org/courses/course-v1:GTx+ISYE6669+2T2018/jump_to/block-v1:GTx+ISYE6669+2T2018+type@vertical+block@3247cb48d14b4f1e97bb9dd74d1ec8a2'
Processing 'https://courses.edx.org/courses/course-v1:GTx+ISYE6669+2T2018/jump_to/block-v1:GTx+ISYE6669+2T2018+type@vertical+block@c49832c367cc47be96ba15a3ce5e9d8c'
Removed 0 duplicated urls from 0 in total
Output directory: Downloaded

[dorianherle]

I have the same issue:

edx_dl version 0.1.10
Password:
Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Downloading Introduction to Discrete Choice Models [course-v1:EPFLx+DiscreteChoiceX+3T2017/co]
Downloading 0 section(s)
Extracting all units information in parallel.
No downloadable video found.

[mor3dr3ad]

So, I've dug into the code a bit and I think I found the issue: for some courses, edx has again updated the structure of their website. The issue is with line 397 in /edx-dl/.parsing.py

    sections_soup = soup.find_all('li', class_='outline-item section')

In the new format, the sections have a different class, namely "outline-item section scored".

Should be easily fixed. will try to hack sth together, but this better be checked by so experienced.

[mor3dr3ad]

Alright, quick fix:

replace as follows in /edx_dl/parsing.py:

Line 385:
subsections_soup = section_soup.find_all('li', class_='vertical outline-item focusable') with subsections_soup = section_soup.find_all('li', class_=['vertical outline-item focusable', 'vertical outline-item focusable scored'])

and line 397:

sections_soup = soup.find_all('li', class_='outline-item section') with sections_soup = soup.find_all('li', class_=['outline-item section', 'outline-item section scored'])

This should work for both the 'old' and new format. Will try to run some tests and create a merge request sometime this week.

[not-lucky]

Thanks a lot.
Its working now.

[malawadd]

thank you it works now

[malawadd]

Alright, quick fix:

replace as follows in /edx_dl/parsing.py:

Line 385:
subsections_soup = section_soup.find_all('li', class_='vertical outline-item focusable') with subsections_soup = section_soup.find_all('li', class_=['vertical outline-item focusable', 'vertical outline-item focusable scored'])

and line 397:

sections_soup = soup.find_all('li', class_='outline-item section') with sections_soup = soup.find_all('li', class_=['outline-item section', 'outline-item section scored'])

This should work for both the 'old' and new format. Will try to run some tests and create a merge request sometime this week.

this partially works , it still misses some weeks and module i tried it on this course

https://courses.edx.org/courses/course-v1:CurtinX+MKT1x+1T2019/course/

and the entire module 3 didnt download