0

[Mohamed-SayedAlAhl]

[anilabhadatta]

@Mohamed-SayedAlAhl

1. Small Request is to add functionality of copying code from code blocks with the copy button, that will be lovely!!
   -- This feature is already added from the initial version. Please check again
2. This request is not important if it affects the performance of the scraper. The quizes are scraped not by order. The last question appears first, then the first, and so on, and then the last repeated again at the end.
   -- The Last Quiz will always show at first since it is the actual quiz container. At most i can remove it at end of scraping but that might cause issues in next page. Secondly the rest of the quizzes are basically screenshots.
3. I would like to see it scraping the solution of puzzles
   -- Send link of those topics that has puzzle. Not sure if i can implement but will see.
4. That is an important request, so I hope you consider investigating it. The scraper doesn't scrape the tests at all. I tried it with Python test on the platform, and it only scraped the overview of the exam and skipped it, which is bad. I would like to know what inside it!!
   -- Send link of those topics.
5. That is not related to the scraper, but the viewer has a technical issue (we can't call it an issue), as its hyperlinks of the lesson are not working.
   -- Send me the HTML, if it doesn't work then it should a screenshot probably. SingleFile scraps everything including hyperlinks.

anilabhadatta/educative-viewer#8
This issue is not solvable, instead use the text file that has the codes inside it.

[Mohamed-SayedAlAhl]

Hi @anilabhadatta,

Thanks for replying. For the first request of the copy, it is still not working, and I am here taking about the button itself, not the ability to copy the code by highlighting the text and copying. I will give you a screen of it.
.

2- Okay, no problem being able to do so, even if they are screenshots, is appreciated, so I can't say anything else.

3- Okay, I will send a link of one of the puzzles here. I don't know how many they are and in which lessons, but I will send you what I came across. here:
https://www.educative.io/module/page/Y6GKZ1igOAW0XJlmE/10370001/6235804428140544/6162077808066560

4- I will send links for this too, here they are:
https://www.educative.io/courses/learn-python-3-from-scratch/R1xBQBMxxmw/assessment
https://www.educative.io/courses/learn-python-3-from-scratch/BnwkPnm7VKx/assessment

And I think they are more than just these two, in other tracks, of course.

5- Again, you misunderstood my request. I am here taking about the viewer itself, not individual HTML files, which are working well. As you said, instead I am taking about when opening any of those HTML, but using the viewer tool. You can try it yourself with any HTML file. The hyperlink will not move you across the parts of the page like it does when you open the HTML file the regular way. And by parts here, I mean move you across the parts of the same lesson, not around the Html files.
Here screen to know what I mean:

Clearly in the screenshot, I clicked the hyperlink, but instead it didn't move me to the position. And yes, if I closed the viewer and simply opened the HTML file by hand, it works perfectly.

The last one you can tell me is that issue inside Chrome itself or what? As it works well on Firefox. I usually use Chrome, so I raised the issue.

[anilabhadatta]

@Mohamed-SayedAlAhl the first request cannot be fullfilled since code containers doesn't contain the complete code while scraping. This is why i scraped the codes beforehand and saved it in a txt file.
Also singlefile html cannot be parsed using bs4

Rest i will see.

[Mohamed-SayedAlAhl]

this the error that appears when trying to scrape this page:
Found Issue, Going Next Course Message: JavaScript error: Cannot read properties of null (reading 'click')
(Session info: headless chrome=96.0.4664.110)

[anilabhadatta]

@Mohamed-SayedAlAhl Fixed, clone version 8.2

[Mohamed-SayedAlAhl]

okay

[Mohamed-SayedAlAhl]

Hi @anilabhadatta ,
I scraped this course and it was all great, except a weird issue with the viewer for one page, as it is not shown in the viewer, and gave an error:
Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

I tried several times, but no luck. All the course content is shown well in the viewer, except this one:
https://www.educative.io/courses/grokking-the-machine-learning-interview/xoxmjX77JGP
This link of the page in the course I am talking about. The page shows well. If I opened it outside the viewer, but it breaks the viewer!!

Also, a weird HTML code appears in the console when the viewer attempts to open it.

[Mohamed-SayedAlAhl]

@anilabhadatta
Still not showing all answers for puzzles, don't know why! It skipped two here, then showed the solution for the last one
https://www.educative.io/module/page/Y6GKZ1igOAW0XJlmE/10370001/6235804428140544/6162077808066560

[Mohamed-SayedAlAhl]

Hi @anilabhadatta,
An infinite loop problem in scraper occurs when scraping content of this course.
Here is an example of one of the pages in the course that make an infinite loop when trying to scrape it:
https://www.educative.io/courses/grokking-coding-interview-patterns-python/7DyPRrN5OG8

This course is good, so I hope you solve the infinite loop, and it will be good to allow it to scrape test cases along with the code.

[anilabhadatta]

@Mohamed-SayedAlAhl I don't know from where these arrows are coming from.
I have fixed the issues related to educative scraper.
The codes might not be saved locally in some cases , will look into this issue later

[anilabhadatta]

@Mohamed-SayedAlAhl https://www.educative.io/courses/grokking-the-machine-learning-interview/xoxmjX77JGP
This is not fixable as of now since the HTML contains Katex code and jinja is unable to render it properly, will see in future if that can be fixed.

[Mohamed-SayedAlAhl]

@anilabhadatta
Thanks a lot ❤
Keep the great work💪

[Mohamed-SayedAlAhl]

@anilabhadatta
Small thing here is the scraper didn't scrape the code, and there are two tabs, and one of them is of course not showing.
And it would be good if you add the ability to scrape test cases with them. I mean to run the code if there is a test case window, and scrape the tabs of the results of each one as screenshots, like the quizzes. That will be awesome!!

https://www.educative.io/courses/grokking-coding-interview-patterns-python/7DyPRrN5OG8

The grokking courses are the best in educative, so I hope you fix these ones especially

[Mohamed-SayedAlAhl]

@anilabhadatta
Can you add the ability to start assessment and scrape the questions:
https://www.educative.io/courses/learn-python-3-from-scratch/R1xBQBMxxmw/assessment
I know the answers won't show, but it's okay, we just want the questions. You can even make a text note with the assessment folder saying that those answers are random and can't rely on them!!

[Mohamed-SayedAlAhl]

Hi @anilabhadatta,
This one is important. The scraper didn't scrape the code blocks, it only scraped the main.py and the same thing in the solution.
Please check if you can fix it, and if possible add the text case request I asked above.
Here is a link for example of the pages with the problem. I think all grokking course pages have the same thing, they have the same style.
https://www.educative.io/courses/grokking-coding-interview-patterns-python/qVvv9BMnE52
And in the solution page after it, it scraped the main.py more than once and didn't scrap the other codes as well

[anilabhadatta]

@Mohamed-SayedAlAhl I will probably add a new system to scrape the code containers in upcoming version.
Regarding the Assessment, there are some issues with scraping that topic - mainly because the course name could not be found if the first URL to scrape is the assessment link - will cause directory issues. If that problem can be fixed then the assessment and projects can be scraped.

[Mohamed-SayedAlAhl]

@anilabhadatta
That will be cool, as the scraping of the code sometimes is not working at all.
I found out this yesterday when trying to scrape the grokking courses.
And try to add features of scraping test cases too.
I found that interview courses that focus on problem solving skills, or the grokking courses with tones of code with test case examples, to ensure the algorithm is running as expected. So try to add a way to scrape those test cases, as text or image anyway will be fine.

[Mohamed-SayedAlAhl]

@anilabhadatta
Chrome flashing when refreshing the page in the viewer, or clicking next or back buttons or opening code in Windows Explorer. All of them make Chrome flash and load the page again. Can you fix this small issue?
I know it's no thing, but it really discomforts my eye

[Mohamed-SayedAlAhl]

@anilabhadatta
I didn't manage to solve it totally.
I only solved the issue of the white frame, but the blinking still happens.
I don't know why it happens in Chrome, not Firefox. In Firefox, it works great.
Here is how it works on Chrome:
https://app.screencast.com/wfSKz101KZPKh
and here on Firefox:
https://app.screencast.com/OeEG9yzwr131v

I just merged the CSS with the custom CSS in the local version and removed the line of calling the style from online. I did that so the CSS was consistent even when I was not using the Internet, and I usually don't use other devices, so that works for me. the problem of the white frame was conflicting in the CSS.
The blinking I don't know the reason, but I came across some blogs that say it's related to JS
And maybe related to how Chrome treats local hosts as insecure, while Firefox treats them as secure.

[anilabhadatta]

@Mohamed-SayedAlAhl Clone v2.7 of Educative-viewer, the jittering issue is not fixed at the expense of redesigning the UI which i think is much better than the previous one.

[anilabhadatta]

@Mohamed-SayedAlAhl Upto version 2.9 the jittering is fixed, In version 3.0 I have added a sidebar for navigation which may cause jittering. In firefox the jittering isn't happening to I will recommend you use that browser.

[Mohamed-SayedAlAhl]

There is another issue that appears in Firefox. Random "< >" across the text.

[Mohamed-SayedAlAhl]

@anilabhadatta
Why you added this slide!!
It is hiding the content and doesn't have any purpose!!
Why add it!!
If someone wants to go to the other page, he will use the next and back or return to the menu!!
The idea is great, but it doesn't seem great as of now, holding info beneath it!

[Mohamed-SayedAlAhl]

@anilabhadatta
I recommend you remove this useless side bar, as it blocks the content.
The app was near perfect, but you destroyed it with this side bar!!

[Mohamed-SayedAlAhl]

@anilabhadatta
And why does when using the next button, then refreshing the refresher instead, go to the next page!!

[anilabhadatta]

@Mohamed-SayedAlAhl

The single file extension adds the br tags. So i can't fix it as of now.

[Mohamed-SayedAlAhl]

@anilabhadatta
Okay thanks

[anilabhadatta]

@Mohamed-SayedAlAhl gildas-lormeau/SingleFile#1164
This problem should be fixed now. Try scraping it again. Clone latest repo if possible although not needed.

[Mohamed-SayedAlAhl]

@anilabhadatta

I tried to scrape the content again without changing anything and it worked perfectly now!!
thanks!!

[anilabhadatta]

@Mohamed-SayedAlAhl There is no change in the repo because the single-file script was updated in an external repo

[Mohamed-SayedAlAhl]

Hi @anilabhadatta ,
Have you found a way to solve the issue of the images text being altered?

[Mohamed-SayedAlAhl]

And sometimes the images are just scraped as black colored images and the text inside it is also black , so it is hard to read. this is less common, but it is annoying too.

[Mohamed-SayedAlAhl]

@anilabhadatta
this course has a lot of images for illustration and they are mostly corrupt by the scraper so, please fix this as quick as possible.
https://www.educative.io/courses/grokking-modern-system-design-interview-for-engineers-managers

[anilabhadatta]

@Mohamed-SayedAlAhl This is not solvable through single-file as this issue relates to the Single-File repo. Maybe creating a screenshot may help but I am unsure about the implementation.
I am busy for a few weeks, so I hardly get time to look into this issue.

[Mohamed-SayedAlAhl]

@anilabhadatta
The issue of the images being blacked is more common than I thought.
The distortion makes it impossible to read any text inside them. I had to open
the site again!!
Please, try to fix these issues.
I have noticed that all photos of the slides panels in educative are scraped perfectly. The issue appears in the single images only.

[Mohamed-SayedAlAhl]

@anilabhadatta
your code works, but I would prefer if this option is only applied on photos that has a background black.
I tried these codes but they Don't seem to work any idea why?

if (svg_element.style.backgroundColor === "rgb(0, 0, 0)" || svg_element.style.backgroundColor === "#15151E") {{
                                        svg_element.style.backgroundColor = "white";
                                    }}

if (svg_element.style.backgroundColor =="black") {{
                                        svg_element.style.backgroundColor = "white";
                                    }}

[anilabhadatta]

@Mohamed-SayedAlAhl It won't work because you have to look into the original webpage. There is no style in SVG element so the condition will not hold true.
According to my logic, this svg will work at any picture because these tags contains text that are black in color so background white will only make them visible.

[Mohamed-SayedAlAhl]

@anilabhadatta

Did you find out why this happens to these specific images?
Is this really the issue "I think the scraper is too fast that it scrapes the image too quickly that doesn't even load yet."
If so can we make a delay so that the scraper waits a bit for the image to load properly?

[Mohamed-SayedAlAhl]

I am using this color as it is usually used in educative as background and it is not glowing, so it won't hurt the eye. and it will work well if there is a white object in the image like here for example:
https://www.educative.io/courses/grokking-modern-system-design-interview-for-engineers-managers/B8R4pw6XyQW

this is the color degree I am using now and it is great:

svg_element.style.backgroundColor = "#E5E5E5";

[anilabhadatta]

@Mohamed-SayedAlAhl will look into it later on.

[Mohamed-SayedAlAhl]

@anilabhadatta

If I can ask you where did you learn Web Scraping? Which Tutorial or Platform did you learn a lot from?

[anilabhadatta]

@Mohamed-SayedAlAhl I learned it on my own actually

[Mohamed-SayedAlAhl]

Hi @anilabhadatta,

This course has different ways of teaching. Sometimes it has a dialogue that appears explaining the concept more when you click on the sentence with a line under it. like in this page:
https://www.educative.io/courses/operating-systems-virtualization-concurrency-persistence/7AOmG87QDg1
the scraping tool is not catching the dialogues do you have a way of making it catch them?

[Mohamed-SayedAlAhl]

and of course to make them only appear when clicking them, not always, as this will hide the other lines

[anilabhadatta]

@Mohamed-SayedAlAhl not planned as of yet, It is not possible as it hides the content behind it

[Mohamed-SayedAlAhl]

@anilabhadatta
I am having a problem with the scraper, it just stops suddenly after scraping around 160 lessons. This course has many lessons, but I found it upsetting, especially when I open the log file and find no errors have been recorded there. So I must ask, have you made a limitation to the program or that something related to eductive itself!!

https://www.educative.io/courses/operating-systems-virtualization-concurrency-persistence

[anilabhadatta]

@Mohamed-SayedAlAhl If there is no log present in txt file then look into the terminal output and paste it here.
It is always recommended to paste debug data from terminal as it allows me to see which function was last accessed.
And you will find the topic url of each topic in that debug info.
Probably will be skipping assignment /project or something like that.

[Mohamed-SayedAlAhl]

@anilabhadatta
Hi,
I hope you are okay.
Educative has changed their platform a little and now every lesson has a mark complete button beside the next button.
The scraper was working great ,but now it keeps repeating the same lesson more than once before skipping to the next lesson.
I hope you solve this small issue. I really like your tool and always using it ,so I would like to see it working again.

[anilabhadatta]

@Mohamed-SayedAlAhl
f22caaf
fixed in the latest commit
I was out of the station for a few days.

[Mohamed-SayedAlAhl]

Thanks a lot

[Mohamed-SayedAlAhl]

Hi @anilabhadatta ,
There is a weird issue I have encountered now. some slides have been bypassed without being scraped.
this lesson : https://www.educative.io/courses/beginner-to-advanced-computing-and-logic-building/JPLj96ZGAoo
if you tried to scrape it alone it will be okay and the slides will be scraped ,but if you try to scrape starting from the first lesson which is what I usually do , It won't scrape the slides. Can you solve this situation and why it skipped it in the first place!!

[Mohamed-SayedAlAhl]

@anilabhadatta
can u make the text after scanning appears a little , it appears a little pale ( the white text is almost faded )?

[anilabhadatta]

@Mohamed-SayedAlAhl Not planned as of yet since SingleFile may not work correctly on some pages and inserting the images and then using singleFile to get the page Data will cause high memory consumption and timeout issues. I think you can visit the questions after reading the whole HTML, it won't cause problems in readability.

To compare the diff from 8.5 to 9.1, mostly there were fixes in the quiz right button, quiz screenshots, demark button function, next button function, Html2Canvas screenshot feature implementation for quiz screenshots, and minor changes.
https://github.com/anilabhadatta/educative.io_scraper/compare/master%40%7B50day%7D...master?diff=split

The only new issue that I recently faced was a CORS issue while injecting a third-party script like singleFile or Html2Canvas.
Although the script was injecting successfully and the error was Report-Only type. Maybe in the near future, I will fix it or maybe the scraper will lose major implementations like quiz images, and single file, basically the features that are heavily dependent on third-party scripts.
Previous:

Now:

I have no idea how to fix/bypass it successfully without errors. Maybe some help is needed.
Also, I have noticed that a single file is working a bit slowly sometimes or maybe not working at all - It will basically create the same multiple folders of the same topic because the single file script may be in an infinite loop condition so all JS calls will stop working.
Possible Fix: Change the Internet or connect to VPN and restart the scraper.

Thank you.

[Mohamed-SayedAlAhl]

@anilabhadatta
the quiz here cannot be scraped, it appears as empty !!
https://www.educative.io/courses/grokking-computer-networking/gkz48qD8Y8Z

[anilabhadatta]

@Mohamed-SayedAlAhl pull latest commit, should fix the issues, also added the white color.

[Mohamed-SayedAlAhl]

@anilabhadatta
It works perfect now!
Thank you!!

[Mohamed-SayedAlAhl]

@anilabhadatta
can u fix this error:
the scraper stops and this appears on the chrome driver console

[1692285128.633][SEVERE]: Timed out receiving message from renderer: 30.000
[1692285158.643][SEVERE]: Timed out receiving message from renderer: 30.000

[anilabhadatta]

@Mohamed-SayedAlAhl educative has changed something in their website causing the single file extension to not work properly.
Yet to find a fix

[Mohamed-SayedAlAhl]

@anilabhadatta

I hope you find an alternative way as this stops the scraper from scraping which is really annoying specially when you are scraping a large module and this happens!!

[anilabhadatta]

@Mohamed-SayedAlAhl url?

[Mohamed-SayedAlAhl]

@anilabhadatta
after some time of scraping the scraper suddenly stops with this message appearing in the console:

Checking Login Function
Checking for captcha Function...
Scrolling Page
Getting File Name

This is a module page
Get File name module
Found Issue, Going Next Course Expected to find two og:title elements.
Exception, Driver exited list index out of range
Main Exception list index out of range
Press Enter to continue

[anilabhadatta]

@Mohamed-SayedAlAhl at this moment i am very busy so development will take time.
Provide the link will look into the fix later.

[Mohamed-SayedAlAhl]

@anilabhadatta

it happened more than once in different lessons.
last time it happened on this lesson: https://www.educative.io/courses/grokking-the-machine-learning-interview/xo0ZRvABl3q

but when I restart the scraper and then scraped the lesson , the scraper works but some times it persist on this error and I don't know what is causing it.

the same error appears in every lesson that is at the end of a module that is because the scraper cannot find the next button ,but this error doesn't bother me so much as you have already scraped the whole module, but it bothers me when it appears in lessons that are not in the end of the module!

[anilabhadatta]

@Mohamed-SayedAlAhl didnt get any error here

[Mohamed-SayedAlAhl]

@anilabhadatta

cannot scrape the page
https://www.educative.io/courses/grokking-the-behavioral-interview/YQXMlDkqpoA

Found Issue, Going Next Course Message:
Stacktrace:
Backtrace:
Ordinal0 [0x005F6903+2517251]
Ordinal0 [0x0058F8E1+2095329]
Ordinal0 [0x00492848+1058888]
Ordinal0 [0x004BD448+1233992]
Ordinal0 [0x004BD63B+1234491]
Ordinal0 [0x004E7812+1406994]
Ordinal0 [0x004D650A+1336586]
Ordinal0 [0x004E5BBF+1399743]
Ordinal0 [0x004D639B+1336219]
Ordinal0 [0x004B27A7+1189799]
Ordinal0 [0x004B3609+1193481]
GetHandleVerifier [0x00785904+1577972]
GetHandleVerifier [0x00830B97+2279047]
GetHandleVerifier [0x00686D09+534521]
GetHandleVerifier [0x00685DB9+530601]
Ordinal0 [0x00594FF9+2117625]
Ordinal0 [0x005998A8+2136232]
Ordinal0 [0x005999E2+2136546]
Ordinal0 [0x005A3541+2176321]
BaseThreadInitThunk [0x7564FA29+25]
RtlGetAppContainerNamedObjectPath [0x76ED7A7E+286]
RtlGetAppContainerNamedObjectPath [0x76ED7A4E+238]

Exception, Driver exited list index out of range
Main Exception list index out of range
Press Enter to continue

[Mohamed-SayedAlAhl]

@anilabhadatta

I installed the libraries again as there were two not in the old repo, but this error appears in 3-dev app when openning it. and close quickly I could only get a screenshot.

[anilabhadatta]

@Mohamed-SayedAlAhl pyinstaller wont work due to multiprocessing.
Use the steps provided in Readme.
Do clean install as well

[Mohamed-SayedAlAhl]

@anilabhadatta
I managed to open the gui but it is not scraping and not using Internet,also in the previous version I was knowing what is happening , but here the progress is always full. i go to the save directory that I added I found the new log file but no folders for the course has been added, I waited 5 min nothing, I thought using headless make this problem ,but It doesn't . What should I do to make it work.
Here is screenshot of the app.

[anilabhadatta]

@Mohamed-SayedAlAhl The scraper function is not yet implemented, you need to wait for few more days.
The progress bar is only shown for downloading the latest chromedrivers or chrome binaries. It doesn't mean scraping progress.
As of now only login feature will work.

[anilabhadatta]

@Mohamed-SayedAlAhl Clone latest repo for latest bug fixes. Still Scraper is not implemented. Also create a new Issue/ Discussion for v3 and rename this as v2

[anilabhadatta]

@Mohamed-SayedAlAhl Test Latest version, should fix most of the issues.
Also I will prefer using VPN for better singleFile responses.