Skip to content

Adds downloading user games from Chess.com and cleans up downloading user games from Lichess.org #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pywz wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Adds downloading user games from Chess.com and cleans up downloading user games from Lichess.org #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e1bc934
Delete .DS_Store
pywz May 5, 2022
42642b4
Adds downloading from Chess.com and other edits
pywz May 6, 2022
62bc377
Remove comment
pywz May 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed .DS_Store
View file
Open in desktop
Binary file not shown.
54 changes: 38 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

## About

This is a python application dedicated to creating chess puzzles/tactics from a pgn file.
Also it can download your games from lichess.org and use that file.
This is a Python application dedicated to creating chess puzzles/tactics from a pgn file.
Also, it can download your games from Lichess.org or Chess.com and use that file to create the chess puzzles/tactics.

It's based on the great [https://github.com/clarkerubber/Python-Puzzle-Creator](https://github.com/clarkerubber/Python-Puzzle-Creator) by @clarkerubber
It is based on the great [https://github.com/clarkerubber/Python-Puzzle-Creator](https://github.com/clarkerubber/Python-Puzzle-Creator) by @clarkerubber

Things that I changed:
- Use a local pgn file with games as a source.
Expand All @@ -21,14 +21,14 @@ It uses a different approach to create tactics, so probably it will generate a d

## Installation

This script requires the *Requests* and *Python-Chess* libraries to run, as well as a copy of *Stockfish*
Is recommended that you use Python 3 and pip3. But it could work with Python 2.7 and pip (probably you will need to install futures `pip install futures` )
This script requires a copy of `Stockfish` and the `Requests`, `Python-Chess`, and `Lichess` libraries to run.
Is recommended that you use Python 3 and pip3, but it could also work with Python 2.7 and pip (you will likely need to install futures `pip install futures`).

Please, take a look at [development doc](DEVELOPMENT.md) for details.

### Install requirements

`pip3 install -r requirements.txt --user`
`pip3 install -r requirements.txt`

### Setup

Expand All @@ -37,22 +37,27 @@ MacOS / Linux : `sh build-stockfish.sh` to obtain the current lichess Stockfish
## Launching Application

### Downloading games for a specific user
You can download games from a specific user using this command:
`python3 download_games.py <lichess username>`
You can download games from a specific user from either Lichess.org or Chess.com.

For a Chess.com user:
- `python3 download_games.py <chessdotcom username> --site chessdotcom`

By default, it will download the last 60 games from blitz, rapid and classical.
<br>

**Arguments**
For a Lichess user:
- `python3 download_games.py <lichess username> --site lichess`

You can use the `max` argument to get more games and use the lichess api token with the `token` argument to make the download faster. https://lichess.org/api#operation/apiGamesUser
or

It will save the games in the `games.pgn` file
- `python3 download_games.py <lichess username>`

<br>

**Arguments**

**Example to get 100 games using the token**
You can use the `max` argument to adjust the amount of games to query for Lichess.org (by default the max is 60 games)

`python3 download_games.py <lichess username> --max 100 --token 123456789`
`python3 download_games.py <lichess username> --max 150`

### Downloading games from tournaments
You can download games from multiple tournaments using this command:
Expand Down Expand Up @@ -94,8 +99,25 @@ The `result header` is the tactic result and not the game result. It can be load

## Problems?

#### Stockfish errors
### Stockfish errors
- If you have problems building Stockfish try downloading Stockfish directly https://stockfishchess.org/download/

### ssl.SSLCertVerificationError
If you receive the following error:
```
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed: unable to get local issuer certificate
```
**MacOS Solution**
1. Check which Python version you're using
`$ python3 --version`
2. Go to the current Python version's directory like so:

"Macintosh HD" >> "Applications" >> folder of Python version from last step

(If you are having trouble finding the Macintosh HD directory read [this](https://discussions.apple.com/thread/5207145))
3. Double click on `Install Certificates.command` file


## Want to see all my chess related projects?
Check [My projects](http://vitomd.com/blog/projects/) for a full detailed list.
Check [my projects](http://vitomd.com/blog/projects/) for a full detailed list.
46 changes: 37 additions & 9 deletions download_games.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
#!/usr/bin/env python

"""Downloading chess puzzles for lichess.org"""
"""Downloading chess puzzles for Lichess.org"""

import argparse
import logging
import sys
import urllib

import requests
import lichess

parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--token", metavar="TOKEN", default="",
Expand All @@ -18,15 +20,41 @@
help="substantially reduce the number of logged messages")
parser.add_argument("--max", metavar="MAX", default="60",
help="max number of games")
parser.add_argument("--site", metavar="SITE", default="lichess", help="Website to query user game data")

settings = parser.parse_args()
logging.basicConfig(format="%(message)s", level=settings.loglevel, stream=sys.stdout)

logging.basicConfig(format="%(message)s", level=settings.loglevel, stream=sys.stdout)
logging.debug("Downloading games from: " + settings.username)

response = requests.get(
'https://lichess.org/api/games/user/' + settings.username + '?max=' + settings.max + '&token=' + settings.token + '&perfType=blitz,rapid,classical&opening=true')
pgn = str(response.text)
all_games = open("games.pgn", "w")
all_games.write(pgn)
all_games.close()
logging.debug("Finished. Pgn is in games.pgn ")

if settings.site == "lichess":
with open("games.pgn", "w") as new_file:
myclient = lichess.Client()
new_file.write(myclient.export_by_user(settings.username, max_games=settings.max))
new_file.close()
logging.debug("Finished. Pgn is in games.pgn ")
elif settings.site == "chessdotcom":
url_chessdotcom = "https://api.chess.com/pub/player/" + \
settings.username.lower() + "/games"
url_archive = url_chessdotcom + "/archives"
http_response = urllib.request.urlopen(url_archive)

archives = http_response.read().decode("utf-8")
archives = archives.replace("{\"archives\":[\"", "\",\"")
archive_dates = archives.split("\",\"" + url_chessdotcom)
Copy link
Owner

@vitogit vitogit May 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be better if the dates are sorted from recent to older. So you can always download the latest games by default.

archive_dates[len(archive_dates) - 1] = archive_dates[
len(archive_dates) - 1].rstrip("\"]}")

with open("games.pgn", "w") as new_file:
Copy link
Owner

@vitogit vitogit May 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will need the max param also for chess.com
One way to do it is use parse the json from https://api.chess.com/pub/player/hikaru/games/2017/04 to count the number of games. and get the pgn with .pgn

for i in range(len(archive_dates) - 1):
cur_url = url_chessdotcom + archive_dates[i + 1] + "/pgn"
cur_filename = archive_dates[i + 1].replace("/", "-")
response = requests.get(cur_url, "./" + cur_filename + ".pgn")
new_file.write(response.text)
new_file.close()
logging.debug("Finished. Pgn is in games.pgn ")
else:
logging.debug("Invalid argument for site: only arguments allowed are lichess and chessdotcom")


1 change: 1 addition & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
colorama==0.4.4
chess==1.5.0
requests==2.20.0
lichess>=0.1.9
Comment on lines 1 to +4
Copy link
Owner

@vitogit vitogit May 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a problem installing this and had to change it to:

Suggested change
colorama==0.4.4
chess==1.5.0
requests==2.20.0
lichess>=0.1.9
colorama==0.4.4
chess==1.5.0
requests==2.27.1
lichess>=0.1.9

This was the error

ERROR: Cannot install -r requirements.txt (line 4) and requests==2.20.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested requests==2.20.0
    lichess 0.1.9 depends on requests==2.27.1