DIS export implementation (export_courses.sh) (#53)

* add sheet to yadisk table adding logic

* integrate yadisk logic to moodle export

* dont push to google if no table/sheet id

* dont push to google if no google_token in args

* if cls.args specified write data to sheets document

* move utils from scripts to moodle_export

* add openpyxl to moodle_export requirements

* cp yadisk_manager.py to moodle_export

* update yadisk version

* yatoken as args to write_sheet_to_file

* yatoken as args to DiskManager

* update default CSV_DELIMITER in moodle_export

* change write mode for logfile

* container stdout to file (error to console)

* fix indent for google sheet export

* update check_export yo yadisk

* update required args for checker_export

* add openpyxl to requirements (checker_export)

* update yadisk version (checker_export requirements)

* update load_data_from_dis

* update add_csv_to_table logic

* update add_csv_to_table (split rows)

* DIS export to csv file

* Update moodle_export.utils.py (clear sheet instead remove)

* Update checker_export.utils.py (clear sheet instead remove)

* clear sheet (insert_rows with 0)

* download_sheet_to_pdf -> download_sheet (file extension as arg)

* update download_sheet args

* update duplicateSheetsToYadisk logs

* update wget for download_sheet

* update abs_disk_path in upload_file_to_disk

* update duplicateSheetsToYadisk

* update duplicateSheetsToYadisk

* fix duplicateSheetsToYadisk

* rm debug logs

* add error handling in export_courses.sh

* if to case (export_courses.sh)

* checker_export: sheet_id to sheet_name

* improve export_courses.sh (implement checker export and rename env var)

* add workflows

* rm comment in Dockerfile (stepik_export)

* rename sheet_id to sheet_name

Author: HadronCollider

.github/workflows/build.yml

@@ -0,0 +1,23 @@
+name: CI
+
+on: pull_request
+
+jobs:
+  build:
+    runs-on: ubuntu-22.04
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build stepik_export image
+        if: ${{ !cancelled() }}
+        run: |
+          ./build.sh stepik_export
+      - name: Build moodle_export image
+        if: ${{ !cancelled() }}
+        run: |
+          ./build.sh moodle_export
+      - name: Build checker_export image
+        if: ${{ !cancelled() }}
+        run: |
+          ./build.sh checker_export

.github/workflows/label_merge_conflicts.yml

@@ -0,0 +1,15 @@
+name: 'Check for merge conflicts'
+
+on: [push]
+
+jobs:
+  find_conflicts:
+    runs-on: ubuntu-22.04
+
+    steps:
+      - uses: mschilde/auto-label-merge-conflicts@master
+        with:
+          CONFLICT_LABEL_NAME: "has conflicts"
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          MAX_RETRIES: 5
+          WAIT_MS: 5000

checker_export/README.md

@@ -3,5 +3,5 @@
 ## Инструкция по запуску скрипта
 
 ```bash
-python3 exporter.py [-h] [--google_token GOOGLE_TOKEN] --checker_token CHECKER_TOKEN [--checker_filter CHECKER_FILTER] --table_id TABLE_ID --sheet_id SHEET_ID [--yandex_token YANDEX_TOKEN --yandex_path YANDEX_PATH]
+python3 exporter.py [-h] [--google_token GOOGLE_TOKEN] --checker_token CHECKER_TOKEN [--checker_filter CHECKER_FILTER] --table_id TABLE_ID --sheet_name SHEET_NAME [--yandex_token YANDEX_TOKEN --yandex_path YANDEX_PATH]
 ```

checker_export/exporter.py

@@ -23,7 +23,7 @@ def parse_args():
     parser.add_argument('--checker_token', type=str, required=True, help='Specify session cookie for slides-checker')
     parser.add_argument('--checker_filter', type=str, required=False, help='Specify filter for slides-checker')
     parser.add_argument('--table_id', type=str, required=False, help='Specify Google sheet document id (can find in url)')
-    parser.add_argument('--sheet_id', type=str, required=False, help='Specify title for a sheet in a document in which data will be printed')
+    parser.add_argument('--sheet_name', type=str, required=False, help='Specify title for a sheet in a document in which data will be printed')
     parser.add_argument('--yandex_token', type=str, required=False, help='Specify Yandex token from https://oauth.yandex.ru/client/new application')
     parser.add_argument('--yandex_path', type=str, required=False, help='Specify output filename on Yandex Disk')
     args = parser.parse_args()
@@ -47,14 +47,14 @@ def load_data_from_dis(checker_filter, checker_token):
     return csv_path, df_data
 
 
-def write_data_to_table(checker_token, checker_filter, google_token, table_id, sheet_id, yandex_token=None, yandex_path=None):
+def write_data_to_table(checker_token, checker_filter, google_token, table_id, sheet_name, yandex_token=None, yandex_path=None):
     csv_path, df_data = load_data_from_dis(checker_filter, checker_token)
 
-    if google_token and sheet_id and table_id:
+    if google_token and sheet_name and table_id:
         gc = pygsheets.authorize(service_file=google_token)
         sh = gc.open_by_key(table_id)
 
-        wk_content = sh.worksheet_by_title(sheet_id)
+        wk_content = sh.worksheet_by_title(sheet_name)
 
         wk_content.set_dataframe(df_data, 'A1', copy_head=True)
 
@@ -69,7 +69,7 @@ def write_data_to_table(checker_token, checker_filter, google_token, table_id, s
 
 def main():
     args = parse_args()
-    write_data_to_table(args.checker_token, args.checker_filter, args.google_token, args.table_id, args.sheet_id, args.yandex_token, args.yandex_path)
+    write_data_to_table(args.checker_token, args.checker_filter, args.google_token, args.table_id, args.sheet_name, args.yandex_token, args.yandex_path)
 
 
 if __name__ == "__main__":

moodle_export/README.md

@@ -18,7 +18,7 @@
 
 Получение данных и в файл, и в Google таблицу:
 
-```python3 grades_parser.py grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_id "name of sheet"[,"name",..]] ```
+```python3 grades_parser.py grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_name "name of sheet"[,"name",..]] ```
 
 Получение данных и в файл, и на Яндекс Диск:
 
@@ -36,7 +36,7 @@
 
 Запуск докер контейнера с параметрами:
 
-``` docker run --rm grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_id "name of sheet"[,"name",..]] ```
+``` docker run --rm grades_parser --moodle_token "your token" --url http://e.moevm.info --course_id "your course id"[,"id",..] --csv_path "name of file" [--percentages] [--options github] --google_token "your google token" --table_id "your table id"[,"id",..] [--sheet_name "name of sheet"[,"name",..]] ```
 
 
 ## Получение Moodle Токена.

moodle_export/args_parser.py

@@ -9,7 +9,7 @@ def arg_parser():
     parser.add_argument('--csv_path', type=str, required=True, help='Specify path to output csv file')
     parser.add_argument('--google_token', type=str, required=False, help='Specify path to google token file')
     parser.add_argument('--table_id', type=lambda s: [i for i in s.split(',')], required=False, help='Specify Google sheet document id (can find in url)')
-    parser.add_argument('--sheet_id', type=lambda s: [i for i in s.split(',')], required=False, help='Specify title for a sheet in a document in which data will be printed')
+    parser.add_argument('--sheet_name', type=lambda s: [i for i in s.split(',')], required=False, help='Specify title for a sheet in a document in which data will be printed')
     parser.add_argument('--yandex_token', type=str, required=False, help='Specify Yandex token from https://oauth.yandex.ru/client/new application')
     parser.add_argument('--yandex_path', type=str, required=False, help='Specify output filename on Yandex Disk')
     parser.add_argument('--percentages', required=False, action='store_true', help='If set then grades will be printed as percentages')

moodle_export/grades_parser.py

@@ -153,17 +153,17 @@ def main(cls):
                         elif i == len(cls.args.table_id) - 1:
                             table_id = cls.args.table_id[i]
 
-                    if cls.args.sheet_id:
-                        for i in range(0, len(cls.args.sheet_id)):
+                    if cls.args.sheet_name:
+                        for i in range(0, len(cls.args.sheet_name)):
                             if cls.args.course_id[i] == course_id:
-                                sheet_id = cls.args.sheet_id[i]
+                                sheet_name = cls.args.sheet_name[i]
                                 break
                             else:
-                                sheet_id = cls.args.sheet_id[i] + ' ' + course_id
+                                sheet_name = cls.args.sheet_name[i] + ' ' + course_id
                     else:
-                        sheet_id = 'course ' + course_id
+                        sheet_name = 'course ' + course_id
 
-                    sheets.write_data_to_table(df, cls.args.google_token, table_id, sheet_id)
+                    sheets.write_data_to_table(df, cls.args.google_token, table_id, sheet_name)
 
                 # write data to yandex disk
                 if cls.args.yandex_token and cls.args.yandex_path:

moodle_export/sheets.py

@@ -3,24 +3,24 @@
 import args_parser
 
 
-def write_data_to_table(df_data, google_token, table_id, sheet_id):
-    if google_token and sheet_id and table_id:
+def write_data_to_table(df_data, google_token, table_id, sheet_name):
+    if google_token and sheet_name and table_id:
         gc = pygsheets.authorize(service_file=google_token)
         sh = gc.open_by_key(table_id)
 
     try:
-        sh.worksheets('title', sheet_id)
+        sh.worksheets('title', sheet_name)
     except:
-        sh.add_worksheet(sheet_id)
+        sh.add_worksheet(sheet_name)
 
-    wk_content = sh.worksheet_by_title(sheet_id)
+    wk_content = sh.worksheet_by_title(sheet_name)
 
     wk_content.set_dataframe(df_data, 'A1', copy_head=True)
 
 
 def main():
     args = args_parser.arg_parser()
-    write_data_to_table(args.csv_path, args.google_token, args.table_id, args.sheet_id)
+    write_data_to_table(args.csv_path, args.google_token, args.table_id, args.sheet_name)
 
 
 if __name__ == "__main__":

scripts/export_courses.sh

@@ -7,6 +7,7 @@
 # )
 # system: "moodle" - Предмет,1zG21U9zJHIkfAM5ejd8WBw,Онлайн-курс,moodle,course_id
 # system: "stepik" - Предмет,1zG21U9zJHIkfAM5ejd8WBw,Онлайн-курс,stepik,course_id,class_id
+# system: "dis" - Предмет,1zG21U9zJHIkfAM5ejd8WBw,Онлайн-курс,dis,filter
 # exportCourses "${exports[@]}"
 
 
@@ -32,31 +33,36 @@ function exportCourses() {
 
           case "${current_export[3]}" in
             "moodle")
-                docker run --rm -v $google_conf:/app/conf.json moodle_export_parser:latest \
-                --moodle_token $moodle_token --url https://e.moevm.info \
-                --csv_path grades --google_token conf.json \
-                --course_id ${current_export[4]} \
-                --table_id ${current_export[1]} \
-                --sheet_id ${current_export[2]} \
-                --options github >> $log_file
+                docker run --rm -v $EXPORTER_GOOGLE_CONF:/app/conf.json moodle_export_parser:latest \
+                    --moodle_token $MOODLE_TOKEN --url https://e.moevm.info \
+                    --csv_path grades --google_token conf.json \
+                    --course_id ${current_export[4]} \
+                    --table_id ${current_export[1]} \
+                    --sheet_name ${current_export[2]} \
+                    --options github >> $log_file
 
                 return_code=$?
                 ;;
             "stepik")
-                docker run --rm -v $google_conf:/app/conf.json stepik_export_parser:latest \
-                --client_id $stepik_client_id --client_secret $stepik_client_secret \
-                --url https://stepik.org:443/api \
-                --csv_path grades --google_token conf.json \
-                --course_id ${current_export[4]} \
-                --class_id ${current_export[5]} \
-                --table_id ${current_export[1]} \
-                --sheet_id ${current_export[2]} >> $log_file
+                docker run --rm -v $EXPORTER_GOOGLE_CONF:/app/conf.json stepik_export_parser:latest \
+                    --client_id $STEPIK_CLIENT_ID --client_secret $STEPIK_CLIENT_SECRET \
+                    --url https://stepik.org:443/api \
+                    --csv_path grades --google_token conf.json \
+                    --course_id ${current_export[4]} \
+                    --class_id ${current_export[5]} \
+                    --table_id ${current_export[1]} \
+                    --sheet_name ${current_export[2]} >> $log_file
 
                 return_code=$?
                 ;;
-            "checker")
-                echo "Not implemented"
-                return_code=1
+            "dis")
+                docker run --rm -v $EXPORTER_GOOGLE_CONF:/app/conf.json checker_export_parser:latest \
+                    --checker_filter "${current_export[4]}" \
+                    --checker_token $DIS_ACCESS_TOKEN \
+                    --table_id ${current_export[1]} \
+                    --sheet_name ${current_export[2]} >> $log_file
+
+                return_code=$?
                 ;;
             *)
                 echo "Недопустимое значение: '${current_export[3]}'"

stepik_export/Dockerfile

@@ -1,4 +1,3 @@
-# syntax=docker/Dockerfile:1
 FROM python:3.8-slim
 
 WORKDIR /app