Merge pull request #55 from OidaTiftla/onedrive

Onedrive

Author: niqdev

.gitignore

@@ -9,6 +9,7 @@
 config/prod.cfg
 config/client_secrets.json
 config/auth_token.json
+config/session.onedrive.pickle
 config/lastNewsletterUrl
 
 ebooks/

README.md

@@ -9,15 +9,15 @@ This crawler automates the following step:
 * parse title, description and useful information
 * download favorite format *.pdf .epub .mobi*
 * download source code and book cover
-* upload files to Google Drive or via scp
+* upload files to Google Drive, OneDrive or via scp
 * store data on Firebase
 * notify via email, IFTTT or Join (on success and errors)
 * schedule daily job on Heroku or with Docker
 
 ### Default command
 ```bash
-# upload pdf to drive, store data and notify via email
-python script/spider.py -c config/prod.cfg -u drive -s firebase -n gmail
+# upload pdf to googledrive, store data and notify via email
+python script/spider.py -c config/prod.cfg -u googledrive -s firebase -n gmail
 ```
 
 ### Other options
@@ -33,9 +33,13 @@ python script/spider.py --config config/prod.cfg -t pdf --extras
 # equivalent (default is pdf)
 python script/spider.py -c config/prod.cfg -e
 
-# download and then upload to Drive (given the download url anyone can download it)
-python script/spider.py -c config/prod.cfg -t epub --upload drive
-python script/spider.py --config config/prod.cfg --all --extras --upload drive
+# download and then upload to Google Drive (given the download url anyone can download it)
+python script/spider.py -c config/prod.cfg -t epub --upload googledrive
+python script/spider.py --config config/prod.cfg --all --extras --upload googledrive
+
+# download and then upload to OneDrive (given the download url anyone can download it)
+python script/spider.py -c config/prod.cfg -t epub --upload onedrive
+python script/spider.py --config config/prod.cfg --all --extras --upload onedrive
 
 # download and notify: gmail|ifttt|join
 python script/spider.py -c config/prod.cfg --notify gmail
@@ -64,42 +68,80 @@ Now you should be able to claim and download your first eBook
 python script/spider.py --config config/prod.cfg
 ```
 
-### Drive
+### Google Drive
 
-From the documentation, Drive API requires OAuth2.0 for authentication, so to upload files you should:
+From the documentation, Google Drive API requires OAuth2.0 for authentication, so to upload files you should:
 
-* Go to [Google APIs Console](https://code.google.com/apis/console) and create a new [Drive](https://console.developers.google.com/apis/api/drive/overview) project named **PacktpubDrive**
+* Go to [Google APIs Console](https://code.google.com/apis/console) and create a new [Google Drive](https://console.developers.google.com/apis/api/drive/overview) project named **PacktpubDrive**
 * On *API manager > Overview* menu
   * Enable Google Drive API
 * On *API manager > Credentials* menu
   * In *OAuth consent screen* tab set **PacktpubDrive** as the product name shown to users
   * In *Credentials* tab create credentials of type *OAuth client ID* and choose Application type *Other* named **PacktpubDriveCredentials**
 * Click *Download JSON* and save the file `config/client_secrets.json`
-* Change your Drive credentials in the config file
+* Change your Google Drive credentials in the config file
 
 ```
-[drive]
+[googledrive]
 ...
-drive.client_secrets=config/client_secrets.json
-drive[email protected]
+googledrive.client_secrets=config/client_secrets.json
+googledrive[email protected]
 ```
 
-Now you should be able to upload your eBook to Drive
+Now you should be able to upload your eBook to Google Drive
 ```
-python script/spider.py --config config/prod.cfg --upload drive
+python script/spider.py --config config/prod.cfg --upload googledrive
 ```
 
 Only the first time you will be prompted to login in a browser which has javascript enabled (no text-based browser) to generate `config/auth_token.json`.
 You should also copy and paste in the config the *FOLDER_ID*, otherwise every time a new folder with the same name will be created.
 ```
-[drive]
+[googledrive]
 ...
-drive.default_folder=packtpub
-drive.upload_folder=FOLDER_ID
+googledrive.default_folder=packtpub
+googledrive.upload_folder=FOLDER_ID
 ```
 
 Documentation: [OAuth](https://developers.google.com/api-client-library/python/guide/aaa_oauth), [Quickstart](https://developers.google.com/drive/v3/web/quickstart/python), [example](https://github.com/googledrive/python-quickstart) and [permissions](https://developers.google.com/drive/v2/reference/permissions)
 
+### OneDrive
+
+From the documentation, OneDrive API requires OAuth2.0 for authentication, so to upload files you should:
+
+
+* Go to the [Microsoft Application Registration Portal](https://apps.dev.microsoft.com/?referrer=https%3A%2F%2Fdev.onedrive.com%2Fapp-registration.htm).
+* When prompted, sign in with your Microsoft account credentials.
+* Find **My applications** and click **Add an app**.
+* Enter **PacktpubDrive** as the app's name and click **Create application**.
+* Scroll to the bottom of the page and check the **Live SDK support** box.
+* Change your OneDrive credentials in the config file
+  * Copy your **Application Id** into the config file to **onedrive.client_id**
+  * Click **Generate New Password** and copy the password shown into the config file to **onedrive.client_secret**
+  * Click **Add Platform** and select **Web**
+  * Enter **http://localhost:8080/** as the **Redirect URL**
+  * Click **Save** at the bottom of the page
+
+```
+[onedrive]
+...
+onedrive.client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+onedrive.client_secret=XxXxXxXxXxXxXxXxXxXxXxX
+```
+
+Now you should be able to upload your eBook to OneDrive
+```
+python script/spider.py --config config/prod.cfg --upload onedrive
+```
+
+Only the first time you will be prompted to login in a browser which has javascript enabled (no text-based browser) to generate `config/session.onedrive.pickle`.
+```
+[onedrive]
+...
+onedrive.folder=packtpub
+```
+
+Documentation: [Registration](https://dev.onedrive.com/app-registration.htm), [Python API](https://github.com/OneDrive/onedrive-sdk-python)
+
 ### Scp
 
 To upload your eBook via `scp` on a remote server update the configs
@@ -136,7 +178,7 @@ firebase.url=https://PROJECT_NAME.firebaseio.com
 
 Now you should be able to store your eBook details on Firebase
 ```
-python script/spider.py --config config/prod.cfg --upload drive --store firebase
+python script/spider.py --config config/prod.cfg --upload googledrive --store firebase
 ```
 
 ### Gmail notification

config/dev.cfg

@@ -20,8 +20,8 @@ path.ebooks=ebooks
 path.extras=ebooks/extras
 #path.group=true
 
-[drive]
-drive.oauth2_scope=https://www.googleapis.com/auth/drive
-drive.client_secrets=config/client_secrets.json
-drive.auth_token=config/auth_token.json
-drive.gmail[email protected]
+[googledrive]
+googledrive.oauth2_scope=https://www.googleapis.com/auth/drive
+googledrive.client_secrets=config/client_secrets.json
+googledrive.auth_token=config/auth_token.json
+googledrive.gmail[email protected]

config/prod_example.cfg

@@ -18,13 +18,20 @@ path.ebooks=ebooks
 path.extras=ebooks/extras
 #path.group=true
 
-[drive]
-drive.oauth2_scope=https://www.googleapis.com/auth/drive
-drive.client_secrets=config/client_secrets.json
-drive.auth_token=config/auth_token.json
-drive.gmail[email protected]
-drive.default_folder=packtpub
-#drive.upload_folder=FOLDER_ID
+[googledrive]
+googledrive.oauth2_scope=https://www.googleapis.com/auth/drive
+googledrive.client_secrets=config/client_secrets.json
+googledrive.auth_token=config/auth_token.json
+googledrive.gmail[email protected]
+googledrive.default_folder=packtpub
+#googledrive.upload_folder=FOLDER_ID
+
+[onedrive]
+onedrive.api_base_url=https://api.onedrive.com/v1.0/
+onedrive.client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+onedrive.client_secret=XxXxXxXxXxXxXxXxXxXxXxX
+onedrive.session_file=config/session.onedrive.pickle
+onedrive.folder=packtpub
 
 [scp]
 scp.host=SCP_HOST

requirements.txt

@@ -12,3 +12,5 @@ python-firebase==1.2
 paramiko==2.0.2
 cryptography==1.6
 scp==0.10.2
+onedrivesdk==1.1.8
+

script/drive.py script/googledrive.pyscript/drive.py renamed to script/googledrive.py

@@ -9,13 +9,13 @@
 from utils import thread_loader
 from logs import *
 
-class Drive(object):
+class GoogleDrive(object):
     """
     """
 
     def __init__(self, config):
         self.__config = config
-        self.__drive_service = None
+        self.__googledrive_service = None
         self.info = {}
 
     def __guess_info(self, file_path):
@@ -27,11 +27,11 @@ def __guess_info(self, file_path):
             'name': file_path.split('/')[-1],
             'mime_type': magic.from_file(file_path, mime=True),
         }
-        log_info('[+] new file upload on Drive:')
+        log_info('[+] new file upload on Google Drive:')
         # log_dict(self.file_info)
 
     def __init_service(self):
-        auth_token = self.__config.get('drive', 'drive.auth_token')
+        auth_token = self.__config.get('googledrive', 'googledrive.auth_token')
 
         if not exists(auth_token):
             self.__save_credentials(auth_token)
@@ -41,12 +41,12 @@ def __init_service(self):
 
         http = httplib2.Http()
         http = credentials.authorize(http)
-        self.__drive_service = build('drive', 'v2', http=http)
+        self.__googledrive_service = build('drive', 'v2', http=http)
 
     def __save_credentials(self, auth_token):
         flow = flow_from_clientsecrets(
-            self.__config.get('drive', 'drive.client_secrets'),
-            self.__config.get('drive', 'drive.oauth2_scope'),
+            self.__config.get('googledrive', 'googledrive.client_secrets'),
+            self.__config.get('googledrive', 'googledrive.oauth2_scope'),
             OOB_CALLBACK_URN)
 
         authorize_url = flow.step1_get_authorize_url()
@@ -63,35 +63,35 @@ def __save_credentials(self, auth_token):
 
     def __create_folder(self): #Create folder with provided name
         try: #Check default folder name
-            default_folder_name = self.__config.get('drive', 'drive.default_folder')
+            default_folder_name = self.__config.get('googledrive', 'googledrive.default_folder')
         except:
             default_folder_name = 'packtpub'
 
         metadata = {
             'title': default_folder_name,
             'mimeType' : 'application/vnd.google-apps.folder'
         }
-        folder = self.__drive_service.files().insert(body = metadata).execute()
-        self.__config.set('drive', 'drive.upload_folder', folder['id'])
+        folder = self.__googledrive_service.files().insert(body = metadata).execute()
+        self.__config.set('googledrive', 'googledrive.upload_folder', folder['id'])
         log_success('[+] creating new directory...')
         print '[+] updating folder permissions...'
         permissions = {
             'role': 'reader',
             'type': 'anyone',
-            'value': self.__config.get('drive', 'drive.gmail')
+            'value': self.__config.get('googledrive', 'googledrive.gmail')
         }
-        self.__drive_service.permissions().insert(fileId=folder['id'], body=permissions).execute()
+        self.__googledrive_service.permissions().insert(fileId=folder['id'], body=permissions).execute()
         log_dict({'folder_name':  default_folder_name,
                   'id': folder['id'],})
                   #'share_link': folder['webContentLink']}) #TODO Fix
-        log_success('[+] Please add this line after [drive] in your configuration file:')
-        log_info('drive.upload_folder=' + folder.get('id'))
+        log_success('[+] Please add this line after [googledrive] in your configuration file:')
+        log_info('googledrive.upload_folder=' + folder.get('id'))
 
         return folder.get('id') #Return folder object ID
 
     def __get_folder(self): #Get folder name settings
         try:
-            return self.__config.get('drive', 'drive.upload_folder')
+            return self.__config.get('googledrive', 'googledrive.upload_folder')
         except:
             return self.__create_folder() #new folder ID
 
@@ -105,18 +105,18 @@ def __insert_file(self):
             'mimeType': self.info['mime_type'],
             'parents': [{'id': self.__get_folder()}]
         }
-        file = self.__drive_service.files().insert(body=body, media_body=media_body).execute()
+        file = self.__googledrive_service.files().insert(body=body, media_body=media_body).execute()
         # log_dict(file)
 
         print '[+] updating file permissions...'
         permissions = {
             'role': 'reader',
             'type': 'anyone',
-            'value': self.__config.get('drive', 'drive.gmail')
+            'value': self.__config.get('googledrive', 'googledrive.gmail')
         }
-        self.__drive_service.permissions().insert(fileId=file['id'], body=permissions).execute()
+        self.__googledrive_service.permissions().insert(fileId=file['id'], body=permissions).execute()
 
-        # self.__drive_service.files().get(fileId=file['id']).execute()
+        # self.__googledrive_service.files().get(fileId=file['id']).execute()
 
         self.info['id'] = file['id']
         self.info['download_url'] = file['webContentLink']