Plugin fails to load on Windows

[barnumbirr]

Hey,

so as suggested, I tried to run calibre in debug mode. Here's the outut:

C:\Users\<user>>calibre-debug -g
calibre 5.26 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (7, 1, 0) && Remarkable Plugin (1, 1, 0)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 92.89142857142856 x 93.78461538461538
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.86] splash screen shown
[0.86] Initializing db...
[0.91] db initialized
[0.91] Constructing main UI...
Startup method for device RemarkablePlugin threw exception
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 378, in run_startup
  File "calibre_plugins.remarkable_plugin.__init__", line 33, in startup
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\remarkable_fs\__init__.py", line 1, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\remarkable_fs\connection.py", line 5, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\__init__.py", line 22, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\transport.py", line 89, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\dsskey.py", line 37, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\pkey.py", line 31, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\bcrypt\__init__.py", line 25, in <module>
ImportError: cannot import name '_bcrypt' from 'bcrypt' (C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\bcrypt\__init__.py)
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000024215260190>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
QWindowsWindow::setGeometry: Unable to set geometry 2560x1389-2560+29 (frame: 2578x1436-2569-9) on QWidgetWindow/"MainWindowWindow" on "\\.\DISPLAY2". Resulting geometry: 2560x1361-2560+29 (frame: 2578x1408-2569-9) margins: 9, 38, 9, 9 minimum size: 583x390 MINMAXINFO maxSize=0,0 maxpos=0,0 mintrack=601,437 maxtrack=0,0)
[1.67] main UI initialized...
[1.67] Hiding splash screen
Starting QuickView
[2.14] splash screen hidden
[2.14] Started up in 2.14 seconds with 28 books
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000024215260190>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000024215260190>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000024215260190>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'

The last stacktrace gets repeated every couple of seconds. I can SSH into the Remarkable and the plugin settings are set correctly.

Cheers.

[r0bbie]

Getting the same issue as described above. Checked my setting are correct, and I can successfully SSH in. However I'm unable to connect to my Remarkable 2 in Calibre, and get the same stacktrace as described.

[naclander]

So its failing to initialize the plugin.

The first error I see in the trace is:

ImportError: cannot import name '_bcrypt' from 'bcrypt' (C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\bcrypt\__init__.py)

I don't really understand why the the import is failing on your machine. The bcrypt dependency is included in the zip. Can you try installing the latest release zip?

All the subsequent errors are most likely due to the fact that startup failed, so I think fixing the above bcrypt import error should fix the subsequent ones.

[naclander]

@r0bbie what OS are you getting the error on, and can you paste the stack trace.

[julianprester]

Thanks for the plugin and looking into this @naclander ! I get the same error on a Windows machine. It seems to have something to do with the import of bcrypt library. This is the stack trace.

calibre 5.26 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (7, 2, 1) && Remarkable Plugin (1, 1, 0) && Zotero Metadata Importer (1, 0, 70)
Turning on automatic hidpi scaling
devicePixelRatio: 3.0
logicalDpi: 96.0 x 96.0
physicalDpi: 92.1829931972789 x 92.36363636363636
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[1.42] splash screen shown
[1.42] Initializing db...
[1.48] db initialized
[1.48] Constructing main UI...
Startup method for device RemarkablePlugin threw exception
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 378, in run_startup
  File "calibre_plugins.remarkable_plugin.__init__", line 33, in startup
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\remarkable_fs\__init__.py", line 1, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\remarkable_fs\connection.py", line 5, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\__init__.py", line 22, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\transport.py", line 89, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\dsskey.py", line 37, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\pkey.py", line 31, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\bcrypt\__init__.py", line 25, in <module>
ImportError: cannot import name '_bcrypt' from 'bcrypt' (C:\Users\julian\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\bcrypt\__init__.py)
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x000001B8732AF2B0>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
[3.34] main UI initialized...
[3.36] Hiding splash screen
Starting QuickView
[4.14] splash screen hidden
[4.14] Started up in 4.14 seconds with 174 books

[naclander]

So I think I may have found the reason that this issue is happening.

I didn't realize bcrypt ( a dependency of paramiko, used to ssh into the remarkable ) is a python library with C extensions. I'm fairly sure that means I can't include it in the Plugin zip file and have users on Windows use the dependency.

I would say as a workaround to use the bcrypt on your system, but as we've found from issue #1 calibre on Windows doesn't use the standard python interpreter like on Linux.

Calibre also does not make it easy ( and I couldn't find any documentation ) on how to include dependencies in plugins. The only information about this is in a random forum post by the creator. So I am a little at a loss on how to make this plugin compatible with Windows. Actually, I am still waiting on confirmation that this plugin successfully works on macOS, or even on other Linux machines.

If someone has any recommendation as to how to make the plugin dependencies compatible with Windows please let me know.

[naclander]

calibre-remarkable-device-driver-plugin.zip

I've made a test plugin with py-bcrypt instead of bcrypt. Apparently py-bcrypt is bcrypt implemented in pure python, so I'm hoping this will at least solve the immediate issue you're seeing. I have bcrypt installed globally on my machine so I can't test this, but would appreciate if you'd tell me if this works for you.

[zelomel]

@naclander

I tried you last release (1.2.2) but I did not achieve to access my rM1.
Here stack trace in debut mode:

`C:\Users<user>>ssh root@10.11.99.1
root@10.11.99.1's password:
ｒｅＭａｒｋａｂｌｅ
╺━┓┏━╸┏━┓┏━┓ ┏━╸┏━┓┏━┓╻ ╻╻╺┳╸┏━┓┏━┓
┏━┛┣╸ ┣┳┛┃ ┃ ┃╺┓┣┳┛┣━┫┃┏┛┃ ┃ ┣━┫┗━┓
┗━╸┗━╸╹┗╸┗━┛ ┗━┛╹┗╸╹ ╹┗┛ ╹ ╹ ╹ ╹┗━┛
reMarkable: ~/ exit
logout
Connection to 10.11.99.1 closed.
C:\Users<user>>calibre-debug -g
calibre 5.28 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.17134 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.17134')
Python 3.8.5
Windows: ('10', '10.0.17134', '', 'Multiprocessor Free')
Interface language: fr
Successfully initialized third party plugins: DeDRM (7, 2, 1) && Remarkable Plugin (1, 2, 2)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 165.87755102040816 x 166.25454545454545
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.39] splash screen shown
[0.39] Initializing db...
[0.39] db initialized
[0.39] Constructing main UI...

c:\users<user>\calibre_plugins.remarkable_plugin.init(34)startup()
(Pdb)
Startup method for device RemarkablePlugin threw exception
Traceback (most recent call last):
File "calibre\gui2\device.py", line 378, in run_startup
File "calibre_plugins.remarkable_plugin.init", line 34, in startup
File "calibre_plugins.remarkable_plugin.init", line 34, in startup
File "bdb.py", line 88, in trace_dispatch
File "bdb.py", line 113, in dispatch_line
bdb.BdbQuit
[29.69] main UI initialized...
[29.69] Hiding splash screen
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000020E911A5790>:
Traceback (most recent call last):
File "calibre\gui2\device.py", line 293, in detect_device
File "calibre_plugins.remarkable_plugin.init", line 51, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
[30.70] splash screen hidden
Traceback (most recent call last):
File "calibre\gui2\main.py", line 379, in initialize
File "calibre\gui2\main.py", line 358, in initialize_db
File "calibre\gui2\main.py", line 315, in initialize_db_stage2
File "calibre\gui2\main.py", line 259, in start_gui
File "calibre\gui2\ui.py", line 246, in initialize
File "calibre\gui2\device.py", line 907, in init_device_mixin
File "threading.py", line 558, in wait
File "threading.py", line 302, in wait
KeyboardInterrupt
`

I had to kill the process when it blocked at (PdB).

[naclander]

@zelomel your issue is not related to the original problem in this thread ( bcrypt failing to load ).

The current issue for @barnumbirr and @r0bbie is that bcrypt is failing to load during plugin initialization, and I made a plugin for them to test here

I am not sure what your issue is, but 1.2.2 is definitely not going to work for you.

If your problem on your windows device is different than the original issue in this thread, please open a new issue.

EDIT: The breakpoint was a bug, it was fixed in 1.2.3. But regardless, that release is not going to work for you, if you're on windows try the test release mentioned above.

[barnumbirr]

Version 1.2.2 produced the same stacktrace and behaviour as posted by zelomel above. Here's the output of version 1,2,3:

C:\Users\martin>calibre-debug -g
calibre 5.27 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (7, 1, 0) && Remarkable Plugin (1, 2, 3)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 92.89142857142856 x 93.78461538461538
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.41] splash screen shown
[0.41] Initializing db...
[0.41] db initialized
[0.41] Constructing main UI...
Startup method for device RemarkablePlugin threw exception
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 378, in run_startup
  File "calibre_plugins.remarkable_plugin.__init__", line 33, in startup
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\remarkable_fs\__init__.py", line 1, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\remarkable_fs\connection.py", line 5, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\__init__.py", line 22, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\transport.py", line 89, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\paramiko\dsskey.py", line 25, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\cryptography\hazmat\primitives\serialization\__init__.py", line 15, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\cryptography\hazmat\primitives\serialization\base.py", line 11, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\cryptography\hazmat\primitives\asymmetric\types.py", line 7, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\cryptography\hazmat\primitives\asymmetric\dsa.py", line 12, in <module>
  File "zipimport.py", line 259, in load_module
  File "C:\Users\martin\AppData\Roaming\calibre\plugins\Remarkable Plugin.zip\cryptography\hazmat\primitives\asymmetric\utils.py", line 6, in <module>
ImportError: cannot import name 'asn1' from 'cryptography.hazmat.bindings._rust' (unknown location)
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000026310500280>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
[0.95] main UI initialized...
[0.95] Hiding splash screen
Starting QuickView
[1.23] splash screen hidden
[1.23] Started up in 1.23 seconds with 28 books
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000026310500280>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x0000026310500280>:
Traceback (most recent call last):
  File "calibre\gui2\device.py", line 293, in detect_device
  File "calibre_plugins.remarkable_plugin.__init__", line 50, in detect_managed_devices
AttributeError: 'RemarkablePlugin' object has no attribute 'seen_device'

Seems bcrypt issue is fixed but now pyca/cryptography misbehaves.

[brett-schneider]

zipimport seems to have trouble with stubs

[naclander]

@barnumbirr The versions publicly released don't solve the problem on Windows.

Can you please try the plugin I have made custom for your use case here ( If that's the version you did use in the stacktrace, please confirm as I may have misunderstood )

Yes the issue with bcrypt is expected to go away when using py-bcrypt. I do not yet know what:

ImportError: cannot import name 'asn1' from 'cryptography.hazmat.bindings._rust' (unknown location)

is referring to. I suppose it is another c extensions dependency that we need to get rid of.

Again to summarize, and C extension dependency of this plugin needs to be replaced with a pure python alternative or this plugin will not work on Windows. If anyone knows how other Calibre plugins have gotten around this please let me know.

[barnumbirr]

@naclander my bad for the shoddy explanation.
I've been using the "custom" plugins you've posted, just used the version reported by the plugin.

[zelomel]

@barnumbirr > I've been using the "custom" plugins you've posted, just used the version reported by the plugin.

Is it working ?

[goldenstarshine]

calibre-remarkable-device-driver-plugin.zip

I've made a test plugin with py-bcrypt instead of bcrypt. Apparently py-bcrypt is bcrypt implemented in pure python, so I'm hoping this will at least solve the immediate issue you're seeing. I have bcrypt installed globally on my machine so I can't test this, but would appreciate if you'd tell me if this works for you.

@naclander I wanted to let you know that I had a comparable issue while trying to install the plugin in Calibre on macOS, it failed with an error reporting that the file init.py could not be found, though it was present.
This version could be installed flawlessly, but calibre does not find my reMarkable 2.
I tried to connect to the device as root user using SSH from the Terminal, which worked flawlessly.

Please find the error protocol below:

calibre Fehlerdiagnoseprotokoll
calibre 5.29  embedded-python: True is64bit: True
macOS-10.15.7-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '19.6.0', 'Darwin Kernel Version 19.6.0: Thu Sep 16 20:58:47 PDT 2021; root:xnu-6153.141.40.1~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.15.7', ('', '', ''), 'x86_64')
Interface language: de
Successfully initialized third party plugins: Remarkable Plugin (1, 2, 1)
calibre 5.29  embedded-python: True is64bit: True
macOS-10.15.7-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '19.6.0', 'Darwin Kernel Version 19.6.0: Thu Sep 16 20:58:47 PDT 2021; root:xnu-6153.141.40.1~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.15.7', ('', '', ''), 'x86_64')
Interface language: de
Successfully initialized third party plugins: Remarkable Plugin (1, 2, 1)
devicePixelRatio: 1.0
logicalDpi: 72.0 x 72.0
physicalDpi: 109.00000163701577 x 109.00000163701577
Using calibre Qt style: True
[0.00] Starting up...
[0.02] Showing splash screen...
[0.31] splash screen shown
[0.31] Initializing db...
[0.33] db initialized
[0.33] Constructing main UI...
> /calibre_plugins.remarkable_plugin.__init__(34)startup()
-> import remarkable_fs
(Pdb) 
Startup method for device RemarkablePlugin threw exception
Error during device detection for <calibre_plugins.remarkable_plugin.RemarkablePlugin object at 0x110aeffa0>:
[1.17] main UI initialized...
[1.17] Hiding splash screen
Starting QuickView