Kernel Crash when opening "corrupted" DXD-Files

[pirtak]

Dear dev,

for work, we use multiple DXD files for longterm measurements (~2 years). The DewesoftX Software is not made for handling that amount of multifiles. Therefore I need to crawl through the files with my python software to enable quick analytics.

However, files that have the status being flagged as "corrupted" in DewesoftX lead to a Kernel Crash when using the open()-method from the dwdatareader-module.

I use:

from dxmp.utils import get_dxd
from dxmp.property import Property
property = Property()

import dwdatareader as dw
filename = "MID_2022_03_23_22_12_31.dxd"
filepath = os.path.join(property.path.dxd_folder, filename)

dxfile = dw.DWFile(filepath)

which leads to the crash.

I have no idea, how I could implement a previous check for the files state.

Best wishes!

[pirtak]

A solution, I have found, is, to create a subprocess:

import subprocess
import json

def is_valid_dxd_file(filepath):
    result = subprocess.run(
        ["python", "check_dw_open.py", filepath],
        capture_output=True,
        text=True
    )
    try:
        return json.loads(result.stdout).get("status") == "ok"
    except:
        return False
    
is_valid_dxd_file(filepath)

which uses the following code in check_dw_open.py:

import sys
import json
import dwdatareader as dw

filepath = sys.argv[1]

try:
    dw.open(filepath)
    print(json.dumps({"status": "ok"}))
except Exception as e:
    print(json.dumps({"status": "error", "message": str(e)}))

Maybe you could implement this. Have a nice week!

[costerwi]

What is dw.get_version() ?
Do you have a (small?) example file you can share which causes the crash?
Thanks!

[Bojan023]

Moveover, what OS are you using? Are you sure that filepath is a valid and accessible file path?

[pirtak]

Greetings and please excuse the long time without any response. Thank you for the quick response from your side.

1. @costerwi getversion() returns 4020031. The files are used in current research and are confidential, but I am allowed to share a file privately with you -> Please contact me via email: patrick.noah.mueller@iwes.fraunhofer.de.
2. @Bojan023 the folder path is correct. The OS is (edit: Linux!). I access the data on a Linux server via SSH but the processing of the other DXD files doesn't lead to any issue.

[Bojan023]

I don't understand why getversion() returns this value. Are you using a old version of dwdatareader (maybe because you are using also an older version of Python?)? The recent versions of dwdatareader (>1.0.0) should have a signature like major.minor.patch (e.g. 1.2.0) as per

dwdatareader/dwdatareader/__init__.py

Line 868 in 968ad71

return f"{ver_major.value}.{ver_minor.value}.{ver_patch.value}"

It looks like that you are using an old version of Dewesoft's Data Reader Library or dwdatareader. The latest version of dwdatareader (>1.0.0) requires Python >3.10. Older versions of dwdatareader are not supported. I also had stability/memory issues with dwdatareader < 1.0.0, especially with parallel read operations on Linux. I also wrapped the read operation in separate processes to let the process crash instead of the kernel, but this is an ugly workaround if you'd ask me.

I would suggest to upgrade the Python environment to >3.10 and use one of the latest versions of dwdatareader. Stability, speed and parallelism are hugely improved partially due to the release of a rewritten Dewesoft Data Reader Library from 1.0.0 onward. It is also compatible with older measurement files.