29 generate contributed defs

[RussBerg]

Made changes to process the contributed_definitions directory also extended README.md.

[prjemian]

I want to walk through the code with a file to review further.

[prjemian]

Check the indentation on lines 27-39 for an extra leading space.

[prjemian]

When testing from linux command line (in conda environment with all dependencies included):

(nxexamples) prjemian@zap:~/.../NeXus/exampledata$ ./nxdl/nxdl_to_hdf5.py -h
./nxdl/nxdl_to_hdf5.py: line 1: import: command not found
./nxdl/nxdl_to_hdf5.py: line 2: import: command not found
./nxdl/nxdl_to_hdf5.py: line 3: import: command not found
./nxdl/nxdl_to_hdf5.py: line 4: import: command not found
./nxdl/nxdl_to_hdf5.py: line 5: import: command not found
./nxdl/nxdl_to_hdf5.py: line 6: import: command not found
./nxdl/nxdl_to_hdf5.py: line 7: import: command not found
from: can't read /var/mail/lxml
./nxdl/nxdl_to_hdf5.py: line 9: import: command not found
from: can't read /var/mail/tinydb
./nxdl/nxdl_to_hdf5.py: line 11: $'\r': command not found
./nxdl/nxdl_to_hdf5.py: line 12: $'\r': command not found
./nxdl/nxdl_to_hdf5.py: line 13: readme_string: command not found
./nxdl/nxdl_to_hdf5.py: line 40: h5py_script_lst: command not found
./nxdl/nxdl_to_hdf5.py: line 41: nxsfrmt_script_lst: command not found
./nxdl/nxdl_to_hdf5.py: line 42: db: command not found
./nxdl/nxdl_to_hdf5.py: line 43: query: command not found
./nxdl/nxdl_to_hdf5.py: line 44: tables_dct: command not found
./nxdl/nxdl_to_hdf5.py: line 45: $'\r': command not found
./nxdl/nxdl_to_hdf5.py: line 46: syntax error near unexpected token `('
'/nxdl/nxdl_to_hdf5.py: line 46: `def init_database():

Since debugging in VSCode with that environment passes these imports, I suspect the CRLF line endings. The debugger session ran as far as files = sorted(os.listdir(pathlib.PurePath(def_dir, def_subdir))) and then failed to find any of the definitions.

Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: '/home/prjemian/Documents/projects/NeXus/exampledata/../../definitions/applications'
  File "/home/prjemian/Documents/projects/NeXus/exampledata/nxdl/nxdl_to_hdf5.py", line 1651, in <module>
    files = sorted(os.listdir(pathlib.PurePath(def_dir, def_subdir)))

Here's my directory structure:

(nxexamples) prjemian@zap:~/.../NeXus/exampledata$ ll /home/prjemian/Documents/projects/NeXus/
total 12K
drwxrwxr-x 15 prjemian prjemian 4.0K Dec  8 11:11 _definitions/
drwxrwxr-x 16 prjemian prjemian 4.0K Dec 13 09:53 definitions/
drwxrwxr-x 16 prjemian prjemian 4.0K Dec 14 11:37 exampledata/

Muyst be off by one ..

[prjemian]

Should test that this directory exists earlier. Might be best if I propose some changes in a sub-branch?

[prjemian]

note: https://stackoverflow.com/questions/57351833/why-does-pathlib-have-both-purepath-path#61436677

[prjemian]

Getting further now. First exception location moved forward:

Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: '/home/prjemian/Documents/projects/NeXus/exampledata/../autogenerated_examples/nxdl/python_scripts/h5py/ex_h5py_/home/prjemian/Documents/projects/NeXus/definitions/applications/NXarchive.py'
  File "/home/prjemian/Documents/projects/NeXus/exampledata/nxdl/nxdl_to_hdf5.py", line 1521, in write_script
    f = open(pathlib.PurePath(path, 'ex_%s_%s.py' % (mod_name,class_nm)), 'w')
  File "/home/prjemian/Documents/projects/NeXus/exampledata/nxdl/nxdl_to_hdf5.py", line 1503, in write_script_files
    write_script(mod_name,
  File "/home/prjemian/Documents/projects/NeXus/exampledata/nxdl/nxdl_to_hdf5.py", line 1472, in make_class_as_nf_file
    write_script_files(class_nm)
  File "/home/prjemian/Documents/projects/NeXus/exampledata/nxdl/nxdl_to_hdf5.py", line 1573, in process_nxdl
    make_class_as_nf_file(class_nm, dest_dir, symbol_dct=sym_args_dct)
  File "/home/prjemian/Documents/projects/NeXus/exampledata/nxdl/nxdl_to_hdf5.py", line 1661, in <module>
    process_nxdl(str(class_path), def_subdir)

Is that a consequence of this code block?

exampledata/nxdl/nxdl_to_hdf5.py

Lines 872 to 875 in a92773b

	#merge all into one dct, syms, docs and return
	#dct, syms, docs = merge_extends_results(def_lst)
	#return(dct, syms, docs)
	return (None, None, None)

[prjemian]

This block of code will become much stronger with the use of a context.

exampledata/nxdl/nxdl_to_hdf5.py

Lines 1377 to 1467 in 5ead471

	nf = h5py.File(fpath, 'w')
	sym_dct = {}
	# process SYMBOLS
	for d in tables_dct['symbols'].all():
	sym_nm = d['name']
	if (True):
	#sym_nm = d['name']
	val = int(d['value'])
	else:
	#sym_nm = d['name']
	val = default_symbol_val
	sym_dct[sym_nm] = val
	#create the symbol string attrs in the root of the file
	_string_attr(nf, sym_nm, d['doc'])
	for l in tables_dct['dimensions'].all():
	if 'rank' in l['attrib'].keys():
	if l['attrib']['rank'] == sym_nm:
	# substitute the value
	l['attrib']['rank'] = val
	for l in tables_dct['dim'].all():
	if 'value' in l['attrib'].keys():
	if l['attrib']['value'] == sym_nm:
	#substitute the value
	l['attrib']['value'] = val
	for l in tables_dct['dim'].all():
	if 'value' in l['attrib'].keys():
	val = l['attrib']['value']
	if not has_numbers(val):
	# does it also contain spaces? if so then it is not a symbol
	if val.find(' ') == -1:
	if val not in sym_dct.keys():
	print('\t-Symbol Warning: the symbol [%s] is being used but has not been defined in the Symbols table, setting to default value of 1' % val)
	sym_dct[val] = 1
	l['attrib']['value'] = 1
	print_script_start(class_nm)
	# create GROUPs
	create_groups(nf)
	# # create FIELDs
	res = create_fields(nf, sym_dct, category)
	# create Links
	create_links(nf)
	# add the docs from fields and groups now that they exist
	add_docs(nf, tables_dct['doc'].all())
	h5py_script_lst.append(' ')
	nxsfrmt_script_lst.append(' ')
	if(res):
	# create Attributes
	create_attributes(nf)
	_string_attr(nf, 'file_name', fpath.replace('\\', '/'), do_print=False)
	_string_attr(nf, 'file_time', make_timestamp_now(), do_print=False)
	_string_attr(nf, 'HDF5_Version', h5py.version.hdf5_version, do_print=False)
	_string_attr(nf, 'h5py_version', h5py.version.version, do_print=False)
	#_string_attr(nf, 'NEXUS_release_ver', rel_ver)
	entry_grp, entry_nm = get_entry(nf)
	if entry_grp is None:
	fatal_error('File does not contain an NXentry group')
	return(None)
	#ensure the definition is correct
	entry_grp['definition'][()] = get_cur_def_name()
	_string_attr(nf, 'default', entry_nm)
	nx_data_grp, nx_data_nm = get_NXdata_nm(nf)
	if (nx_data_nm):
	_string_attr(entry_grp, 'default', nx_data_nm)
	dset_nm = get_NXdataset_nm(nx_data_grp)
	if (dset_nm):
	_string_attr(nx_data_grp, 'signal', dset_nm)
	_string_attr(nx_data_grp[dset_nm], 'signal', '1')
	_dataset(nf, 'README', readme_string % (rel_ver, class_nm), 'NX_CHAR', nx_units='NX_UNITLESS', dset={}, do_print=False)
	prune_extended_entries(nf)
	nf.close()
	#print('finished exporting to [%s]' % fpath)
	else:
	print('Failed exporting [%s]' % fpath)
	nf.close()

Change 1377 to:

    with h5py.File(fpath, 'w') as nf:

indent the remaining lines in the block, and remove the two lines that read: nf.close() (since closing is handled by the context manager). With an HDF5 file open and so many lines of codes with branches and decisions, it is possible that the code may exity before calling nf.close() directly. The context manager protects against those possibilities.

Read here for more details on context managers.

[prjemian]

Also, if you are writing for Python 3.7 and above, you should consider using Python f strings. Replace a line such as this:

exampledata/nxdl/nxdl_to_hdf5.py

Line 419 in 4298d62

print('Error: invalid definition category [%s]' % (category))

with

     print(f'Error: invalid definition category [{category}]')

Another example:

exampledata/nxdl/nxdl_to_hdf5.py

Line 86 in 4298d62

h5py_script_lst.append('root[\'%s\'].attrs[\'%s\'] = \'%s\'' % (nxgrp.name, name, s_data.split('/')[-1]))

would be changed (including a swap of single to double outer quotes) to

            h5py_script_lst.append(f"root['{nxgrp.name}'].attrs[{name}'] = '{s_data.split('/')[-1]}'")

[prjemian]

Since I can't run the code locally on Linux, some changes are needed as noted. Additional changes may also be needed.

[prjemian]

The import failures reported above are due to how the module is started from the command line. While the file is marked executable, for this to work in UNIX (and linux), a shebang line is needed to inform UNIX how this text file should be run. The most durable shebang line for Python modules is:

#!/usr/bin/env python

which informs the OS to inspect the current environment and start this code with the first python executable that it finds.

I was able to start this module directly as:

python ./nxdl/nxdl_to_hdf5.py

with no such errors about import statements.

[RussBerg]

#30 (comment)
I think the problem was in the README as it should have read python nxdl_to_hdf5.py -d applications as that's how I had been running it on windows and linux. I have had mixed results using the shebang, to me it will work more consitantly (now that I have updated the README) for people if they are expected to specify python explicitly.

Since I can't run the code locally on Linux, some changes are needed as noted. Additional changes may also be needed.