updated dbprocessing code for sqlalchemy 2.0 #137

ezd1000 · 2024-02-02T21:12:00Z

Updated a series of errors so that dbprocessing code works for both sqlalchemy 2.0 and sqlalchemy 1.4. Closes #135.

PR Checklist

Pull request has descriptive title
Pull request gives overview of changes
(N/A) New code has inline comments where necessary
(N/A) Any new modules, functions or classes have docstrings consistent with dbprocessing style
(N/A) Major new functionality has appropriate Sphinx documentation
(N/A) Added an entry to release notes if fixing a major bug or providing a major new feature
(See below) New features and bug fixes should have unit tests
Relevant issues are linked in the description (use Closes # if this PR closes the issue, or some other reference, such as See # if it is related in some other way)

*We don't yet have unit tests to make sure warnings aren't raised, the tests now pass without warnings on Python 3.10, sqlalchemy 1.4, and 2.0.

* doesn't work yet but fixes some errors

unit_tests/dbp_testing.py

jtniehof · 2024-02-06T18:09:15Z

This would close #135, so should edit the PR description to say that. (Last item in the PR checklist, see #120 for an example, also some information here.)

jtniehof

Functionality looks good. Just to make history less arduous, can you do some rebasing and squash some commits together?

the four commits, three "added bind to drop_all" and "removed self from dbu_engine" should be one
"fixed warning message" and "fixed all warning messages" should be one
the "added bind=self.engine" and "took out bind=self.engine" should be combined; really this is "move the bind from Metadata constructor to table creation"
probably the two "fixed another join" should be one

jtniehof

I did a few quick checks for places which aren't in the unit tests but show a similar pattern to what we had to change.

create_all requiring the bind now: CreateDBsabrs line 321 (@dnadeau-lanl , is CreateDBsabrs now out of date, or is it superseded now that we can create postgres databases? Maybe we should be removing it instead.)

execute no longer taking a direct SQL string: scrubber line 27, clean_test_db line 119. To test the scrubber, we're actually kind of short on canned test databases (the psp one is enormous now). You can get an old one from history by git show e3c67f3feabdc62bec2ffabba78bceb3d8dd5d89:./functional_test/testDB.sqlite > foo.sqlite. You can get the old mageis database for testing clean_test_db with git show 18469404b10ece57051b9b2391c1ea3c0741aefe:./unit_tests/RBSP_MAGEIS.sqlite > RBSP_MAGEIS.sqlite

MetaData no longer taking a bind argument (having to bind separately): CreateDBsabrs line 40 (like in other places where we've made that change, that line needs to be fixed and specifying the bind may have to happen in multiple places) EDIT: bind needs to be added to the reflect call in init_db (right after it was removed from the instantiation of self.metadata, and somewhat related the create_all call around line 321 already has the bind, but should be self.engine not engine.

The functional test is also not running. Unfortunately it doesn't run in the CI (#68) so it's possible this has just drifted out of date. Running it is pretty simple, run python ./functionalTest.py in the functional_test directory. Unfortunately debugging it is going to be harder; it makes a temp directory where all the scripts and such live, and for some reason when the codes fail to run, the moveToError that copies the error log to the error directory is also failing.

For finding these various scripts, use find in the dbprocessing checkout.

dbprocessing/DButils.py

docs/CONTRIBUTORS.rst

scripts/scrubber.py

developer/scripts/clean_test_db.py

* fixes sqlalchemy 2.0

… call)

jtniehof

Few fairly minor changes suggested

jtniehof · 2025-03-27T16:41:54Z

functional_test/scripts/run_rot13_L0toL1.py

-    print "infiles", infiles
-    print "outfile", outfile
+    print("infiles + ", infiles)
+    print("outfile + ", outfile)


The plus should be outside the quote here, since it's concatanation...this will put in an extra comma and parens.

jtniehof · 2025-03-27T16:42:16Z

functional_test/scripts/run_rot13_L1toL2.py

-            output.write(infile.read().encode('rot13'))
-
+            output.write(codecs.encode(infile.read(), 'rot_13'))
+            


Extra whitespace here

jtniehof · 2025-03-27T16:42:46Z

functional_test/scripts/run_rot13_L1toL2.py

-    print "infile", infile
-    print "outfile", outfile
+    print("infile + ", infile)
+    print("outfile + ", outfile)


Same comment as for the L0 to L1 (concatenate, not print a literal +)

jtniehof · 2025-03-27T16:42:52Z

functional_test/scripts/run_rot13_RUN_timebase.py

 	infile = args[0]

-	print "infile", infile
+	print("infile + ", infile)


jtniehof · 2025-03-27T16:45:19Z

unit_tests/test_Inspector.py

+        if sys.version_info < (3, 4):
+            import imp # Depracated in Python 3.4
+            inspect = imp.load_source('inspect', os.path.join(dbp_testing.testsdir, 'inspector', 'rot13_L1_dlevel.py'))
+        else:
+            import importlib.util
+            import importlib.machinery
+            loader = importlib.machinery.SourceFileLoader('inspect', os.path.join(dbp_testing.testsdir, 'inspector', 'rot13_L1_dlevel.py'))
+            spec = importlib.util.spec_from_file_location('inspect', os.path.join(dbp_testing.testsdir, 'inspector', 'rot13_L1_dlevel.py'), loader=loader)
+            inspect = importlib.util.module_from_spec(spec)
+            loader.exec_module(inspect)
+        with warnings.catch_warnings(record=True) as w: 
+            self.assertEqual(repr(Diskfile.Diskfile(goodfile, self.dbu)), repr(inspect.Inspector(goodfile, self.dbu, 1,)()))


Can we break this out into a function, or static method, that gets called both here and the setup? Or I wonder about putting the function in dbprocessing/Utils.py, since this is also used by the code in dbprocessing/dbprocessing.py.

jtniehof · 2025-03-27T16:47:09Z

dbprocessing/dbprocessing.py

+                if sys.version_info < (3,4):
+                    import imp # imp is deprecated at python 3.4
+                    inspect = imp.load_source('inspect', code)
+                else:
+                    import importlib.util  # used in python 3.4 and above
+                    import importlib.machinery
+                    loader = importlib.machinery.SourceFileLoader('inspect', code)
+                    spec = importlib.util.spec_from_file_location('inspect', code, loader=loader)
+                    inspect = importlib.util.module_from_spec(spec)
+                    loader.exec_module(inspect)


This is used in several places, see comments below.

I'm always wary of explicit self.version_info checks and would prefer we check, say, if importlib.util exists. However, the very next thing we're going to do is rip out support for old versions of Python, so I'm inclined to leave this as you wrote it.

jtniehof

Small stuff, can probably just do a single cleanup commit and call it good.

jtniehof · 2025-05-12T16:16:46Z

dbprocessing/Utils.py

                else:
                    ans[section][item] = (ans[section][item], 0, 0)
    return ans
+def load_source(modname, filepath, module):


PEP8 specifies space (i.e. one empty line) between the functions.

Can you document parameters/returns as in the other functions?

The module argument doesn't appear to be used--can it be removed? (And then update calls to it, of course).

jtniehof · 2025-05-12T16:17:37Z

dbprocessing/dbprocessing.py

        for code, desc, arg, product in act_insp:
            try:
-                inspect = imp.load_source('inspect', code)
+                fname = code


Is this assignment necessary? Can't we just pass "code" directly to load_source?

jtniehof · 2025-05-12T16:17:56Z

dbprocessing/dbprocessing.py

-                inspect = imp.load_source('inspect', code)
+                fname = code
+                inspect = None
+                inspect = Utils.load_source('inspect',fname, inspect)


space after the comma per PEP-8

jtniehof · 2025-05-12T16:22:21Z

functional_test/scripts/run_rot13_L1toL2.py

-            output.write(infile.read().encode('rot13'))
-
+            output.write(codecs.encode(infile.read(), 'rot_13'))            


go ahead and retain the empty line at the end of the function

jtniehof · 2025-05-12T16:23:13Z

unit_tests/test_DBRunner.py

-            self.assertEqual(
-                '{}: error: {}'.format(os.path.basename(sys.argv[0]), msg), err)
-
-
+            running_script = os.path.basename(sys.argv[0])
+            self.assertTrue(err.startswith("{}: error: argument".format(running_script)))


Per PEP 8, keep two blank lines after the class definition

jtniehof · 2025-05-12T16:24:28Z

unit_tests/test_Inspector.py

-            dbp_testing.testsdir, 'inspector', 'rot13_L1.py'))
-
+        filename = os.path.join(dbp_testing.testsdir, 'inspector', 'rot13_L1.py')
+        self.inspect = None


If you update load_source not to take the module argument, all these None assignments can go away (several in this file)

jtniehof · 2025-05-12T16:25:50Z

unit_tests/test_Utils.py

+    def test_load_source(self):
+        """Testing load_source in Utils.py"""
+        filename = "temp_testfile.py"
+        with open(filename, "w") as f:
+            f.write("def hello():\n    return 'Hello, world!'\n")
+        inspect = None
+        module = Utils.load_source("temp_testfile", filename, inspect)
+        self.assertEqual(module.hello(), 'Hello, world!')
+


Nice test. Do leave a blank line before it, and two after (because it's the end of the class). Use tempfile to make a temporary directory and put your test file in there, then clean it up after (with a try/finally)--that way you're not leaving it in the current directory.

ezd1000 added 4 commits January 10, 2024 13:59

DButils: changed to 2.0 metadata binding syntax

7aae917

* doesn't work yet but fixes some errors

DButils: changed to declarative mapping

38f12f2

dbp_testing.py: add bind to table drop

6099814

DButils.getTraceback: get the sqlalchemy 2.0 join syntax right

9903e22

jtniehof reviewed Feb 2, 2024

View reviewed changes

unit_tests/dbp_testing.py Show resolved Hide resolved

jtniehof requested changes Feb 22, 2024

View reviewed changes

ezd1000 added 7 commits March 4, 2024 14:40

DButils.py: fixed join methods

8eb5697

moved the bind from Metadata constructor to table creation

028dbf5

DButils.py: fixed assertion error

9c54844

DButils.py: fixed all warning messages

4becd0d

undid change so CI will run 2.0

812baa8

added bind to drop_all and removed self from dbu.engine

7a12790

casted string with sqlalchemy.sql.text()

83c4964

ezd1000 force-pushed the sqlalchemy20 branch from bc9696b to fbb0ebd Compare March 4, 2024 20:05

jtniehof requested changes Mar 27, 2024

View reviewed changes

dbprocessing/DButils.py Outdated Show resolved Hide resolved

docs/CONTRIBUTORS.rst Outdated Show resolved Hide resolved

jtniehof reviewed May 20, 2024

View reviewed changes

scripts/scrubber.py Outdated Show resolved Hide resolved

jtniehof reviewed May 20, 2024

View reviewed changes

developer/scripts/clean_test_db.py Outdated Show resolved Hide resolved

ezd1000 added 6 commits September 18, 2024 10:28

added my name to contributors

fbe6787

added bind to create_all

2e1cb80

removed bind from metadata

bde52aa

scrubber: fix raw SQL syntax in version_number_check

aca843c

* fixes sqlalchemy 2.0

clean_test_db: fix raw SQL syntax in vacuum

8002ce0

* fixes sqlalchemy 2.0

removed extra parentheses

34a04ab

jtniehof force-pushed the sqlalchemy20 branch from dfe33cc to 34a04ab Compare September 18, 2024 14:33

sqlalchemy.sql was used without it imported

edc8b6a

ezd1000 added the admin Administrative issue not related to the codebase (e.g. access requests) label Jan 9, 2025

ezd1000 added 3 commits January 9, 2025 12:54

move the bind from Metadata constructor to table creation (to reflect…

80c5eab

… call)

execute no longer taking direct SQL string

ac70274

adding self to bind argument

e05685a

jtniehof removed the admin Administrative issue not related to the codebase (e.g. access requests) label Mar 10, 2025

ezd1000 force-pushed the sqlalchemy20 branch 2 times, most recently from f7ef949 to 66df48d Compare March 18, 2025 09:51

ezd1000 added 2 commits March 18, 2025 05:55

Creates error directory if it doesn't exist

ba5c4bf

Updating python print syntax to a function call

403d1c7

ezd1000 force-pushed the sqlalchemy20 branch 2 times, most recently from 77c64fb to e5c4e55 Compare March 18, 2025 10:44

ezd1000 added 2 commits March 18, 2025 07:19

Updated imp to importlib for load_source past Python 3.4

587b029

Updated error message checking to comply with Python 3.12 syntax

f92c10c

ezd1000 force-pushed the sqlalchemy20 branch from e5c4e55 to f92c10c Compare March 18, 2025 11:23

jtniehof requested changes Mar 27, 2025

View reviewed changes

ezd1000 added 3 commits April 1, 2025 08:31

Updating concatanation

c862aed

Creating function to handle the update of imp to importlib

62ed2ee

Test for load_source function

d60fac1

ezd1000 force-pushed the sqlalchemy20 branch from e6cb5b0 to d60fac1 Compare April 1, 2025 18:50

jtniehof requested changes May 12, 2025

View reviewed changes

		output.write(infile.read().encode('rot13'))

		output.write(codecs.encode(infile.read(), 'rot_13'))

updated dbprocessing code for sqlalchemy 2.0 #137

Are you sure you want to change the base?

updated dbprocessing code for sqlalchemy 2.0 #137

Uh oh!

Conversation

ezd1000 commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jtniehof commented Feb 6, 2024

Uh oh!

jtniehof left a comment

Choose a reason for hiding this comment

Uh oh!

jtniehof left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jtniehof left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtniehof left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ezd1000 commented Feb 2, 2024 •

edited

Loading

jtniehof left a comment •

edited

Loading