Skip to content

SqliteQueryGenerator._gen_sql_array_sub_selection() incompatibility with recent SQLite #21

@haatveit

Description

@haatveit

When running the following select query test against recent versions of SQLite

pysquril/pysquril/tests.py

Lines 164 to 167 in af1a58c

# selecting keys inside an array slice
out = run_select_query('select=x,c[1|h,p]')
assert out[0][1] is None
assert out[1][1] == [32, 0]

it fails with exception

AssertionError: assert '[32,0]' == [32, 0]

The SQL used for this select query is generated here:

def _gen_sql_array_sub_selection(
self,
term: SelectTerm,
parsed: Union[
ArraySpecificSingle,
ArraySpecificMultiple,
ArrayBroadcastSingle,
ArrayBroadcastMultiple,
],
) -> str:
if (
isinstance(parsed, ArraySpecificSingle)
or isinstance(parsed, ArraySpecificMultiple)
):
fullkey = f"and fullkey = '$.{term.bare_term}[{parsed.idx}]'"
vals = 'vals'
else:
fullkey = ''
vals = 'json_group_array(vals)'
temp = []
for key in parsed.sub_selections:
temp.append(f"json_extract(value, '$.{key}')")
sub_selections = ','.join(temp)
sub_selections = f'json_array({sub_selections})' if len(temp) > 1 else f'{sub_selections}'
selection = f"""
(case when json_extract(data, '$.{term.bare_term}') is not null then (
select {vals} from (
select
{sub_selections} as vals
from (
select key, value, fullkey, path
from {self.table_name}, json_tree({self.table_name}.data)
where path = '$.{term.bare_term}'
{fullkey}
)
)
)
else null end)
"""
return self._maybe_apply_function(term, selection)

Generated SQL:

select json_array(json_extract(data, '$.x'),
                (case when json_extract(data, '$.c') is not null then (
                    select vals from (
                        select
                            json_array(json_extract(value, '$.h'),json_extract(value, '$.p')) as vals
                        from (
                            select key, value, fullkey, path
                            from test_table, json_tree(test_table.data)
                            where path = '$.c'
                            and fullkey = '$.c[1]'
                            )
                        )
                    )
                else null end)
            ) from test_table   

Value we're getting back from run_select_query('select=x,c[1|h,p]'):

[[1900, None], [None, '[32,0]'], [88, None], [107, None], [10, None]]

So far I've confirmed this behavior when using SQLite 3.39.5 and 3.40.1. After skimming the SQLite release notes, my hunch is that JSON behavior most likely changed in 3.38.0, where JSON support was promoted to built-in and some effort was made to align behavior with PostgreSQL/MySQL, but I have yet to confirm anything.

### Tasks
- [ ] Confirm which exact SQLite version introduces this new behavior
- [ ] Add support for new behavior without breaking backwards compatibility

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions