GeoPackage reader #183

jph6366 · 2025-11-18T22:52:44Z

No description provided.

Will comb through for various edges cases (empty, 3D, etc.) once the writer is gpkg spec-compliant implementation

…d import DBInterface

… commits and removing unneccesary branches

…idiomatic modifications, and map do syntax for meshes collection

… abstract WKBGeometry type added, handling failing tests (1 erroring gisissues.jl, 1 failing novalues.jl), fixed bugs gpkgmeshattrs and gpkgmesh

… function for ring vs rope extents

…ed, idiomatic refactor of SQLite.load! internal transactions, small fixes

…ite.Stmt and Transaction

…coords already float

…nd table inserts, updated geopackage test conform to specification req 29

…ures... and other cases

…kg_io

jph6366 · 2025-11-18T23:08:34Z

src/extra/gpkg/read.jl

+  code = metadata.code
+  srsid = metadata.srsid
+  if srsid == 0 || srsid == 4326
+    crs = isone(metadata.zextent) ? LatLonAlt{WGS84Latest} : LatLon{WGS84Latest}


I bounded crs conditionally to LatLonAlt if z field in the GeoPackage gpkg_geometry_columns table has a 1.

If LatLon is bounded to a crs that has an axis values recorded for {x,y,{z}}.
The outputted Geometry is reduced by one axis to fit LatLon{Datum, 2} instead of including the z axis.

This removal of the z axis happens in getcoord in conversion.jl where its calls raw(coords(p))

The method overload to retain the z/alt value

raw(coords::CRS) = coords.x, coords.y raw(coords::LatLon) = coords.lon, coords.lat raw(coords::LatLonAlt) = coords.lon, coords.lat, coords.alt

Is this not an appropriate solution to handling the z-dimension?

Should there also be {x,y,{z}} method overload for raw(coord::CRS) = coords.x, coords.y, coords.z as well?

You need to adhere to the GPKG spec. The srdid == 4326 probably means LatLon{WGS84Latest}, but you are using the zextent to switch to another type, which is probably incorrect.

If you use the spec as your only source of truth, we will be much more safe.

jph6366 · 2025-11-18T23:14:30Z

src/extra/gpkg/read.jl

+  table = map(gpkgbinary) do row
+    # According to https://www.geopackage.org/spec/#r30
+    # A feature table or view SHALL have only one geometry column.
+    geomindex = findfirst(==(Symbol(geomcolumn)), keys(row))


Per the previous comments on simplifying this check for the geomcolumn:

The main conflict is the test for duplicate fieldnames in the values Tuple, otherwise we could simplify this iteration to conditionally check for geomcolumn by its name rather than using its geomindex

The main conflict is the test for duplicate fieldnames in the values Tuple, otherwise we could simplify this iteration to conditionally check for geomcolumn by its name rather than using its geomindex

Is that possible? I thought that the rows of this table contained the attributes of the geometry as named tuples, which by definition are unique names. You can't have duplicate field names in a named tuple.

src/extra/gpkg/read.jl

jph6366 · 2025-11-19T16:38:44Z

src/extra/gpkg/read.jl

+  # calculate GeoPackageBinaryHeader size in byte stream given extent of envelope:
+  # envelope is [minx, maxx, miny, maxy, minz, maxz], 48 bytes or envelope is [minx, maxx, miny, maxy], 32 bytes or no envelope, 0 bytes
+  # byte[2] magic + byte[1] version + byte[1] flag + byte[4] srs_id + byte[(8*2)×(x,y{,z})] envelope
+  skiplen = iszero(envelope) ? 4 : 4 + 8 * 2 * (envelope + 1)


I'm swapping to using skip (before seek) and starting from byte[4] given previous reads.

if we have no envelope then we will skip srs_id byte[4]

otherwise we calculate the amount of bytes needed to skip srs_id and envelope bytes

juliohm · 2025-11-20T18:50:21Z

src/extra/gpkg/read.jl

+  # According to https://www.geopackage.org/spec/#r6 and https://www.geopackage.org/spec/#r7
+  # PRAGMA integrity_check returns a single row with the value 'ok'
+  # PRAGMA foreign_key_check (w/ no parameter value) returns an empty result set
+  if first(DBInterface.execute(db, "PRAGMA integrity_check;")).integrity_check != "ok" ||


Suggested change

if first(DBInterface.execute(db, "PRAGMA integrity_check;")).integrity_check != "ok" ||

if only(DBInterface.execute(db, "PRAGMA integrity_check;")).integrity_check != "ok" ||

When the query returns a single entry, we can replace first by only to get a safer check.

Please split these two checks into separate paragraphs.

juliohm · 2025-11-20T18:53:02Z

src/extra/gpkg/read.jl

+  # According to https://www.geopackage.org/spec/#r10 and https://www.geopackage.org/spec/#r13
+  # A GeoPackage SHALL include a 'gpkg_spatial_ref_sys' table and a 'gpkg_contents table'
+  if first(DBInterface.execute(
+    db,
+    """
+    SELECT COUNT(*) AS n FROM sqlite_master WHERE 
+    name IN ('gpkg_spatial_ref_sys', 'gpkg_contents') AND 
+    type IN ('table', 'view');
+    """
+  )).n != 2
+    throw(ErrorException("missing required metadata tables in the GeoPackage SQL database"))
+  end


Is this the most idiomatic query to find out if a given table is present in the database? It seems you are using a count to find out if the table is present. I wonder if we couldn't be more explicit with some special query that simply says yes or no when a table is present.

We should probably split this check into two checks, one for each required table.

juliohm · 2025-11-20T18:54:56Z

src/extra/gpkg/read.jl

+# gpkg_contents table.
+function gpkgextract(db; layer=1)
+  # get the first (and only) feature table returned in sqlite query results
+  metadata = first(DBInterface.execute(


Suggested change

metadata = first(DBInterface.execute(

metadata = only(DBInterface.execute(

ERROR: LoadError: TypeError: non-boolean (Missing) used in boolean context

Each row is only valid when called and for some reason only invalidates the forward-only iterator context

SQLite.Row{false}: :tablename missing :geomcolumn missing :srsid missing :org missing :orgcoordsysid missing

first retains the cursor's current step after it calls iterate as I understand it

juliohm · 2025-11-20T18:57:11Z

src/extra/gpkg/read.jl

+# According to Geometry Columns Table Requirements
+# https://www.geopackage.org/spec/#:~:text=2.1.5.1.2.%20Table%20Data%20Values
+#------------------------------------------------------------------------------
+# Requirement 16: https://www.geopackage.org/spec/#r16 
+# Values of the gpkg_contents table srs_id column 
+# SHALL reference values in the gpkg_spatial_ref_sys table srs_id column
+#
+# Requirement 18: https://www.geopackage.org/spec/#r18
+# The gpkg_contents table SHALL contain a row 
+# with a lowercase data_type column value of "features" 
+# for each vector features user data table or view.
+#
+# Requirement 22: gpkg_geometry_columns table
+# SHALL contain one row record for the geometry column
+# in each vector feature data table
+#
+# Requirement 23: gpkg_geometry_columns table_name column
+# SHALL reference values in the gpkg_contents table_name column
+# for rows with a data_type of 'features'
+#
+# Requirement 24: The column_name column value in a gpkg_geometry_columns row
+# SHALL be the name of a column in the table or view specified by the table_name
+# column value for that row.
+#
+# Requirement 25: The geometry_type_name value in a gpkg_geometry_columns row
+# SHALL be one of the uppercase geometry type names specified
+#
+# Requirement 146: The srs_id value in a gpkg_geometry_columns table row
+# SHALL match the srs_id column value from the corresponding row in the
+# gpkg_contents table.


Please try to place comments right before the lines of code where they are needed. Right now it is difficult to see which lines of code require some of these assumptions.

juliohm · 2025-11-20T19:01:50Z

src/extra/gpkg/read.jl

+
+function wkbgeombuffer(row, geomcolumn)
+  # get the column of SQL Geometry Binary specified by gpkg_geometry_columns table in column_name field
+  buff = IOBuffer(getproperty(row, Symbol(geomcolumn)))


What type of object do you get if you don't wrap the result with IOBuffer? Isn't it a valid IO stream already? Or a Vector{UInt8}? Depending on the type we can decide what to do.

it is a Vector{UInt8}
sample:

UInt8[0x47, 0x50, 0x00, 0x01, 0xe6, 0x10, 0x00, 0x00, 0x01, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00]

juliohm · 2025-11-20T19:10:19Z

src/extra/gis.jl

+  elseif endswith(fname, ".gpkg")
+    return gpkgread(fname; layer, kwargs...)


Notice that you are adding the function call in the wrong place. The gpkgread function already returns a GeoTable, it needs to be called in the body of gisread. If you call it inside gistable, then the result will be forwarded to asgeotable which is a duplication of work.

Perhaps we should even consider moving this call to the root load function. The pipeline we are discussing for gpkg is very different from the other GIS backends.

jph6366 · 2025-11-24T14:41:58Z

src/extra/gpkg/read.jl

+  aspatial = getindex.(table, 1)
+
+  # check if type of aspatial elements generated are calls to the macro for declaring `NamedTuple` types
+  eltype(aspatial) != @NamedTuple{} || return nothing, getindex.(table, 2)


returns here if aspatial = [NamedTuple(), ..., NamedTuple()]

…g_read

jph6366 and others added 30 commits August 4, 2025 09:07

initial commit

6aae933

rigorously tested yet no valid gpkg writer just yet.

53a4b88

SQLite package install, read/write includes, SQLite.DBInterface

43a87af

Will comb through for various edges cases (empty, 3D, etc.) once the writer is gpkg spec-compliant implementation

formatter installed, addressing comments and some varname changes, an…

91ed1ef

…d import DBInterface

addressing commit suggestions

3166a1c

missed this one

0255232

another mistake missed

5e60488

Reader is sufficiently spec-compliant for further testing, addressing…

39730ce

… commits and removing unneccesary branches

removed newline

26b7602

removed returns for code style

0077fef

creation of wkb.jl, removing/recycling redundant and/or unused code, …

de56781

…idiomatic modifications, and map do syntax for meshes collection

formatting and include(wkb.jl)

6c3c505

incoporated custom SQLite data types, annotated type for wkbgeometry,…

5dc18fb

… abstract WKBGeometry type added, handling failing tests (1 erroring gisissues.jl, 1 failing novalues.jl), fixed bugs gpkgmeshattrs and gpkgmesh

removed minor version

46b20b9

passing all tests, added branch for novalues, and added wkblinearring…

6538130

… function for ring vs rope extents

missing blob error handling

e93f91c

removed unnecessary arrays

d73a169

added necessary arrays comprehensions back, database needs to be clos…

1cf927e

…ed, idiomatic refactor of SQLite.load! internal transactions, small fixes

fixing Ring and Rope expressions to include splat and rearranging SQL…

4201eb2

…ite.Stmt and Transaction

add Random, some capi sqlite, sql faster and safer in a transaction, …

f401635

…coords already float

Merge branch 'master' into gpkg_io

82d635a

removed unnecessary Random and Dates, and reorganized create tables a…

acc948c

…nd table inserts, updated geopackage test conform to specification req 29

removed id ok autoincrement, undo my evil test changes

dc722cc

tests passing! simple features supported! need to test for multi feat…

f7e4e85

…ures... and other cases

multi. fix, added util to infer Multi const geometry,

b2451b1

Merge branch 'master' into gpkg_io

31d9cba

formatting

a884e88

Merge branch 'gpkg_io' of https://github.com/jph6366/GeoIO.jl into gp…

e9cd713

…kg_io

mit license header

6d2244c

resolving reviewed changes

7cd4851

juliohm and others added 13 commits November 11, 2025 13:47

More adjusments

435f6bb

More adjustments

7751cd1

More adjustments

c1a2ea7

simply select stmt; zextent refactor

7399fa2

Refactor wkb2geom

ef477bb

More adjustments

b912fa8

More adjustments

2aa6ce0

More adjustments

5efb822

wkb refactors

f0ff9b2

simplify and rewrite headerskip

2a012bc

batch wkb2points

bd2a0ed

refactor wkbtype z mask

b154726

tested on 3dLineString

f2fcb78

jph6366 marked this pull request as ready for review November 18, 2025 22:59

jph6366 commented Nov 18, 2025

View reviewed changes

src/extra/gpkg/read.jl Outdated Show resolved Hide resolved

jph6366 commented Nov 19, 2025

View reviewed changes

juliohm reviewed Nov 20, 2025

View reviewed changes

jph6366 force-pushed the gpkg_read branch from 0ead8f2 to f2fcb78 Compare November 22, 2025 16:19

jph6366 added 3 commits November 22, 2025 13:07

hoisted gpkgread out of gis.jl; clean code; novalues condition

6f62749

clean code

5da73de

format

6f4273b

jph6366 commented Nov 24, 2025

View reviewed changes

Merge branch 'master' of https://github.com/jph6366/GeoIO.jl into gpk…

64b2b35

…g_read

	if first(DBInterface.execute(db, "PRAGMA integrity_check;")).integrity_check != "ok" \|\|
	if only(DBInterface.execute(db, "PRAGMA integrity_check;")).integrity_check != "ok" \|\|

	metadata = first(DBInterface.execute(
	metadata = only(DBInterface.execute(

		elseif endswith(fname, ".gpkg")
		return gpkgread(fname; layer, kwargs...)

GeoPackage reader #183

Are you sure you want to change the base?

GeoPackage reader #183

Conversation

jph6366 commented Nov 18, 2025

Uh oh!

jph6366 Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

The method overload to retain the z/alt value

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jph6366 Nov 18, 2025 •

edited

Loading