Repository navigation
ValueError: Can't create geoarrow.types.Crs from 4326 #73
Description
This might be expected if geoarrow-types hasn't yet updated to the latest support for non-projjson CRS representations in 0.2?
If it's not expected, I can create a more detailed repro. I don't have an easy repro here because passing a crs argument to geoarrow-rust will automatically use pyproj to convert to PROJJSON. In geodatafusion I'm assigning a CRS with SRID 4326 when calling ST_Point(x, y, 4326). You can see my SQL call here: https://github.com/developmentseed/lonboard/blob/751f1a1a3d35d078758f51c4ef422e04dd45bd01/examples/nyc_taxi_trips.py#L137-L138
Traceback (most recent call last):
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/marimo/_output/formatting.py", line 231, in try_format
mimetype, data = formatter(obj)
^^^^^^^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/marimo/_output/formatters/repr_formatters.py", line 107, in f_repr
contents = method()
^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/datafusion/dataframe.py", line 327, in _repr_html_
return self.df._repr_html_()
^^^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/datafusion/dataframe_formatter.py", line 303, in format_html
html.extend(self._build_table_header(schema))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/datafusion/dataframe_formatter.py", line 379, in _build_table_header
for field in schema:
^^^^^^
File "pyarrow/types.pxi", line 2907, in __iter__
File "pyarrow/types.pxi", line 2903, in pyarrow.lib.Schema.__getitem__
File "pyarrow/types.pxi", line 3209, in pyarrow.lib.Schema._field
File "pyarrow/public-api.pxi", line 185, in pyarrow.lib.pyarrow_wrap_field
File "pyarrow/types.pxi", line 2495, in pyarrow.lib.Field.init
File "pyarrow/public-api.pxi", line 129, in pyarrow.lib.pyarrow_wrap_data_type
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/geoarrow/types/type_pyarrow.py", line 62, in __arrow_ext_deserialize__
return _deserialize_storage(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/geoarrow/types/type_pyarrow.py", line 619, in _deserialize_storage
spec = TypeSpec.from_extension_metadata(extension_metadata).with_defaults(spec)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/geoarrow/types/type_spec.py", line 275, in from_extension_metadata
out_crs = crs.create(metadata["crs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/.cache/uv/archive-v0/N2xYiaCnXio3dQ-xoXvod/lib/python3.12/site-packages/geoarrow/types/crs.py", line 322, in create
raise ValueError(f"Can't create geoarrow.types.Crs from {obj}")
ValueError: Can't create geoarrow.types.Crs from 4326
Activity
paleolimbot commentedon Sep 18, 2025
Thanks! That should definitely work.
In SedonaDB we prefix it with
EPSG:4326(i.e., we only support SRIDs that are EPSG codes, and we make that assumption explicit in the scalar function implementation rather than assume something downstream will interpret it correctly). We also have ST_Crs() and ST_SetCrs() to make it a bit easier to work with non EPSG things.kylebarron commentedon Sep 18, 2025
Hmm, perhaps I should assume that input is always EPSG and assign the CRS as such.
Then I would have:
{"crs_type": "authority_code", "crs": "EPSG:4326"}I believe currently I'm creating
{"crs_type": "srid", "crs": "4326"}I can verify this. But is this representation invalid? In terms of the spec there's not much validation around what the crs string should be formatted like here.
paleolimbot commentedon Sep 18, 2025
Yes, I think that was on purpose (on my part, anyway), to force that assumption to be made explicit. It was added to ensure that when we passed on SRIDs from other specifications where this wasn't explicit (cough, Parquet), we didn't insert any assumptions.
kylebarron commentedon Sep 18, 2025
You're saying yes that is invalid? In the spec or in your implementation, or both?
Is there some spec text we can change to clarify this?
So you're saying we should essentially always use
authority_codeandEPSG:4326, and essentially never usesrid?paleolimbot commentedon Sep 18, 2025
Sorry!
"EPSG:1234"(i.e., resolve and document that assumption in your function rather than assume on an interpretation somewhere else).kylebarron commentedon Sep 19, 2025
Ok, thanks! That's good info.
I'm still trying to ascertain if there's any bug here in
geoarrow-types.Perhaps I'm incorrectly saving it as an integer 👀
paleolimbot commentedon Sep 19, 2025
Ahh! That would make sense. geoarrow-types should still probably handle that even though it's theoreetically out of spec.
kylebarron commentedon Sep 19, 2025
Hmm
https://github.com/datafusion-contrib/datafusion-geo/blob/171bdaf31069efe618371249e051e5b8eaa109d1/rust/geodatafusion/src/udf/native/constructors/point.rs#L80-L81
https://github.com/geoarrow/geoarrow-rs/blob/338a08cd2f5f2b35df36453ec3ad6ef031dfb702/rust/geoarrow-schema/src/crs.rs#L101-L107
I sure thought I was saving it as a string.
This crashes the interpreter when trying to access the table schema. (Run with
uv run script.py)But this works just fine when I load it into arro3:
Run with
uv run script.pyI see:Is the issue that it should be
{'ARROW:extension:metadata': {"crs":"4326","crs_type":"srid"}, 'ARROW:extension:name': 'geoarrow.point'}without encoding the JSON as a string?
But the spec says
paleolimbot commentedon Sep 19, 2025
I think you're doing the right thing here...I bet something funny is going on somewhere you're not in control of (hopefully not geoarrow-c, since theoretically that shouldn't be in play here). With that script I can probably get a backtrace (if the crash is happening in geoarrow-c or pyarrow).
kylebarron commentedon Sep 19, 2025
This is happening just with
geoarrow-types, wheregeoarrow-cisn't even a dependency I don't think?paleolimbot commentedon Sep 19, 2025
In theory, it's not a dependency!
kylebarron commentedon Sep 19, 2025
Just verified it's not.
uv lock --script script.py; cat script.py.lockgives the following, and nogeoarrow-cCyberNativeAI commentedon Oct 16, 2026
I think this is where the integer comes from: geoarrow-types turns the SRID string into a JSON number when it re-serializes the type.
StringCrs.__geoarrow_crs_json_values__(geoarrow-types/src/geoarrow/types/crs.py, same in 0.4.1 and current main) callsjson.loads(self._crs)so that valid JSON is not escaped into a JSON string.json.loads("4326")succeeds and returns the int4326, soextension_metadata()writes{"crs": 4326}and dropscrs_type. The nextfrom_extension_metadata()passes that int tocrs.create(), which raises. So geodatafusion was writing a string, as you thought.With geoarrow-types 0.4.1 alone:
Your first traceback shows that ValueError propagating normally, so I haven't pinned down exactly how it becomes the interpreter crash in your second script; the Parquet crash below does stop with the change at the end. The same path is now reachable from plain Parquet: pyarrow 26 maps a Parquet GEOMETRY column whose
crsissrid:2193to{"crs": "2193", "crs_type": "srid"}. Withgeoarrow.pyarrowimported,pq.read_table()on a small synthetic file like that exits with SIGSEGV (3 of 3 runs; pyarrow 26.0.0, geoarrow-pyarrow 0.3.0, geoarrow-types 0.4.1, Linux x86-64).srid:0crashes the same way, and GeoParquet 2.0.0-rc.1 says writers SHOULD usesrid:0for an unknown CRS. The same column withEPSG:2193or inline PROJJSON reads correctly.Separately, and not specific to srid: with the same versions I also saw an intermittent SIGABRT at interpreter exit (
terminate called without an active exception) after reads that had already returned correct results, EPSG files included. pyarrow 26 without thegeoarrow.pyarrowimport did not show it. I haven't looked into that one; mentioning it so it is not mistaken for the srid crash, which is a SIGSEGV with the ValueError traceback every time.A possible fix that keeps the existing unwrapping of JSON objects and quoted strings:
With that change, the snippet prints
{"crs": "4326"}and round-trips,pytest geoarrow-types/testsstill passes (79 passed at 65cf79c, before and after), and thesrid:2193andsrid:0Parquet files read instead of segfaulting (extension metadata{"crs": "2193"}and{"crs": "0"}).crs_typeis still not carried through, so after the round trip the value is a bare"2193"; whetherStringCrsshould keepcrs_type: "srid"is a separate decision.Prepared by CyberNative AI LLC, an AI-run company, using AI agents. Corrections: hello@cybernative.ai.