SNOW-2916697: KeyError: \'command\'

[blotero]

Python version

3.12.9

Operating system and processor architecture

macOS-15.7.2-arm64-arm-64bit

Installed packages

I'm not using pip, but uv instead.

This is my pyproject.toml: https://github.com/fluidattacks/universe/blob/b6c79b7d0a410c5c29ff516dffb84f1eaa6f5949/sifts/pyproject.toml

And this is my uv.lock: https://github.com/fluidattacks/universe/blob/b6c79b7d0a410c5c29ff516dffb84f1eaa6f5949/sifts/uv.lock

What did you do?

The execution runtime failed when running:

mv.run(sample_df, service_name=PREDICTIONS_MODEL_MAP[PREDICTION_VERSION])

(full source code at https://github.com/fluidattacks/universe/blob/f7df308765f4db7fe8ab69aec48a6c5a219753ea/sifts/sifts/validation/pipeline.py#L392)

Failing with the following error:

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/_internal/telemetry.py", line 576, in execute_func_with_statement_params
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/model/_client/model/model_version_impl.py", line 549, in run
    return self._model_ops.invoke_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/model/_client/ops/model_ops.py", line 1007, in invoke_method
    s_df = snowpark_handler.SnowparkDataFrameHandler.convert_from_df(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/model/_signatures/snowpark_handler.py", line 129, in convert_from_df
    sp_df = session.create_dataframe(df)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/_internal/utils.py", line 1112, in call_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/session.py", line 3477, in create_dataframe
    table = self.write_pandas(
            ^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/_internal/utils.py", line 1106, in call_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/session.py", line 3270, in write_pandas
    success, _, _, ci_output = write_pandas(
                               ^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/pandas_tools.py", line 445, in write_pandas
    cursor._upload(
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/cursor.py", line 1836, in _upload
    file_transfer_agent.execute()
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/file_transfer_agent.py", line 401, in execute
    self._parse_command()
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/file_transfer_agent.py", line 928, in _parse_command
    self._command_type = response["command"]
                         ~~~~~~~~^^^^^^^^^^^
KeyError: 'command'

Likely to be an uncaught permissions issue, since the execution succeeds with different permissions sets.

1. With ownership on the model: success
2. With READ and USAGE permissions on the model: fail with KeyError

READ and USAGE are meant to be enough permissions according to documentation https://docs.snowflake.com/en/developer-guide/snowflake-ml/model-registry/overview#required-privileges

What did you expect to see?

I would have expected either a success when trying to run the model inference or a more insightful error in case of missing permissions.

Can you set logging to DEBUG and collect the logs?

025-12-23 07:56:16,139 - asyncio_12 network.py:880 - _request_exec_wrapper() - DEBUG - remaining request timeout: 5000 ms, retry cnt: 1
2025-12-23 07:56:16,139 [DEBUG] [snowflake.connector.network] remaining request timeout: 5000 ms, retry cnt: 1
2025-12-23 07:56:16,139 - asyncio_12 network.py:861 - add_request_guid() - DEBUG - Request guid: 0bb59c4e-40b5-4bd0-9aad-db29bf645e73
2025-12-23 07:56:16,139 [DEBUG] [snowflake.connector.network] Request guid: 0bb59c4e-40b5-4bd0-9aad-db29bf645e73
2025-12-23 07:56:16,139 - asyncio_12 network.py:1065 - _request_exec() - DEBUG - socket timeout: 60
2025-12-23 07:56:16,139 [DEBUG] [snowflake.connector.network] socket timeout: 60
2025-12-23 07:56:16,310 - asyncio_12 connectionpool.py:544 - _make_request() - DEBUG - https://pekojyy-zwb14441.snowflakecomputing.com:443 "POST /telemetry/send?request_guid=0bb59c4e-40b5-4bd0-9aad-db29bf645e73 HTTP/1.1" 200 None
2025-12-23 07:56:16,310 [DEBUG] [snowflake.connector.vendored.urllib3.connectionpool] https://pekojyy-zwb14441.snowflakecomputing.com:443 "POST /telemetry/send?request_guid=0bb59c4e-40b5-4bd0-9aad-db29bf645e73 HTTP/1.1" 200 None
2025-12-23 07:56:16,310 - asyncio_12 network.py:1097 - _request_exec() - DEBUG - SUCCESS
2025-12-23 07:56:16,310 [DEBUG] [snowflake.connector.network] SUCCESS
2025-12-23 07:56:16,311 - asyncio_12 network.py:740 - _post_request() - DEBUG - ret[code] = None, after post request
2025-12-23 07:56:16,311 [DEBUG] [snowflake.connector.network] ret[code] = None, after post request
2025-12-23 07:56:16,311 - asyncio_12 telemetry.py:198 - send_batch() - DEBUG - Successfully uploading metrics to telemetry.
2025-12-23 07:56:16,311 [DEBUG] [snowflake.connector.telemetry] Successfully uploading metrics to telemetry.
Traceback (most recent call last):
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/bin/sifts-validation", line 10, in <module>
    sys.exit(validation_cli())
             ^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/__init__.py", line 63, in validation_cli
    asyncio.run(_run_config_validation(config, skip_pipeline=skip_pipeline))
  File "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/__init__.py", line 165, in _run_config_validation
    await _run_validation_pipeline_steps(config, working_dir, group_name, root_nickname)
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/__init__.py", line 123, in _run_validation_pipeline_steps
    await run_validation_pipeline(
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/pipeline.py", line 456, in run_validation_pipeline
    await generate_predictions(db_backend, snowflake_session, group_name, root_nickname)
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/pipeline.py", line 380, in generate_predictions
    count = await db_backend.batch_insert_predictions(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blotero/Projects/universe/sifts/common/sifts_io/db/sqlite_backend.py", line 888, in batch_insert_predictions
    async for prediction in predictions:
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/pipeline.py", line 372, in prediction_generator
    async for pred in _classify_batch(embeddings, snowflake_session, taxonomy_index):
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/pipeline.py", line 435, in _classify_batch
    predictions = await asyncio.to_thread(_classify_blocking)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blotero/Projects/universe/sifts/sifts/validation/pipeline.py", line 404, in _classify_blocking
    mv.run(sample_df, service_name=PREDICTIONS_MODEL_MAP[PREDICTION_VERSION])
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/_internal/telemetry.py", line 611, in wrap
    return ctx.run(execute_func_with_statement_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/_internal/telemetry.py", line 576, in execute_func_with_statement_params
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/model/_client/model/model_version_impl.py", line 549, in run
    return self._model_ops.invoke_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/model/_client/ops/model_ops.py", line 1007, in invoke_method
    s_df = snowpark_handler.SnowparkDataFrameHandler.convert_from_df(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/ml/model/_signatures/snowpark_handler.py", line 129, in convert_from_df
    sp_df = session.create_dataframe(df)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/_internal/utils.py", line 1112, in call_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/session.py", line 3477, in create_dataframe
    table = self.write_pandas(
            ^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/_internal/utils.py", line 1106, in call_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/snowpark/session.py", line 3270, in write_pandas
    success, _, _, ci_output = write_pandas(
                               ^^^^^^^^^^^^^
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/pandas_tools.py", line 445, in write_pandas
    cursor._upload(
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/cursor.py", line 1836, in _upload
    file_transfer_agent.execute()
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/file_transfer_agent.py", line 401, in execute
    self._parse_command()
  File "/nix/store/x64185xsqfzr6359adc1cfw8ds3mfsrg-sifts-unified/lib/python3.12/site-packages/snowflake/connector/file_transfer_agent.py", line 928, in _parse_command
    self._command_type = response["command"]
                         ~~~~~~~~^^^^^^^^^^^
KeyError: 'command'
2025-12-23 07:56:16,322 [INFO] [snowflake.snowpark.session] Closing session: 2082230315931982
2025-12-23 07:56:16,323 [INFO] [snowflake.snowpark.session] Canceling all running queries
2025-12-23 07:56:16,323 - MainThread cursor.py:952 - execute() - DEBUG - executing SQL/command
2025-12-23 07:56:16,323 [DEBUG] [snowflake.connector.cursor] executing SQL/command
2025-12-23 07:56:16,323 - MainThread cursor.py:971 - execute() - DEBUG - query: [select system$cancel_all_queries(2082230315931982)]
2025-12-23 07:56:16,323 [DEBUG] [snowflake.connector.cursor] query: [select system$cancel_all_queries(2082230315931982)]

[sfc-gh-sghosh]

Hello @blotero ,

Thanks for raising the issue.

The documentation just mentions about the USAGE and READ on the model, but that is not sufficient to run the model because internally it also creates a temp table while doing write_pandas...so you need following privileges

On the role that only has READ / USAGE on the model, also grant:

USAGE on the warehouse they’re using
USAGE on the database and schema where inference is running
CREATE TEMPORARY TABLE (or CREATE TABLE (for mv.run)), CREATE STAGE ( for write_pandas_ on that schema
This aligns with what session.write_pandas(...) needs to create a temp table and upload data.

Regards,
Sujan

[blotero]

Thanks @sfc-gh-sghosh, will give it a try and let you know.

Regards,