Converting Python dictionary into an internal data structure

[inikolaev]

I'm new both Rust and PyO3 and maybe this is somewhat obvious, but could not figure this out yet.

I'm trying to create a Python extension around a Rust library that accepts its own context object that I have to construct from a dictionary that I pass from Python. Here's an example code:

fn evaluate(&mut self) -> PyResult<bool> {
    let context = Context::default();
    let value = self.program.execute(&context).unwrap();

    match value {
        Value::Bool(v) => Ok(v),
        _ => Ok(false),
    }
}

Now I'm trying to change that so that it accepts a Python dictionary and I need to convert values from the dictionary to the correct type. Something like this:

fn evaluate(&mut self, py: Python, ctx: HashMap<String, &PyObject>) -> PyResult<bool> {
    let context = Context::default();

    for (key, value) in ctx {
        let type_of_value = // how would I get the type of `value` here?
        match type_of_value {
            bool => context.add_variable(key, Value::Bool(value)),
            String => context.add_variable(key, Value::String(value)),
        }
    }

    let value = self.program.execute(&context).unwrap();
        
    match value {
        Value::Bool(v) => Ok(v),
        _ => Ok(false),
    }
}

For the Python side this would look like this:

program.evaluate({
  "string1": "value1",
  "bool1": True,
})

The Value is an enum, so maybe instead I could somehow specify how it could be serialised between Rust and Python or maybe a better option would be to create custom classes to represent the values I want to pass from Python?

Now that I think of it I feel like creating an internal enum that would be used from Python side as a value could be a better option instead of being able to accept any Python object 🤔

I would appreciate any pointers into the right direction.

[adamreichold]

You have multiple options: You could derive FromPyObject for your enum and consume it directly, e.g.

#[derive(FromPyObject)]
enum Value {
  String(String),
  Bool(bool),
}

#[pyfunction]
fn evaluate(&mut self, py: Python, ctx: HashMap<String, Value>) -> PyResult<bool> {
    let context = Context::default();

    for (key, value) in ctx {
        context.add_variable(key, value);
    }

    let value = self.program.execute(&ctx).unwrap();
        
    match value {
        Value::Bool(v) => Ok(v),
        _ => Ok(false),
    }
}

But you could also try to extract the types required, i.e. use the FromPyObject trait manually, e.g.

#[pyfunction]
fn evaluate(&mut self, py: Python, ctx: HashMap<String, PyObject>) -> PyResult<bool> {
    let context = Context::default();

    for (key, value) in ctx {
        if let Ok(value) = value.extract::<bool>() {
            context.add_variable(key, Value::Bool(value));
        } else if let Ok(value) = value.extract::<String>() {
            context.add_variable(key, Value::String(value))
        } else {
            // Handle case of unsupported type, e.g. return an error.
        }
    }

    let value = self.program.execute(&context).unwrap();
        
    match value {
        Value::Bool(v) => Ok(v),
        _ => Ok(false),
    }
}

[inikolaev]

Thank you! This is really helpful.
I had to make a slight change - the extract needs an instance of Python to be passed as an argument, however if I replace HashMap with &PyDict - then I don't have to do that. I guess this is because &PyDict is pointing to the actual dictionary in Python, while HashMap<String, PyObject> is already a copy which is detached?

Regarding enums - as I understood from your example this enum is not exposed to Python - it merely helps to deserialise dictionary values. Would this still work if I were to expose it to Python?

For example I would like to enforce restrictions on the types on values that could be put into the dictionary to the ones supported. Those could be numbers, booleans, strings, lists and nested dictionaries. E.g. consider the following Python signature:

MyList = list[Union[MyStr, MyInt, MyBool, "MyList", "MyDict"]]
MyDict = dict[str, Union[MyStr, MyInt, MyBool, MyList, "MyDict"]]
MyValue = MyStr | MyInt | MyBool | MyList | MyDict


def evaluate(context: dict[str, MyValue]):
    pass

[adamreichold]

Regarding enums - as I understood from your example this enum is not exposed to Python - it merely helps to deserialise dictionary values. Would this still work if I were to expose it to Python?

By "exposing" you mean using it as a #[pyclass]? I think we support only fieldless enums as pyclasses, c.f .https://pyo3.rs/v0.19.2/class.html#pyclass-enums

[adamreichold]

(But since type checking is an optional add in Python, I think you can still use your type signatures and unpack them using the internal enum. Consider it as a type-driven short hand for the hand-rolled extract-based code.)

[inikolaev]

Alright, thank you.

I think I have some ideas to try out:

• create #[pyclass]es to use for type checking and then add some signatures to the methods and also define type aliases that could be included into the wheel with maturin
• create a separate context #[pyclass] and use that to construct the context directly, without trying to parse a dictionary.