pydantic support

[henderiw]

did anyone try pydantic with componentize-py?

I ran into this issue and was told to raise an issue here.

I am not able to compile the code to a wasm binary
error:

(.venv) (base) wasm/component-reconciler - (result) > just build-guest-python                                    
/opt/homebrew/bin/uv
==> building python guest component...
Traceback (most recent call last):
  File "/Users/henderiw/.cache/uv/archive-v0/_oAvmxPnx7MxwA070F1Ji/bin/componentize-py", line 10, in <module>
    sys.exit(script())
             ~~~~~~^^
AssertionError: Traceback (most recent call last):
  File "/0/rec.py", line 3, in <module>
    from apis.topo.v1alpha1.topology_types import Topology 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/0/apis/topo/v1alpha1/topology_types.py", line 1, in <module>
    from pydantic import BaseModel, Field, ValidationError, model_validator
  File "/1/pydantic/__init__.py", line 421, in __getattr__
    module = import_module(module_name, package=package)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/1/pydantic/main.py", line 29, in <module>
    import pydantic_core
  File "/1/pydantic_core/__init__.py", line 6, in <module>
    from ._pydantic_core import (
ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'


Caused by:
    ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'

I installed the modules in my venv
(.venv) (base) wasm/component-reconciler - (result) > uv pip show typing-extensions
Name: typing-extensions
Version: 4.12.2
Location: /Users/henderiw/code/wasm/component-reconciler/.venv/lib/python3.11/site-packages
Requires:
Required-by: pydantic, pydantic-core
(.venv) (base) wasm/component-reconciler - (result) >
(.venv) (base) wasm/component-reconciler - (result) > uv pip show pydantic_core
Name: pydantic-core
Version: 2.27.2
Location: /Users/henderiw/code/wasm/component-reconciler/.venv/lib/python3.11/site-packages
Requires: typing-extensions
Required-by: pydantic
Not sure what is going on? (edited)

[henderiw]

the code with the error is in this PR

henderiw/component-reconciler#3

[dicej]

Hi @henderiw. Thanks for reporting this.

I just tried making a simple app that uses Pydantic, and it seems to work. Here's what I did:

mkdir foo
cd foo
python3 -m venv .venv
source .venv/bin/activate
cargo install wasmtime-cli
pip install pydantic==2.5.3 componentize-py
curl -OL https://github.com/WebAssembly/wasi-cli/archive/refs/tags/v0.2.2.tar.gz
tar xf v0.2.2.tar.gz
curl -OL https://github.com/dicej/wasi-wheels/releases/download/latest/pydantic_core-wasi.tar.gz
tar xf pydantic_core-wasi.tar.gz
cat >app.py <<EOF
from command import exports
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel

class User(BaseModel):
    id: int
    name: str = 'John Doe'
    signup_ts: Optional[datetime] = None
    friends: List[int] = []

class Run(exports.Run):
    def run(self) -> None:
        external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
        user = User(**external_data)
        print(user)
EOF
componentize-py -d wasi-cli-0.2.2/wit -w command componentize app -o cli.wasm
wasmtime run cli.wasm

Note that pydantic depends on pydantic-core, which is written in Rust and compiled as a native extension. When you run pip install pydantic, pip will download and install a copy of pydantic-core for your native platform (e.g. MacOS/ARM64 or whatever) -- not wasm32-wasi, which is what we actually want. And since there's not yet an official way to publish WASI wheels for Python, we must use a build from the wasi-wheels repo instead. And since that build is about a year old now, we need to also use an old pydantic package which matches (v2.5.3). If you want to use the latest version, you'll need to build the corresponding version of pydantic-core for wasm32-wasi yourself (e.g. using this build script).

BTW, @benbrandt is working on improving the WASI wheel situation. The goal is to create an "official" package index to which you'll be able to point pip so it can download and install WASI wheels itself. That will remove the need to download and unpack a .tar.gz file manually.

Hope that helps!

[henderiw]

I can now build indeed the component. but when I run it I get this error

(.venv) (base) wasm/component-reconciler - (result) > just run-guest-python
==> running python guest component...
Finished dev profile [unoptimized + debuginfo] target(s) in 0.10s
Running target/debug/reconciler
Running iteration: 1
Traceback (most recent call last):
File "/0/rec.py", line 15, in reconcile
File "/1/pydantic/_internal/_model_construction.py", line 92, in new
File "/1/pydantic/_internal/model_construction.py", line 308, in inspect_namespace
ModuleNotFoundError: No module named 'pydantic.fields'
thread '' panicked at runtime/src/lib.rs:499:25:
Python function threw an unexpected exception
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Reconcile Iteration 0 failed: ReconcileError { code: 500, message: "Failed to call reconcile: error while executing at wasm backtrace:\n 0: 0xc1a397 - libcomponentize_py_runtime.so!__rust_start_panic\n 1: 0xc1a208 - libcomponentize_py_runtime.so!rust_panic\n 2: 0xc1a1db - libcomponentize_py_runtime.so!std::panicking::rust_panic_with_hook::h683e04a3e610de41\n 3: 0xc1263d - libcomponentize_py_runtime.so!std::panicking::begin_panic_handler::$u7b$$u7b$closure$u7d$$u7d$::h55bf48e5a4dad3ba\n 4: 0xc12565 - libcomponentize_py_runtime.so!std::sys::backtrace::__rust_end_short_backtrace::hc61383746b72f4f5\n 5: 0xc19aa1 - libcomponentize_py_runtime.so!rust_begin_unwind\n 6: 0xc3d60f - libcomponentize_py_runtime.so!core::panicking::panic_fmt::h6c2ed8e979e576c4\n 7: 0xb93b2f - libcomponentize_py_runtime.so!componentize-py#Dispatch\n 8: 0x22d21cc - libcomponentize_py_bindings.so!reconcile" }

this is the code I used

import reconciler
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel

class Reconciler(reconciler.Reconciler):
  def reconcile(self, object: str) -> reconciler.ReconcileResult:
    # Example values for the result
    requeue = False  # Whether to requeue
    requeue_after = 30  # Requeue after 30 seconds
    #response_object = f"Processed: {object}"  # Return a string with processed information

    class User(BaseModel):
      id: int
      name: str = 'John Doe'
      signup_ts: Optional[datetime] = None
      friends: List[int] = []
    
    external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
    user = User(**external_data)
    print(user)
    
    # Return the result with appropriate values
    return reconciler.ReconcileResult(
        requeue=requeue,
        requeue_after=requeue_after,
        object=f"Processed: {object}"
    )

without the pydantic code it runs fine

import reconciler

class Reconciler(reconciler.Reconciler):
  def reconcile(self, object: str) -> reconciler.ReconcileResult:
    # Example values for the result
    requeue = False  # Whether to requeue
    requeue_after = 30  # Requeue after 30 seconds
    
    # Return the result with appropriate values
    return reconciler.ReconcileResult(
        requeue=requeue,
        requeue_after=requeue_after,
        object=f"Processed: {object}"
    )

(.venv) (base) wasm/component-reconciler - (result) > just run-guest-python
==> running python guest component...
Finished dev profile [unoptimized + debuginfo] target(s) in 0.09s
Running target/debug/reconciler
Running iteration: 1
Reconcile Iteration 0 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 0 elaspetime 1.548375ms
Running iteration: 2
Reconcile Iteration 1 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 1 elaspetime 24.042µs
Running iteration: 3
Reconcile Iteration 2 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 2 elaspetime 15.917µs
Running iteration: 4
Reconcile Iteration 3 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 3 elaspetime 19.875µs
Running iteration: 5
Reconcile Iteration 4 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 4 elaspetime 14.584µs
Running iteration: 6
Reconcile Iteration 5 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 5 elaspetime 11.875µs
Running iteration: 7
Reconcile Iteration 6 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 6 elaspetime 12.875µs
Running iteration: 8
Reconcile Iteration 7 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 7 elaspetime 18.625µs
Running iteration: 9
Reconcile Iteration 8 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 8 elaspetime 11.667µs
Running iteration: 10
Reconcile Iteration 9 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}

[henderiw]

I can also confirm your example works for me. but in my runtime it doesn't.

this is my current runtime for reference

use std::path::PathBuf;
use anyhow::{Context, Result};
use wasmtime::component::{bindgen, Component, Linker};
use wasmtime::{Engine, Store};
use wasmtime_wasi::{ResourceTable, WasiCtx, WasiView};
use std::time::Instant;


bindgen!({path: "../../../wit", world: "reconciler", async: false});

/// This state is used by the Runtime host,
/// we use it to store the WASI context (implementations of WASI)
/// and resource tables that components will use when executing
///
/// see:
/// - https://docs.rs/wasmtime-wasi/latest/wasmtime_wasi/trait.WasiView.html
/// - https://docs.rs/wasmtime-wasi/latest/wasmtime_wasi/fn.add_to_linker_sync.html
struct HostState {
    ctx: WasiCtx,
    table: ResourceTable,
}

impl HostState {
    pub fn new() -> Self {
        let ctx = WasiCtx::builder().inherit_stdio().build();
        Self {
            ctx,
            table: ResourceTable::default(),
        }
    }
}

impl ReconcilerImports for HostState {
   fn get(&mut self, name: String) -> String {
        println!("Host received name: {}", name);
        format!("Hello, {}!", name)
    }
}

impl WasiView for HostState {
    fn ctx(&mut self) -> &mut WasiCtx {
        &mut self.ctx
    }
    fn table(&mut self) -> &mut ResourceTable {
        &mut self.table
    }
}

/// load the WASM component and return the instance
fn load_reconciler_instance(
    path: PathBuf,
) -> Result<(Store<HostState>, Reconciler)> {
    // Initialize the Wasmtime engine
    let engine = Engine::default();

    // Load the WASM component
    let component = Component::from_file(&engine, &path).context("Component file not found")?;

    // Create the store to manage the state of the component
    let states: HostState = HostState::new();
    let mut store = Store::<HostState>::new(&engine, states);

    // Set up the linker for linking interfaces
    let mut linker = Linker::new(&engine);

    // Add WASI implementations to the linker for components to use
    wasmtime_wasi::add_to_linker_sync(&mut linker)?;
    // Add the `Reconciler` interface to the linker
    Reconciler::add_to_linker(&mut linker, |state| state)?;


    // Instantiate the component
    let instance = Reconciler::instantiate(&mut store, &component, &linker)
        .context("Failed to instantiate the reconciler world")?;

    Ok((store, instance))
}

// call the reconcile function
fn call_reconcile(
    store: &mut Store<HostState>,
    instance: &Reconciler,
    input_json: String,
) -> std::result::Result<ReconcileResult, ReconcileError> {
    // Call the reconcile function
    let result = instance
        .call_reconcile(store, &input_json)
        .map_err(|e| ReconcileError {
            code: 500,
            message: format!("Failed to call reconcile: {}", e),
        })??;

    Ok(result)
}


fn main() -> Result<()> {
    let wasm_path = PathBuf::from(
        std::env::var_os("GUEST_WASM_PATH")
            .context("missing/invalid path to WebAssembly module (env: GUEST_WASM_PATH)")?,
    );

    // Input JSON
    let input_json = r#"{"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}"#.to_string();

    //load the instance
    let (mut store, instance) = load_reconciler_instance(wasm_path)
        .map_err(|e| anyhow::anyhow!("Error loading reconciler instance: {}", e))?;

   // Measure time taken to run the instance 10 times
    let start = Instant::now();

    for i in 0..10 {
        println!("Running iteration: {}", i + 1);
         // Measure iteration time
        let iteration_start = Instant::now();
        match call_reconcile(&mut store, &instance, input_json.clone()) {
            Ok(result) => {
                let iteration_duration = iteration_start.elapsed();
                println!("Reconcile Iteration {} succeeded with output: {:#?}", i, result);
                println!("Reconcile Iteration {} elaspetime {:?}", i, iteration_duration);
            }
            Err(e) => {
                let iteration_duration = iteration_start.elapsed();
                eprintln!("Reconcile Iteration {} failed: {}", i, e);
                println!("Reconcile Iteration {} elaspetime {:?}", i, iteration_duration);
            }
        }
    }

    let duration = start.elapsed();
    println!("Time taken for 10 iterations: {:?}", duration);

    Ok(())
}

[dicej]

ModuleNotFoundError: No module named 'pydantic.fields'

I suspect this is related to the issue described here and here. I would recommend moving the class User(BaseModel): declaration to the top level of the file. Another option might be to add a from pydantic import fields to the top level of the file (and iterate if there are other modules which need to be explicitly pulled in).

[henderiw]

confirmed.

this works

import reconciler
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel

class User(BaseModel):
    id: int
    name: str = 'John Doe'
    signup_ts: Optional[datetime] = None
    friends: List[int] = []

class Reconciler(reconciler.Reconciler):
  def reconcile(self, object: str) -> reconciler.ReconcileResult:
    # Example values for the result
    requeue = False  # Whether to requeue
    requeue_after = 30  # Requeue after 30 seconds
    
    external_data = {'id': '123', 'signup_ts': '2017-06-01 12:22', 'friends': [1, '2', b'3']}
    user = User(**external_data)
    print(user)
    
    # Return the result with appropriate values
    return reconciler.ReconcileResult(
        requeue=requeue,
        requeue_after=requeue_after,
        object=f"Processed: {object}"
    )

Running iteration: 1
id=123 name='John Doe' signup_ts=datetime.datetime(2017, 6, 1, 12, 22) friends=[1, 2, 3]
Reconcile Iteration 0 succeeded with output: ReconcileResult {
requeue: false,
requeue-after: 30,
object: "Processed: {"apiVersion":"topo.kubenet.dev/v1alpha1","kind":"Topology","metadata":{"name":"kubenet","namespace":"default"},"spec":{"defaults":{"type":"7220ixr-d3l","provider":"srlinux.nokia.com","version":"24.7.2"},"nodes":[{"name":"node1"},{"name":"node2"}],"links":[{"endpoints":[{"node":"node1","port":1,"endpoint":1},{"node":"node2","port":1,"endpoint":1}]}]}}",
}
Reconcile Iteration 0 elaspetime 3.163375ms