Validate target environments #2806

itowlson · 2024-09-06T00:05:25Z

~~EXTREMELY WIP.~~

There are plenty of outstanding questions here (e.g. ~~environment definition,~~ Redis trigger, ~~hybrid componentisation~~) ~~, and reintegration work (e.g. composing dependencies before validating)~~ (ETA: component dependencies now work). ~~In its current state, the PR is more for visibility than out of any sense of readiness. Also, it has not had any kind of tidying pass, so a lot of naming is not great.~~

But it does work, sorta kinda, some of the time. So we progress.

cc @tschneidereit

[edit(lann): Retroactive SIP here: #3217]

itowlson · 2024-09-17T03:16:04Z

Checking environments adds an appreciable bump to the time to run spin build. I looked into caching the environments but a robust way didn't really help - the bulk of the time, in my unscientific test, was in the initial registry request, which retrieval of the Wasm bytes (for the Spin environment) about a quarter of the time. (In my debug build, from a land far away from Fermyon Cloud and GHCR where my test registry is hosted: approx 2500ms to get the digest, then 600ms to get the bytes. The cache eliminated only the 600ms.)

We could, of course, assume that environments are immutable, and key the cache by package reference instead of digest. But that would certainly be an assumption and not guaranteed to be true.

itowlson · 2024-09-18T23:56:04Z

This is now sort of a thing and can possibly be looked at.

Outstanding questions:

Where shall we publish the initial set of environments?
- The wkg configuration will need to reflect this. At the moment it uses the user's default registry. Yeah nah.
Where shall we maintain the environment WITs?
- Separate repo in the Fermyon org?
What does that initial set contain?
- Currently I've created a Spin CLI "2.5" world with just the HTTP trigger (WASI 0.2 and WASI RC).
Testing

Possibly longer term questions:

How can we manage environments where the set of triggers is variable - specifically the CLI with trigger plugins?
How to avoid a lengthy network round-trip on every build
Better error reporting for environments where a trigger supports multiple worlds (like, y'know, the Spin CLI).

If folks want to play with this, add the following to your favourite spin.toml:

[application]
targets = ["spin:[email protected]"]

and set your wkg config (~/.config/wasm-pkg/config.toml) to:

default_registry = "registrytest-vfztdiyy.fermyon.app"

[registry."registrytest-vfztdiyy.fermyon.app"]
type = "oci"

(No, that is not the cat walking across the keyboard... this is my test registry which backs onto my ghcr.)

lann · 2024-09-19T13:15:52Z

Where shall we publish the initial set of environments?

fermyon.com?

Where shall we maintain the environment WITs?

I'd suggest "next to the code that implements them"; ideally generated by that code.

crates/build/src/manifest.rs

rylev · 2024-09-19T14:37:33Z

crates/build/src/manifest.rs

+            let dt = deployment_targets_from_manifest(&manifest);
+            Ok((bc, dt, Ok(manifest)))
+        }
+        Err(e) => {


It's not obvious to me what's happening here. It reads like we're trying to get build and deployment configs even when we can't parse the manifest? I think some terse one-line comments on each branch of this match would go a long way to helping this be a bit more understandable.

Reading the code more, I'm unsure why we go through all these great lengths to read the targets config here when later on we only seem to run the targets check if the manifest was successfully parsed. Won't this information just be thrown away?

@rylev You are absolutely right - this was a holdover from an earlier iteration before I realised I was going to need to full manifest - thanks for catching it. I've pared this back to "has deployment targets", which I think is worth keeping so we can warn if the manifest errors have caused us to bypass checking.

rylev · 2024-09-19T14:39:39Z

crates/build/src/manifest.rs

+            Ok((bc, dt, Ok(manifest)))
+        }
+        Err(e) => {
+            let bc = fallback_load_build_configs(&manifest_file).await?;


It seems like there's some need for better error messages here through liberal use of .context. As the code stands now, component_build_configs function might return an error saying only "expected table found some other type" which would be very confusing.

On reflection these should preserve (and immediately return) the original manifest load error rather than blatting it with whatever went awry during the fallback attempt.

rylev · 2024-09-19T14:40:20Z

crates/build/src/manifest.rs

+            let table: toml::value::Table = toml::from_str(&manifest_text)?;
+            let target_environments = table
+                .get("application")
+                .and_then(|a| a.as_table())
+                .and_then(|t| t.get("targets"))
+                .and_then(|arr| arr.as_array())
+                .map(|v| v.as_slice())
+                .unwrap_or_default()
+                .iter()
+                .filter_map(|t| t.as_str())
+                .map(|s| s.to_owned())
+                .collect();


Would serializing to a type through serde::Deserialize making this easier to read?

rylev · 2024-09-19T14:42:00Z

crates/build/src/manifest.rs

@@ -57,6 +75,30 @@ async fn fallback_load_build_configs(
    })
 }

+async fn fallback_load_deployment_targets(


Nit: is "deployment targets" the right nomenclature? That sounds like what you would be targeting for spin deploy and not the environment you're targeting. Perhaps we could play with the wording here?

@rylev I'm very open to naming on "deployment targets." The current name emerged in a woolly manner from a sense of "these are possible targets for deployment" or "the target environments we want to be able to deploy to." I do feel, albeit not strongly, that it's worth specifying 'deployment' (cf. e.g. 'compilation target') - but for sure let's improve this!

"runtime targets"?

crates/build/src/manifest.rs

rylev · 2024-09-19T15:05:50Z

crates/compose/src/lib.rs

+    async fn load_component_source(&self, source: &Self::Component) -> anyhow::Result<Vec<u8>>;
+    async fn load_dependency_source(&self, source: &Self::Dependency) -> anyhow::Result<Vec<u8>>;


Are we implementing ComponentSourceLoader more than once? I'm a bit lost why we need the flexibility of defining the type of dependency and component instead of hard coding. Can you explain what this buys us?

The current build system doesn't create a lockfile. The current composition implementation depends on the locked types (because it runs at trigger time). This allows us to implement composition on the raw AppManifest as well as on the LockedApp.

rylev · 2024-09-19T15:10:22Z

crates/environments/src/lib.rs

+    "#
+    );
+
+    let doc = wac_parser::Document::parse(&wac_text)


We really should implement a programatic API in wac for this so that we don't need to manipulate a wac script.

Yeah! I tried constructing a wac document AST, which seemed cleaner than parsing a document, but I foundered on - if memory serves - something as trivial as the miette spans. I tried it with dummy spans but something internal seems to have been trying to use the length of the name in error or something and anyway there was anguish. Using the underlying model would have been awesome, but was (for me) hard to understand and to line up inputs for.

crates/environments/src/lib.rs

rylev · 2024-09-19T15:15:22Z

crates/environments/src/loader.rs

+}
+
+pub async fn load_and_resolve_all<'a>(
+    app: &'a spin_manifest::schema::v2::AppManifest,


Nit: seems like we pass app and resolution_context around a lot. It might be nicer to put those into a Resolver struct and implement these functions as methods on that struct.

I had an explore of this, and have pushed something like it as a separate commit for easy reversion. I ended up naming the proposed struct ApplicationToValidate because otherwise I ended up with no "application" visible in a module that was ostensibly all about validating applications. I still have mixed feelings but there's definitely some nice aspects to encapsulating the component loading stuff - I was able to tuck some more clutter away and split things up in a way that's hopefully more readable...

tschneidereit · 2024-09-20T17:15:11Z

We could, of course, assume that environments are immutable, and key the cache by package reference instead of digest. But that would certainly be an assumption and not guaranteed to be true.

@itowlson I think that's an okay assumption to make, personally. Perhaps paired with a way to flush the local cache explicitly?

tschneidereit · 2024-09-20T17:18:49Z

We could also consider introducing something like target-environments.lock which would be stored next to Spin.toml, and which would contain the environments. Then that explicit way to flush the cache would become "remove the file".

itowlson · 2024-09-24T03:47:28Z

Okay I have pencilled in fermyon.com as the default registry for environment definitions (with !NEW! the option to override it or use a local path !NEW!). One thing that occurred to me though was that, at the moment, an environment reference is a package name. Are we okay with environments occupying the same namespace as worlds and components? Or should we adopt an implicit prefix e.g. instead of spin:[email protected] have users write [email protected] and translate this to e.g. environment:[email protected]? I'm not sure I have an opinion, but hopefully someone else does...

tschneidereit · 2024-09-24T09:49:06Z

One thing that occurred to me though was that, at the moment, an environment reference is a package name.

Aren't environments also just packages though? Ones that we apply some additional semantics to, yes, but not ones that are someone non-standard.

Which actually raises the question: could we use the exact packages to generate bindings for Spin and the SDKs as well? (Not necessarily as part of this effort, but as a future thing.)

itowlson · 2024-10-04T01:21:19Z

Implemented caching on the basis of immutability as per Till's comment #2806 (comment) (and double checked on Windows).

I'll have a think about the lockfile idea. A lockfile mapping environment IDs to digests would allow us to reuse the existing Wasm by-digest caching infra instead of having something custom for name-based lookup. If we generated it into .spin then it would get picked up by .gitignore which is probably desirable, although it would be less discoverable for deletion purposes (.spin already contains a bunch of other stuff of varying deletability).

ETA: I implemented Till's lockfile-based cache strategy. It does make some things cleaner for sure. I've done it as a separate commit for now, but will squash once we agree on it.

itowlson · 2025-05-05T23:35:10Z

I did some unscientific performance testing with this. Validation for local or HTTP components is quick once registry targets are cached. If they have local dependencies, then composition before validation is also quick. However, if a component has a registry dependency, then that seems to be not cacheable, and that does introduce a perceptible delay. I recall in the previous work on this I tested whether I could speed it up by checking the registry digest and re-downloading only if changed, but it turned out that even just checking fetching the digest was slow. So I am not sure of the way round this.

(ETA: for clarity: the initial download of a registry env is a perceptible delay; and we do not currently check for updates (because of the perf hit). You need to delete the lockfile to force re-download. So environments once published must be stable.)

itowlson · 2025-05-06T00:21:32Z

@tschneidereit

Aren't environments also just packages though? Ones that we apply some additional semantics to, yes, but not ones that are someone non-standard.

Yes and no. I found that the canonical spin:up package from the Spin WIT folder didn't accept the fileserver component (at least the version I was testing), because the fileserver exported an RC version of wasi:http/incoming-handler - which Spin supports, but which is not listed in the top-level Spin world. Maybe that's an error in the spin:up package in that it fails to represent all worlds accepted by spin up? But it's not necessarily one we want to remedy in Spin "canon," because the RC world is for back compat with Spin 2, not one we want to view as a primary entry point going forwards.

We may also find it tricky to manage trigger plugins if we try to unify. E.g. would we want the world in the Spin repo to list the cron trigger export? But the target environments thing is going to have to allow for it I expect.

Signed-off-by: itowlson <[email protected]>

This SIP describes and motivates the functionality introduced in spinframework#2806. Signed-off-by: Till Schneidereit <[email protected]>

tschneidereit · 2025-08-07T10:02:54Z

@itowlson I put together a SIP describing this functionality, and hope that it reflects the reality of the implementation reasonably well. It certainly is at least pretty much what I had in mind with this feature.

tschneidereit · 2025-08-07T10:05:20Z

As an implementation this looks great to me! Things I noticed would be good to expand:

Include tests with actual target world definitions in the spin.toml
Change templates to start including targets, so that from hereon out people don't accidentally start targeting new environments when updating the Spin CLI tool
Documentation

itowlson · 2025-08-08T01:48:19Z

@tschneidereit

Change templates to start including targets

There are two challenges here:

We first of all need to build some definitions and put them in a registry. I've not included that as part of this PR (because it involves deployment to registries) but it will be needed for templates.
It's not clear what targets we should put in the templates. I mean, okay, we could put the current version of the command line, but:
a. That's not particularly useful - the value comes from validating against your deployment environment(s)
b. That'll be annoying when they upgrade Spin to use a new API and get told "nope that doesn't exist in your declared target"

So I'd like to exclude that from this PR.

Documentation

Is this Spin project docs, or user docs? User docs go in a separate repo and we usually write them once a feature is in main and a bit more stable than a PR. If project docs that should be part of this PR, can you elaborate on what you're looking for?

itowlson · 2025-08-08T01:53:02Z

Checking over the code:

const DEFAULT_ENV_DEF_REGISTRY_PREFIX: &str = "ghcr.io/spinframework/environments";
const DEFAULT_PACKAGE_REGISTRY: &str = "spinframework.dev";

So we would need to:

Set up a wkg registry reference on spinframework.dev and populate its backing OCI registry with WITs
Upload TOML files to GHCR

(We probably also need to confirm that everyone is happy with these proposed default locations.)

tschneidereit · 2025-08-08T12:49:24Z

Thank you for the explanations regarding templates and all that. I suppose we could in principle land this without any of those changes, but I'm not sure how valuable it'd be.

And thinking about it a bit more, I don't think we should land it for now: before doing so, we should have had the opportunity to at least try it out ourselves and see if and how it works.

What we could at the very least already do though is to add end-to-end tests based on local files, right?

lann · 2025-08-08T14:28:17Z

const DEFAULT_ENV_DEF_REGISTRY_PREFIX: &str = "ghcr.io/spinframework/environments";
const DEFAULT_PACKAGE_REGISTRY: &str = "spinframework.dev";
(We probably also need to confirm that everyone is happy with these proposed default locations.)

With the registry config in place for spinframework.dev we could at least look up the OCI prefix (ghcr.io/spinframework/) via RegistryMetadata::fetch.

Signed-off-by: itowlson <[email protected]>

itowlson · 2025-08-12T02:26:05Z

Thank you for the explanations regarding templates and all that. I suppose we could in principle land this without any of those changes, but I'm not sure how valuable it'd be.

Again, the trouble I'm finding with putting it in templates is we don't know what targets to put in. We could prompt, but then we risk facing new users with the dreaded Question You Cannot Answer. Could you perhaps propose more precisely what you feel we should add to the templates? You mention "so that from hereon out people don't accidentally start targeting new environments when updating the Spin CLI tool" but I'm not sure what you're envisaging here. Thanks!

itowlson · 2025-08-13T00:17:22Z

The wasm32-wasip2 / WASI 0.2.3 debacle has thrown up another obstacle:

Spin CLI accepts components that target WASI 0.2.3
WAC does not accept such components as compatible with the Spin world
So target environment validation against the Spin WITs ends up rejecting a component which would run

This seems like it needs a fix upstream in WAC - I vaguely recall @fibonacci1729 looking into this, and I'm not sure if anything came of it, but it's not in WAC 0.7 unfortunately.

(Yes, we might be able to patch this by revving Spin to WASI 0.2.3 but that just kicks the can down the road for when someone comes up with WASI 0.2.4!)

fibonacci1729 · 2025-08-13T16:26:04Z

@itowlson i'm going to look into fixing this in wac over the next few days. I'll report back!

itowlson added the do-not-merge label Sep 6, 2024

itowlson force-pushed the validate-target-environment branch 6 times, most recently from 30aa1fc to 3b25dad Compare September 9, 2024 22:10

itowlson force-pushed the validate-target-environment branch 2 times, most recently from e74895b to 828cb70 Compare September 17, 2024 00:57

itowlson marked this pull request as ready for review September 18, 2024 23:56

rylev reviewed Sep 19, 2024

View reviewed changes

itowlson force-pushed the validate-target-environment branch from 4870bba to 531e4fb Compare September 24, 2024 00:27

itowlson force-pushed the validate-target-environment branch from 51a09f4 to 76cb8af Compare September 24, 2024 21:08

itowlson force-pushed the validate-target-environment branch 7 times, most recently from ae77cd9 to 86f61b8 Compare May 5, 2025 04:01

itowlson force-pushed the validate-target-environment branch from 5c099d2 to 4fb5c7a Compare May 12, 2025 21:54

itowlson marked this pull request as ready for review May 13, 2025 22:04

itowlson removed the do-not-merge label May 13, 2025

itowlson force-pushed the validate-target-environment branch 3 times, most recently from 89663c8 to e14b9d0 Compare May 14, 2025 01:39

itowlson force-pushed the validate-target-environment branch from e14b9d0 to e4438e9 Compare June 23, 2025 01:52

itowlson force-pushed the validate-target-environment branch from 49444d3 to 45ec77e Compare July 14, 2025 22:38

itowlson requested a review from tschneidereit July 14, 2025 22:38

itowlson force-pushed the validate-target-environment branch from 45ec77e to e10150a Compare July 14, 2025 23:36

itowlson added 2 commits July 31, 2025 10:40

Validate against target environments during build

69e9f39

Signed-off-by: itowlson <[email protected]>

Validate host requirements as part of target environments

8847533

Signed-off-by: itowlson <[email protected]>

itowlson force-pushed the validate-target-environment branch from e10150a to 8847533 Compare July 31, 2025 00:47

fibonacci1729 added the spin-3.4 Label to track items to land in the release of Spin v3.4 label Aug 5, 2025

macolso added this to Spin Triage Aug 5, 2025

macolso moved this to In Progress in Spin Triage Aug 5, 2025

tschneidereit added a commit to tschneidereit/spin that referenced this pull request Aug 7, 2025

Add SIP for target world validation

ad2a273

This SIP describes and motivates the functionality introduced in spinframework#2806. Signed-off-by: Till Schneidereit <[email protected]>

tschneidereit mentioned this pull request Aug 7, 2025

Add SIP for target world validation #3217

Open

tschneidereit added a commit to tschneidereit/spin that referenced this pull request Aug 7, 2025

Add SIP for target world validation

be389fb

This SIP describes and motivates the functionality introduced in spinframework#2806. Signed-off-by: Till Schneidereit <[email protected]>

Tests and contexting some file-not-found errors

acb84b5

Signed-off-by: itowlson <[email protected]>

itowlson force-pushed the validate-target-environment branch from b77d25d to acb84b5 Compare August 12, 2025 02:19

		async fn load_component_source(&self, source: &Self::Component) -> anyhow::Result<Vec<u8>>;
		async fn load_dependency_source(&self, source: &Self::Dependency) -> anyhow::Result<Vec<u8>>;

Validate target environments #2806

Are you sure you want to change the base?

Validate target environments #2806

Conversation

itowlson commented Sep 6, 2024 • edited by lann Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itowlson commented Sep 17, 2024

Uh oh!

itowlson commented Sep 18, 2024

Uh oh!

lann commented Sep 19, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tschneidereit commented Sep 20, 2024

Uh oh!

tschneidereit commented Sep 20, 2024

Uh oh!

itowlson commented Sep 24, 2024

Uh oh!

tschneidereit commented Sep 24, 2024

Uh oh!

itowlson commented Oct 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itowlson commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itowlson commented May 6, 2025

Uh oh!

tschneidereit commented Aug 7, 2025

Uh oh!

tschneidereit commented Aug 7, 2025

Uh oh!

itowlson commented Aug 8, 2025

Uh oh!

itowlson commented Aug 8, 2025

Uh oh!

tschneidereit commented Aug 8, 2025

Uh oh!

lann commented Aug 8, 2025

Uh oh!

itowlson commented Aug 12, 2025

Uh oh!

itowlson commented Aug 13, 2025

Uh oh!

fibonacci1729 commented Aug 13, 2025

Uh oh!

itowlson commented Sep 6, 2024 •

edited by lann

Loading

itowlson commented Oct 4, 2024 •

edited

Loading

itowlson commented May 5, 2025 •

edited

Loading