loading files via go 1.16 embed directives

[yassinm]

related: #1117

I tried loading files via go embed directives. I am actually not able to make this work. Here is the playground link i was able to make. The playground is not working now because it is timing out.

What i am trying to do is load a start file from the local file system than depends on files that are shipped with the binary as embed in go 1.16. main.cue should be able to reference pipeline.Schema

[roylisto]

Have you tried this?
load.Instances([]string{}, &load.Config{Dir: "/path/to/cue/files"})

[myitcv]

@yassinm please can you provide a reproduction of what is not working?

[verdverm]

The ovpath has 2 scripts in it, pathing is off

[ymolists]

Correct me if i am wrong but i dont think the location in the file system should matter. Meaning the file is loaded into memory in a map.

Can you confirm its ending up in the correct location in the overlay ?

[ymolists]

@verdverm Mea culpa

I read your comment as two script files are in the file system.
The code in the repo was correct. What i pasted was wrong. I edited the comment above to reflect what i am actually getting.

[verdverm]

You may need to set some of the path variables in the Config object passed to load. You'll have to deal with the absolute path of the overlay and the CWD path of the first arg passed to the load function. The absolute path is outside of the working directory. You could deal with it by making everything absolute or relative to CWD. I'd go absolute so that you don't overlap with files on disk. Using a temp dir is recommended

[ymolists]

@verdverm thank you so much for all your help ! I was able to figure it based on what you mentioned.

This test code was also quite helpful. I am leaving what i did in here for posterity. I might remove that repo at some point. Basically we are tricking cue in loading a file from memory while it is mapped to a path that looks like a file from the local file system. As you can see cue.mod is also not necessary. I don't know how that works but it does work . That is for maybe another day ! Cheers !

main.go


const (
	Root = "play.ground"
)

func Run() error {
	cwd, err := os.Getwd()
	if err != nil {
		return err
	}

	overlay, err := loadFiles(scripts.StaticFs, cwd)
	if err != nil {
		return err
	}

	config := &load.Config{
		Module:  Root,
		Overlay: overlay,
	}

	insts := load.Instances([]string{"./local/main.cue"}, config)

	ctx := cuecontext.New()
	for _, instIdx := range insts {
		inst := ctx.BuildInstance(instIdx)

		val := inst.Value()
		fmt.Printf("%v\n", val)
	}

	return nil
}

func loadFiles(fsys fs.FS, rDir string) (map[string]load.Source, error) {
	overlay := make(map[string]load.Source)

	abs := func(dpath string) string {
		return path.Join(rDir, dpath)
	}

	err := fs.WalkDir(fsys, ".", func(dpath string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}

		if d == nil || d.Type().IsDir() || !d.Type().IsRegular() {
			return nil
		}

		ovpath := abs(path.Join("scripts", dpath))
		if !strings.HasSuffix(dpath, ".cue") {
			return nil
		}

		f, err := fsys.Open(dpath)
		if err != nil {
			return err
		}

		buf, err := io.ReadAll(f)
		if err != nil {
			return err
		}

		fmt.Println("dpath ", dpath)
		fmt.Println("ovpath ", ovpath)
		fmt.Println("")

		overlay[ovpath] = load.FromBytes(buf)

		return nil
	})

	if err != nil {
		return nil, err
	}

	return overlay, nil
}

-- scripts/main.cue --
package main

import (
  "play.ground/scripts/pipeline"
)

s : pipeline.#Schema & {
  Name: "jdoe"
  Age: 2000
  bebe: 11
}


-- scripts/pipeline/schema.cue --

package pipeline

#Schema: {
	Name: string
	Age:  >1000
	bebe: >1
}

[myitcv]

#607 (comment) is relevant here, specifically:

Put another way, an overlay, in its current form, exists in order to complement what is on disk, rather than wholesale replace its contents.

The comment goes on to suggest how we will like add support for loading from an io/fs.FS in the future, via a cuelang.org/go/cue/load.Config field or similar.

Until that time, you need to follow an approach that, as @verdverm points out above, runs from an empty directory to be sure that the load does not interact with files on disk. This effectively becomes equivalent to writing the your io/fs.FS to a temporary directory FWIW. As @verdverm also mentioned, a cue.mod/module.cue will be required in the FS for non-builtin imports to work.

Hence I have the following example, which happens to use an overlay (but could just as easily write the io/fs.FS files to the temporary directory and dispense with the overlay). The goal of this example is to load the configuration that corresponds to the main CUE package, i.e. scripts/main.cue, from the go:embed FS, and then print that value:

# Populate our go.sum
go mod tidy

# Verify that the cue in play.ground/scripts:main evaluates as expected
exec cue export --out cue play.ground/scripts:main
cmp stdout cue.golden

# Build our main program
go build

# Verify that our main program gives the same output
exec ./play.ground
cmp stdout go.golden

# Remove the scripts directory to prove that we are not cheating
# by loading the CUE files from there
rm scripts
exec ./play.ground
cmp stdout go.golden

-- cue.mod/module.cue --
module: "play.ground"
-- go.mod --
module play.ground

go 1.16

require cuelang.org/go v0.4.0
-- main.go --
package main

import (
	"embed"
	"fmt"
	"io"
	"io/fs"
	"log"
	"os"
	"path/filepath"

	"cuelang.org/go/cue/cuecontext"
	"cuelang.org/go/cue/load"
)

// Define our embed.FS.
//
//go:embed cue.mod/module.cue scripts/*.cue scripts/pipeline/*.cue
var staticFS embed.FS

func main() {
	// Loading staticFS via CUE we have two options:
	//
	// 1. Write staticFS to a temporary directory and
	//    call plain cue/load.Instances
	// 2. Create an overlay using the contents of staticFS
	//    and call cue/load.Instances with a config that
	//    uses the overlay and an empty directory as the
	//    working directory equivalent
	//
	// For this example we use option 2

	// Create a temporary directory so that the load of the overlay is
	// guaranteed to happen within an empty directory. An overlay complements
	// what is on disk, so we need to be sure that load doesn't pick up anything
	// other than the overlay.
	td, err := os.MkdirTemp("", "")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(td)

	// Create the overlay. Note the overlay is a mapping of absolute filepaths
	// to contents. So we need to prepend the filepath from the embed.FS with
	// our temporary directory, the directory in which we are performing the
	// load.
	overlay := make(map[string]load.Source)
	err = fs.WalkDir(staticFS, ".", func(p string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		if !d.Type().IsRegular() {
			return nil
		}
		f, err := staticFS.Open(p)
		if err != nil {
			return err
		}
		byts, err := io.ReadAll(f)
		if err != nil {
			return err
		}
		op := filepath.Join(td, p)
		overlay[op] = load.FromBytes(byts)
		return nil
	})
	if err != nil {
		log.Fatal(err)
	}

	// Load the build instances. Attempting to load play.ground/scripts:main
	// will succeed because Dir is set to the temporary directory, a directory
	// structure that is augmented by the overlay, and the overlay contains
	// cue.mod/module.cue, which defines the module path to be play.ground.
	// Notice that we need the :main package selector, because
	// play.ground/scripts does not identify the main package.
	config := &load.Config{
		Overlay: overlay,
		Dir:     td,
	}
	inst := load.Instances([]string{"play.ground/scripts:main"}, config)[0]
	if err := inst.Err; err != nil {
		log.Fatal(err)
	}
	ctx := cuecontext.New()
	v := ctx.BuildInstance(inst)
	fmt.Printf("%v\n", v)
}
-- scripts/main.cue --
package main

import (
	"play.ground/scripts/pipeline"
)

s: pipeline.#Schema & {
	Name: "jdoe"
	Age:  2000
	bebe: 11
}
-- scripts/pipeline/schema.cue --
package pipeline

#Schema: {
	Name: string
	Age:  >1000
	bebe: >1
}
-- cue.golden --
s: {
	Name: "jdoe"
	Age:  2000
	bebe: 11
}
-- go.golden --
{
	s: {
		Name: "jdoe"
		Age:  2000
		bebe: 11
	}
}

[verdverm]

There are directories and files either way, so the "FS" abstraction seems unavoidable. How are modules and packages to be organized otherwise?

Again the Golang FS package is an abstraction (set of interfaces) that is not tied to an OS filesystem. It could be disk, backed, memory based, or some sort of resolver based system that you are describing. You could easily implement a resolver based system using this interface and pass that fs.FS type to the CUE API

[myitcv]

@yassinm I'm still not entirely clear why io/fs.FS cannot be that abstraction you are talking about. It is, in effect, almost exactly the "callback" you are defining above.

type FS interface {
	// Open opens the named file.
	//
	// When Open returns an error, it should be of type *PathError
	// with the Op field set to "open", the Path field set to name,
	// and the Err field describing the problem.
	//
	// Open should reject attempts to open names that do not satisfy
	// ValidPath(name), returning a *PathError with Err set to
	// ErrInvalid or ErrNotExist.
	Open(name string) (File, error)
}

Right now the caller cannot decide the order in which file system resolution happens

Is there a reason this is significant?

I am asking if it should be via that interface or a resolver interface ?

I think we would be willing to consider an alternative if there are problems with io/fs.FS, or clear benefits of the alternative. io/fs.FS is just an interface. Whether it maps to a real file system, something in memory, a hybrid, something HTTP-based... is entirely undefined and left up to implementations.

The one clear benefit of using io/fs.FS to my mind is that it limits the API surface of the CUE Go API, by reusing a well defined, well known concept from the Go standard library. The cognitive load is therefore lower etc.

[yassinm]

@myitcv that makes a lot of sense thank you :-)

I think the only reservation i have left is about the order of the resolution and restricting one FS to a prefix path. In my use case i need to make sure a particular path like "play.ground/scripts/*" is only resolved by my fs implementation and not loaded from the disk. I can move that discussion to where the FS interface implementation is happening which makes more sense.

[myitcv]

In my use case i need to make sure a particular path like "play.ground/scripts/*" is only resolved by my fs implementation and not loaded from the disk.

And that's entirely doable with io/fs.FS. Because you pass the io/fs.FS value to cue/load.Config and hence you control the implementation.

[myitcv]

For anyone following along here, we have just created a proposal for io/fs.FS support within cue/load: #4285