RFC: Language and IDE ideas

[Irev-Dev]

From the inception of this language, when I initially presented a basic demo, a significant portion of the demo was dedicated to justifying our decision to develop our own domain-specific language (DSL). While I won't delve into those details at this moment, the extended explanation was because language development shouldn't be taken lightly. But since we're commit to this endeavor, the language should fulfill the following objectives:

1. Kick butt
2. Establish distinct boundaries to aid us in trimming its scope
3. Cater well to our users, that is folks who might not have programmed before

Here are some ideas for your consideration.

1. App will have a default directory, When users create a new project it will add a new folder to this directory with its name and create the structure of the project (project.toml with name = <name>, some folders including a components folder and components/main.pf)
2. A project will just be a directory, but the project.toml will be what defines a project.
3. The app will search the default directory for all project.tomls and grab the project's name and display a list of those projects in the app's home screen for the user to select from
4. Of course users can still open a file browser to select a project/repo that's not in their default directory.
5. project.toml will function much the same as a package.json or cargo.toml, listing dependencies, and any other metadata about the project.
6. Lang will change from having the show function to exports/public entities. I think this is a better pattern for defining what 3d artifacts are displayed in the scene, and separately supports building utils/modules in separate files.
7. In general, we'll aim for, convention over configuration. Want to make all projects feel familiar as will help foster an ecosystem.
8. The lang will support special comments for documentation, both of Rusts ///for documenting specific things and//!` should be borrowed.
9. We can also borrow Go's convention of function names prefixed with example to become examples for the suffix (exampleSomeFn is an example for someFn)
10. It would be good for our documentation to have syntax for putting screenshots into the docs (possibly with annotations) This is a departure from other languages' docs system, but reflects the visual nature of the lang
11. Reserved folder names, so far I can think of three, examples, components and lib

• examples: (self-explanatory, become examples in the docs)
• components: each file in the folder is expected to export 3d entities (not functions, data or other). When a user selects one of these files they will execute and display in the 3d-scene. components is probably not the best name, the name needs to be fairly generic as needs to cover piece-parts and assemblies. any suggestions?. options: Elements, Structures, Units, Constructs, Modules
• lib: just a place for general utils and abstractions that might be used by multiple components. When a user selects one of these, the editor won't change.

12. project.toml will be treated as the central file that defines the project. When the user selects it the right-hand pane (usually the 3d-scene) will instead render the documentation for the project.
13. parts/main.pf will be a special name that we'd expect to normally be the assembly for an entire project. //! comments at the start of this file form the start of the documentation when the user selects the project.toml.
14. In the short term project.toml can list dependencies as file paths, URLs and git-urls. While a centralised package repository would be nice at some point, allowing the above gives a lot of bang for buck when it comes to having some dependencies support and suits our "local first" approach.
15. Opinionated formatting: this is already the case since it had to be built for the recasting ability. At some point, we might want to allow some preferences here, but for now, keeping it strict is less of a maintenance burden.
16. Dev tools: Big big topic especially since the code-gen that makes up the core of this app is arguably a dev tool. I'll link to Bret Victor's Learnable Programming that @franknoirot found as there's lots there to learn from.
17. Walk through onboarding, Getting set up with our mouse preferences etc, as well as walking through how to make their first part.

Needs more consideration

• Errors, how we expose them to user's and how we can best help users resolve them.

Further in the future

• built-in testing framework
• types
• Plugin system, for example, I don't particularly want to build git support, but maybe someone else can.

not doing

• no async, or concurrency support
• no running processes. i.e. no puffin servers.
• I don't think either of the above belongs as capability in the lang, we should stay focused on the use case of script executes to output an artifact.
• No HTTP, I can see us budging on this, but to begin within at least we shouldn't support it. If data needs to be fetched, it should be done beforehand by some other lang/process and left in a csv/json or something to be imported by our lang.
• Exploding scope by supporting other langugaes: Many of the limitations we decide to put on the language will have an easy workaround if executing scripts from other langs is supported.

[Irev-Dev]

@adamchalmers @iterion @alteous @franknoirot @greg-kcio @jgomez720 @jessfraz @mlfarrell @gserena01 Calling the brain trust 🗣️.

Any other good ideas?
Any points above you disagree with? (I added numbers to the main points to make back and forth easier).

[Irev-Dev]

for 11. The naming issue of "components", could be "pufflings" but probably a bit too cutesy.

[franknoirot]

These are all naming convention-related thoughts first.

for 11. The naming issue of "components", could be "pufflings" but probably a bit too cutesy.

Aw "pufflings" is such a cute baby animal name, I didn't know that's what they were called. I love the metaphors around wax and puffins, but I would recommend we keep them to branded aspects and out of the DSL if possible. I think "components" or "parts" will be easier for mechie types to adopt.

12. wax.toml will be treated as the central file that defines the project.

I also think main.toml or project.toml will be more immediately grokkable, as much as I like wax.toml.

Random thought on file format name

How do you feel about .pf, standing for Paraffin File? that way we still lightly evoke puffins (I'll still say "puff file" in my head) but don't have to have the whole word? Thinking .rs for Rust. Unless it would be confusing for users with a lot of .pdfs.

[Irev-Dev]

I'll stick with components, parts get confusing because they can be assemblies and parts sounds like it's a piece-part.

The rest sounds good.

[alteous]

Is it necessary to separate project configuration from the source code? Perhaps the configuration could be part of the source code, for example:

package(name = "app", version = "0.1.0", description = "A Tauri app", authors = ["you"])

How do you feel about .pf, standing for Paraffin File?

I like the terseness of the .pf extension but I'm concerned it may get confused with other formats in a web search. The same can be said for .puf. .puff seems OK is this regard.

I love the metaphors around wax and puffins, but I would recommend we keep them to branded aspects and out of the DSL if possible. I think "components" or "parts" will be easier for mechie types to adopt.

I agree. As a layperson, the association between wax and paraffin was not intuitive.

---

I would like to understand better the relationship between this file format Kurt is developing here and the one I am developing. For context, the format I am developing is based on glTF, notably adding boundary representations.

I'm thinking of the relationship between .puff and .gltf files as analogous to that of .c (C source code) and .o (compiled object) files, where .puff is the source code and .gltf is the compiled output. With this in mind, perhaps we could have a 'compiled output' view like godbolt.org does. @Irev-Dev, are we on the same page?

[Irev-Dev]

Is it necessary to separate project configuration from the source code

I think so, having metadata in the source code seems like a lot of clutter, especially since I assume you would need to list it in every file? I can also easily see us coming up with more things to put in the toml files in future, which potentially means more and more clutter.

I would like to understand better the relationship between this file format Kurt is developing here and the one I am developing

Compiled vs non-compiled is definitely a legitimate way of conceptualizing the two, though I'm not sure if that will be a mental model that will be useful to our users. The app will also support exporting other files too (mesh, step and maybe even file formats like solidworks 😉). The compiled output like godbolt speaks to the nerd in me, though I'm I think this might fall into the camp of devtools for those looking to contribute to the lang itself, and not something the average ME would be interested in, I might be wrong though.

[Irev-Dev]

I'm also not worried about the names of things as that's also a branding question that hasn't been settled yet anyway.

[franknoirot]

6. Lang will change from having the show function to exports/public entities. I think this is a better pattern for defining what 3d artifacts are displayed in the scene, and separately supports building utils/modules in separate files

Am I understanding the role of exports fully?

1. Export geometry piece-parts or assemblies ("components")
2. Export callable functions ("utils/modules")
3. Export constants (not sure about this one?)

Then I'll have one question on this point:

14. In the short term project.toml can list dependencies as file paths, URLs and git-urls. (Emphasis mine)

Are all exported elements within the same directory implicitly available for import by files within the same project directory, like Yarn/NPM projects? And then this "file paths" dependency type is for using modules on your local machine but not in the current project?

[adamchalmers]

Replying to Kurt

3, 4, 7, 8, 10, 13, 15: Love this.

1. This is really niche but please make the default directory configurable via some env var. My main Linux partition is tiny and I configure almost everything to run on my secondary SSD :)

2. show and public/private seem kinda orthogonal. public/private is meant to hide implementation details from consumers of libraries, but show is controlling what actually gets rendered in the UI. Should we have both?

3. Beyond just still screenshots, let's support live 3D renders, so the user can move it around with their mouse. Or animations, like the cube slowly spinning. Of course, screenshots first, and the other stuff goes on the roadmap :)

4. Instead of the compiler treating certain function names specially (e.g. exampleSomeFn() I prefer function annotations, like

#[example]
fn someFn()

Why? Because the former is not composable. What happens if we want to configure how the example is run? For example, say we add serial examples (that cannot be run simultaneously) or remote examples (that are run on a different server). Are we going to support exampleSerialSomeFn() and exampleRemoteSomeFn()? This gets hairy fast. Or what if we decide to use public/private signifiers? We can't use capitalization of function names now, because function names are already special for examples. So you can't specify a pub or private example.

Annotations are inherently composable, because you can add more annotations, or more details to existing annotations. Here's how I'd represent those concerns above:

#[example]
fn someFn()

#[example(serial)]
fn someFn()

#[example(serial, remote)]
fn someFn()

#[example]
pub fn someFn()

I think this is simpler and cleaner than relying on special function names.

11. Reserved folders sounds fine, but as an alternative, maybe they should be default folders, with configurable overrides in the project.toml? That seems a bit more extensible. I don't really mind one way or the other.

12. Two thoughts.

A1: URLs as imports can be a bit dangerous because URL contents can change at any time, causing build errors even when you haven't updated anything. There's no version or content hash built into URLs.

A2: Do we want a project.lock like how Cargo, Go and some Python projects work? That way if users specify a vague dependency (like a git branch) we can resolve to a specific dependency (like a git revision/commit/hash) and make sure we consistently keep using it, even if a new commit is added to the branch.

Not doing

• Async: Agree that async won't be something users can choose, or even really think about. But that's because the whole app is inherently async (it's being rendered remotely).
• HTTP: If we aren't doing HTTP, how does untitled-app map git/URL dependencies to local files?

Other

A3: Will users be able to use their own text editor?

On one hand, it's a bad idea, because you're building deep integration between the 3D view and the language code view (e.g. clicking a line in the 3D editor will highlight its code in the source editor).

But on the other hand, my local editors are very powerful -- they have all kinds of features like regex find/replace, vim navigation, being able to manage my git diff etc. And it'll take a long time before we can build those features into untitled app.

Maybe we could support this with an Untitled App "local files" mode, where it doesn't spawn a two-paned window. Instead, it spawns a window with only the 3D view, and watches a local directory for changes. This way, users can open their preferred code editor on the left, and the 3D view in a new window on the right.

But it's OK if you don't want to support this, for the above reasons.

A4: In addition to embedding screenshots into docstrings (point 10), what if users could embed normal 2D image files (e.g. a JPG of the existing, pre-kittycad engineering diagrams) too? The project structure could have a /media directory, and you could insert a reference to an image under /media by using normal markdown syntax, like ![Engineering diagram for the door](/media/door.png) I think this would help people link their new KittyCAD files to their existing legacy engineering documentation.

[adamchalmers]

Is it necessary to separate project configuration from the source code

I think so, having metadata in the source code seems like a lot of clutter, especially since I assume you would need to list it in every file? I can also easily see us coming up with more things to put in the toml files in future, which potentially means more and more clutter.

Note: Rust is currently experimenting with exactly this idea -- put the Cargo.toml metadata into a special comment block at the top of your source code file. See its RFC. This is a pretty convenient idea, because it helps beginners get onboarded quickly. No need to navigate a file system, and you can easily copy/paste your source code into a DM or a Discord room or whatever to get help with it. I actually think it might be a really good idea for our target audience, who aren't as technical as the average Rust user.

[Irev-Dev]

@franknoirot

Am I understanding the role of exports fully?

What I was thinking is you can export values, any value you like, that is any variable you like. So whatever is in that variable gets exported, it could be a number, it could be a piece of geometry (component), or it could be a function. Does that sound okay to you? It's just like a normal language, you can export whatever you want from a module, I wouldn't get tied up on components somehow being special, as far as the language is concerned, it's just an id that points to something the engine is taking care of.

Are all exported elements within the same directory implicitly available . . .

Yeah sounds like you're one the same page as what I was thinking. Just some basic ways for users to manage dependencies without a full-blown package manager. URLs are going to be more useful IMO, but I think file-paths should be included. But yes you should also be able to from './rel/path/in/same/project' import rockerGasket.

@adamchalmers

1. Might as well, for app settings we probably want put it into JSON (or similar) file like VS-Code, Edit through UI to the JSON directly. Maybe it would be worth making the default directory value an array of strings? if you want projects scattered in multiple places, maybe that's silly actually.

2. I had a few attempts at writing something here, I think I'd prefer to chat about it, I think it would help clear my thinking on it.

3. That all sounds really dope, though thinking through how this would get implemented from a publishing point of view would give me more confidence here, nothing solid here just a few scenarios and questions.

• Someone clones a colleague's repo, this person has untitled-app all setup with an api-key etc, when they look at the docs for this repo within untitled-app it renders all the 3d goodness using his personal key. It costs him a little but probably not much is that okay? Or Do we pretender the images and store them in the repo somehow, do 3d objects get exported to a format so they can be shown locally without spinning up KittyCAD GPUs
• The guy who created the repo has put it on github, and wants a good read me there, does he create it themselves, or does KittyCAD have a handy dandy GH action for rendering //! comments to markdown for github, in which case what do we do images or worse 3d things (I don't think actions can upload to GH media upload, might be wrong)
• KittyCAD by this point has built untitled-lang-components.io where users can publish projects, assuming we don't require people to sign in to browse projects how do we show them 3d renders? the cheap answer is pre-rendered meshes shown with three.js, but maybe some features are lost doing that, do we just wear that GPU cost?
  I don't have answers here, but yeah we should figure out a way to make 3d publishing work because that would be so cool. Maybe gracefully fallbacks for those not signed into untitled-lang-components.io etc would be a decent compromise.

4. You make a compelling case for annotations. Does allowing annotations have more implications, would annotations only ever be the ones we baked into the language or would we be forced into making part of the language? They're connected to macros in Rust right?

5. That sounds pretty reasonable, but at the same time on the principle of "erring on strict", gives us room to ease up if people are asking for a feature like that (harder to go the other way).

A1 Yeah URLs have problems, but it's kinda a "in lieu of a package manager". I would much rather have URLs than nothing, and if you pick your URL well they shouldn't change.
A2 Maybe 🤷 , I was thinking this made more sense once a proper package manager had been built. To be clear is what your suggesting is that if someone puts https://github.com/some-org/some-repo as a dependency, we'd add https://github.com/some-org/some-repo/tree/<commit-hash> to the lock file for them? That's a pretty neat idea. My only other thought is maybe it's better to have something that's dead simple over something in between a package manager and not. That would require us to have special logic around github URLs, and similar for gitlab, bitbucket, when do we stop? Vs just telling people to use a specific to commit git URL if they want to be sure it doesn't change.

Not doing HTTP, sorry I guess I mean I don't think putting a fetch type of function where you can get random data from the internet during execute is a good idea. Dependencies are a little different.

A3 I don't think we should encourage or discourage using other editors. The UI direct-manipulation <-> code-gen interope is the core innovation of untitled app, and that's going to be hard to bring to existing editors, so I think untitled-lang will be greatly diminished without untitled-app. Having said that if we allow hiding the editor in untitled-app, with your side by side suggestion, that will still allow of the codegen, what they will loose doing that is the ability to right click on a feature in 3d and select, "go to definition". Oh just had and idea, we could have "copy relative definition path" and get something like like "./path/to/file:34", to easily get to that spot again in another editor.

A4 hmm, yeah I like this, I think that might work well for "graceful fallbacks" I mentioned earlier, maybe whenever folks put screenshots in their docs than we generate an image in media so that it is easier to publish elsewhere (github etc) and if they want to embed something 3d in the docs, we also screenshot the first frame and put it in media as the fallback?

[adamchalmers]

3. Maybe we export the 3D images at creation time into gltf, which David says is like png for 3d graphics, and view them in whatever browser 3D rendering tech exists.

4. Firstly, I incorrectly called #[example] an annotation but the actual Rust term is an attribute. Basically they're just metadata describing some item -- a function, type, field, etc. Sometimes this metadata is used by a macro to customize its output. Other times, the metadata is used directly by the compiler itself, e.g. for conditional compilation on different platforms. For now, we could support only the attributes we (KittyCAD) want to program in. I don't think we'll have macros in Untitled App. For more, read https://doc.rust-lang.org/reference/attributes.html

5 (which I think you meant as 11): Yeah, agree, let's use reserved dirs for now.

A3: Agree

A4: Agree

[Irev-Dev]

3. Yup I agree.

4. Yeah agree that macros are probably a bad idea for us, thanks for the clarification

Thanks @adamchalmers for linking to the cargo-script RFC, finally read it. And sorry @alteous, I'm definitely warming to the idea. I think I was hung up on too much clutter, but the cargo-script basically only lets uses to put dependencies up top and a bunch of the fields get sane defaults, and you would have to break it up into a toml file to change those fields.
David made the point that it would be easier for beginners, but I hadn't considered how this will help with folks sharing quick snippets, or minimal bug-reproduction code. some questions:

• Should we block the publishing of the single-file scripts? they don't for the cargo-scripts but, the versions is implicitly locked to v0.0.0, I think blocking publishing to our central package repository makes sense (when we get to that) but listing a single-file script as a dependency just via url/git is probably good?
• Should we make it single-file only? the moment you want to import another local file into the single-file script, you have to break them apart? (we'll give users a way to do that for them)

[adamchalmers]

Both suggestions sound good. We should also have an easy button to convert a single-file script into a proper project. And to be clear I think the single-file script mode can be low-priority for now.