Some thoughts on language ecosystem

[gulshan]

This can regarded as just some thoughts based on my personal experience and not some language design idea/proposal. I think when I am developing, I am in one of these three modes:

• Explore: Here I have a somewhat rough idea of what I want to achieve. Then I explore various ways of achieving that goal. I may switch how am I going to do it. I may experiment with different libraries. Until the goal is reached, I remain in this mode. In this mode, I want the language/ecosystem to stand out of my way. But I also want the language/ecosystem to support me with fast feedback loops.
• Develop: In this mode, I actually have a clear idea and plan to get the end product. So I shall invest in the structure of the code. The code needs to work properly, handle all the corner cases. The best practices would be followed, the units tests will be used al the way. The failures would be neat and clean. The application/program will be monitored. etc. I think you get the idea.
• Optimize: In the previous phase/mode, optimization was not my concern. My concern was correctness of the program I am developing. But once the program is working fine, I would shift my attention to optimization. What are the bottlenecks in the program? Is it robust enough? Can I save some bytes here, some milliseconds there? And other things like that.

I am sure that, there would be other things not conforming to any of these modes. But I am focusing on these three for now. I think no one single language can be a good fit for all three modes.

For the first mode, I would like to use a optionally typed, error-forgiving, interpreted language with great type inference and tooling support. I have found non-strict TypeScript or type checked JavaScript a good fit in this category. I have not used Python that much. But I am sure it would also be a good choice here. Beginners and data scientists will most probably choose this category of language. Also the scripting languages like PowerShell fall into this category.

C# is kind of the language needed for the second mode. Newer languages like Swift and Kotlin also are nice choices. Strict TypeScript can also go with this category. Here I would like to have strong and strict typing, reliable application development frameworks, testing and other tooling. Also I think the language has to be managed or garbage collected for this mode.

For the Optimize mode, my top choice would be Rust. C++/C are also in this category. Developer needs to have full control over the bits and bytes. There would be no runtime. The program will be compiled to a native artifact for the target platform. And to have this control, some agility will be sacrificed.

But what I would love have is an ecosystem having languages for all three modes. Some points I would like to see are-

• The syntax will be similar but not same for the languages.
• The language features will be selected based on the mode. Features will be deliberately omitted from a language if not fits well within the intended mode. Which will help to keep these languages small and simple.
• The three languages will share type system, most of the standard library and package ecosystem (like nuget/npm).
• There would be seamless interop among the languages. Like all three language can be part of same compilation unit (project).
• There would be tooling support for migration from exploration language(mode 1) to development language (mode 2) or development to performance (mode 3) language. I may not go all the way, but migrating 80% using the tools would be a great help.
• For the exploration language, I would like to see optional typing, interpreter mode, automatic async operation handling, implicit interface detection etc. It should support scripting scenarios. Traspilation to JS would also be desirable. It cannot define any generic functions or data types. But can use them, while the type is inferred.
• For the development language, I would like to have everything statically typed. Built-in error, nullability and validity checking in the type system would be nice. Extension everything, built-in support for composition (over inheritance) are desirable. If the Razor/XAML like sub-languages can become built-into this language, it'll be great. Also this would be a managed/garbage-collected language, requiring a runtime.
• As I have said, I would like have a rust-like language. Not going further here.

I feel now ecosystems are becoming language-centric. Rust community wants to do everthing in Rust. C# community wants C# everywhere. Same goes for Javascript/Typescript and other languages. But having different languages focusing on different use-cases, while still being part of the same ecosystem will be nicer. I would expect the libraries and frameworks mostly being built with perf-lang and some may be in dev-lang. But consumed from dev-lang or explore-lang. Application development would mostly be targeted to dev-lang, with basic support for explore-lang.

This idea is so large that, I don't expect much in terms of implementation. But I am just shareing my thoughts.

Lastly some name suggestions for languages :). How about EScript, D# and Perf?

[jnm2]

If there is a language for each mode, does that mean switching between modes is a full transliteration between languages each time? Also there is no terminal point that I see, since any code that was last written in mode 3 may need mode 2 attention next or even mode 1.

[gulshan]

• The transition/transliteration/transpilation/migration will be one way- from script-lang to dev-lang or dev-lang to perf-lang. The other direction should not be supported. But they can keep calling the new code like the old code.
• The transition will be on a best-effort basis as it is not possible to guarantee that same code will not generate any error while transpiled into a stricter mode. But having an machine-driven heads-up in the transition phase helps the developer a lot. Kotlin has a built-in best-effor Java-to-Kotlin transpiler.
• As I want all the language can be used in a single project/compilation-unit, transitions can happen gradually instead of doing it all at once. Many projects transitioned form JS to TS in this way.

We can use the above mentioned Java->Kotlin migrations or JS -> TS migrations can be taken as examples. Those migrations support multiple languages in the same compilation unit- Java+Kotlin or JS+TS. The transition can happen one file/class at a time. The previous Java code can keep calling migrated Kotlin code. And old JS code can keep calling new TS code. But the other way transition will be a lossy one, so not desired.

[HaloFour]

Some reading regarding mixing languages within a project: dotnet/roslyn#21024

[gulshan]

@HaloFour Is languages are not designed to support this kind of interop, it becomes almost impossible once the languages mature. Kotlin and Typescript were designed to be interoperable with Java and JavaScript respectively.

[gulshan]

For multiple languages in a single codebase, I think some sort of language hierarchy is needed during referencing. This may be something like-

• perf-lang can only reference other perf-lang code, cannot reference dev-lang or script-lang code.
• dev-lang can reference perf-lang or dev-lang code, but not script-lang code.
• script-lang code can reference any code. But as per the above restrictions, cannot itself be referenced from dev-lang or perf-lang.

This means, when all 3 language codes exist in the same project/compilation unit, at first the perf-lang codes will be compiled into a single intermediate artifact. Then the all the dev-lang code along with the perf-lang artifact will be compiled into another artifact. And then script codes will be compiled/interpreted along with the last artifact to create final lib/executable file/artifact.

This means the libs will mostly be written in perf-lang but some may be in mixed with dev-lang or in dev-lang. If a library is written in script-lang (or in multiple languages including script-lang), it can only be referenced by other script-lang files/projects.

[YairHalberstadt]

Rust is both performant and safe because it has ownership is a fast class concept.

Ownership is one of the things that makes rust difficult to learn and unwieldy for certain types of problems.

A language without ownership would not be able to safely use rust data structures.

Similar would apply to other features that make languages perf oriented.

As a result I can't really see how you could have one ecosystem that is usable in a pleasant way from all 3 languages.

[gulshan]

Rust has interop libraries for many languages like JS, Python or Ruby. Some glue code is required for the interop. And the type systems are not same between languages on the two side. So, a Rust code can access a JS number or string, manipulate them and return them. Rust can convert it's own numbers and strings into JS numbers and strings. Then the JS code can call the Rust code.

What if they all shared the same type system, but the view is different from different languages. Then the glue codes and type conversion will not be needed. So, say perf-lang can see whether a data is on stack or heap, managed attributes of the data, lifetimes etc. But the dev-lang only sees the data view of the same type and would not know about the memory details. When dev-lang creates an object, it is always on the heap, managed by the gc and perf-lang can see it. But perf-lang can create object of the same type on stack/heap, make it ref-counted or gc-managed etc. When perf-lang wants to expose something to the dev-lang, it has to follow certain rules. Otherwise dev-lang would not be able to access the data. My thinking was something like this.