In-Editor AI generated GDScript for devs to use as guidance, fed by (opt-in) user submitted code.

[RabbitB]

AI generated source-code isn't quite there yet, but it is coming and we would be wise to at least acknowledge that our development process as programmers is going to change as AI progresses further. Given the current state of AI generated source-code, we're unlikely to find much use for the end-product in our projects, but something I have realized from experimenting, is that there is value in using generated source as a guide for writing our own. Even code that may not compile/run, often is structurally sound or makes use of concepts/designs that are outside our personal toolbox. That is to say, we all have our way of writing code that we're comfortable with, but doesn't always encompass the best way to organize or structure that particular code.

Infrastructure Requirements

AI models are not cheap to use and actively expensive to train. This will change in the future as AI dedicated hardware continues to advance and develop, but we aren't quite there yet. This means that the Godot Foundation would need to actively manage funds to procure and operate the model. Actively funding such an endeavor today, entirely through donations is untenable. I do however see three ways in which we could go about this:

1. Funding grants through OpenAI and other AI development projects. As I'll outline later, Godot presents some unusual opportunities that could see it as a preferred development platform for these technologies.
2. Subscription fee through W4 Games. In this scenario, W4 Games would be the actual operator of the model and Godot users would need to pay a subscription fee to use the model within the Godot engine. This would limit the amount of users that would benefit, but would provide a direct funding source that would minimize the volatility of finding funding.
3. Wait long enough that advancements in said technology limits the cost of operating an AI model, such that it can be easily funded from the Godot Foundation coffers without being a detriment to funding for other needs.

Training the Model

This would be an opt-in procedure. As users write code, they can submit a script, method, or even a highlighted selection of GDScript to be used in training the model. The submission would require a brief description of what the code does, and optionally a longer description of how it works (or does not work). The natural inclination is to only train on working code, but it would be just as valuable to submit bugged code, along with a description of what the bug is and why it doesn't work.

Training Feedback
As code is generated, users should be able to provide feedback to the model. Is it actually valid code; does it run without errors; does it actually do what you want it to do? The beauty of language models is that we don't need to be shy about providing feedback. The more we write and describe what's happening, the better the model will become, as long as the given information is accurate.

Using the Model

There are many ways to go about this and it would absolutely change over-time, but short-term I envision it as a diff-like helper. As you're writing a script, you open a prompt within the script editor that asks you to submit a request.

Movement code for a top-down character, where it can move in any direction, but input is from the four cardinal directions. Movement is relative to the local transform and accelerates, decelerates and turns at a user-definable rate.

When you make a request, your script is submitted as part of the prompt, as a baseline to build from. Not much different from how you can generate AI images using an existing image as part of your prompt to direct the AI in what you expect to see it generate.

This would split the editor, the left-side being your actual script and the right being the generated code superimposed over your script, and the editor providing a diff-like view allowing you to see what was actually added, removed or modified. From there, you can make manual changes, accept changes or reject them. Alternatively, you can submit a follow-up prompt on a portion (or all) of the generated code, which details the issue and the model will attempt to regenerate the resubmitted section.

How it Works
If implemented today, we would likely use an older model, but realistically if such a system was implemented in Godot, there would be enough time passed that the newer model designs will be available to us. Newer designs differ from older designs in that they don't just have a fixed data-set they were trained from, but instead are continuously trained from feedback to their generated output. Every time a user accepts, rejects or requests changes to the output, especially when given a brief description of why, the model learns from that interaction. As more users interact, the better our model will be and the more likely users will continue to interact and at increasing rates. This will generate a feedback loop that hopefully will accelerate substantially the capabilities of our model.

Opportunities Within Godot

Godot and its community is facing, if not unique, certainly an uncommon opportunity. That is, Godot is a self-contained development environment; with its own language; particular/preferred ways of implementing various functionality; strong community support and a fleet of built-in tools that can be automated on the backend to provide additional information to train the model. These are all boons to training.

• Self-Contained Development Environment: The model can be tightly integrated into the Godot Engine, either core or as a GDExtension. This streamlines development using the tool and removes as many friction points as possible, making its use more likely than inside other dev environments.
• Our Own Language: GDScript being a dedicated language means that all submitted code is designed to work and function within the Godot ecosystem. You aren't going to get erroneous code because the model has also been trained on completely unrelated code that just happens to be written in the same language. We can ensure that output is specific to our use-case.
• How Code is Structured: Similar to the previous point, code from one project to the next is generally structured within a familiar way, as the engine and its API inherently defines ways that are easier and/or more efficient to work with it. If you have a million objects to track, you're likely going to use a server instead of a million nodes, and the model will reflect this when generating code.
• Strong Community Support: This community has always and continues to stand as what I envision a supportive and active development community should be. We aren't without our flaws, but the interaction rate is incredible and its likely such a model would receive copious amounts of feedback and support that may not be seen with other communities.
• Built-In Tools: If you're developing in Rust, building an environment that will check output for valid code, benchmark it, etc. is much more difficult than doing so with Godot, which provides all the tools out of the box, even if we may or may not need to provide additional command-line hooks to utilize them to their full potential.

Legality (and why we lucked out)

There's a lot of questions regarding the legality of generated output from different models, as they're almost universally trained on datasets that include proprietary or copyrighted information. Our model would be opt-in and thus trained wholly on a dataset that has been provided for explicitly such a task. Thus a 'Terms of Service' can be written that directly addresses this issue, and certifies that anyone to have opted-in has authorized their code for both training and generation purposes, while protecting end-users and the Godot community as a whole from any legal issues that potentially could arise with traditionally trained models.

The Future is Bright

For now we're just looking at bolstering our own code development with AI support, but a day will come where the majority of our code can simply be generated and hand-editing is only required for edge and outlying cases. Beyond that, given our self-contained development environment as stated above regarding opportunities open to us, we could potentially see a point where our AI model is capable of understanding our scenes, our resources and everything else involved in the development of our projects. If, as a community we play our cards right and invest ourselves into this project, we could stand to see ourselves as a platform from which anyone can develop games and see their ideas come to fruition. Unconstrained by their technical and artistic abilities and to be only limited by their imagination.

Bright indeed. You better wear sunglasses 😎

[Zireael07]

Godot's core is supposed to be lean. ANY AI model is HUGE, on the order of GBs. Something for an addon, possibly, and even still doubtful due to unclear copyright status of AI generated stuff

[AThousandShips]

I think in US law it's been at least partially established that copyright is void, at least partially, and I suspect that might be the direction things will take at least for a while

[theraot]

@AThousandShips The AI cannot be an author because it is not human. And the generated code cannot be copyrighted, because the generation is not a creative process. But it might still violate copyright.

Errata: as pointed by the replies, what I claim here depends on the jurisdiction.

[AThousandShips]

That has not been established yet and many jurisdictions have no legal framework yet, but yes it can violate copyright, which is important to consider if W4 takes on responsibility for the model

[AThousandShips]

One of those problems of the internet: If some country establishes that AI models trained on copyrighted code violates copyright themselves it becomes a problem, and I think it'll be years until law on this becomes mature enough for an organization not equipped with an army of lawyers can risk it

[Zireael07]

@AThousandShips I fully expect different countries and jurisdictions to have differing stances on this.

[AThousandShips]

I believe you severely underestimate the volume of material required to build an AI model, the time and effort required to train it, and the capabilities of such a model

[AThousandShips]

Any model will also be so slow in training for quite a while yet that it can't be relevant for a still changing language like GDScript

[AThousandShips]

And to be honest this feels like putting the cart before the horse, the legal, technological, and moral aspects of AI are still very much in their infancy and there's a lot of very unhelpful hype about it, it has matured a lot in recent years but a lot of aspects are severely lagging behind, and one can set oneself up for a lot of problems because of the very hard to predict changes

[AThousandShips]

For example (I am not a lawyer, this is not legal advice, please consult a copyright lawyer before using AI generated materials in your project) the legal protections of AI produced materials

The jury is still out, and jurisdictions will almost certainly differ on their rulings, both between themselves and over time, on something as simple as if AI produced intellectual property can be copyrighted

What becomes even more muddy is the derived materials from AI derived material, for example if you generate a basic setup, and then heavily modify it, is it also not copyrightable? How much do you have to alter it? How much of your project has to be produced by a human to count?

But to me the biggest risk and worry is contamination, for example what happens if the AI just spits out parts of material scraped off the internet that is protected by copyright, even just behind a permissive license like MIT? A human arriving at the same results as someone else, without having seen their results, is just normal, and while you might have to prove that you didn't, in fact, copy someone else (which can be tricky) you are allowed to reach the same conclusion. But with AI the jury is still out on this, and there is a very real risk of you being liable for having violated someone's copyright by just using an AI model that spits out someone else's materials.

The last point also becomes much much more messy when the code isn't just copyrighted but patented, in that case much of the copyright related rules do not apply, and you can be in serious trouble, though of course this is much less of a risk as much less code is protected in this way.

All of these fall under varying jurisdictions and changing laws, and there are also risks involving control of publication. For example there are differing laws on when local laws apply to a creator in a specific jurisdiction, it can be when you explicitly publish your work there, or it can be when someone else distributes it there. This has been used in the past as a way to attack controversial authors, when someone simply imports their books to some country and then sue them over it, even though the author never published their work in that jurisdiction.

With online distribution services this becomes even more of a mess

Further, the risk also reflects back on Godot and W4 if any official product was provided, which complicates things

[theraot]

I can see it working as a W4Games plugin for the engine. Furthermore, game developers might prefer to be able to choose different tooling (e.g. Facebook Llama or OpenChatKit), which also supports the idea of making it using plugins. But not as part of the engine itself. As such, I don't think it is something to handle by the Godot project.

The things I could see the Godot side doing are:

• Continue to improve the documentation.
• Continue to make templates.

Which are training data.

---

I want to point out that there is another interested group which is the people making and monetizing Godot learning material, as integrating AI based code generation on the engine might impact their revenue.

Similarly, I feel the AI integration would also impact the promised asset store.

[bossoverdose]

🐑

• https://github.com/tatsu-lab/stanford_alpaca
• https://crfm.stanford.edu/2023/03/13/alpaca.html

[azur-wolve]

in the future when AI is more polished will have stuff like that, it will come eventually by itself, its too soon rn

in any case the thing would ideally be using an open-source model that is maintained by a third party, to dont overload the Godot project with such complex task. Also fixing bugs and all that, its too much.

btw the best you can do is instead of don't knowing coding and try to do whatever you can by intuition, actually learn how to code
data structures, algorithms, design patterns and principles, OOP, CG, etc

the problem here is that there's much people that want to make games, without having enough knowledge to do it, the fast and effortless way
can't even anticipate where this mindset leads to

[AThousandShips]

Agreed, there's a danger in tools that allow you to achieve basic functionality, like piecing together building blocks with no understanding of the details, or copy-pasting tutorials, while learning exactly nothing on the way

While it can be a slog to have to learn how to do a thing on the way, and the risk is that some people end up giving up before they learn, the same people would most likely end up never moving on from having someone else do it for them to actually learning themselves when it's required

[ckcollab]

I think this is a fantastic idea. For example, the GPT4-Turbo release has 128k context, I believe that's enough for most projects to live in. This is just the beginning, as well, and we may have plenty of context to store all Godot docs + your project + your plugins + ...

Not sure there will be a need fine tuning or anything?

[ChildLearningClub]

AI is an extremely powerful tool that, if not utilized, will put all Godot users at a huge disadvantage to other game engines and tools. I'm a huge advocate for not taking shortcuts to learning, but even though I might like walking, I don't think it's fair to expect the same of the next person. We can all sit idle, waiting to find out that @RabbitB was right, or embrace what is inevitable if we want to stay relevant.

[emircanerkul]

Could we implement https://codeium.com/ ?

[jabrek]

With external editor set to VS code - codeium has a plugin there - worth trying one (will do that too)

[emircanerkul]

But I do not like external window. Could we inject vscode window into godot app?

[polymorphicshade]

What about https://github.com/janhq/jan? It can run 100% offline (and even works well without a GPU), has a simple REST API, and includes a super easy way to download/use models (Llama 3 support was added very recently).

(not an ad, I just really like this project)

Perhaps we could hook Godot up with it with a simple UI tab somewhere.

Example use case: a game development team can host a server running Jan, and each could interface with it in their own network, letting them use the AI without needing very powerful development workstations.

[Calinou]

I feel this is really add-on territory, especially since Jan is AGPL-licensed (which isn't a license that we can integrate in Godot).