Package publishing support

[Lazauya]

I'm confused about the status of package publishing. It looks like you had a publish command a while back, but it seems that it was removed. I notice that you more recently added search, which searches the limited packages on the site. I also couldn't find any reference to publishing in TODO's or anything. Is this planned/are there any active efforts in this area? If so I'd like to contribute.

[ken-matsui]

Thank you for your interest. There is a lot of room to design/implement package publishing, such as poac.toml schema, database schema, etc. I'm happy to discuss further if you are interested.

[Lazauya]

I would be interested. I guess the poac.toml schema would be a good place to start. Crates.io uses a single Github repo for their entire package index, so I guess we can just copy them. 😅.

On the DB side, there are already a few C++ package repos, and I think we should look into storing our packages there. It might not be feasible, but it would make startup a little faster, and people would be able to benefit from poac packages right away. I assume we can include a JSON/toml file in the package that would identify it as poac-compatible.

Those are just my two cents. I'd like to know what's been the current discussion.

[ken-matsui]

I would be interested. I guess the poac.toml schema would be a good place to start. Crates.io uses a single Github repo for their entire package index, so I guess we can just copy them. 😅.

Not sure if we can just copy it, but it would be great if we could replicate their way to index packages.

On the DB side, there are already a few C++ package repos, and I think we should look into storing our packages there. It might not be feasible, but it would make startup a little faster, and people would be able to benefit from poac packages right away. I assume we can include a JSON/toml file in the package that would identify it as poac-compatible.

Yes, we can use the existing database, but I guess we need some tweaks.

Those are just my two cents. I'd like to know what's been the current discussion.

My current conclusion is to implement a Cargo-like package registry (rather than relying on GitHub) to avoid issues like the ones discussed here. However, I haven't had enough time to fully investigate its implementation.

Unlike the Rust ecosystem, where libraries are created within the Cargo framework, C++ already has a large number of existing libraries. This means we need to prevent malicious users from uploading packages with the same names as well-known libraries. To address this, we can use GitHub authentication for the Poac registry's credentials and require users to use their GitHub username by default. For example, if I want to upload a SemVer package, I would need to name it ken-matsui/semver. However, users may prefer shorter, more familiar names, like fmt instead of fmtlib/fmt. For widely recognized packages, we could allow official aliases in the database, which would be displayed on the website as official packages. The challenge will be in defining the criteria for these official aliases, but we can tackle that later.

Another challenge is protecting the service from malicious users. While GitHub authentication can filter out some malicious actors, it won't prevent all issues, such as namesquatting (can be prevented by the solution I've shown above), typosquatting (can be prevented by lev distance?), or misuse of the registry for non-Poac packages (e.g., using it as storage). We'll need a way to verify that packages are legitimate Poac packages.

Finally, there's the issue of cost. All services should remain free and based on donations. For example, if I receive $10 per month in donations, I will allocate that $10 to upgrading servers, but I won’t spend more than the amount donated.

Here's my current thoughts. Please let me know if you have any ideas!

[Lazauya]

That all makes sense, I think.

For hosting, I have a server rack with a lot of bandwidth. It's not the most powerful machine but it should work for the immediate future. I can provide more details if you're interested.

I definitely think prefixing the packages with Github usernames is a good idea. We should probably just do that automatically, i.e. you name a package as "semver" but it becomes "ken-matsui/semver" in the index. The namesquatting and typosquatting issue is tricky. I'm of the opinion that namesquatting is unavoidable. We could manually prune the index to solve conflicts, though. I think it will be less of an issue if we enforce GH username prefixes, though, even when declaring a dependency. Like literally write "ken-matsui/semver" = 1.0.0 in the toml. If someone wants to include a package and yours is the only version, then you just write poac add semver and it will still autoresolve. If yours isn't the only one maybe poac could prompt you for clarification? Cargo doesn't have anything like this really so they don't have to worry about conflicts, but I do agree it's more tricky with how developed the C++ ecosystem already is.

If there aren't any complaints, I'll try making a MVP for the indexing portion of the server using some of the things talked about here.

P.S.
I wonder if there's a truly serverless way to do everything using only Github and existing CPP repos. (I don't mean using AWS Lambda functions or anything, but literally just the poac client making all the GH and repo upload calls itself). I don't really think there is due to safety concerns, but I'll keep thinking about that.

[ken-matsui]

That all makes sense, I think.

For hosting, I have a server rack with a lot of bandwidth. It's not the most powerful machine but it should work for the immediate future. I can provide more details if you're interested.

Thank you for your cooperation. Unfortunately, we can't use a personal server rack because we need to ensure the server is running without any modifications to the original source code. Instead, we'd prefer to use a free hosting service with a direct deployment.

I definitely think prefixing the packages with Github usernames is a good idea. We should probably just do that automatically, i.e. you name a package as "semver" but it becomes "ken-matsui/semver" in the index.

I believe it's more straightforward to enforce the use of a prefix directly in poac.toml. This approach simplifies local parsing of the poac.toml files when necessary.

The namesquatting and typosquatting issue is tricky. I'm of the opinion that namesquatting is unavoidable. We could manually prune the index to solve conflicts, though.

Our first step should be implementing a registry with the prefixing feature. We can then consider more user-friendly solutions, such as offering official names.

I think it will be less of an issue if we enforce GH username prefixes, though, even when declaring a dependency. Like literally write "ken-matsui/semver" = 1.0.0 in the toml. If someone wants to include a package and yours is the only version, then you just write poac add semver and it will still autoresolve. If yours isn't the only one maybe poac could prompt you for clarification?

This is something we can revisit later as we refine the system.

Cargo doesn't have anything like this really so they don't have to worry about conflicts, but I do agree it's more tricky with how developed the C++ ecosystem already is.

If there aren't any complaints, I'll try making a MVP for the indexing portion of the server using some of the things talked about here.

I'm not sure we have a fully defined concept of the server's indexing portion yet, but I'm happy to review the MVP once it's ready.

P.S. I wonder if there's a truly serverless way to do everything using only Github and existing CPP repos. (I don't mean using AWS Lambda functions or anything, but literally just the poac client making all the GH and repo upload calls itself). I don't really think there is due to safety concerns, but I'll keep thinking about that.

Using only GitHub and existing C++ repositories, safety concerns remain. While verifying packages with sha256 could mitigate some risks, the potential for package deletion and its impact on dependents' installations is a critical issue. That's why Cargo maintains its own registry, and we may need to do something similar.

[ken-matsui]

I wanted to provide some additional details about my infrastructure to ensure a smooth migration of your code, as you mentioned that you've been working on the server implementation here. Currently, I'm using the free version of Hasura, so I don't have a dedicated backend server. However, for package publishing, we may require custom, exhaustive validations, which I think Hasura Actions handle effectively. Please keep these infrastructure details in mind to save time and effort as you work. Feel free to reach out for more details.