🛡️ Decision Guide: Running Internal Automated Tests in S-CORE

[AlexanderLanin]

🛡️ Decision Guide: Running Internal Automated Tests on Public Repositories

Last updated: 2025-04-23
_Status: looks good 👍
This is a living document. 🤝 Discussions and contributions welcome!

---

⚡ Executive Summary

Open source repositories thrive on public collaboration, but internal teams often rely on proprietary tests that cannot be shared publicly. This guide helps you decide how to integrate internal automated tests with public repositories to ensure internal stability without hindering open source contributions.

You’ll find:

• A comparison of two main strategies: Event-Driven vs. Nightly Testing
• Key security considerations when handling untrusted code
• Actionable next steps for implementation

👉 Recommendation: For most teams, adopting a simple Nightly Build strategy provides the best balance between internal safety and open source collaboration.

---

🎯 Context & Problem Statement

Public collaboration is essential for open source projects. However, internal systems may depend on private verification steps, such as automated tests tied to proprietary infrastructure or sensitive data.

In short, we need a way to ensure changes in open source software do not break internal usage, while keeping collaboration open and seamless.

---

⚖️ Options Overview

There are two primary strategies to integrate internal tests with public repositories:

Option	🚀 Pros	⚠️ Cons	🎯 Best For
1️⃣ Event-Driven (Pre-Merge, -Release etc.)	Real-time feedback	Complex setup	Repositories with clear ownership
2️⃣ Nightly (Post-Merge, -Release etc.)	Minimal setup. No delays for contributors.	Slightly delayed feedback	Most open source & collaborative repositories

👉 In general, Nightly Builds are the preferred starting point unless your project has strict pre-merge validation needs.

---

🛠️ Details

1️⃣ Event-Driven Testing (Pre-Merge, -Release etc.)

Trigger internal pipelines based on GitHub events to perform real-time internal testing. This aligns with standard practices like verifying pull requests before merging.

Trigger Mechanisms

Option	🚀 Pros	⚠️ Cons
GitHub Actions with self-hosted runners	Quick setup. Outbound connections only.	Requires changes in public workflows. Runner maintenance.
GitHub App	Scales well across repositories.	Development effort. Requires public workflow adjustments.
Periodic Pulling	No changes to public setup. Simpler integration.	Delayed feedback compared to event-driven triggers.

Evaluation

Pros:

• Immediate detection of breaking changes
• Prevents faulty code from being merged

Cons:

• Complex setup and ongoing maintenance
• Limited automated feedback to contributors ("Internal test failed" isn’t helpful without context)
• Requires self-hosted infrastructure or custom GitHub Apps

Example Feedback

Our internal tests have failed. Please give us 48h to look into it, before merging this PR.

👉 Use Event-Driven Testing when fast feedback is critical and your team has full ownership and capacity to maintain the integration.

---

2️⃣ Nightly Builds (Post-Merge, -Release etc.)

Running internal tests on a scheduled basis after changes are merged is often the simplest and most effective approach for integrating internal verification with public repositories.

While traditionally used for long-running tests, Nightly Builds are particularly well-suited for open source projects, where contributor experience, low maintenance, and minimal disruption are priorities.

For many teams, this strategy strikes the best balance between safeguarding internal systems and fostering open collaboration.

Evaluation

Pros:

• Low complexity: Minimal setup and maintenance
• Contributor-friendly: No delays or workflow changes for public contributors
• Infrastructure light: No need for self-hosted runners, GitHub Apps, or public workflow modifications
• Good enough protection for fast-moving projects

Cons:

• Feedback is delayed, but manageable with good monitoring
• Post-merge fixes are necessary if issues arise

➡️ Unless immediate blocking feedback is essential, Nightly Builds should be your default choice.

---

🔐 Security Considerations (Handling Untrusted Code)

• Always treat public contributions as untrusted code.
• Prefer verifying the main branch, as it has passed maintainer review.
• Never run untrusted code on sensitive infrastructure without isolation (e.g., disposable VMs or containers).
• Prevent bots from leaking internal information in public logs or comments.
• Monitor integrations for unusual behavior.
• Use GitHub Environments cautiously—they mitigate some risks but require public workflow changes.

Follow your company’s security policies and ensure infrastructure is designed to handle the risks inherent in integrating public code.

---

📝 Summary & Next Steps

• Start with Nightly Builds unless your project clearly requires pre-merge validation.
• Engage with the maintainers infrastructure team for setup guidance.
• Review security implications thoroughly before connecting internal systems.
• Explore Public Test Contracts where applicable to minimize internal complexity.

---

🤝 Contributions Welcome!

This guide will evolve as new patterns and technologies emerge.
If you have suggestions, improvements, or lessons learned—open a PR and help streamline internal testing for everyone! 🚀