Architectural Refactor for Project Structure #1155
Replies: 13 comments
-
|
I think in general, more structure is better.* I personally find it easier to find things in a project when I don't have to guess as to where it is, and one of the easiest ways to achieve this is with a highly structured hierarchical setup. If we are optimizing for approachability from a new contributor perspective, I think it would make the most sense to decouple technical details from the structuring as much as possible, but the way described in the A combination of the self-contained packages (9) and hyper-specific split (3) is what I would lean towards, though there are definitely examples of the other structures working out if designed properly. The core/features split (1) also looks appealing, but may need some care when implementing. * Of course, to a certain degree- it's very easy to get caught up in restructuring and forget to write any meaningful code. I do still think that being as specific as possible is definitely worth it though ^w^ |
Beta Was this translation helpful? Give feedback.
-
|
I personally really like Proposal 3, albeit with a few changes. For one, I also feel that discord/ui and discord/handlers should be moved to a I don't think we should shoehorn everything into the In essence, what I am saying is, I do think the |
Beta Was this translation helpful? Give feedback.
-
|
For proposal 2, I do like that idea. When it comes to organizing, I believe that moving files around to the appropriate directories would look great and understandable for new maintainers. Same goes for proposal 1. Even though it can be quite vague in some cases, I don't think it's a bad idea. |
Beta Was this translation helpful? Give feedback.
-
|
I find the structure in #7 - The Hybrid "Monorepo-Ready" structure provides the most value for the bot's current status and makes room for long-term goals. #1 and #2 are simpler, great for contribution and is easier to get acquainted with but it lacks the clarity and flexibility that #7 offers. For example, dropping "everything" (for a lack of a better word) into a general-purpose folders like features or services might be okay in the short term but it will become tech debt as we grow. It's easy to start treating them as catch-all folders, iykwim (which makes navigation and maintenance harder over time). what I like about the hybrid structure is its explicitness and it's intentional. So there is a top-level folder that separates between core components such as bot/, services/, and shared/. The subfolders inside them make it easy to locate what we're looking for. If we need to ever add another top-level component like api/ or web/, this structure allows for that growth without needing to restructure again. i also agree with the concern about verbose imports. That said, we can manage this with consistent naming conventions or even import aliases to avoid excessive verbosity while keeping things... readable. that's my 2 canadian cents. |
Beta Was this translation helpful? Give feedback.
-
|
I love 9 for the clearly encapsulated code and division of responsibility. making the project more loosely integrated and having every bit of code related to a module right there inside it does, in my opinion, make it far easier for contributors to understand and work with the modules. the discovery step would add some overhead, though not enough to be that inconvenient, check https://github.com/anemoijereja-eden/chandragen to see some examples (the formatter system loads every single bit of formatting code as modules and still runs pretty fast) I do, however, understand not wanting to go for such a drastic architecture shift. for a simple organization change rather that a full design paradigm shift I personally find 7 the most attractive. it cleanly divides responsibilities for sections of the code and leaves plenty of room for expansion. getting some things out of the tux/ dir is definitely a huge plus. |
Beta Was this translation helpful? Give feedback.
-
|
Proposal 2 would be my go-to. In my opinion, splitting the two components apart really helps take the pressure off when you're just focused on a specific part of the bot (like a command). You don’t have to worry about all the backend you might not be touching, and I think it'll just make everything feel more organized and easier to get around. Then again, structure 9 basically does the same thing, with a more detailed and descriptive base layout. |
Beta Was this translation helpful? Give feedback.
-
PrefaceFirst, I want to start by saying I'm not a super-active contributor to this project. So I don't have a large intuition on what is tightly coupled. Maybe y'all (@kzndotsh and @electron271) can chime in on that part and help answer questions. I don't want to "rollplay" as some active contributor as I've only sent in a few PRs and have a shallow view of the codebase. I want to give a more conditional response because of my absense of experience and understanding. Especially since there's really two major contributors to this project, I really believe that the others commenting on this issue (especially those that have made 0 contributions) should do the same. From all of the projects that I've maintained, shipped, etc, there's been one common theme. Coupling and abstraction go hand and hand. As you abstract two or more components, they become tightly coupled together. And vice versa, components that are less abstracted or generalized together will have a looser coupling. If you want a visual explanation of this idea, I deeply recommend watching CodeAesthetic's video on "Abstraction". My idea here is that things that naturally end up very tightly coupled should be organized closely together. And things that are less coupled should naturally be more separate. Now whether it's best to make these "less coupled" components into packages, sub-directories, entirely different repos, etc, I can chime a lot more on. That said, it's conditional. ProposalIn reality, no single proposal alone will be decided on. It will most likely be a little mix of a few proposals. So I think it is important to address each one. FeaturesIf your "features" mentioned in Proposal 1 are less coupled together, I think it would be best to have them as each as individual sub-directories under the parent directory I do not believe it is beneficial to have it under a separate features directory unless there is some sort of abstraction or generalization connecting them together. Regardless, even if there needed to be a features directory to hold them together, I think it's still best under the Moreover, if there is some shared logic and abstraction, I would create the features directory and try to reuse that amongst all of the features. ServicesThe "services" mentioned in Proposal 2 seem to be like wrappers of external processes, systems, etc or just API handling for external services. That said, I'm not sure if there's anything coupling them together. They mostly seem like independent parts. I can imagine that if there is some coupling, it isn't even between all the services. Related services that show some coupling should be organized together. Hyper-SpecificI don't think there should be a "core" directory. Just put it under I would really go and question if "utils" is even utilities. Again, if ExtensionsI don't have much to say here besides there's no runtime plugin system so I don't think this matters. All pros found here can be more or less found in the other proposals. If you are actually wanting to design a "framework" design, I'd actually go and make a discord bot framework and have that as a separate thing. I would never go and half-bake that idea. And I really only see benefit to that if you're reusing that framework across multiple bots. Maybe you guys have a long term goal of doing that. But if that's the case, that framework, per se, is a separate project that should be in its own repo. It would be best if it was not a part of a single specifc bot. It would also be awkward for other bots to have to use a framework that is part of some monolith deeply integrated into some discord bot repository. So go 100% or don't at all. DomainsAgain, I would go with what's naturally coupled here instead of enforcing arbitrary organization. My intuition is that I would also assume that See "Web App" for my thoughts on "Web App" likeSee my comments on "Extensions". If it's a framework, make it a separate, external, generalized framework. If not, don't lean into it at all. However, I do like the idea of a Monorepo-readySee my concern for "Services" on CleanI think this aims to create separation of concerns, but in reality, it separates coupled ideas and separates things with the same concern. If anything, I think this is separating more arbitrarily and less about "separation of concerns". This enforced de-abstracting things which naturally are related and would force you to repeat yourself instead of promoting code reuse. You'll also have longer tree traversals to access similar things. Side note: I think it's funny that this is a "Clean Code" architecture b/c it goes against some of the "Clean Code" principles like DRY. Self-Contained PackagesIf you have packages, actually make separate python packages and import them. They should be version tracked separately in separate repositories. If that is too much separation b/c of how little the functionality is or because of how coupled it is, I wouldn't make separate packages at all. ConclusionPlease ping / tag me and respond with the missing information that my response requires. I think the next step would be to take all of the ideas from these proposals we like (not necessarily the proposals as a whole) and plan next on how to mix and merge these ideas into a cohesive mass. |
Beta Was this translation helpful? Give feedback.
-
|
I'm thinking that a hybrid between 7 and 9 would actually be great, as a "tightly integrated monorepo with support for fully modular extensions"
effectively, it's #9 but with the scope of the "core" bot system expanded out. we could ship a few modules with the base bot, but then leave the rest of it up to the end user. we could maintain a few official modules, while keeping the actual scope of what the modules do fairly small. the use of a unified API for it makes contributing easier, because a contributor would not need to have an understanding of the full scope of the bot itself, just the module layout and extension APIs. |
Beta Was this translation helpful? Give feedback.
-
|
the module loading system would also ideally have some configuration options per-module that could be changed at runtime for things like channel restrictions, role level restrictions, etc. |
Beta Was this translation helpful? Give feedback.
-
|
I'm most in favor for proposal 9 i think having a modular system would benefit any self-hosters who wants only some features and not others while also allowing for custom features that might not fit into the main bot/repo for example something like a "fishboard" separate from the starboard. and to add onto this a module skeleton or example repo (using git submodule) possibly to show how one could be made |
Beta Was this translation helpful? Give feedback.
-
|
yes i think proposal 9 would be best, may also let us expand on #704 i also do like the idea from @anemoijereja-eden about a hybrid of proposal 9 and 7 |
Beta Was this translation helpful? Give feedback.
-
|
A mix of 7 and 9 looks best to me. A monorepo but instead of "features" to replace cogs there are installable "packages". |
Beta Was this translation helpful? Give feedback.
-
Tux Architecture: Modular, Scalable, and MinimalIssue: #924 Architectural Refactor for Project Structure Introduction & GoalsTux is designed for clarity, extensibility, and ease of contribution. The architecture separates core logic, infrastructure, features, and user add-ons, while keeping the root directory minimal and clean. Finalized Directory StructureRationale for the Structure
Proposals ReviewedAs Tux has grown, the need for a more intentional, scalable, and contributor-friendly project structure has become clear. Multiple proposals were presented, ranging from simple splits to modular plugin architectures. This document summarizes the discussion and feedback from contributors and presents a final, consensus-driven proposal. Extensibility Model: Modules & Custom ModulesTux supports two types of extensions: 1. Modules (Official/Core Features)
2. Custom Modules (User/Server-Specific Add-ons)
How Modules Work (Official Features)
How Custom Modules Work (Self-Hoster Add-ons)
Loader Logic Example: import os
import glob
from discord.ext import commands
bot = commands.Bot(command_prefix="!")
# Load official modules
for mod in ["modules.moderation", "modules.fun", "modules.starboard"]:
bot.load_extension(mod)
# Load all custom modules (if any)
if os.path.isdir("custom_modules"):
# Load .py files
for file in glob.glob("custom_modules/*.py"):
modname = f"custom_modules.{os.path.splitext(os.path.basename(file))[0]}"
bot.load_extension(modname)
# Load packages (folders with __init__.py)
for folder in os.listdir("custom_modules"):
folder_path = os.path.join("custom_modules", folder)
if os.path.isdir(folder_path) and os.path.isfile(os.path.join(folder_path, "__init__.py")):
modname = f"custom_modules.{folder}"
bot.load_extension(modname)
# IMPORTANT: Also load all cogs/extensions in infra/handlers/ just like modules and custom_modules
if os.path.isdir("infra/handlers"):
for file in glob.glob("infra/handlers/*.py"):
modname = f"infra.handlers.{os.path.splitext(os.path.basename(file))[0]}"
bot.load_extension(modname)
for folder in os.listdir("infra/handlers"):
folder_path = os.path.join("infra/handlers", folder)
if os.path.isdir(folder_path) and os.path.isfile(os.path.join(folder_path, "__init__.py")):
modname = f"infra.handlers.{folder}"
bot.load_extension(modname)How to add a custom module as a submodule: git submodule add https://github.com/yourorg/tux-my-cool-module custom_modules/my_cool_module
git submodule update --init --recursiveAfter updating or cloning, run: git submodule update --init --recursiveUnified Summary Table
Handlers and Cog Loader
Database Considerations
Tests
ConclusionThis structure provides a clean, scalable, and contributor-friendly foundation for Tux, supporting both official features and self-hosted extensions, with clear separation of concerns and minimal root-level clutter. References: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Architectural Refactor for Project Structure
The Problem
As Tux has grown, our project structure within the
tux/directory has evolved organically. While functional, it has reached a point where related components are spread out, and the separation between core logic, user-facing features, and utilities is not as clear as it could be. This can increase the cognitive load for new contributors and make maintenance and scaling more difficult in the long run.The Goal
The purpose of this discussion is to collaboratively decide on a new, more intentional architecture for the project. The ideal structure should:
The Proposals
Below are nine various proposals for restructuring the
tux/directory. They range from simple refactors to patterns inspired by community standards, modern web apps, and formal software architecture. These proposals do not imply automatic consideration because of their existence, they are simply a means of inspiration/brainstorm/example to get the discussion flowing.Please review each one. They are collapsed for readability.
Proposal 1: The "core" and "features" Split
A straightforward approach that divides the application into two main packages:
corefor backend logic andfeaturesfor user-facing components.Structure (Proposal 1)
Pros (Proposal 1)
Cons (Proposal 1)
featuresis a bit vague and could become a miscellaneous dumping ground.Proposal 2: The "bot" and "services" Split
This option frames the structure around the two main parts of the application: the
botitself (all Discord-related code) and the backendservicesthat support it.Structure (Proposal 2)
Pros (Proposal 2)
botandservicesis conceptually clean and easy to grasp.utils: A dedicated, top-levelutilsdirectory solves the problem of common, shared code.Cons (Proposal 2)
Proposal 3: The Hyper-Specific Functional Split
This is the most granular approach, creating specific top-level directories for each major function of the application. It follows a more rigorous domain-driven design philosophy.
Structure (Proposal 3)
Pros (Proposal 3)
Cons (Proposal 3)
Proposal 4: The "app" Monolith with "ext"
This model keeps most of the application together in a main
appdirectory but moves all external-facing components (like Discord cogs and the CLI) into anext(extensions) directory. This is similar to the pattern used by frameworks like Flask or Django.Structure (Proposal 4)
Pros (Proposal 4)
extcontains all the ways a user or developer "enters" the application.Cons (Proposal 4)
app: Theappdirectory itself can become large and less organized over time.appandextmay not be as immediately clear asbotvs.services.Proposal 5: The "src" Layout with Domain Grouping
This approach uses a standard
"src"layout and groups code by its domain or high-level feature, rather than its technical function.Structure (Proposal 5)
Pros (Proposal 5)
Cons (Proposal 5)
/cogs,/services).common.Proposal 6: The "Web App" / Next.js Inspired Structure
This model borrows concepts from modern web frameworks like Next.js, organizing the project by features ("routes") and co-locating related code. It emphasizes a strong separation between feature logic, reusable UI components, and shared libraries.
Structure (Proposal 6)
Pros (Proposal 6)
app), reusable UI (components), shared logic (lib), and application setup (core).Cons (Proposal 6)
app/might require creating several files (commands.py,views.py, etc.), which could feel repetitive.Proposal 7: The Hybrid "Monorepo-Ready" Structure (Community Inspired)
This proposal is a synthesis of the most common and scalable patterns observed across dozens of other major Discord bots, combined with modern architectural practices. It is designed to be immediately familiar, highly organized, and ready to scale into a full-stack application (e.g., with a website or API).
Structure (Proposal 7)
Pros (Proposal 7)
bot/directory provides a clean namespace, making it trivial to add awebsite/orapi/directory at the same level in the future.bot/,services/, andshared/are unambiguous.bot/features/is self-contained, but it has easy access to reusable UI frombot/components/and logic fromservices/.Cons (Proposal 7)
from bot.features import moderation), but this explicitness is generally considered a good thing for clarity.Proposal 8: The "Clean/Hexagonal" Architecture
This is a highly disciplined, academic approach based on patterns like Clean Architecture. The goal is maximum separation of concerns, making the core application logic completely independent of external frameworks (like Discord.py) and services (like the database). The "core" of the application knows nothing about the "outside world."
Structure (Proposal 8)
How it Works (Proposal 8)
infrastructure/discord/cogs/moderation.pyis invoked.application/services/moderation.pyservice.domain/moderation/entities.pyobjects and calls methods on a repository interface defined indomain/moderation/repository.py.infrastructure/database/repositories/, which was "injected" into the service when the bot started.Pros (Proposal 8)
Cons (Proposal 8)
Proposal 9: The "Self-Contained Packages" (Modular Plugins) Structure
This approach treats every feature of the bot as a complete, self-contained Python package. The main application is merely a lightweight shell responsible for discovering and loading these "plugins."
Structure (Proposal 9)
How it Works (Proposal 9)
core/bot.pyscans thepackages/directory.moderation), it imports it and calls a well-defined entry point function, likesetup(bot).moderationpackage'ssetupfunction is responsible for adding its own cogs, views, and listeners to the bot instance. It gets any dependencies like a database session from thebot.Pros (Proposal 9)
packagesdirectory.Cons (Proposal 9)
Request for Feedback
We invite all maintainers and contributors to review these proposals and provide your thoughts. To help guide the discussion, please consider the following:
Your feedback is crucial for setting a clear direction for the future of Tux. Thank you!
Beta Was this translation helpful? Give feedback.
All reactions