Working groups

[csbarnet]

Hello all,

At PEARC25 and at the latest community call, we talked about forming a few working groups to discuss features and topics concerning the Coldfront ecosystem.

@ANekhai offered the community call zoom as a 'place' for us to meet.

What are topics of interest for initial working groups? Some that have been mentioned:

• plugin architecture
• storage plugin
• governance
• code standards
• documentation (user-facing and developer-facing)

Please comment with any areas of interest. @dsajdak also mentioned that she will be sending out an email to community call registrants.

[csbarnet]

Some prior work on plugin architecture mentioned by @dsajdak

#627
#697

[Eric-Butcher]

I suggest that the storage plugin and the plugin architecture groups be merged together.

My thinking is that there is only so much progress that anyone can make by just sitting down and thinking how plugins should work. Many important details and concerns will only come up by actually going through the process of building one.

My thinking is the process could go something like this:

1. Discuss current pain points and desires for how ColdFront plugins should work more generally. Define a set of goals that the community wants for plugins.
2. Discuss specifically the goals and scope of the storage plugin.
3. Start creating the storage plugin without making changes to how ColdFront currently works. Everyone working on the storage plugins would take notes on any pain points they encounter while writing, testing, integrating, and deploying the plugin.
4. Come back to re-evaluate the goals defined in step 1 with lessons learned from step 3 and create a new detailed design specification and architecture for how plugins should work. Create documentation so that everyone is on the same page.
5. Refactor the plugin system in ColdFront to accomplish the spec outlined in step 4.
6. Refactor the storage plugin already written to work with the new plugin system created in step 5.
7. Test the storage plugin with the new system. Evaluate whether the goals have been actually accomplished and loop back through the refactoring process until a system is accomplished that is suitable.
8. Release the new system and provide developer documentation clearly describing how the system works and pointing new plugin authors to the storage plugin as the reference implementation.

After this decisions could be made about what should be done with pre-existing plugins.

Of course this is only one idea on how to handle this. This proposal would probably only work well if there is significant overlap between the groups of people interested in plugin architecture vs. the storage plugin specifically. Doing these two things together might also stall progress since it would prevent these discussions from otherwise happening in parallel.

I believe others may have already started working on the storage plugin? So maybe that throws a wrench in this as well.

I am interested to see what others think.

[csbarnet]

@Eric-Butcher I think it makes sense for the storage plugin and plugin architecture groups to at least start as the same group and I think that your proposed process is reasonable. I agree that doing both in tandem will slow down progress on a storage plugin.

I'm sure that there are others, but one key pain point that I'm experiencing is the added friction of having the plugins folder embedded in the directory structure of the project (and the necessity of having the associated file in the 'settings' directory). It makes working with git more cumbersome and also makes adding and activating plugins in container deployments or pip install-type deployments difficult.

Possibly, this could be solved relatively simply by extricating the plugins folder to the top level and using importlib.

Longer term, perhaps, I think it would be fantastic to have a simplified version of what Apache Airflow offers. Essentially, there are a number of connectors that handle low level details, and the user just writes high level workflows in python that are triggered by a signal or are scheduled. The plugins/connectors are specified in the requirements and pip installed and the directory with the workflow code can be specified.

One thing that I would add is that I think it will be useful to look at other django based projects to see if the difficulties we're seeing have been solved to our satisfaction elsewhere already. I certainly agree that it is important for us to define our requirements before making that determination.

Since we have a community meeting coming up next Tuesday, perhaps we can use the opportunity to set up a first meeting time to start outlining pain points, requirements, etc.

[ANekhai]

That's a good point on the plugin bundling pain point. I've been looking at some other extensible Django projects to see how they handle the issue of 3rd party plugins. Looking at projects like Wagtail, they've got comprehensive developer docs on how to extend the app coupled with a website pointing towards a bunch of plugins hosted on GitHub with some subset being pip installable through PyPI. This seems like the design we should be moving towards.

All of this is to say I think moving the plugins out of the project directory might be a sensible course of action.