discuss perpl/propel

[smhg]

This is meant as as continuation of mringler's comment on an issue in the Propel2 repo:

I wanted to see what happens when I cleanup the worst parts in Propel Runtime. The conclusion is that it is semi-enjoyable work, but the code becomes much more maintainable. Now it builds queries 30-50% faster (microseconds, not noticeable irl), has type-preserving query methods, typed collections, and some other things I was missing for a long time. Tests are still working too.
I understand this is never going to find its way back to Propel2 - it's too much to review, even if there were still reviews. Yet it feels awkward doing it just for myself, and I miss getting feedback here. So I pushed it to composer as perplorm/perpl.
I like working on Propel and I would enjoy to keep collaborating. So if these changes sound interesting, or you just need an actively-maintained fork, have a look, let me know what you think.
Cheers.

I will soon add some feedback. But thank you for your efforts!

[smhg]

From the top of my mind. I think for a project to succeed it comes down to:

1. Updates are as painless as possible. Propel is/was very good at this if you ask me.
2. Changes are released swiftly. Propel got this very wrong (no blaming). Nothing kills interest like unused code (for maintainers and users alike).

Those create trust. Which is what this fork would need IMHO.

I understand this is never going to find its way back to Propel2 - it's too much to review.

That sentence scared me more than switching to a fork after ~15y of using Propel :)

You should be able to release anything of course.
But if you want to continue Propel and convince people to move along, I suggest to release the current Propel main branch's code as version X. (I'd say 2.0.0, but that doesn't matter too much). So switching to Perpl means updating a line in composer.json.
And after that, anything is possible.

But please stick to the first 2 points above.

[smhg]

I personally love semver. At least, as a user, I know broadly speaking what time I might need to invest in an update.
So let's say you release current Propel as Perpl 2.0.0. If your current changes might break things, please do release them as 3.0.0. And next week 4.0.0 if applicable. Propel's history with their major version release attempts is irrelevant at that point.

[smhg]

Take @oojacoboo's feedback as an example:

We do still have a fair amount of other Propel code that isn't covered though, and won't be able to commit to using it until that can be further tested.
...
What commitment are you making to maintaining Perpl?

I think the 2 points above tackle those (valid) worries.

I don't think any additional commitment is necessary. Most projects aren't able to provide that. Propel definitely didn't for many years.

[smhg]

FWIW: Propel indeed seems to be stuck with Spryker as you mentioned. Not in any evil way, but they can't do point 2 (swift releases).

Their releases in the past have however been very thorough (leading to point 1, painless updates) as far as my experience goes.

While I'd avoid months-long release schedules, it would be awesome to still have a chance to merge with Propel at some point.

[oojacoboo]

So switching to Perpl means updating a line in composer.json

@smhg this was actually true for me. Propel 2 has been in beta for a long time. Version 2.0.0 here is basically like saying it's coming out of beta. Switching to another repo should be treated like a version update as well, a major one too. So, I don't see any issues here with switching. For me, I was able to run perpl on our application without any errors/issues - just one line change in composer.json.

Regarding my comments on not all of our code being tested. We are working on some new Cypress testsuites to increase coverage of our non-API based routes (it's mostly controller coverage of concern with Propel). After that's been implemented, we'd probably have enough confidence to deploy perpl. It's also possible we could make it an initiative to do some manual testing for a deployment.

The issue I have with Propel is that there isn't any appetite to move the lib forward at all. We've already started refactoring out of Propel (into Doctrine, as it's a better fit for our needs), and honestly we'll probably continue to do so. But, there are a lot of improvements that can/should be made to the ORM that aren't likely to get merged into Propel.

[mringler]

Thank you for the feedback, @smhg and @oojacoboo! Sorry for the delayed response - apparently I have to watch the repository to get notifications, even if it is my own.

I understand this is never going to find its way back to Propel2 - it's too much to review.

That sentence scared me more than switching to a fork after ~15y of using Propel :)

I know :) Since you have been with Propel from around the beginning, do you know why everybody pretty much stopped working on it between 2013 and 2020? I always wondered.

Let me say, these large changes I have been doing (and want to keep doing) are not optimal. Ideally, they wouldn't be necessary, however, I fully agree with @oojacoboo that Propel cannot be improved by working around the edges. The way it is, Propel is simply not maintainable. It is code from a different era, I'm sure the people who wrote it wouldn't write it like this today. I have come across similar code in my professional life, quite recently too - but I have never seen it in a library (I'm happy to point out a few places if you are interested). Currently I am looking at cross-relations because of this issue, and it is pretty bad.

When they all but stopped maintaining Propel, I though about switching to Doctrine, too. But it is not realistic at the moment (how is the switch going on your end, @oojacoboo?). I discarded the idea to maybe merge the two, i.e. run Propel on top of Doctrine to then gradually remove the Propel parts. It's too much work I though, but this has been extensive, too. Now I hope there is another side to come out at, where it is actually easy and convenient to work with (and inside) Propel/Perpl.

Reliability has (for me) always been an issue with Propel, so I really don't want to make it worse. At the same time, making backwards compatibility the main goal means leaving all the crazy stuff in. It's a conundrum.
My hope is that we can rely on the tests to some extend, which are running through fine. And that through proper versioning (as you suggested too) and reasonably quick patching, impact of bugs can be minimized.

That said, I have updated my two projects to Perpl, found it easier than installing Propel2 (with the explicit vcs import of the dev branch, no versioning, switching back and forth between repos while PRs were in limbo, etc), and it seems to be running nicely so far. I do enjoy finally being able to do some real changes with the code. I wonder what it means with getting merges from Propel, at least in core this will be very limited - then again, Propel/Spryker hasn't really touched that in the last five years, I don't expect them do it now.

All that is to say that I agree with you, things should be easy, swift and clean. Let's see how that goes. In case you try Perpl, let me know how it goes!

[oojacoboo]

how is the switch going on your end, @oojacoboo?

We started this process years ago. We first created an EntityPersister that replicated the interface for Doctrine's EntityManager (this is an addition to Perpl that'd I'd recommend over the persistence methods in base classes!). We then implemented interfaces between Doctrine entities and Propel models. This actually allowed for them to be fairly interchangeable. You still have to manage state though between Doctrine's EntityManager and Propel. That part is a bit tricky, even requiring factories that will convert from one to the other.

That said, we found parts we could swap iteratively and have been making progress. All of our new stuff is on Doctrine.

do you know why everybody pretty much stopped working on it between 2013 and 2020? I always wondered.

Yes, Doctrine became the standard for Symfony and everyone ditched ActiveRecord. Not sure about 2020, but I suspect that was probably around the Laravel Eloquent rise, which is also ActiveRecord.

All-in-all, I really don't think Propel is that bad. If it was cleaned up, modernized and a new focus was placed on actually using it in a more moden fashion, I suspect there could be some decent interest as another RAD ActiveRecord ORM. But that will never happen with Propel. It will just continue to fizzle out, as there isn't any interest from Spryker to make larger changes.

For us, it's unlikely we'll get all of our code refactored over to Doctrine anytime soon. It'll be years for sure.

[mringler]

Interesting. So you are using Doctrine to save Propel models and you read them as Doctrine objects, which you turn into Propel models when necessary? Have you though about integrating Doctrine into Propel models, so you can hydrate them through Doctrine? Is that even possible?
Having spent a lot of time working on those Propel Criteria classes, I feel they are getting better, but still, I wouldn't mind getting rid of the whole goddamn thing. In general, making the actual database logic exchangeable would be a huge thing.

Doctrine became the standard for Symfony and everyone ditched ActiveRecord

It's amazing how much mileage Doctrine got out of declaring they are not ActiveRecord. I sometimes wonder how many people actually know what that means.
Still incredible that pretty much everybody left. You'd think there were some projects that could not change and maybe a few devs who keep it going. Apparently, they all dropped it like a hot potato.

I started using Propel in 2020, in a project where I had to exchange a self-built interface, and on paper, Propel sounded incredible: builds the schema file for you, builds the classes, an extensive, opinionated interface, comes with migration management, you can tweak and replace most of the functionality pretty easily. Took the first obstacles like a champ (more like an idiot, considering how they were warnings), like having to use the dev branch, getting the base configuration right, figuring out what the generic error messages meant, having five ways for everything and none really works, and so on. And when I had it set up beyond the point of no return, I began leaning into the features, and ran into bug after bug after bug. Really everything seemed to come apart the moment it is used beyond the one main, non-specific use case. At least this is how it felt. And when I looked into the code, it all made sense. Started to commit the generated model files, because I wanted to keep up with changes as I wasn't confident they'd work. After a while, I started to push my fixes back to Propel, and here we are. Still, it is pretty apparent how much work has not been going into the library.

That said, I also think Propel isn't that bad, particularly after you got it running. Conceptually, I like it a lot, and I didn't mind the bugs, as long as I could simply fix them. But now, without maintenance, it's starting to feel like some poor old package you find on the net and adjust for your own needs.

If this Perpl helps to make things more orderly, maybe even get away from the complicated history and reputation of Propel for some of the remaining users, I'd be happy.

[smhg]

@oojacoboo switching one composer line indeed is always the case (and yes, we are all using some kind of branch alias by now I guess). But I meant Perpl immediately containing significant changes, is huge commitment (independent of their necessity and value - reviewing them is hard). While I think you could take many more users along if the first step is as small as possible. And yes, changing things in small releases after, if possible. I want to switch like that :)

[smhg]

@mringler I started using Propel around 2008 I guess. And a few years (2011?) after that in the project I'm currenlty maintaining. Which was built on Zend Framework 1 (!). I'm halfway through converting that to Symfony 7. Propel (through skyfox/propel-bundle) still works quite well with current Symfony. It's great to have such a central part of my project being this trustworthy and adapting to the environment around it changing through all these years.

But Propel is much older. I think it was based on Apache Torque (a Java ORM). I think by Hans Lellelid (and others). When I picked it up, it was being maintained by Francois Zaninotto and he handed it over to William Durand. They did an amazing job if you ask me. After that it indeed lost its momentum. You can read Francois' opinion about what went wrong here.
But it truly got some boost again around 2020-ish. I think mainly because of @dereuromark's guidance? But also your own efforts! Thank you for this!

It is brave you went with Propel in 2020 :) But I hope what was possible around that time can be achieved again. Forks are always a last resort, no? Something seems to happen in the Propel repository now. Maybe it can start again? Maybe there is a way to bring your work along?

[mringler]

Thank you for the background information, @smhg! I know the names from the code, but I didn't know that they passed Propel on to each other. Very interesting to read Francois' perspective, it's pretty close to what I figured - basically a falling out over where to go, between working on Propel2 and branching out into Propel3 at the same time to then both just not going anywhere. And then Spryker and particularly Mark reviving the project, doing tons of work consolidating the forks, establishing workflows and building up a community again. And now it has fallen apart once more. Funny how that goes.
Yes, let's see how it continues.

[smhg]

basically a falling out over where to go, between working on Propel2 and branching out into Propel3 at the same time to then both just not going anywhere. And then Spryker and particularly Mark reviving the project

Mark worked on Propel between William and Spryker. He lead the Propel 3 rewrite. I think that was an attempt to replace Active Record with Datamapper. Without agreeing or disagreeing, that is the part Francois mentioned in his blog. Things definitely stalled during that period and people moved on.
I don't know how Propel moved from Mark to Spryker. I guess Spryker took the initiative? But someone please fill in the blanks :)

[oojacoboo]

Interesting. So you are using Doctrine to save Propel models and you read them as Doctrine objects, which you turn into Propel models when necessary? Have you though about integrating Doctrine into Propel models, so you can hydrate them through Doctrine? Is that even possible?
Having spent a lot of time working on those Propel Criteria classes, I feel they are getting better, but still, I wouldn't mind getting rid of the whole goddamn thing. In general, making the actual database logic exchangeable would be a huge thing.

No, we're not doing this, although that might be possible, if you could get them to follow a clean interface. Unfortunately, that's not possible, and we actually already discussed this @mringler propelorm#1656 lol.

The issue preventing that is passing around this ConnectionInterface (connection object). That's a very poor design choice. That, alone, prevents you from having clean models that aren't littered with Propel particulars.

But beyond that, even if you fixed that issue, you're still extending a pretty nasty base class. However, that's how ActiveRecord works.

What we did, was create a Propel specific version of Doctrine's EntityManager. We essentially replicated the interface and logic to work with Propel. We don't use save() for instance on our models. We'll use EntityPersister::persist and EntityPersister::flush. This comes with the added benefit that your code is cleaner and you can "stage" your persistence.

So, by making this change, it means our instantiation and getting/setting are nearly the same, and our persistence logic is basically the same, only using a different persistence service (EntityPersister instead of EntityManager).

Despite this and the fact that we have a pretty well oiled situation, we still do not have clean interfaces. That's due to the above issue on the ConnectionInterface parameter, as well as a lack of proper typing on base Propel models. Our Doctrine entities are all properly typed (strict).

[oojacoboo]

@smhg honestly, the tagged version of 2.0 is mostly compatible with the current version of Propel for userland. There is a list of breaking changes on the README. If you're not using those, it's unlikely you'd have any issues at all. We have one instance of Criteria::condition and Criteria::addCond. Those are easily refactored.

@mringler can speak more to this, but it looks like Perpl forked a while back, so I don't think it'd be easy to tag a version without the referenced BC breaks, not without losing some of the other fixes and commits that have come sense, even on Propel.

[mringler]

@smhg

Mark worked on Propel between William and Spryker

I think that's Marc, not Mark :)

@oojacoboo

we actually already discussed this

Quite detailed too. Thanks for the reminder! I have opened an issue to discuss this further at #41.

it looks like Perpl forked a while back

It's currently four commits behind Propel and 154 ahead. The changes to the interface are absolutely required (in my rigid mind), but my estimate is that the impact is quite minimal, as it affects lots of functionality that probably shouldn't have been public in the first place. Notable exception are Criteria::condition() and Criteria::addCond() which was required as a workaround, and setSelectQuery().
Seems like this is exactly the impact on @oojacoboo's side - thank you for confirming!

However, note that you can still use all of them, they are still resolved through magic __call() - they just don't show up on the interface and IDE and static analysis will complain - as they should, it's all crap :)

But I completely see your point, @smhg, a changing interface is a problem for an existing Propel project. I am aware of that and intent to keep the impact as minimal as possible (like by not just removing the old interface but by resolving it through __call() as a fallback).
If you have ideas how this can be realized further, I am thankful to hear them! Your point about adhering to strict versioning is well-taken. I am currently looking at cross-refs, and this will be version 2.1 as it affects the way relations are named in some cases (for the better, I am fixing a bug after all, but still).