Proposal: Consolidate spark-bebe into spark-daria as a Subproject

[zeotuan]

Background

• spark-daria is our primary Spark utility library, containing general-purpose functions and enhancements.
• spark-bebe was originally created to house sql functions that the Spark team decided not to expose in the public Scala/Python APIs, as well as time manipulation utilities.
• spark-bebe has not been actively maintained in some time.
• spark-daria now includes a subproject (currently named unsafe) which contain custom native Catalyst expressions

Proposal

• Rename the existing spark-daria extension/unsafe subproject to bebe
• Use this subproject as the new home for spark-bebe functionality. Include expression in daria-extension as part of bebe

Rationale

• Unifies efforts and single API under one actively maintained repository. We can release all custom native sql expression in a single maven package
• Clearly separates concerns:
  • daria/: Safe, general-purpose utilities based on more stable spark sql functions API
  • bebe/: Custom native Spark Expression and API

Note: Alternatively, we can move these native spark sql expression into bebe git repository. But the reason why the above method is proposed is due to spark-daria already have actively maintained CI. As for maven release, it will remain two separate projects

Please let me know what you think about this proposal

[alfonsorr]

Let me check the use of bebe, but I don't like the idea to reuse a library already published. It won't be a new version, it will be a refurbish of the name...

[zeotuan]

This is a fair point. I think there is two alternatives:

• Have a distinct package name for daria-extension. we can still port existing bebe functionality into daria if bebe is not going to be actively maintained
• Move the native SQL expression and API from daria extension into bebe and breathe new life into the repository.

The one main goal I am looking at is to have 2 separate packages housing:

1. All spark version independent utility - Daria
2. All Spark version dependent utility - Bebe or Daria-extension (Whatever works best)

[zeotuan]

After taking another look at the scala API exposed by bebe, daria and official spark. I found that many of bebe functions has actually been expose as Scala API by spark.

Daria already has somewhat richer datetime functions than bebe and some are overlapped as well.

• bebe beginningOfMonth vs daria beginningOfMonthDate/Time
• bebe endOfMonth vs daria endOfMonthDate/Time

One exception is beginningOfDay and endOfDay (can easily port these)

Therefore, I have come to the conclusion that it is not necessary to "consolidate" Bebe into Daria and we can release daria extension as a separate package

[MrPowers]

@zeotuan - thanks for the proposal!

We've always worked really hard to maintain backward compatibility and I think this proposal may be entirely backward compatible, so it could be great!

Could you provide some snippets on whether users would need to change their code or if they would maintain the same import patterns with this proposal? I am interested in learning more about this proposal! Also, did Spark end up exposing a lot of these APIs that bebe was designed to expose?

[zeotuan]

Hi @MrPowers, it is possible to make this completely backward compatible thus no change required from current user of bebe. Thing we will need to do:

• rename the package name and structure of the current extension subproject within daria.
• Rename data-extension-functions to BebeFunctions and port all bebe functions to it. We could also rename this extension sub project to bebe completely to reflect the migration
• Update documentation on both repositories about this migration
• Optional: Soft deprecated bebe functionality that has already been exposed by official Spark Scala API (bebe_approx_percentile vs percentile_approx)

[zeotuan]

Hi @MrPowers, I’ve submitted PR #177 to demonstrate porting bebe into this repo.

The file unsafe/src/test/scala/org/apache/spark/sql/BebeFunctionsSpec.scala is a 1:1 migration of the existing bebeFunctionSpec, which shows how the functionality can be used. This migration is fully backward compatible and requires no changes from users.