Release 0.22 tracking

[bkamins]

Here I list things to do before 0.22 can be released.

A must (breaking):

• [BREAKING] Multicolumn transformations for GoupedDataFrame #2481 multiple columns in transformation functions)
• [BREAKING] deprecate DataFrame constructors #2464 (data frame constructors)
• [BREAKING] Refactor unstack #2494 (clean up unstack)
• [BREAKING] remove CategoricalArrays dependency from joins #2505 (categorical in joins)
• Explicitly handling missingness in join columns #2499 (missings in joins)
• add Cols support #2495 (add Cols support)
• require DataAPI.jl version 1.4 #2514 (up DataAPI.jl dependency)
• [BREAKING] Add PrettyTables.jl backend for printing DataFrames #2429 (change printing backend - it is breaking as allowed kwargs will change)
• Switch from Coveralls to Codecov #2520 (bump coverage tests)
• remaining dependencies on CategoricalArrays #2506, avoid CategoricalArrays dependency in aggregates #2519, make hashrows_col! not depend on CategoricalArrays.jl #2518 (remaining categorical dependencies)
• Allow CategoricalArrays 0.9 #2521 (up CategoricalArrays.jl)
• move All(args...) tests to deprecated.jl #2515 (clean up deprecated tests)
• Update Categorical test #2523 (CategoricalArrays.jl integration)
• Update tests to Tables.jl v1.2 #2530 (sync with Tables 1.2)
• Enable precompilation #2456 (improve precompilation)
• code layout improvements #2536 (code layout)
• update deprecated tests #2538 (clean up tests)
• Post PrettyTables.jl housekeeping #2525 (improve float printing in text/plain)
• update manual and docstrings to PrettyTables.jl #2522 (update docstrings and manual to new printing backend)
• bump DataFrames.jl version
  Add subset #2496 (add where function) (removed from the list, this will be decided before 1.0 release)
  Add standard deviation and 25% and 75% quantiles to describe :detailed #2459 (describe update),(it is non breaking in the end so it can be done later)

@nalimilan, @pdeffebach - if you see something that is required more for 0.22 then please comment.

The objective is that after 0.22 release we should aim at 1.0 release soon without any breaking changes (0.22 should serve as testing period only + new functionality).

[nalimilan]

Just a note that we should decide whether we change the way missing values are handled by default. Not saying that we should, but we must be aware that if we release 0.22/1.0 without it then it won't happen until 2.0 (if at all).

[bkamins]

Just a note that we should decide whether we change the way missing values are handled by default.

My understanding was the conclusion was to add where as I described above and this is the only change we need. What additional changes do you see required? (in particular I understand that we concluded that adding a functionality that conditionally updates/creates a column only on a subset of rows in a data frame is something that we decided is not crucial to have)

[bkamins]

What additional changes do you see required?

@nalimilan + @pdeffebach : do you see any more changes required in 0.22 (except for the described above) in the area of handling missing values?

[nalimilan]

My understanding was the conclusion was to add where as I described above and this is the only change we need. What additional changes do you see required? (in particular I understand that we concluded that adding a functionality that conditionally updates/creates a column only on a subset of rows in a data frame is something that we decided is not crucial to have)

One potential issue is that having where skip missing values but other functions like combine, select and transform not skip them would be a bit inconsistent. And while we can add where without breaking anything, changing the behavior of other functions would be breaking. So at least we should decide whether that inconsistency would be a problem before 0.22. Probably better keep discussing this at #2314.

I'm not sure the conclusion regarding what we should do was so clear (you're referring to #2314, right?). One simple and consistent solution that wouldn't break anything would be to say that by default missing values should never be skipped in DataFrames (not even with where), but that they should be skipped by all functions in DataFramesMeta.

[bkamins]

OK - all required PRs for 0.22 release are now open

[bkamins]

We should also decide on #2499 before 0.22 release.

[bkamins]

Today we made a major step towards 0.22. I assume @nalimilan will now work on #2459 and I rewrite manual and docstrings using the new backend. After these two PRs are merged I will open a PR bumping DataFrames.jl to 0.22 to check again if all works as expected.

From this moment, except #2459 we do feature freeze for 0.22 release so testing is welcome!

[bkamins]

There is a proposal (which I like) that for 0.22 release all deprecated functionality should have force=true set (so that we make sure users see deprecation warnings). @deprecate does not support force=true, but I can make a substitute for this case. The question is if we agree with this approach.

@oxinabox if I recall correctly, you recommended to drop printing depwarns by default in Julia Base, so your opinion here would be welcome.

[oxinabox]

I don't love deprecation warnings being turned off.
It's not my preferred solution.
But the problem was that deprecation warnings can cause immense slow down.
It was effectively breaking for users of DataStructures.jl (LightGraphs and JuMP) because it was so much slower.
I actually ended up having to undeprecate it for them.

If you do force there is no way for the user to save themselves from it. Even if they know it isn't a problem.

They also are transitive to the user.
It makes sense to see the deprecation warnings for your direct dependencies.
It doesn't make sense for your indirect dependencies.
(I would really like to fix that using a LoggingExtras filter. Though I think Base needs to be changed to make sure to set the group on deprecation warnings so they can be identified. But it's not something I have had time to do yet).
A particularly bad case is any kind of long running production system with logs that people need to read.
Deprecation warnings are turned off for a production system, since it's a frozen Manifest in a docker container -- no chance of anything being updated by mistake.
And being flooded by deprecation warnings when you are trying to deal with a production crash is noone's idea of a good time.
(Not too hard to work around since such a logging system has its own filters hopefully. I know SumoLogic does)

Also: it's not a problem to use deprecated methods.
They still work.
We follow semver.
Users should check for deprecations before updating to a new breaking change.
Plus we have depwarn turn on in tests -- which is the place you go to check.

Also allows users to turn off deprecations for tests if they don't want to be spammed so they focus on their actual errors not on the deprecation spam.
Using force is going to be really confusing for folks at Invenia as we run two seperate nightly CI jobs for everything.
One with deprecations turned off, so we can focus on the package itself, and one with it set to error so we can know if it's to update anything.