Plotting categorical values as colors

[juliohm]

I have plot recipes that try to plot the categorical values as colors in a geographic map. For example, the crop type in this plot: https://juliaearth.github.io/GeoStats.jl/stable/workflow.html#Plotting-solutions

I was doing some manual pre-processing by depending on CategoricalArrays.jl and converting to a vector of level codes manually. Given that the latest CategoricalArrays.jl support plot recipes already, I wanted to stop doing this manual fix. What is the appropriate method to pass categorical arrays to be interpreted as colors with an appropriate legend containing the levels?

I tried to pass the categorical array as marker_z -> array but it didn´t work. Appreciate any help as this is the last issue I need to solve before releasing a new version of the project.

[juliohm]

cc: @daschw

[juliohm]

Very practically, my question is the following:

How can I plot a scatter of points where colors are levels without depending on CategoricalArrays.jl?

using Plots
using CategoricalArrays

c = categorical([1,2,3])

scatter([1,2,3], [1,2,3], marker_z=c)

[juliohm]

Appreciate any help regarding this issue. Maybe it should be moved to Plots.jl?

[bkamins]

I think you need to wait for @nalimilan to have time to look at the issue, as he implemented the recipe AFAICT.

[nalimilan]

I have no idea, I just copied the definition provided by @daschw. Maybe @mkborregaard could help too?

[juliohm]

Any help would be great. The issue is specific to the maker_z option. I can use categorical arrays in heatmaps for example.

[juliohm]

I guess I will have to revert the changes in downstream projects in order to release? Who is leading Plots.jl nowadays? Should the Julia community pay a software engineer to maintain and fix these issues? It is really hard to progress otherwise.

I will start a thread on Discourse to see what people think about starting a group to split the payment of a salary to a free lancer.

[daschw]

Unfortunately, Recipes only apply to input data and not to attributes like marker_z. This would require major changes and additions in RecipesBase.jl and RecipesPipeline.jl, which I don't have the capacity to tackle right now. However PRs are very welcome.

If I try to run your example I get the following error:

julia> scatter([1,2,3], [1,2,3], marker_z=c)
Error showing value of type Plots.Plot{Plots.GRBackend}:
ERROR: MethodError: no method matching get(::ColorSchemes.ColorScheme{Vector{RGBA{Float64}}, String, String}, ::CategoricalValue{Int64, UInt32}, ::Tuple{Float64, Float64})
Closest candidates are:
  get(::CategoricalPool, ::Any, ::Any) at /home/dani/.julia/packages/CategoricalArrays/rDwMt/src/pool.jl:55
  get(::DataStructures.RobinDict{K, V}, ::Any, ::Any) where {K, V} at /home/dani/.julia/packages/DataStructures/ixwFs/src/robin_dict.jl:384
  get(::Test.GenericDict, ::Any, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1663
  ...
Stacktrace:
  [1] get(::PlotUtils.ContinuousColorGradient, ::CategoricalValue{Int64, UInt32}, ::Tuple{Float64, Float64})
    @ PlotUtils ~/.julia/packages/PlotUtils/es5pb/src/colorschemes.jl:18

So it seems like there is no method for get(colorscheme, categoricalvalue, range). Having a look at https://github.com/JuliaGraphics/ColorSchemes.jl/blob/a50d23f9ba76bc4811fbebd94dd309d6750abe6d/src/ColorSchemes.jl#L234 I see that they only implement methods for AllowedInput, which apparently is Union{Real, AbstractArray{<:Real}}. So it seems that CategoricalValue{Int} is not a subtype of Real.

I tried to overcome this by loosening the type restrictions for get in Colorschemes locally. However, then I run into the following error:

julia> scatter([1,2,3], [1,2,3], marker_z=c)
Error showing value of type Plots.Plot{Plots.GRBackend}:
ERROR: ArgumentError: cannot compare a `CategoricalValue` to value `v` of type `CategoricalValue{Int64, UInt32}`: wrap `v` using `CategoricalValue(v, catvalue)` or `CategoricalValue(v, catarray)` first
Stacktrace:
  [1] <(x::CategoricalValue{Int64, UInt32}, y::Float64)
    @ CategoricalArrays ~/.julia/packages/CategoricalArrays/rDwMt/src/value.jl:176
  [2] <(y::Float64, x::CategoricalValue{Int64, UInt32})
    @ CategoricalArrays ~/.julia/packages/CategoricalArrays/rDwMt/src/value.jl:180
  [3] >(x::CategoricalValue{Int64, UInt32}, y::Float64)
    @ Base ./operators.jl:305
  [4] clamp(x::CategoricalValue{Int64, UInt32}, lo::Float64, hi::Float64)
    @ Base.Math ./math.jl:65
  [5] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
  [6] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
  [7] getindex
    @ ./broadcast.jl:575 [inlined]
  [8] copy
    @ ./broadcast.jl:898 [inlined]
  [9] materialize
    @ ./broadcast.jl:883 [inlined]
 [10] get(cscheme::ColorSchemes.ColorScheme{Vector{RGBA{Float64}}, String, String}, x::CategoricalValue{Int64, UInt32}, rangescale::Tuple{Float64, Float64})
    @ ColorSchemes ~/.julia/dev/ColorSchemes/src/ColorSchemes.jl:240

I'm not really sure what we in Plots can do here without depending on CategoricalArrays.

[juliohm]

Would you reconsider the dependency on CategoricalArrays.jl? Without an explicit treatment of categorical variables we won't be able to generate correct legend elements for nominal/ordered variables for example.

[bkamins]

Without an explicit treatment of categorical variables we won't be able to generate correct legend elements for nominal/ordered variables for example.

Agreed. Also given that CategoricalArrays.jl is now compiler friendly and Plots.jl is compiler-heavy anyway I would vote to add this integration. An alternative would be to add appropriate methods to DataAPI.jl and use the interface.

[nalimilan]

The methods that error really look like they need a Real value, so it's not surprising that they fail for CategoricalValue and I don't see how they could work with them. Maybe what's needed is rather a fallback method for non-Real types that would attribute a real value to each unique value?

@juliohm How do you expect categorical values to be translated to colors? Should that be equivalent to passing the result of levelcode(v)?

[bkamins]

I was referring to the general approach. In this case (the mapping @juliohm wants) - I do not know enough about the problem to informatively comment.

[juliohm]

@juliohm How do you expect categorical values to be translated to colors? Should that be equivalent to passing the result of levelcode(v)?

Or a method that maximizes the visual discrepancy between the categories. When the notion of order is important, it could produce a sequential colormap for example. Think of a cloropleth maps like this one: https://en.wikipedia.org/wiki/Choropleth_map#/media/File:Countries_by_mean_wealth_per_adult_in_2018.png

[nalimilan]

The approach I mentioned would definitely work if you ignore the order and just choose a qualitative palette. To take into account order, the implementation would have to find a reliable way of checking whether the input values are ordered or not (see JuliaData/DataAPI.jl#26).

I'm not sure whether it should be Plots or ColorSchemes' job to do this, but probably worth filing an issue in one of these packages?

[daschw]

The marker_z attribute in Plots is a mapping from numerical values to colors in a ColorGradient or ColorScheme. So this only makes sense for <: Real values. I would not really know right now, how to handle this generally for CategoricalValues if they are not numeric.
We could take the CategoricalArrays dependency in Plots, as @bkamins suggested, if it is compiler-friendly now and fallback to levelcode. However, automatic matching with the legend entries in Plots is probably more involved, than adding this fallback.
@juliohm why do you want to remove the CategoricalArrays dependency from your recipes?