[ZTF][Rubin]Add ELEPHANT hostless detection as a science module

[utthishtastro]

#1111

Also updated the ZTF FINK URL for the Telegram channel

[JulienPeloton]

Thanks @utthishtastro ! I left a few comments, but it is in good shape :-)

One more comment. This change adds 2 new fields: elephant_kstest and finkclass (see comments). They also need to be defined at the level of the database ingestion script to make sure you can access them in the DB. This happens at the level of fink_broker/ztf/hbase_utils. But as elephant_kstest returns an array, this is not a trivial addition (I handle currently primitives and maptype, but not arraytype).

We have three options here:

1. update the science module to return maptype (easy, but break backward compatibility)
2. update the DB ingestion script to handle arraytype (hard, but keep backward compatibility)
3. instead of returning elephant_kstest, create another column with booleans: hostless/not hostless

Personally, I prefer the third option -- as it is does not break compatibility (no change on science module), and it is much easier to interpret than the kstest numbers. This would mean:

# after you compute elephant_kstest (without the slice function)

cond_science_low = df["kstest_static"][0] >= 0.0
cond_science_high = df["kstest_static"][0] <= 0.5
cond_template_low = df["kstest_static"][1] >= 0.0
cond_template_high = df["kstest_static"][1] <= 0.85
cond_max_detections = F.size(F.array_remove("cmagpsf", np.nan)) <= 20

df = df.withColumn("is_hostless", cond_science_low & cond_science_high & cond_template_low & cond_template_high & cond_max_detections)

expanded.extend(["elephant_kstest"])

This way you add a boolean column on which you can filter later on:

df.filter("is_hostless")

What do you think?

[JulienPeloton]

tracklet is not yet defined here as the algorithm to link alerts runs only at the end of the night. You can replace it by the default value: F.lit("").

[utthishtastro]

@JulienPeloton Thank you for reviewing and your suggestions :) I discussed this with Emille and the third option with a boolean output for ZTF sounds good to us 👍. I have defined new columns in fink_broker/ztf/hbase_utils please check if they are fine.

However for Rubin, we would like to output the kstest_scores. I hope this is still possible. Can I make the changes in fink_broker/rubin/hbase_utils already?

[JulienPeloton]

Thanks @utthishtastro !

However for Rubin, we would like to output the kstest_scores. I hope this is still possible. Can I make the changes in fink_broker/rubin/hbase_utils already?

Yes. The best would be to change ArrayType to MapType (returning a dictionary with named fields).

[utthishtastro]

@JulienPeloton I have updated the PR to support MapType. I have also updated the return type in fink-science. Please let me know if further changes are required. Thanks a lot :)

[utthishtastro]

Also added SLSN score to ZTF hostless Telegram bot astrolabsoftware/fink-science#582