API endpoint and portal data mismatch for SSOs.

[Birr0]

I noticed that for some SSOs the portal does not return the same data as the API. For example:

The 'ssoft' endpoint (https://api.ztf.fink-portal.org/api/v1/ssoft?sso_number=831069) and the data from the SSO bulk ('ssobulk') endpoint return the same data.

However, this differs from the data from the 'sso' endpoint (https://api.ztf.fink-portal.org/api/v1/sso?n_or_d=831069) and the portal (https://ztf.fink-portal.org/ZTF25ackensd). These sources seem to be consistent with each other.

One effect of this is that there are less observations in the ssoft and ssobulk endpoint for the same object. It appears that the fitted parameters from the phase functions are also different which affects the reported residuals for these objects.

Let me know if this is intentional or I should look elsewhere for bulk downloads.

Thanks!

[JulienPeloton]

Hello @Birr0 ,

Thanks for reporting! I apologize for the delay in my answer, the launch of LSST alerts took all my attention...

What you see is expected, although badly documented, hence the confusion. The SSO endpoint is updated everyday with nightly data, and on the portal when you fit for phase curve parameters, it uses all data available (+/- last observing night). On the contrary, the SSOFT is updated once a month (on the 1st day of each month -- see https://doc.ztf.fink-broker.org/en/latest/services/search/solar_system/#ssoft-solar-system-object-fink-table) -- hence the difference. But I believe, for long lasting events, the SSOFT parameters should have more or less the same order of magnitude if you add a few observations (unless unusual behaviour appears!).

[Birr0]

Hi @JulienPeloton,

Thanks for your reply! That makes sense for the updates to data available on the endpoints. With the monthly updates, I would expect only the most recent observations to not be matched. Precisely, if there are more recent observations than the last update to the SSO endpoint then these would not yet available. However, I would expect the first observations of the object to match from both API endpoints. (If this isn't true, just let me know). I will provide a minimal example here to show the issue I am facing on this point.

For a number of objects, the first object observation from the ssobulk response does not match the first observation date from the 'sso' endpoint. It appears that in general, the 'sso' endpoint has more historical observations than the same object retrieved from the bulk endpoint. I don't think this can be explained by the monthly update unless some historical observations are dropped from the ssobulk endpoint.

Here is the minimal example that shows for these select objects that the historical observations do not align from the ssobulk and sso endpoints.

*Note: these are some of the unusual phase curves I've found from the machine learning approach developed at the sprint.

Input example

# download sso's from the bulk endpoint
r = requests.post(
  "https://api.ztf.fink-portal.org/api/v1/ssobulk",
  json={
    "output-format": "parquet"
  }
)
df_sso = pd.read_parquet(BytesIO(r.content))

# example ids for which the historical observations do not align.
example_ids = ['831069', '812101', '836513', '819749', '843079', '847304','830760', '833358', '793578', '806503']
for id_ in example_ids:
    # request the sso endpoint
    r = requests.get(
        f'https://api.ztf.fink-portal.org/api/v1/sso?n_or_d={id_}'
    )
    
    # sort the dates returned by both endpoints in ascending order.
    dates = [item['i:jd'] for item in r.json()]
    api_dates = np.sort(dates)

    bulk_object_query = df_sso.loc[df_sso["ssnamenr"] == id_]
    bulk_dates = np.sort(bulk_object_query["cjd"].to_numpy()[0])
    
    try:
        # check the minimum dates available. Return these and the number of available observations.
        assert bulk_dates.min() == api_dates.min(), (
            f"Earliest date mismatch for {id_}:\n"
            f"  bulk min jd = {bulk_dates.min()}, no. of observations: {len(bulk_dates)}\n"
            f"  api  min jd = {api_dates.min()}, no. of observations: {len(api_dates)}"
        )
        print(f"{id_}: ✓ OK")

    except AssertionError as e:
        print(f"{id_}: ❌ {e}")

Output

831069: ❌ Earliest date mismatch for 831069:
  bulk min jd = 2460922.8167014, no. of observations: 70
  api  min jd = 2460861.8713194, no. of observations: 131
812101: ❌ Earliest date mismatch for 812101:
  bulk min jd = 2460882.9208565, no. of observations: 69
  api  min jd = 2460861.8694329, no. of observations: 81
836513: ❌ Earliest date mismatch for 836513:
  bulk min jd = 2460923.8826389, no. of observations: 57
  api  min jd = 2460878.9025347, no. of observations: 74
819749: ❌ Earliest date mismatch for 819749:
  bulk min jd = 2460882.8331713, no. of observations: 50
  api  min jd = 2460875.8690162, no. of observations: 63
843079: ❌ Earliest date mismatch for 843079:
  bulk min jd = 2460923.8617477, no. of observations: 55
  api  min jd = 2459085.9228356, no. of observations: 117
847304: ❌ Earliest date mismatch for 847304:
  bulk min jd = 2460922.7736227, no. of observations: 54
  api  min jd = 2460878.9129051, no. of observations: 88
830760: ❌ Earliest date mismatch for 830760:
  bulk min jd = 2460922.8298495, no. of observations: 54
  api  min jd = 2459523.7830093, no. of observations: 96
833358: ❌ Earliest date mismatch for 833358:
  bulk min jd = 2460923.9019676, no. of observations: 57
  api  min jd = 2458993.984919, no. of observations: 280
793578: ❌ Earliest date mismatch for 793578:
  bulk min jd = 2460841.8373958, no. of observations: 61
  api  min jd = 2460821.820706, no. of observations: 71
806503: ❌ Earliest date mismatch for 806503:
  bulk min jd = 2460844.9014236, no. of observations: 51
  api  min jd = 2458963.8534259, no. of observations: 58

[JulienPeloton]

Thanks for detailed report, I will inspect!