On-Demand-Druid Exhaust Job on Obsrv Building Block

[Shakthieshwari]

Hi ,

We got to know from Sunbird-Lern BB Team, that to all the data products they are adding a security audit feature. As per the Design Detailed out in this https://docs.google.com/document/d/1Pq0oWdeSNWWw2mGQelXpGmj8b_NS5Q3ndt0folpj2r4/edit# doc

Today we have a Generic On-Demand-Druid Exhaust Data Product which extracts the data from druid, ML Totally have 10 CSV's being extracted based on the different model configs. Hence we wanted to know , if security audit enhancement can impact (or) need any changed to this data product present in the Sunbird-Obsrv BB.

@SanthoshVasabhaktula @reshmi-nair @rhwarrier Please let us know your availability to connect on this as soon as possible ....

Cc- @aishwaryashikshalokam @Ashwiniev95 @Prateek-slokam @aks30 @kiranharidas187 @vijiurs

Please do the needful at the earliest....

Thanks

[Shakthieshwari]

@SanthoshVasabhaktula @anandp504 Can you please let us know your availability to connect on this ?

[SanthoshVasabhaktula]

@Shakthieshwari - There should not be any impact to OnDemandDruidExhaustJob as by design we don't store any PII information in druid. All data in Druid is anonymous and used for aggregates and analytics purposes only and is not meant to store transactional data along with any PII.

If ML team has gone ahead and used Druid incorrectly, then go ahead and create a wrapper around this job. This you can review it internally with @aks30 and @vijiurs

[Shakthieshwari]

@SanthoshVasabhaktula We donot get or store any PII Information using OnDemandDruidExhaustJob. We are using this Data Product to get only Projects,Observation and Survey Level Data.

Since there were different SecurityLevels mentioned by reshmi in the design doc of Lern, we are just confirming if there would be any impact on the OnDemandDruidExhaustJob present in Sunbird Obsrv BB ? should we change anything regarding SecurityLevels in Obsrv Data Product ?

[Shakthieshwari]

And also we don't have PII Related information stored in druid as well

[SanthoshVasabhaktula]

There is no impact to OnDemandDruidExhaustJob.

[Shakthieshwari]

Thanks for Clarifying @SanthoshVasabhaktula .

@aks30 @vijiurs FYI