[Task]: Create a script to train sklearn model for IT test.

[AnandInguva]

What needs to happen?

Sklearn doesn't offer backward compatibility wrt models on newer versions. In the Sklearn IT tests, we use models trained manually and this can get outdated when there is an update to sklearn.

So to tackle this, we need to create a script which trains the sklearn models on the data and then publish this model to a GCS bucket, once this is done we can use this model to run Sklearn IT test.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[nervoussidd]

Hey Anand,
Can you please assign me this issue as i would like to contribute.And if you can also guide me it will be perfect.

[AnandInguva]

You can comment .take-issue for self assigning.

Let me take a look on what we need for this task.

[AnandInguva]

The script here trains the Sklearn model required to run inference on Japanese housing dataset IT test.

Now, the problem is we have a set of Sklearn models trained and saved in GCS path. So we use that models for IT testing. But these models are not forward compatible with newer version of sklearn. So to resolve this issue, we would like to

• create a script that could run and train the script
• Save the trained models on a GCS location
• Run the pipeline in the example as IT test using the above saved model.
• Add this setup to the Beam infra and make sure this gets run as part of postcommit tests(I can help here once the above thing is done)

[AnandInguva]

You can go through the Sklearn beam example and the sklearn training script. If you are ready to do it, please let me know and we can forward with setting up data.

[nervoussidd]

Thank You for the reply.YES i am ready to work under your guidance.But please give me atleast 2 days to go through the Sklearn beam example and the sklearn training script..

…

On Fri, Feb 3, 2023 at 12:27 AM Anand Inguva ***@***.***> wrote: The script here <https://gist.github.com/AnandInguva/1ce58c7df98ade4efe6bc95a157b82da> trains the Sklearn model required to run inference on Japanese housing dataset IT test <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py> . Now, the problem is we have a set of Sklearn models trained and saved in GCS path. So we use that models for IT testing. But these models are not forward compatible with newer version of sklearn. So to resolve this issue, we would like to - create a script that could run and train the script <https://gist.github.com/AnandInguva/1ce58c7df98ade4efe6bc95a157b82da> - Save the trained models on a GCS location - Run the pipeline in the example <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py> as IT test using the above saved model. - Add this setup to the Beam infra and make sure this gets run as part of postcommit tests(I can help here once the above thing is done) — Reply to this email directly, view it on GitHub <#24903 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUA2XLT34RXU7SD7ENEPOADWVP7Q5ANCNFSM6AAAAAATSCNU4Y> . You are receiving this because you commented.Message ID: ***@***.***>

[nervoussidd]

Hello Anand, I have gone through the example and the script.And now i am going to learn somethings about the Sklearn as i am new to this things.Keeping you updated with my progress. On Fri, Feb 3, 2023 at 5:34 PM Siddharth Aryan ***@***.***> wrote:

…

Thank You for the reply.YES i am ready to work under your guidance.But please give me atleast 2 days to go through the Sklearn beam example and the sklearn training script.. On Fri, Feb 3, 2023 at 12:27 AM Anand Inguva ***@***.***> wrote: > The script here > <https://gist.github.com/AnandInguva/1ce58c7df98ade4efe6bc95a157b82da> > trains the Sklearn model required to run inference on Japanese housing > dataset IT test > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py> > . > > Now, the problem is we have a set of Sklearn models trained and saved in > GCS path. So we use that models for IT testing. But these models are not > forward compatible with newer version of sklearn. So to resolve this issue, > we would like to > > - create a script that could run and train the script > <https://gist.github.com/AnandInguva/1ce58c7df98ade4efe6bc95a157b82da> > - Save the trained models on a GCS location > - Run the pipeline in the example > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py> > as IT test using the above saved model. > - Add this setup to the Beam infra and make sure this gets run as > part of postcommit tests(I can help here once the above thing is done) > > — > Reply to this email directly, view it on GitHub > <#24903 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AUA2XLT34RXU7SD7ENEPOADWVP7Q5ANCNFSM6AAAAAATSCNU4Y> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[nervoussidd]

Hey Anand,
Sorry but because of my other stuff (my mid sem exams,tutoring and GSOC preparation)i was not able to focus on this project.To inform you about my progress i have already learnt about the respective things to create a script.i need some more days to finish it off.And again i am sorry for not updating you with my progress previously.Hope you understand.

[AnandInguva]

Hey Anand, Sorry but because of my other stuff (my mid sem exams,tutoring and GSOC preparation)i was not able to focus on this project.To inform you about my progress i have already learnt about the respective things to create a script.i need some more days to finish it off.And again i am sorry for not updating you with my progress previously.Hope you understand.

Hi, please take your time and work when you can on Beam. I am here to help you in anyway I can. Thanks for the update,