Skip to content

Commit dc874bd

Browse files
Amar3ttoAmar3tto
authored
Add PyTorch DistilBERT Sentiment Analysis Batch/Streaming pipelines for ML Benchmarks (#34577)
* Add PyTorch DistilBERT Sentiment Analysis streaming pipeline * Improve website ML pipelines metrics pages * Use autoscaling * Add batch sentiment pipeline * Fix Formatter * Fix Formatter
1 parent 4f5b78f commit dc874bd

File tree

17 files changed

+587
-3
lines changed

17 files changed

+587
-3
lines changed

‎.github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,33 @@ jobs:
8484
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Language_Modeling_Bert_Base_Uncased.txt
8585
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Language_Modeling_Bert_Large_Uncased.txt
8686
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Imagenet_Classification_Resnet_152_Tesla_T4_GPU.txt
87+
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Streaming_DistilBert_Base_Uncased.txt
88+
${{ github.workspace }}/.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Batch_DistilBert_Base_Uncased.txt
8789
# The env variables are created and populated in the test-arguments-action as "<github.job>_test_arguments_<argument_file_paths_index>"
8890
- name: get current time
8991
run: echo "NOW_UTC=$(date '+%m%d%H%M%S' --utc)" >> $GITHUB_ENV
92+
- name: run Pytorch Sentiment Streaming using Hugging Face distilbert-base-uncased model
93+
uses: ./.github/actions/gradle-command-self-hosted-action
94+
timeout-minutes: 180
95+
with:
96+
gradle-command: :sdks:python:apache_beam:testing:load_tests:run
97+
arguments: |
98+
-PloadTest.mainClass=apache_beam.testing.benchmarks.inference.pytorch_sentiment_benchmarks \
99+
-Prunner=DataflowRunner \
100+
-PpythonVersion=3.10 \
101+
-PloadTest.requirementsTxtFile=apache_beam/ml/inference/torch_tests_requirements.txt \
102+
'-PloadTest.args=${{ env.beam_Inference_Python_Benchmarks_Dataflow_test_arguments_6 }} --mode=streaming --job_name=benchmark-tests-pytorch-sentiment-distilbert-base-uncased-streaming-${{env.NOW_UTC}} --output_table=apache-beam-testing.beam_run_inference.result_sentiment_distilbert_base_uncased' \
103+
- name: run Pytorch Sentiment Batch using Hugging Face distilbert-base-uncased model
104+
uses: ./.github/actions/gradle-command-self-hosted-action
105+
timeout-minutes: 180
106+
with:
107+
gradle-command: :sdks:python:apache_beam:testing:load_tests:run
108+
arguments: |
109+
-PloadTest.mainClass=apache_beam.testing.benchmarks.inference.pytorch_sentiment_benchmarks \
110+
-Prunner=DataflowRunner \
111+
-PpythonVersion=3.10 \
112+
-PloadTest.requirementsTxtFile=apache_beam/ml/inference/torch_tests_requirements.txt \
113+
'-PloadTest.args=${{ env.beam_Inference_Python_Benchmarks_Dataflow_test_arguments_7 }} --mode=batch --job_name=benchmark-tests-pytorch-sentiment-distilbert-base-uncased-batch-${{env.NOW_UTC}} --output_table=apache-beam-testing.beam_run_inference.result_sentiment_distilbert_base_uncased' \
90114
- name: run Pytorch Vision Classification with Resnet 101
91115
uses: ./.github/actions/gradle-command-self-hosted-action
92116
timeout-minutes: 180

‎.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Batch_DistilBert_Base_Uncased.txt‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
--region=us-central1
18+
--machine_type=n1-standard-2
19+
--num_workers=20
20+
--max_num_workers=250
21+
--disk_size_gb=50
22+
--autoscaling_algorithm=THROUGHPUT_BASED
23+
--staging_location=gs://temp-storage-for-perf-tests/loadtests
24+
--temp_location=gs://temp-storage-for-perf-tests/loadtests
25+
--requirements_file=apache_beam/ml/inference/torch_tests_requirements.txt
26+
--publish_to_big_query=true
27+
--metrics_dataset=beam_run_inference
28+
--metrics_table=torch_sentiment_distilbert_base_uncased_batch
29+
--input_options={}
30+
--influx_measurement=torch_sentiment_distilbert_base_uncased
31+
--device=CPU
32+
--input_file=gs://apache-beam-ml/testing/inputs/sentences_50k.txt
33+
--runner=DataflowRunner
34+
--model_path=distilbert-base-uncased-finetuned-sst-2-english
35+
--model_state_dict_path=gs://apache-beam-ml/models/huggingface.sentiment.distilbert-base-uncased.pth

‎.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Sentiment_Streaming_DistilBert_Base_Uncased.txt‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
--region=us-central1
18+
--machine_type=n1-standard-2
19+
--num_workers=20
20+
--max_num_workers=250
21+
--disk_size_gb=50
22+
--autoscaling_algorithm=THROUGHPUT_BASED
23+
--staging_location=gs://temp-storage-for-perf-tests/loadtests
24+
--temp_location=gs://temp-storage-for-perf-tests/loadtests
25+
--requirements_file=apache_beam/ml/inference/torch_tests_requirements.txt
26+
--publish_to_big_query=true
27+
--metrics_dataset=beam_run_inference
28+
--metrics_table=torch_sentiment_distilbert_base_uncased
29+
--input_options={}
30+
--influx_measurement=torch_sentiment_distilbert_base_uncased
31+
--device=CPU
32+
--input_file=gs://apache-beam-ml/testing/inputs/sentences_50k.txt
33+
--runner=DataflowRunner
34+
--model_path=distilbert-base-uncased-finetuned-sst-2-english
35+
--model_state_dict_path=gs://apache-beam-ml/models/huggingface.sentiment.distilbert-base-uncased.pth

‎.test-infra/tools/refresh_looker_metrics.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@
4040
("78", ["243", "244", "245", "246", "247"]), # PyTorch Resnet 101
4141
("79", ["248", "249", "250", "251", "252"]), # PyTorch Resnet 152
4242
("80", ["253", "254", "255", "256", "257"]), # PyTorch Resnet 152 Tesla T4
43+
("82", ["263", "264", "265", "266", "267"]), # PyTorch Sentiment Streaming DistilBERT base uncased
44+
("85", ["268", "269", "270", "271", "272"]), # PyTorch Sentiment Batch DistilBERT base uncased
4345
]
4446

4547

‎sdks/python/apache_beam/examples/inference/pytorch_language_modeling.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
# limitations under the License.
1616
#
1717

18-
""""A pipeline that uses RunInference to perform Language Modeling with Bert.
18+
"""A pipeline that uses RunInference to perform Language Modeling with Bert.
1919
2020
This pipeline takes sentences from a custom text file, converts the last word
2121
of the sentence into a [MASK] token, and then uses the BertForMaskedLM from

0 commit comments

Comments
 (0)