Skip to content

Commit fbdca81

Browse files
mchoi8739mchoi8739
authored
Refactor the Debugger detect_stalled_training_job_and_stop.ipynb notebook (#1592)
* publish BYOC with Debugger notebook * some test change * revert the kernel names in the metadata * fix typos * incorporate feedback * incorporate comments * pin to pysdk v1 * remove installation output logs * refactor the stalled training job notebook * remove unnecessary module imports / minor fix * incorporate feedback * minor fix * fix typo * minor fix * fix unfinished sentence * incorporate feedback * minor fix Co-authored-by: Miyoung Choi <[email protected]>
1 parent aa5c9c2 commit fbdca81

File tree

3 files changed

+138
-253
lines changed

3 files changed

+138
-253
lines changed

‎sagemaker-debugger/tensorflow_action_on_rule/detect_stalled_training_job_and_stop.ipynb

Lines changed: 104 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -4,30 +4,24 @@
44
"cell_type": "markdown",
55
"metadata": {},
66
"source": [
7-
"# Detect stalled training and stop training job using debugger rule\n",
7+
"# Detect Stalled Training and Stop Training Job Using SageMaker Debugger Rule\n",
88
" \n",
9+
"This notebook shows you how to use the `StalledTrainingRule` built-in rule. This rule can take an action to stop your training job, when the rule detects an inactivity in your training job for a certain time period. This functionality helps you monitor the training job status and reduces redundant resource usage.\n",
910
"\n",
10-
"In this notebook, we'll show you how you can use StalledTrainingRule rule which can take action like stopping your training job when it finds that there has been no update in training job for certain threshold duration.\n",
11+
"## How the StalledTrainingRule Built-in Rule Works\n",
1112
"\n",
12-
"## How does StalledTrainingRule works?\n",
13+
"Amazon Sagemaker Debugger captures tensors that you want to watch from training jobs on [AWS Deep Learning Containers](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html#debugger-supported-aws-containers) or your local machine. If you use one of the Debugger-integrated Deep Learning Containers, you don't need to make any changes to your training script to use the functionality of built-in rules. For information about Debugger-supported SageMaker frameworks and versions, see [Debugger-supported framework versions for zero script change](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md#zero-script-change). \n",
1314
"\n",
14-
"Amazon Sagemaker debugger automatically captures tensors from training job which use AWS DLC(tensorflow, pytorch, mxnet, xgboost)[refer doc for supported versions](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md#zero-script-change). StalledTrainingRule keeps watching on emission of tensors like loss. The execution happens outside of training containers. It is evident that if training job is running good and is not stalled it is expected to emit loss and metrics tensors at frequent intervals. If Rule doesn't find new tensors being emitted from training job for threshold period of time, it takes automatic action to issue StopTrainingJob.\n",
15+
"If you want to run a training script that uses partially supported framework by Debugger or your own custom container, you need to manually register the Debugger hook to your training script. The `smdebug` library provides tools to help the hook registration, and the sample script provided in the `src` folder includes the hook registration code as comment lines. For more information about how to manually register the Debugger hooks for this case, see the training script at `./src/simple_stalled_training.py`, and documentation at [smdebug TensorFlow hook](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/tensorflow.md), [smdebug PyTorch hook](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/pytorch.md), [smdebug MXNet hook](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/mxnet.md), and [smdebug XGBoost hook](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/xgboost.md).\n",
1516
"\n",
16-
"#### With no changes to your training script\n",
17-
"If you use one of the SageMaker provided [Deep Learning Containers](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html). [Refer doc for supported framework versions](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md#zero-script-change), then you don't need to make any changes to your training script for activating this rule. Loss tensors will automatically be captured and monitored by the rule.\n",
18-
"\n",
19-
"You can also emit tensors periodically by using [save scalar api of hook](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#common-hook-api) . \n",
20-
"\n",
21-
"Also look at example how to use save_scalar api [here](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_fit_non_eager.py#L42)"
17+
"The Debugger `StalledTrainingRule` watches tensor updates from your training job. If the rule doesn't find new tensors updated to the default S3 URI for a threshold period of time, it takes an action to trigger the `StopTrainingJob` API operation. The following code cells set up a SageMaker TensorFlow estimator with the Debugger `StalledTrainingRule` to watch the `losses` pre-built tensor collection."
2218
]
2319
},
2420
{
25-
"cell_type": "code",
26-
"execution_count": null,
21+
"cell_type": "markdown",
2722
"metadata": {},
28-
"outputs": [],
2923
"source": [
30-
"! pip install -q sagemaker"
24+
"### Import SageMaker Python SDK"
3125
]
3226
},
3327
{
@@ -36,21 +30,16 @@
3630
"metadata": {},
3731
"outputs": [],
3832
"source": [
39-
"import boto3\n",
40-
"import os\n",
4133
"import sagemaker\n",
4234
"from sagemaker.tensorflow import TensorFlow\n",
4335
"print(sagemaker.__version__)"
4436
]
4537
},
4638
{
47-
"cell_type": "code",
48-
"execution_count": null,
39+
"cell_type": "markdown",
4940
"metadata": {},
50-
"outputs": [],
5141
"source": [
52-
"from sagemaker.debugger import Rule, DebuggerHookConfig, TensorBoardOutputConfig, CollectionConfig\n",
53-
"import smdebug_rulesconfig as rule_configs"
42+
"### Import SageMaker Debugger classes for rule configuration"
5443
]
5544
},
5645
{
@@ -59,24 +48,23 @@
5948
"metadata": {},
6049
"outputs": [],
6150
"source": [
62-
"# define the entrypoint script\n",
63-
"# Below script has 5 minutes sleep, we will create a stalledTrainingRule with 3 minutes of threshold.\n",
64-
"entrypoint_script='src/simple_stalled_training.py'\n",
65-
"\n",
66-
"# these hyperparameters ensure that vanishing gradient will trigger for our tensorflow mnist script\n",
67-
"hyperparameters = {\n",
68-
" \"num_epochs\": \"10\",\n",
69-
" \"lr\": \"10.00\"\n",
70-
"}"
51+
"from sagemaker.debugger import Rule, CollectionConfig, rule_configs"
7152
]
7253
},
7354
{
7455
"cell_type": "markdown",
7556
"metadata": {},
7657
"source": [
77-
"### Create unique training job prefix\n",
78-
"We will create unique training job name prefix. this prefix would be passed to StalledTrainingRule to identify which training job, rule should take action on once the stalled training rule condition is met.\n",
79-
"Note that, this prefix needs to be unique. If rule doesn't find exactly one job with provided prefix, it will fallback to safe mode and not take action of stop training job. Rule will still emit a cloudwatch event if the rule condition is met. To see details about cloud watch event, check [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger/tensorflow_action_on_rule/tf-mnist-stop-training-job.ipynb). "
58+
"### Create a unique training job prefix\n",
59+
"A unique prefix must be specified for `StalledTrainingRule` to identify the exact training job name that you want to monitor and stop when the rule triggers the stalled training job issue.\n",
60+
"If there are multiple training jobs sharing the same prefix, this rule may react to other training jobs. If the rule cannot find the exact training job name with a provided prefix, it falls back to safe mode and does not stop the training job. The rule evaluation process goes on in parallel while the training jobs are running. If you want to access the rule job logs, you will later find how to get the information at [Get a direct Amazon CloudWatch URL to find the current rule processing job log](#cw-url).\n",
61+
"\n",
62+
"The following code cell includes:\n",
63+
"* a code line to create a unique `base_job_name_prefix`\n",
64+
"* a stalled training job rule configuration object\n",
65+
"* a SageMaker TensorFlow estimator configuration with the Debugger `rules` parameter to run the built-in rule\n",
66+
"\n",
67+
"**Note**: Debugger collects `loss` tensors by default every 500 steps."
8068
]
8169
},
8270
{
@@ -85,13 +73,35 @@
8573
"metadata": {},
8674
"outputs": [],
8775
"source": [
76+
"# Append current time to your training job name to generate a unique base_job_name_prefix\n",
8877
"import time\n",
89-
"print(int(time.time()))\n",
90-
"# Note that sagemaker appends date to your training job and truncates the provided name to 39 character. So, we will make \n",
91-
"# sure that we use less than 39 character in below prefix. Appending time is to provide a unique id\n",
9278
"base_job_name_prefix= 'smdebug-stalled-demo-' + str(int(time.time()))\n",
93-
"base_job_name_prefix = base_job_name_prefix[:34]\n",
94-
"print(base_job_name_prefix)"
79+
"\n",
80+
"# Configure a StalledTrainingRule rule parameter object\n",
81+
"stalled_training_job_rule = [\n",
82+
" Rule.sagemaker(\n",
83+
" base_config=rule_configs.stalled_training_rule(),\n",
84+
" rule_parameters={\n",
85+
" \"threshold\": \"120\", \n",
86+
" \"stop_training_on_fire\": \"True\",\n",
87+
" \"training_job_name_prefix\": base_job_name_prefix\n",
88+
" }\n",
89+
" )\n",
90+
"]\n",
91+
"\n",
92+
"# Configure a SageMaker TensorFlow estimator\n",
93+
"estimator = TensorFlow(\n",
94+
" role=sagemaker.get_execution_role(),\n",
95+
" base_job_name=base_job_name_prefix,\n",
96+
" train_instance_count=1,\n",
97+
" train_instance_type='ml.m5.4xlarge',\n",
98+
" entry_point='src/simple_stalled_training.py', # This sample script forces the training job to sleep for 10 minutes\n",
99+
" framework_version='1.15.0',\n",
100+
" py_version='py3',\n",
101+
" train_max_run=3600,\n",
102+
" ## Debugger-specific parameter\n",
103+
" rules = stalled_training_job_rule\n",
104+
")"
95105
]
96106
},
97107
{
@@ -100,41 +110,26 @@
100110
"metadata": {},
101111
"outputs": [],
102112
"source": [
103-
"stalled_training_job_rule = Rule.sagemaker(\n",
104-
" base_config={\n",
105-
" 'DebugRuleConfiguration': {\n",
106-
" 'RuleConfigurationName': 'StalledTrainingRule', \n",
107-
" 'RuleParameters': {'rule_to_invoke': 'StalledTrainingRule'}\n",
108-
" }\n",
109-
" },\n",
110-
" rule_parameters={\n",
111-
" 'threshold': '120',\n",
112-
" 'training_job_name_prefix': base_job_name_prefix,\n",
113-
" 'stop_training_on_fire' : 'True'\n",
114-
" }, \n",
115-
")"
113+
"estimator.fit(wait=False)"
116114
]
117115
},
118116
{
119-
"cell_type": "code",
120-
"execution_count": null,
117+
"cell_type": "markdown",
121118
"metadata": {},
122-
"outputs": [],
123119
"source": [
124-
"estimator = TensorFlow(\n",
125-
" role=sagemaker.get_execution_role(),\n",
126-
" base_job_name=base_job_name_prefix,\n",
127-
" train_instance_count=1,\n",
128-
" train_instance_type='ml.m5.4xlarge',\n",
129-
" entry_point=entrypoint_script,\n",
130-
" #source_dir = 'src',\n",
131-
" framework_version='1.15.0',\n",
132-
" py_version='py3',\n",
133-
" train_max_run=3600,\n",
134-
" script_mode=True,\n",
135-
" ## New parameter\n",
136-
" rules = [stalled_training_job_rule]\n",
137-
")\n"
120+
"## Monitoring Training and Rule Evaluation Status\n",
121+
"\n",
122+
"Once you execute the `estimator.fit()` API, SageMaker initiates a training job in the background, and Debugger initiates a `StalledTrainingRule` rule evaluation job in parallel.\n",
123+
"Because the training scripts has a few lines of code at the end to force a sleep mode for 10 minutes, the `RuleEvaluationStatus` for `StalledTrainingRule` will change to `IssuesFound` in 2 minutes after the sleep mode is on and trigger the `StopTrainingJob` API."
124+
]
125+
},
126+
{
127+
"cell_type": "markdown",
128+
"metadata": {},
129+
"source": [
130+
"### Print the training job name\n",
131+
"\n",
132+
"The following cell outputs the training job name and its training status running in the background."
138133
]
139134
},
140135
{
@@ -143,23 +138,21 @@
143138
"metadata": {},
144139
"outputs": [],
145140
"source": [
146-
"# After calling fit, SageMaker will spin off 1 training job and 1 rule job for you\n",
147-
"# The rule evaluation status(es) will be visible in the training logs\n",
148-
"# at regular intervals\n",
149-
"# wait=False makes this a fire and forget function. To stream the logs in the notebook leave this out\n",
141+
"job_name = estimator.latest_training_job.name\n",
142+
"print('Training job name: {}'.format(job_name))\n",
150143
"\n",
151-
"estimator.fit(wait=True)"
144+
"client = estimator.sagemaker_session.sagemaker_client\n",
145+
"\n",
146+
"description = client.describe_training_job(TrainingJobName=job_name)"
152147
]
153148
},
154149
{
155150
"cell_type": "markdown",
156151
"metadata": {},
157152
"source": [
158-
"## Monitoring\n",
153+
"### Output the current job status and the rule evaluation status\n",
159154
"\n",
160-
"SageMaker kicked off rule evaluation job `StalledTrainingRule` as specified in the estimator. \n",
161-
"Given that we've stalled our training script for 10 minutes such that `StalledTrainingRule` is bound to fire and take action StopTrainingJob, we should expect to see the `TrainingJobStatus` as\n",
162-
"`Stopped` once the `RuleEvaluationStatus` for `StalledTrainingRule` changes to `IssuesFound`"
155+
"The following cell tracks the status of training job until the `SecondaryStatus` changes to `Stopped` or `Completed`. While training, Debugger collects output tensors from the training job and monitors the training job with the rules. "
163156
]
164157
},
165158
{
@@ -168,9 +161,30 @@
168161
"metadata": {},
169162
"outputs": [],
170163
"source": [
171-
"# rule job summary gives you the summary of the rule evaluations. You might have to run it over \n",
172-
"# a few times before you start to see all values populated/changing\n",
173-
"estimator.latest_training_job.rule_job_summary()"
164+
"import time\n",
165+
"\n",
166+
"if description['TrainingJobStatus'] != 'Completed':\n",
167+
" while description['SecondaryStatus'] not in {'Stopped', 'Completed'}:\n",
168+
" description = client.describe_training_job(TrainingJobName=job_name)\n",
169+
" primary_status = description['TrainingJobStatus']\n",
170+
" secondary_status = description['SecondaryStatus']\n",
171+
" print('Current job status: [PrimaryStatus: {}, SecondaryStatus: {}] | {} Rule Evaluation Status: {}'\n",
172+
" .format(primary_status, secondary_status, \n",
173+
" estimator.latest_training_job.rule_job_summary()[0][\"RuleConfigurationName\"],\n",
174+
" estimator.latest_training_job.rule_job_summary()[0][\"RuleEvaluationStatus\"]\n",
175+
" )\n",
176+
" )\n",
177+
" time.sleep(15)"
178+
]
179+
},
180+
{
181+
"cell_type": "markdown",
182+
"metadata": {},
183+
"source": [
184+
"<a class=\"anchor\" id=\"cw-url\"></a>\n",
185+
"### Get a direct Amazon CloudWatch URL to find the current rule processing job log\n",
186+
"\n",
187+
"The following script returns a CloudWatch URL. Copy the URL and Paste it to a browser. This will directly lead you to the rule job log page."
174188
]
175189
},
176190
{
@@ -203,40 +217,34 @@
203217
" result[status[\"RuleConfigurationName\"]] = _get_cw_url_for_rule_job(rule_job_name, region)\n",
204218
" return result\n",
205219
"\n",
206-
"get_rule_jobs_cw_urls(estimator)"
220+
"print(\n",
221+
" \"The direct CloudWatch URL to the current rule job:\", \n",
222+
" get_rule_jobs_cw_urls(estimator)[estimator.latest_training_job.rule_job_summary()[0][\"RuleConfigurationName\"]]\n",
223+
")"
207224
]
208225
},
209226
{
210227
"cell_type": "markdown",
211228
"metadata": {},
212229
"source": [
213-
"After running the last two cells over and until `VanishingGradient` reports `IssuesFound`, we'll attempt to describe the `TrainingJobStatus` for our training job."
230+
"## Conclusion\n",
231+
"\n",
232+
"This notebook showed how you can use the Debugger `StalledTrainingRule` built-in rule for your training job to take action on rule evaluation status changes. To find more information about Debugger, see [Amazon SageMaker Debugger Developer Guide](https://integ-docs-aws.amazon.com/sagemaker/latest/dg/train-debugger.html) and the [smdebug GitHub documentation](https://github.com/awslabs/sagemaker-debugger)."
214233
]
215234
},
216235
{
217236
"cell_type": "code",
218237
"execution_count": null,
219238
"metadata": {},
220239
"outputs": [],
221-
"source": [
222-
"estimator.latest_training_job.describe()[\"TrainingJobStatus\"]"
223-
]
224-
},
225-
{
226-
"cell_type": "markdown",
227-
"metadata": {},
228-
"source": [
229-
"## Result\n",
230-
"\n",
231-
"This notebook attempted to show a very simple setup of how you can use CloudWatch events for your training job to take action on rule evaluation status changes. Learn more about Amazon SageMaker Debugger in the [GitHub Documentation](https://github.com/awslabs/sagemaker-debugger)."
232-
]
240+
"source": []
233241
}
234242
],
235243
"metadata": {
236244
"kernelspec": {
237-
"display_name": "Python 3",
245+
"display_name": "conda_tensorflow_p36",
238246
"language": "python",
239-
"name": "python3"
247+
"name": "conda_tensorflow_p36"
240248
},
241249
"language_info": {
242250
"codemirror_mode": {

0 commit comments

Comments
 (0)