Skip to content

Commit 7514d2b

Browse files
silverlakelifacebook-github-bot
authored andcommitted
Obey training.planner.log_plan flag on if to log sharding plan (#3244)
Summary: Pull Request resolved: #3244 ## Why Sharding plan can be verbose and we used to control if to log it via training.planner.log_plan, which creates NoopEmbeddingStats when is False and valid stats otherwise: https://www.internalfb.com/code/fbsource/[375d786e20a4]/fbcode/apf/rec/distributed/parallel.py?lines=872-882 D74557713 adds a new logging on the raw sharding plan without following this flag. This diffs guards it with the flag. ## What Only log the raw sharding plan when there is valid stats (i.e. not NoopEmbeddingStats). Reviewed By: ge0405 Differential Revision: D79271421 fbshipit-source-id: bc16a5bf4349780a9f316a5f6ccc65489e77672a
1 parent 5c8e5e2 commit 7514d2b

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

torchrec/distributed/planner/planners.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
GridSearchProposer,
3333
UniformProposer,
3434
)
35-
from torchrec.distributed.planner.stats import EmbeddingStats
35+
from torchrec.distributed.planner.stats import EmbeddingStats, NoopEmbeddingStats
3636
from torchrec.distributed.planner.storage_reservations import (
3737
HeuristicalStorageReservation,
3838
)
@@ -513,7 +513,10 @@ def plan(
513513
sharding_plan = to_sharding_plan(best_plan, self._topology)
514514

515515
end_time = perf_counter()
516+
shall_log_sharding_plan = False
516517
for stats in self._stats:
518+
if not isinstance(stats, NoopEmbeddingStats):
519+
shall_log_sharding_plan = True
517520
stats.log(
518521
sharding_plan=sharding_plan,
519522
topology=self._topology,
@@ -528,7 +531,8 @@ def plan(
528531
sharders=sharders,
529532
debug=self._debug,
530533
)
531-
logger.info(f"Found sharding plan {sharding_plan}")
534+
if shall_log_sharding_plan:
535+
logger.info(f"Found sharding plan {sharding_plan}")
532536
return sharding_plan
533537
else:
534538
global_storage_capacity = reduce(

0 commit comments

Comments
 (0)