Nemo-run Processor for TopIPL training

[nune-tadevosyan]

Added three processors for training and inference command generation and one NeMo-Run processor specifically for IPL training.
Config file should be provided for NeMo-Run processor with the following parameters.

# The script to be run.
script: # Script path  to run relative to directory 
script_config: # Training config file for the script. ipl_epoch_stopper_callback should be provided in the config
inference_config: # Inference config file of unlabeled data for transcribe_speech_parallel

exp_name: null  # populated by exp_manager.name if not provided
results_dir: # Where to store the results of the run

nemo_directory: # Nemo directory path
do_average: # Boolean value indicating whether to do average of checkpoints for pseudo-label generation
p_cache: # Probability with which update pseudo-labeled set
num_ipl_epochs: How many epochs do pseudo-labeling

# Optional arguments
num_runs: 
num_gpus: 
num_tasks_per_node: 
max_runtime: # Specify for clusters

########################################################################################################################

executor: slurm # or local

USER: ntadevosyan

# Fields for cluster run
ssh_tunnel:
  host: 
  # ------------------------------- Fill this up! -------------------------------
  user: "${USER}"  # your username; or resolved from ${USER} environment variable ; or can be null which resolved from ${USER} environment variable
  job_dir: ""
  identity: ""
  # -----------------------------------------------------------------------------

account: 
partition:
job_name_prefix: 

containers:
  asr: # Container image


env_vars:
  - 'TOKENIZERS_PARALLELISM='
  - 'AIS_ENDPOINT="
  - 'LHOTSE_AUDIO_DURATION_MISMATCH_TOLERANCE='
  - 'TORCH_CUDNN_V8_API_ENABLED='
  - 'PYTORCH_CUDA_ALLOC_CONF='
  - 'HYDRA_FULL_ERROR=1'

required_env_vars:
  - 'HF_TOKEN='
  - 'WANDB_KEY=' 

mounts:
  # Replace with your own paths in your cluster config
  - /path/to/mount:/where/to/mount/

timeouts:
  partition_name: # Specify time

[karpnv]

Need to add yaml config example and readme

[karpnv]

Add a requirements file with all needed dependencies, including NeMo version

[lilithgrigoryan]

Overall, LGTM. Thank you.
Please add copyright files where needed and pull changes from main.

And please fix DCO test

[Jorjeous]

Good morning Nune!
Please do
'''git checkout origin/main -- docker/Dockerfile'''
For Docker tests to fix

[karpnv]

git pull origin mian should fix tests

[karpnv]

pull the latest vertion in main to pass the tests

[Jorjeous]

config.rst
Please fix warnings in doctests related to file mentioned above
(line interval, etc)

[Jorjeous]

I see that now we have some ubuntu related problems

[Jorjeous]

Please add example section

[Jorjeous]

Please add example section
like in this PR
https://github.com/NVIDIA/NeMo-speech-data-processor/pull/115/files#diff-424af7846bc8503aed8b73bc8e234ce9b50e7fa7ac6f1408cc5fdeedbd4c61b9

[Jorjeous]

Please add example section

[karpnv]

LGTM

[Jorjeous]

We forgot example section...