[Question] Hyperparameter tuning

[VikasChidananda]

Question

Hey everyone,

I am trying add SAC from skrl to quadcopter_env for my experiments. I was wondering how was hyperparameters for other algorithms (PPO, AMP) determined? Is it through extensive hyperparameter tuning for each environment?
I found some cool guides from Raffin's blog post and hyperparameters for TD3 suggested by @Toni-SM which were helpful but I was not successful in getting some signs of life in learning.

I changed a bit from default hyperparameters from skrl and rescaled action_space to [-1,1] (by default action and observation space are (-inf, inf)) which still didn't help me. If anyone has an idea how you did for your own environments, any advice would be helpful.

Cheers!

seed: 42

models:
    separate: True
    policy:
        class: GaussianMixin
        clip_actions: False
        clip_log_std: True
        min_log_std: -20.0
        max_log_std: 2.0
        initial_log_std: 0.0
        network: 
            -   name: net
                input: STATES
                layers: [32, 32]
                activation: elu
        output: tanh(ACTIONS)
    critic_1:
        class: DeterministicMixin
        clip_actions: False
        network:
            -   name: cnet1
                input: concatenate([STATES, ACTIONS])
                layers: [32, 32]
                activation: elu
        output: ONE
    critic_2:
        class: DeterministicMixin
        clip_actions: False
        network:
            -   name: cnet2
                input: concatenate([STATES, ACTIONS])
                layers: [32, 32]
                activation: elu
        output: ONE 
    target_critic_1:
        class: DeterministicMixin
        clip_actions: False
        network: 
            -   name: tnet1
                input: concatenate([STATES, ACTIONS])
                layers: [32, 32]
                activation: elu
        output: ONE
    target_critic_2:
        class: DeterministicMixin
        clip_actions: False 
        network: 
            -   name: tnet2
                input: concatenate([STATES, ACTIONS])
                layers: [32, 32]
                activation: elu
        output: ONE

memory:
    class: RandomMemory
    memory_size: -1

agent:
    class: SAC
    rollouts: 32
    gradient_steps: 64            # gradient steps
    batch_size: 64               # training batch size
    discount_factor: 0.99        # discount factor (gamma)
    polyak: 0.005                # soft update hyperparameter (tau)
    actor_learning_rate: 5.0e-4    # actor learning rate
    critic_learning_rate: 5.0e-4   # critic learning rate
    learning_rate_scheduler: null       # learning rate scheduler class (see torch.optim.lr_scheduler)
    learning_rate_scheduler_kwargs:    # learning rate scheduler's kwargs (e.g. {"step_size": 1e-3})
        kl_threshold: 8e-3
    state_preprocessor: RunningStandardScaler
    state_preprocessor_kwargs: null
    value_preprocessor: RunningStandardScaler
    value_preprocessor_kwargs: null

    random_timesteps: 100          # random exploration steps
    learning_starts: 1000           # learning starts after this many steps
    grad_norm_clip: 1.0            # clipping coefficient for the norm of the gradients

    learn_entropy: True          # learn entropy
    entropy_learning_rate: 0.001  # entropy learning rate
    initial_entropy_value: 0.02   # initial entropy value
    target_entropy: null         # target entropy
    rewards_shaper: None         # rewards shaping function: Callable(reward, timestep, timesteps) -> reward
    mixed_precision: True       # enable automatic mixed precision for higher performance
    experiment: 
        directory: SAC            # experiment's parent directory
        experiment_name: ""      # experiment name
        write_interval: auto   # TensorBoard writing interval (timesteps)
        checkpoint_interval: auto      # interval for checkpoints (timesteps)

trainer:
    class: SequentialTrainer
    timesteps: 4000
    environment_info: log

[RandomOakForest]

Thank you for posting this question. I'll move it to our Discussions for the team to follow up.