Need help with using WandB for hyperparameter optimization.

[kushuu]

Hey there! I'm building a NER model using this amazing library and I'm using weights and biases for hyperparameter optimization (wandb sweeps). My sweeps.yml file looks like this:

command:
  - ${env}
  - python3
  - sweeps_using_config.py
  - ./config.cfg
  - ./training
method: random
metric:
  goal: maximize
  name: ents_f
parameters:
  components.ner.model.state_type:
    distribution: categorical
    values:
      - ner
  corpora.dev.path:
    distribution: categorical
    values:
      - ./validation_file.spacy
  corpora.train.path:
    distribution: categorical
    values:
      - ./training_file.spacy
  nlp.batch_size:
    distribution: int_uniform
    max: 512
    min: 128
  paths.dev:
    distribution: categorical
    values:
      - ./validation_file.spacy
  paths.train:
    distribution: categorical
    values:
      - ./training_file.spacy
  training.batcher.discard_oversize:
    distribution: categorical
    values:
      - "true"
      - "false"
  training.batcher.size.start:
    distribution: int_uniform
    max: 200
    min: 50
  training.batcher.size.stop:
    distribution: int_uniform
    max: 2000
    min: 500
  training.batcher.tolerance:
    distribution: uniform
    max: 0.4
    min: 0.1
  training.dev_corpus:
    distribution: categorical
    values:
      - corpora.dev
  training.dropout:
    distribution: uniform
    max: 0.2
    min: 0.05
  training.max_steps:
    distribution: int_uniform
    max: 40000
    min: 10000
  training.optimizer.beta1:
    distribution: uniform
    max: 1.8
    min: 0.45
  training.optimizer.beta2:
    distribution: uniform
    max: 1.998
    min: 0.4995
  training.optimizer.eps:
    distribution: uniform
    max: 2e-08
    min: 5e-09
  training.optimizer.grad_clip:
    distribution: int_uniform
    max: 2
    min: 1
  training.optimizer.learn_rate:
    distribution: uniform
    max: 0.002
    min: 0.0005
  training.optimizer.use_averages:
    distribution: categorical
    values:
      - "true"
      - "false"
  training.patience:
    distribution: int_uniform
    max: 3200
    min: 800
  training.score_weights.ents_f:
    distribution: int_uniform
    max: 2
    min: 1
  training.train_corpus:
    distribution: categorical
    values:
      - corpora.train
project: address_ent_extraction

I'm getting an error TypeError: can't convert complex to float for this particular (and many other) combination of hyperparameters:

        components.ner.model.extra_state_tokens: true
	components.ner.model.state_type: ner
	corpora.dev.path: ./validation_file.spacy
	corpora.train.path: ./training_file.spacy
	nlp.batch_size: 350
	paths.dev: ./validation_file.spacy
	paths.train: ./training_file.spacy
	training.accumulate_gradient: 2
	training.batcher.discard_oversize: true
	training.batcher.size.compound: 1.0992073097473145
	training.batcher.size.start: 119
	training.batcher.size.stop: 1062
	training.batcher.tolerance: 0.20642158033454755
	training.dev_corpus: corpora.dev
	training.dropout: 0.054338843072065886
	training.max_steps: 29998
	training.optimizer.beta1: 1.6529881168330167
	training.optimizer.beta2: 1.772109466118724
	training.optimizer.eps: 1.409889683900334e-08
	training.optimizer.grad_clip: 2
	training.optimizer.learn_rate: 0.0017509784956132625
	training.optimizer.use_averages: false
	training.patience: 2275
	training.score_weights.ents_f: 2
	training.train_corpus: corpora.train

I'm not able to understand the cause of this error and what hyperparameters I'm tweaking incorrectly (or out of range maybe).
Also, I'm new with deep learning and spacy so kindly help with as many resources (and depth) as possible.
Thanks.

[ljvmiranda921]

Hi @kushuu ,

Can you show a sample traceback when the error occurs? Perhaps to help debug this problem, you can single out which particular hyperparameter causes this TypeError. One thing I'd do is to put all the single-value "hyperparameters" (like the training path, dev path, etc.) in the config file itself and retain only those hyperparameters you want to search upon. I also don't want to rule out the possibility that it may as well be a WandB problem, in that case, it might be better to check-in their forums.