ValueError when running run_connectivity_sampling.py #1
Description
Hi @oozdenizci and thanks for your work on this repo!
Unfortunately, I am currently facing an issue when running run_connectivity_sampling.py that I am struggling to solve. The launched command is:
python -u run_connectivity_sampling.py --data cifar10 --model resnet18 --n_classes 10 -s -pc 0.05 --w_decay 1e-3 --objective "at"
The error happens while the training process is about to start apparently, as no log of ongoing epochs appear.
The line of code from which the error stems from is:
optimizer.apply_gradients(zip(grads, model.param_list(trainable=True)))
Error Log
Traceback (most recent call last):
File "run_connectivity_sampling.py", line 176, in <module>
main()
File "run_connectivity_sampling.py", line 129, in main
train_step(batch_xs_adv, batch_ys)
File "run_connectivity_sampling.py", line 100, in train_step
optimizer.apply_gradients(zip(grads, model.param_list(trainable=True)))
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow_addons/optimizers/weight_decay_optimizers.py", line 154, in apply_gradients
return super().apply_gradients(grads_and_vars, name=name, **kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 504, in apply_gradients
return distribute_ctx.get_replica_context().merge_call(
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2420, in merge_call
return self._merge_call(merge_fn, args, kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2427, in _merge_call
return merge_fn(self._strategy, *args, **kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 282, in wrapper
return func(*args, **kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 591, in _distributed_apply
update_ops.extend(distribution.extended.update(
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2013, in update
return self._update(var, fn, args, kwargs, group)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2659, in _update
return self._update_non_slot(var, fn, (var,) + tuple(args), kwargs, group)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2665, in _update_non_slot
result = fn(*args, **kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 282, in wrapper
return func(*args, **kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 567, in apply_grad_to_update_var
update_op = self._resource_apply_dense(grad, var, **apply_kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow_addons/optimizers/weight_decay_optimizers.py", line 175, in _resource_apply_dense
with tf.control_dependencies([self._decay_weights_op(var)]):
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow_addons/optimizers/weight_decay_optimizers.py", line 159, in _decay_weights_op
self._get_hyper("weight_decay", var.dtype) * var, self._use_locking
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 1072, in _run_op
return tensor_oper(a.value(), *args, **kwargs)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 1009, in r_binary_op_wrapper
x = ops.convert_to_tensor(x, dtype=y.dtype.base_dtype, name="x")
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
return _constant_impl(value, dtype, shape, name, verify_shape=False,
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File "/home/gpiras/anaconda3/envs/bcsp/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (<tensorflow.python.keras.optimizer_v2.learning_rate_schedule.PiecewiseConstantDecay object at 0x7f72b1ee8f10>) with an unsupported type (<class 'tensorflow.python.keras.optimizer_v2.learning_rate_schedule.PiecewiseConstantDecay'>) to a Tensor.
My env
The env should be compliant with the requirements. I'll list here the main packages:
cudatoolkit 10.1.243 h6bb024c_0 anaconda
cudnn 7.6.5 cuda10.1_0 anaconda
foolbox 3.3.1 pyh44b312d_1 conda-forge
keras 2.4.3 pyhd8ed1ab_0 conda-forge
keras-preprocessing 1.1.2 pyhd3eb1b0_0 anaconda
numpy 1.23.5 pypi_0 pypi
pickle-mixin 1.0.2 pypi_0 pypi
pip 23.3 py38h06a4308_0
python 3.8.18 h955ad1f_0
pyyaml 5.3 py38h516909a_0 conda-forge
tensorboard 2.2.0 pypi_0 pypi
tensorboard-data-server 0.7.0 py38h52d8a92_0 anaconda
tensorboard-plugin-wit 1.8.1 py38h06a4308_0 anaconda
tensorflow 2.2.0 gpu_py38hb782248_0 anaconda
tensorflow-addons 0.10.0 pypi_0 pypi
tensorflow-base 2.2.0 gpu_py38h83e3d50_0 anaconda
tensorflow-estimator 2.2.0 pypi_0 pypi
tensorflow-gpu 2.2.0 h0d30ee6_0 anaconda
tensorflow-probability 0.10.0 pypi_0 pypi
Any chance you could help me? Thanks