Skip to content

Commit 029d7c7

Browse files
authored
Revert unresolved changes in Bias initialization
it should be zeros_ or parameterized in future properly.
1 parent cc56df9 commit 029d7c7

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

modules/hypernetworks/hypernetwork.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ def __init__(self, dim, state_dict=None, layer_structure=None, activation_func=N
7575
w, b = layer.weight.data, layer.bias.data
7676
if weight_init == "Normal" or type(layer) == torch.nn.LayerNorm:
7777
normal_(w, mean=0.0, std=0.01)
78-
normal_(b, mean=0.0, std=0.005)
78+
normal_(b, mean=0.0, std=0)
7979
elif weight_init == 'XavierUniform':
8080
xavier_uniform_(w)
8181
zeros_(b)

0 commit comments

Comments
 (0)