Skip to content

Commit 70debdb

Browse files
author
gongenlei
authored
fix: change RELU to GELU for FT gpt (PaddlePaddle#1048)
1 parent da5a62b commit 70debdb

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

paddlenlp/ops/patches/FasterTransformer/open_decoder.h

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,8 @@ class OpenDecoder {
9696
int size_per_head,
9797
int memory_hidden_units,
9898
bool normalization_before = true,
99-
ActivationType act = ActivationType::RELU)
99+
ActivationType act = ActivationType::GELU)
100+
// Activation function default to GELU for GPT.
100101
: batch_size_(batch_size),
101102
max_seq_len_(seq_len),
102103
head_num_(head_num),

0 commit comments

Comments
 (0)