Fix gelu (#1287)

* fix leaky_relu

* update mxnet as 0b20200802

Author: zheyuye

README.md

@@ -20,16 +20,16 @@ First of all, install the latest MXNet. You may use the following commands:
 
 ```bash
 # Install the version with CUDA 10.0
-pip install -U --pre "mxnet-cu100>=2.0.0b20200716" -f https://dist.mxnet.io/python
+pip install -U --pre "mxnet-cu100>=2.0.0b20200802" -f https://dist.mxnet.io/python
 
 # Install the version with CUDA 10.1
-pip install -U --pre "mxnet-cu101>=2.0.0b20200716" -f https://dist.mxnet.io/python
+pip install -U --pre "mxnet-cu101>=2.0.0b20200802" -f https://dist.mxnet.io/python
 
 # Install the version with CUDA 10.2
-pip install -U --pre "mxnet-cu102>=2.0.0b20200716" -f https://dist.mxnet.io/python
+pip install -U --pre "mxnet-cu102>=2.0.0b20200802" -f https://dist.mxnet.io/python
 
 # Install the cpu-only version
-pip install -U --pre "mxnet>=2.0.0b20200716" -f https://dist.mxnet.io/python
+pip install -U --pre "mxnet>=2.0.0b20200802" -f https://dist.mxnet.io/python
 ```
 
 
@@ -42,7 +42,7 @@ pip install -U -e .
 pip install -U -e .[extras]
 
 # In case you are using zsh, try to use the following command for installing
-pip install -U -e ."[extras]" 
+pip install -U -e ."[extras]"
 ```
 
 If you find that you do not have the permission, you can also install to the user folder:
@@ -51,12 +51,12 @@ If you find that you do not have the permission, you can also install to the use
 pip install -U -e . --user
 ```
 
-For Windows users, we recommend to use the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/about). 
+For Windows users, we recommend to use the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/about).
 
 
 # Access the Command-line Toolkits
 
-To facilitate the researcher and the engineers, we provide command-line-toolkits for 
+To facilitate the researcher and the engineers, we provide command-line-toolkits for
 downloading and preprocessing the NLP datasets. For more details, you may refer to
  [GluonNLP Datasets](./scripts/datasets) and [GluonNLP Preprocessing Tools](./scripts/preprocess).
 

src/gluonnlp/layers.py

@@ -356,8 +356,7 @@ def __init__(self, mode='erf'):
 
     def hybrid_forward(self, F, x):
         if self._mode == 'erf':
-            # TODO Investigate the precision of F.npx.leaky_relu(x, act_type='gelu')
-            return x * 0.5 * (1.0 + F.npx.erf(x / math.sqrt(2.0)))
+            return F.npx.leaky_relu(x, act_type='gelu')
         elif self._mode == 'tanh':
             return 0.5 * x\
                    * (1.0 + F.np.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * (x ** 3))))