Skip to content

Blending activation #173

@sdatkinson

Description

@sdatkinson

New activation, "Blend", which is similar to the gating activation.

Input is of dimension $2d$, and output is dimension $d$.

The activation holds two sub-activations. The first is the "base" activation (cf. Tanh in WaveNet), and the other is the "blend" activation (like how a sigmoid is often used in a gating activation).

The output is

$y_i = \alpha x_i + (1-\alpha) a_1(x_i)$,

where $a_1(\cdot)$ is the base activation and $\alpha=a_2(x_{i+d})$ is the blending parameter. A value of $\alpha=0$ removes the activation, and $\alpha=1$ uses the base activation as-is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions