paper url fix

Author: vpj

translate_cache/activations/fta/__init__.ja.json

@@ -1,5 +1,5 @@
 {
- "<h1>Fuzzy Tiling Activations (FTA)</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation/tutorial of <a href=\"https://papers.labml.ai/paper/1911.08068\">Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online</a>.</p>\n<p>Fuzzy tiling activations are a form of sparse activations based on binning.</p>\n<p>Binning is classification of a scalar value into a bin based on intervals. One problem with binning is that it gives zero gradients for most values (except at the boundary of bins). The other is that binning loses precision if the bin intervals are large.</p>\n<p>FTA overcomes these disadvantages. Instead of hard boundaries like in Tiling Activations, FTA uses soft boundaries between bins. This gives non-zero gradients for all or a wide range of values. And also doesn&#x27;t lose precision since it&#x27;s captured in partial values.</p>\n<h4>Tiling Activations</h4>\n<p><span translate=no>_^_1_^_</span> is the tiling vector,</p>\n<p><span translate=no>_^_2_^_</span></p>\n<p>where <span translate=no>_^_3_^_</span> is the input range, <span translate=no>_^_4_^_</span> is the bin size, and <span translate=no>_^_5_^_</span> is divisible by <span translate=no>_^_6_^_</span>.</p>\n<p>Tiling activation is,</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>where <span translate=no>_^_8_^_</span> is the indicator function which gives <span translate=no>_^_9_^_</span> if the input is positive and <span translate=no>_^_10_^_</span> otherwise.</p>\n<p>Note that tiling activation gives zero gradients because it has hard boundaries.</p>\n<h4>Fuzzy Tiling Activations</h4>\n<p>The fuzzy indicator function,</p>\n<p><span translate=no>_^_11_^_</span></p>\n<p>which increases linearly from <span translate=no>_^_12_^_</span> to <span translate=no>_^_13_^_</span> when <span translate=no>_^_14_^_</span> and is equal to <span translate=no>_^_15_^_</span> for <span translate=no>_^_16_^_</span>. <span translate=no>_^_17_^_</span> is a hyper-parameter.</p>\n<p>FTA uses this to create soft boundaries between bins.</p>\n<p><span translate=no>_^_18_^_</span></p>\n<p><a href=\"experiment.html\">Here&#x27;s a simple experiment</a> that uses FTA in a transformer.</p>\n": "<h1>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3 (FTA)</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u3053\u308c\u306f\u3001<a href=\"https://papers.labml.ai/paper/1911.08068\">\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3:\u30b9\u30d1\u30fc\u30b9\u8868\u73fe\u3092\u30aa\u30f3\u30e9\u30a4\u30f3\u3067\u5b66\u7fd2\u3059\u308b\u305f\u3081\u306e\u7c21\u5358\u306a\u30a2\u30d7\u30ed\u30fc\u30c1\u306e <a href=\"https://pytorch.org\">PyTorch</a></a> \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002</p>\n<p>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306f\u3001\u30d3\u30cb\u30f3\u30b0\u306b\u57fa\u3065\u304f\u30b9\u30d1\u30fc\u30b9\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306e\u4e00\u7a2e\u3067\u3059\u3002</p>\n<p>\u30d3\u30cb\u30f3\u30b0\u3068\u306f\u3001\u9593\u9694\u306b\u57fa\u3065\u3044\u3066\u30b9\u30ab\u30e9\u30fc\u5024\u3092\u30d3\u30f3\u306b\u5206\u985e\u3059\u308b\u3053\u3068\u3067\u3059\u3002\u30d3\u30cb\u30f3\u30b0\u306e\u554f\u984c\u306e 1 \u3064\u306f\u3001\u307b\u3068\u3093\u3069\u306e\u5024 (\u30d3\u30f3\u306e\u5883\u754c\u3092\u9664\u304f) \u3067\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u304c\u30bc\u30ed\u306b\u306a\u308b\u3053\u3068\u3067\u3059\u3002\u3082\u30461\u3064\u306f\u3001\u30d3\u30f3\u306e\u9593\u9694\u304c\u5927\u304d\u3044\u3068\u30d3\u30cb\u30f3\u30b0\u306e\u7cbe\u5ea6\u304c\u4f4e\u4e0b\u3059\u308b\u3053\u3068\u3067\u3059</p>\u3002\n<p>FTA\u306f\u3053\u308c\u3089\u306e\u6b20\u70b9\u3092\u514b\u670d\u3057\u307e\u3059\u3002FTA\u306f\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306e\u3088\u3046\u306a\u30cf\u30fc\u30c9\u30d0\u30a6\u30f3\u30c0\u30ea\u30fc\u306e\u4ee3\u308f\u308a\u306b\u3001\u30d3\u30f3\u306e\u9593\u306b\u30bd\u30d5\u30c8\u30d0\u30a6\u30f3\u30c0\u30ea\u30fc\u3092\u4f7f\u3044\u307e\u3059\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u3059\u3079\u3066\u307e\u305f\u306f\u5e83\u7bc4\u56f2\u306e\u5024\u306b\u5bfe\u3057\u3066\u30bc\u30ed\u4ee5\u5916\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u304c\u5f97\u3089\u308c\u307e\u3059\u3002\u307e\u305f\u3001\u90e8\u5206\u7684\u306a\u5024\u3067\u30ad\u30e3\u30d7\u30c1\u30e3\u3055\u308c\u308b\u305f\u3081\u3001\u7cbe\u5ea6\u304c\u5931\u308f\u308c\u308b\u3053\u3068\u306f\u3042\u308a\u307e\u305b\u3093\u3002</p>\n<h4>\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</h4>\n<p><span translate=no>_^_1_^_</span>\u306f\u30bf\u30a4\u30ea\u30f3\u30b0\u30d9\u30af\u30c8\u30eb\u3001</p>\n<p><span translate=no>_^_2_^_</span></p>\n<p>\u3053\u3053\u3067\u3001<span translate=no>_^_3_^_</span>\u306f\u5165\u529b\u7bc4\u56f2\u3001<span translate=no>_^_4_^_</span>\u306f\u30d3\u30f3\u306e\u30b5\u30a4\u30ba\u3001<span translate=no>_^_5_^_</span><span translate=no>_^_6_^_</span>\u3067\u5272\u308a\u5207\u308c\u307e\u3059\u3002</p>\n<p>\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306f\u3001</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>\u3053\u3053\u3067<span translate=no>_^_8_^_</span>\u3001\u5165\u529b\u304c\u6b63\u306e\u304b\u3069\u3046\u304b\u3092\u793a\u3059\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u95a2\u6570\u3068\u3001<span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>\u305d\u3046\u3067\u306a\u3044\u304b\u3069\u3046\u304b\u3092\u793a\u3059\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u95a2\u6570\u304c\u3042\u308a\u307e\u3059\u3002</p>\n<p>\u30bf\u30a4\u30ea\u30f3\u30b0\u3092\u6709\u52b9\u306b\u3059\u308b\u3068\u3001\u5883\u754c\u304c\u56fa\u3044\u305f\u3081\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306f\u30bc\u30ed\u306b\u306a\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n<h4>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</h4>\n<p>\u30d5\u30a1\u30b8\u30fc\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u6a5f\u80fd\u3001</p>\n<p><span translate=no>_^_11_^_</span></p>\n<p><span translate=no>_^_12_^_</span><span translate=no>_^_13_^_</span>\u3053\u308c\u306f\u304b\u3089\u3044\u3064\u307e\u3067\u76f4\u7dda\u7684\u306b\u5897\u52a0\u3057\u3001<span translate=no>_^_15_^_</span> for <span translate=no>_^_14_^_</span> <span translate=no>_^_16_^_</span> \u3068\u7b49\u3057\u304f\u306a\u308a\u307e\u3059\u3002<span translate=no>_^_17_^_</span>\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u3067\u3059</p>\u3002\n<p>FTA \u306f\u3053\u308c\u3092\u4f7f\u3063\u3066\u30d3\u30f3\u306e\u9593\u306b\u30bd\u30d5\u30c8\u306a\u5883\u754c\u7dda\u3092\u4f5c\u308a\u307e\u3059\u3002</p>\n<p><span translate=no>_^_18_^_</span></p>\n<p><a href=\"experiment.html\">\u3053\u308c\u306f\u3001\u5909\u5727\u5668\u3067FTA\u3092\u4f7f\u7528\u3059\u308b\u7c21\u5358\u306a\u5b9f\u9a13\u3067\u3059</a>\u3002</p>\n",
+ "<h1>Fuzzy Tiling Activations (FTA)</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation/tutorial of <a href=\"https://arxiv.org/abs/1911.08068\">Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online</a>.</p>\n<p>Fuzzy tiling activations are a form of sparse activations based on binning.</p>\n<p>Binning is classification of a scalar value into a bin based on intervals. One problem with binning is that it gives zero gradients for most values (except at the boundary of bins). The other is that binning loses precision if the bin intervals are large.</p>\n<p>FTA overcomes these disadvantages. Instead of hard boundaries like in Tiling Activations, FTA uses soft boundaries between bins. This gives non-zero gradients for all or a wide range of values. And also doesn&#x27;t lose precision since it&#x27;s captured in partial values.</p>\n<h4>Tiling Activations</h4>\n<p><span translate=no>_^_1_^_</span> is the tiling vector,</p>\n<p><span translate=no>_^_2_^_</span></p>\n<p>where <span translate=no>_^_3_^_</span> is the input range, <span translate=no>_^_4_^_</span> is the bin size, and <span translate=no>_^_5_^_</span> is divisible by <span translate=no>_^_6_^_</span>.</p>\n<p>Tiling activation is,</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>where <span translate=no>_^_8_^_</span> is the indicator function which gives <span translate=no>_^_9_^_</span> if the input is positive and <span translate=no>_^_10_^_</span> otherwise.</p>\n<p>Note that tiling activation gives zero gradients because it has hard boundaries.</p>\n<h4>Fuzzy Tiling Activations</h4>\n<p>The fuzzy indicator function,</p>\n<p><span translate=no>_^_11_^_</span></p>\n<p>which increases linearly from <span translate=no>_^_12_^_</span> to <span translate=no>_^_13_^_</span> when <span translate=no>_^_14_^_</span> and is equal to <span translate=no>_^_15_^_</span> for <span translate=no>_^_16_^_</span>. <span translate=no>_^_17_^_</span> is a hyper-parameter.</p>\n<p>FTA uses this to create soft boundaries between bins.</p>\n<p><span translate=no>_^_18_^_</span></p>\n<p><a href=\"experiment.html\">Here&#x27;s a simple experiment</a> that uses FTA in a transformer.</p>\n": "<h1>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3 (FTA)</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/activations/fta/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u3053\u308c\u306f\u3001<a href=\"https://arxiv.org/abs/1911.08068\">\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3:\u30b9\u30d1\u30fc\u30b9\u8868\u73fe\u3092\u30aa\u30f3\u30e9\u30a4\u30f3\u3067\u5b66\u7fd2\u3059\u308b\u305f\u3081\u306e\u7c21\u5358\u306a\u30a2\u30d7\u30ed\u30fc\u30c1\u306e <a href=\"https://pytorch.org\">PyTorch</a></a> \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002</p>\n<p>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306f\u3001\u30d3\u30cb\u30f3\u30b0\u306b\u57fa\u3065\u304f\u30b9\u30d1\u30fc\u30b9\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306e\u4e00\u7a2e\u3067\u3059\u3002</p>\n<p>\u30d3\u30cb\u30f3\u30b0\u3068\u306f\u3001\u9593\u9694\u306b\u57fa\u3065\u3044\u3066\u30b9\u30ab\u30e9\u30fc\u5024\u3092\u30d3\u30f3\u306b\u5206\u985e\u3059\u308b\u3053\u3068\u3067\u3059\u3002\u30d3\u30cb\u30f3\u30b0\u306e\u554f\u984c\u306e 1 \u3064\u306f\u3001\u307b\u3068\u3093\u3069\u306e\u5024 (\u30d3\u30f3\u306e\u5883\u754c\u3092\u9664\u304f) \u3067\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u304c\u30bc\u30ed\u306b\u306a\u308b\u3053\u3068\u3067\u3059\u3002\u3082\u30461\u3064\u306f\u3001\u30d3\u30f3\u306e\u9593\u9694\u304c\u5927\u304d\u3044\u3068\u30d3\u30cb\u30f3\u30b0\u306e\u7cbe\u5ea6\u304c\u4f4e\u4e0b\u3059\u308b\u3053\u3068\u3067\u3059</p>\u3002\n<p>FTA\u306f\u3053\u308c\u3089\u306e\u6b20\u70b9\u3092\u514b\u670d\u3057\u307e\u3059\u3002FTA\u306f\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306e\u3088\u3046\u306a\u30cf\u30fc\u30c9\u30d0\u30a6\u30f3\u30c0\u30ea\u30fc\u306e\u4ee3\u308f\u308a\u306b\u3001\u30d3\u30f3\u306e\u9593\u306b\u30bd\u30d5\u30c8\u30d0\u30a6\u30f3\u30c0\u30ea\u30fc\u3092\u4f7f\u3044\u307e\u3059\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u3059\u3079\u3066\u307e\u305f\u306f\u5e83\u7bc4\u56f2\u306e\u5024\u306b\u5bfe\u3057\u3066\u30bc\u30ed\u4ee5\u5916\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u304c\u5f97\u3089\u308c\u307e\u3059\u3002\u307e\u305f\u3001\u90e8\u5206\u7684\u306a\u5024\u3067\u30ad\u30e3\u30d7\u30c1\u30e3\u3055\u308c\u308b\u305f\u3081\u3001\u7cbe\u5ea6\u304c\u5931\u308f\u308c\u308b\u3053\u3068\u306f\u3042\u308a\u307e\u305b\u3093\u3002</p>\n<h4>\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</h4>\n<p><span translate=no>_^_1_^_</span>\u306f\u30bf\u30a4\u30ea\u30f3\u30b0\u30d9\u30af\u30c8\u30eb\u3001</p>\n<p><span translate=no>_^_2_^_</span></p>\n<p>\u3053\u3053\u3067\u3001<span translate=no>_^_3_^_</span>\u306f\u5165\u529b\u7bc4\u56f2\u3001<span translate=no>_^_4_^_</span>\u306f\u30d3\u30f3\u306e\u30b5\u30a4\u30ba\u3001<span translate=no>_^_5_^_</span><span translate=no>_^_6_^_</span>\u3067\u5272\u308a\u5207\u308c\u307e\u3059\u3002</p>\n<p>\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306f\u3001</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>\u3053\u3053\u3067<span translate=no>_^_8_^_</span>\u3001\u5165\u529b\u304c\u6b63\u306e\u304b\u3069\u3046\u304b\u3092\u793a\u3059\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u95a2\u6570\u3068\u3001<span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>\u305d\u3046\u3067\u306a\u3044\u304b\u3069\u3046\u304b\u3092\u793a\u3059\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u95a2\u6570\u304c\u3042\u308a\u307e\u3059\u3002</p>\n<p>\u30bf\u30a4\u30ea\u30f3\u30b0\u3092\u6709\u52b9\u306b\u3059\u308b\u3068\u3001\u5883\u754c\u304c\u56fa\u3044\u305f\u3081\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306f\u30bc\u30ed\u306b\u306a\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n<h4>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</h4>\n<p>\u30d5\u30a1\u30b8\u30fc\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u6a5f\u80fd\u3001</p>\n<p><span translate=no>_^_11_^_</span></p>\n<p><span translate=no>_^_12_^_</span><span translate=no>_^_13_^_</span>\u3053\u308c\u306f\u304b\u3089\u3044\u3064\u307e\u3067\u76f4\u7dda\u7684\u306b\u5897\u52a0\u3057\u3001<span translate=no>_^_15_^_</span> for <span translate=no>_^_14_^_</span> <span translate=no>_^_16_^_</span> \u3068\u7b49\u3057\u304f\u306a\u308a\u307e\u3059\u3002<span translate=no>_^_17_^_</span>\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u3067\u3059</p>\u3002\n<p>FTA \u306f\u3053\u308c\u3092\u4f7f\u3063\u3066\u30d3\u30f3\u306e\u9593\u306b\u30bd\u30d5\u30c8\u306a\u5883\u754c\u7dda\u3092\u4f5c\u308a\u307e\u3059\u3002</p>\n<p><span translate=no>_^_18_^_</span></p>\n<p><a href=\"experiment.html\">\u3053\u308c\u306f\u3001\u5909\u5727\u5668\u3067FTA\u3092\u4f7f\u7528\u3059\u308b\u7c21\u5358\u306a\u5b9f\u9a13\u3067\u3059</a>\u3002</p>\n",
  "<h3>Fuzzy Tiling Activations (FTA)</h3>\n": "<h3>\u30d5\u30a1\u30b8\u30fc\u30bf\u30a4\u30ea\u30f3\u30b0\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3 (FTA)</h3>\n",
  "<h4>Code to test the FTA module</h4>\n": "<h4>FTA \u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u30c6\u30b9\u30c8\u3059\u308b\u30b3\u30fc\u30c9</h4>\n",
  "<h4>Fuzzy indicator function</h4>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h4>\u30d5\u30a1\u30b8\u30fc\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u30fc\u6a5f\u80fd</h4>\n<p><span translate=no>_^_0_^_</span></p>\n",