tdtext.github.io/index.html at master · tdtext/tdtext.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="chrome=1">
    <title>How to create a The DataTank extension by tdtext</title>
    <link rel="stylesheet" href="stylesheets/styles.css">
    <link rel="stylesheet" href="stylesheets/pygment_trac.css">
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
    <script src="javascripts/respond.js"></script>
    <!--[if lt IE 9]>
      <script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
    <![endif]-->
    <!--[if lt IE 8]>
    <link rel="stylesheet" href="stylesheets/ie.css">
    <![endif]-->
    <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">

  </head>
  <body>
      <div id="header">
        <nav>
          <li class="fork"><a href="https://github.com/tdtext">View On GitHub</a></li>
        </nav>
      </div><!-- end header -->

    <div class="wrapper">

      <section>
        <div id="title">
          <h1>How to create a The DataTank extension</h1>
          <p>Extending The DataTank's functionality without touching the core</p>
          <hr>
          <span class="credits left">Project maintained by <a href="https://github.com/tdtext">tdtext</a></span>
          <span class="credits right">Hosted on GitHub Pages &mdash; Theme by <a href="https://twitter.com/michigangraham">mattgraham</a></span>
        </div>

        <h1>
<a name="tdtext" class="anchor" href="#tdtext"><span class="octicon octicon-link"></span></a>tdtext</h1>

<p>Have you as well been dreaming about a world where it is possible for anyone to write a The DataTank extension and amaze the world of Open Data?</p>

<p>With tdtext, this becomes possible. Just like people can easily create ckan extensions (ckanext, see what we did there?) with a minimum of documentation, you are able to create The DataTank extensions which plug into the core workflow. The DataTank extension systems is built for the ease of developers. Once a The DataTank is installed, it should be a breeze to install: scrapers, extra formatters, extra visualizations, strategies to read data, and so on.</p>

<h2>
<a name="architecture" class="anchor" href="#architecture"><span class="octicon octicon-link"></span></a>Architecture</h2>

<p>We have one singleton class called <code>TdtextNotifier</code>. This class will be updated by tdt/core whenever something happens: the routes are loaded, the formatters are loaded, the documentation is updated, and so on. <code>TdtextNotifier</code> will collect all updates and will direct then to registered tdtext classes. These classes can be added to the configuration. What event will go to which class is decided based on the interfaces they inherit from.</p>

<h2>
<a name="a-new-github-organisation" class="anchor" href="#a-new-github-organisation"><span class="octicon octicon-link"></span></a>A new github organisation</h2>

<p><a href="http://github.com/tdtext">http://github.com/tdtext</a></p>

<p>This is the organisation where we will add all official extensions for The DataTank and a barebone repository which everyone can fork to start its own tdtextension.</p>

<p>It also contains these docs at <a href="http://tdtext.github.io/">http://tdtext.github.io/</a> which can be edited at <a href="http://github.com/tdtext/tdtext.github.io">http://github.com/tdtext/tdtext.github.io</a>.</p>

<h2>
<a name="installing-extensions-using-composer" class="anchor" href="#installing-extensions-using-composer"><span class="octicon octicon-link"></span></a>Installing extensions using composer</h2>

<p>If you're not familiar with composer, check out <a href="http://getcomposer.org/doc/00-intro.md">http://getcomposer.org/doc/00-intro.md</a>.</p>

<div class="highlight"><pre><span class="nb">cd</span> /path/to/tdt/root/
composer require tdtext/<span class="k">${</span><span class="nv">extensionname</span><span class="k">}</span>
composer update <span class="c"># this will also trigger the code to enable the extension. You can disable it in your config.</span>
</pre></div>

<h2>
<a name="writing-a-tdt-extension" class="anchor" href="#writing-a-tdt-extension"><span class="octicon octicon-link"></span></a>Writing a tdt extension</h2>

<ul>
<li>Fork the barebone tdtextension at <a href="http://github.com/tdtext/barebone">http://github.com/tdtext/barebone</a> </li>
<li>(Re)Name your git repository towards tdtext-{extensionname} (this will make your extension discoverable through <a href="https://packagist.org">https://packagist.org</a>)</li>
<li>Add the description of your extension in composer.json</li>
<li>run composer update</li>
<li>Edit your class and start creating your extension</li>
</ul><p>Once you have tested it by installing it locally, you can add it to packagist: <a href="https://packagist.org/packages/submit">https://packagist.org/packages/submit</a></p>

<h3>
<a name="the-interfaces-the-hard-way" class="anchor" href="#the-interfaces-the-hard-way"><span class="octicon octicon-link"></span></a>The Interfaces (the hard way)</h3>

<h4>
<a name="tdtcoretdtextirouteseditor" class="anchor" href="#tdtcoretdtextirouteseditor"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/IRoutesEditor</h4>

<p>With the route mapper you can add a controller for a specific route. This comes in handy if you want to go past all tdt controller handling. This is not recommended if you're going to be handling data through here. It is recommended if you want to implement your own brand new functionality for certain URI patterns.</p>

<p>For example:</p>

<div class="highlight"><pre><span class="k">class</span> <span class="nc">MyOwnRouteController</span> <span class="k">implements</span> <span class="nx">IRoutesEditor</span> <span class="p">{</span>

    <span class="k">function</span> <span class="nf">editRoutes</span><span class="p">(</span><span class="o">&amp;</span><span class="nv">$routes</span><span class="p">){</span>
        <span class="c1">//key of routes is divided by a |</span>
        <span class="c1">//the first part is the HTTP Method use, the second part is a regular expression</span>
        <span class="nv">$routes</span><span class="p">[</span><span class="s2">"GET | about/(?P&lt;person&gt;[^?]+)"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">get_class</span><span class="p">(</span><span class="nv">$this</span><span class="p">);</span>
    <span class="p">}</span>

    <span class="k">function</span> <span class="nf">GET</span><span class="p">(</span><span class="nv">$matches</span><span class="p">){</span>
        <span class="k">echo</span> <span class="s2">"A page about "</span> <span class="o">.</span> <span class="nb">urldecode</span><span class="p">(</span><span class="nv">$matches</span><span class="p">[</span><span class="s2">"person"</span><span class="p">]);</span>
    <span class="p">}</span>
<span class="p">}</span>
</pre></div>

<h4>
<a name="tdtcoretdtextidefinitionseditor" class="anchor" href="#tdtcoretdtextidefinitionseditor"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/IDefinitionsEditor</h4>

<p>Called when the documentation in tdt/core is ready</p>

<p>You can add your own documentation for a new resource if you want to have your own configuration in there. You can use it to define new configurations without them being stored in the database.</p>

<p>For example:</p>

<div class="highlight"><pre><span class="k">class</span> <span class="nc">NewDefinitionAdder</span> <span class="nx">IDefinitionsEditor</span> <span class="p">{</span>
    <span class="k">function</span> <span class="nf">editDefinitions</span><span class="p">(</span><span class="o">&amp;</span><span class="nv">$definitions</span><span class="p">){</span>
        <span class="nv">$definitions</span><span class="p">[</span><span class="o">...</span><span class="p">]</span> <span class="o">=</span> <span class="o">...</span><span class="p">;</span>
    <span class="p">}</span>
<span class="p">}</span>
</pre></div>

<h4>
<a name="tdtcoretdtextiformatterseditor" class="anchor" href="#tdtcoretdtextiformatterseditor"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/IFormattersEditor</h4>

<p>Add your own formatter to the list</p>

<div class="highlight"><pre><span class="k">Interface</span> <span class="nx">IFormattersEditor</span> <span class="p">{</span>
    <span class="sd">/**</span>
<span class="sd">    * Add or edit formatters in this array</span>
<span class="sd">    */</span>
    <span class="k">abstract</span> <span class="k">function</span> <span class="nf">editFormatters</span><span class="p">(</span><span class="o">&amp;</span><span class="nv">$formatters</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>

<h4>
<a name="tdtcoretdtextitransformer" class="anchor" href="#tdtcoretdtextitransformer"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/ITransformer</h4>

<p>A transformer transforms an object after it is read into memory.</p>

<div class="highlight"><pre><span class="k">Interface</span> <span class="nx">ITransformer</span> <span class="p">{</span>
   <span class="sd">/**</span>
<span class="sd">    * Add or edit an object from the moment is read into memory</span>
<span class="sd">    * @param $resourceconfiguration contains the identifier of a resource and the configuration</span>
<span class="sd">    * @param $object is the data object</span>
<span class="sd">    */</span>
    <span class="k">abstract</span> <span class="k">function</span> <span class="nf">transform</span><span class="p">(</span><span class="nv">$resourceconfiguration</span><span class="p">,</span> <span class="o">&amp;</span><span class="nv">$object</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>

<p>...WIP</p>

<h3>
<a name="using-an-abstract-class-recommended---its-there-if-you-dont-know-about-the-datatank-internals" class="anchor" href="#using-an-abstract-class-recommended---its-there-if-you-dont-know-about-the-datatank-internals"><span class="octicon octicon-link"></span></a>Using an abstract class (recommended - it's there if you don't know about The DataTank internals</h3>

<h4>
<a name="tdtcoretdtextaformatter" class="anchor" href="#tdtcoretdtextaformatter"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/AFormatter</h4>

<p>Write a new behaviour for a certain format.</p>

<p>For example:</p>

<div class="highlight"><pre><span class="k">class</span> <span class="nc">YAMLFormatter</span> <span class="k">extends</span> <span class="nx">\tdt\core\tdtext\AFormatter</span> <span class="p">{</span>
    <span class="k">public</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(){</span>
        <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">name</span> <span class="o">=</span> <span class="s2">"YAML"</span><span class="p">;</span> <span class="c1">//you can also override the behaviour for for example XML or JSON</span>
    <span class="p">}</span>

    <span class="k">function</span> <span class="nf">getGETParameters</span><span class="p">(){</span>
        <span class="k">return</span> <span class="k">array</span><span class="p">();</span>
    <span class="p">}</span>

    <span class="k">function</span> <span class="nf">print</span><span class="p">(</span><span class="nv">$resourceconfiguration</span><span class="p">,</span> <span class="nv">$parameters</span><span class="p">,</span> <span class="nv">$object</span><span class="p">){</span>
        <span class="c1">//YAML print code</span>
    <span class="p">}</span>
<span class="p">}</span>
</pre></div>

<h4>
<a name="tdtcoretdtextascraper" class="anchor" href="#tdtcoretdtextascraper"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/AScraper</h4>

<p>Writing a scraper can be done easily by extending the AScraper class. Use this class if you want to have a certain URI in The DataTank to scrape data from a certain source.</p>

<p>Example can be found in the barebone repository which can be forked to start your own: <a href="http://github.com/tdtext/barebone">http://github.com/tdtext/barebone</a></p>

<h4>
<a name="tdtcoretdtextastrategy" class="anchor" href="#tdtcoretdtextastrategy"><span class="octicon octicon-link"></span></a>tdt/core/tdtext/AStrategy</h4>

<p>If you want to implement a new strategy next to the standard ones for reading a certain source (e.g. a NoSQL cluster or a certain data format), extend this abstract class.</p>

<p>Draft:</p>

<div class="highlight"><pre><span class="k">abstract</span> <span class="k">class</span> <span class="nc">AStrategy</span> <span class="k">implements</span> <span class="o">...</span> <span class="p">{</span>

    <span class="sd">/**</span>
<span class="sd">     * Returns an array according to the discovery API of parameter objects.</span>
<span class="sd">     * They include the parameters needed to read a resource which's source uses this strategy.</span>
<span class="sd">     */</span>
    <span class="k">abstract</span> <span class="k">function</span> <span class="nf">getGETParameters</span><span class="p">();</span>

    <span class="sd">/**</span>
<span class="sd">     * Returns an array according to the discovery API of parameter objects.</span>
<span class="sd">     * They include documentation about whether the parameter is required or not when configuring a source of this strategy type through a PUT request.</span>
<span class="sd">     */</span>
    <span class="k">abstract</span> <span class="k">function</span> <span class="nf">getConfigParameters</span><span class="p">();</span>

    <span class="sd">/**</span>
<span class="sd">     * when reading the a resource configured with this strategy, this is what's going to happen.</span>
<span class="sd">     * The resourceconfiguration contains the resourceidentifier and the config parameters (as defined by the getConfigParameters() function)</span>
<span class="sd">     */</span>
    <span class="k">abstract</span> <span class="k">function</span> <span class="nf">read</span><span class="p">(</span><span class="nv">$resourceconfiguration</span><span class="p">,</span> <span class="nv">$parameters</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
      </section>

    </div>
    <!--[if !IE]><script>fixScale(document);</script><![endif]-->

  </body>
</html>