You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/contributing-terms.en.md
+3-377Lines changed: 3 additions & 377 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,395 +5,21 @@ weight: 4
5
5
6
6
# Contributing new declarations
7
7
8
-
This is a step by step guide to help you add declarations to the [Contrib collection](https://github.com/OpenTermsArchive/contrib-declarations). This collection is dedicated to volunteer contribution of declarations to Open Terms Archive.
8
+
This is a step by step guide to help you add declarations to the [contrib-declaration](https://github.com/OpenTermsArchive/contrib-declarations) repository. This repository is dedicated for volunteer contribution of declarations to Open Terms Archive.
9
9
10
10
Having understood briefly how a declaration is structured in JSON format, we need to look at concrete steps on how you can add these JSON files to the repository.
11
11
12
-
## Tracking new terms
13
-
14
-
Tracking terms is done by _declaring_ them and the service they are associated with in a collection. Declarations are found in JSON files in the `declarations` folder of the collection.
15
-
16
-
Before adding new terms, open the [`declarations`](https://github.com/OpenTermsArchive/contrib-declarations/tree/main/declarations) folder and check if the service you want to track terms for is already declared. If a JSON file with the name of the service is already present, you can jump straight to [declaring terms](#declaring-terms). Otherwise, keep reading!
17
-
18
12
## Prerequisites
19
13
20
14
In order to add declarations:
21
15
22
-
1. You need to have [Node.js](https://nodejs.org/en/) installed on your machine. If you don't have it, you can [download it](https://nodejs.org/en/download/) from the official website.
23
-
2. You need to have Git installed on your machine. If you don't have it, you can [download it](https://git-scm.com/downloads) from the official website.
16
+
1. You need to have [Node.js](https://nodejs.org/en/) installed on your machine. If you don't have it, you can download it from the official website [here](https://nodejs.org/en/download/).
17
+
2. You need to have git installed on your machine. If you don't have it, you can download it from the official website [here](https://git-scm.com/downloads).
24
18
25
19
## Adding a declaration
26
20
27
21
To add a declaration, you need to follow these steps:
28
22
29
-
1. Clone the [`contrib-declarations`](https://github.com/OpenTermsArchive/contrib-declarations) repository to your local machine.
30
-
2. Create a branch that describes your contribution e.g. `add_open_terms_archive_terms_of_service` or `add_firefox_privacy_policy`
31
-
3. Run `npm install`. This will install all the dependencies including the Open Terms Archive engine which will allow you to test and validate your declaration.
32
-
4. Create a JSON file with, as filename, the [service ID]({{< relref "reference/declaration#service-id" >}}) of the service you are adding the declaration for. This JSON file should be in the `declarations` folder of the repository. To learn more about selecting the right service ID, please read the [declaring a new service]({{< relref "reference/declaration#declaring-a-new-service" >}}) section.
33
-
5. Visit the declaration URL and use [browser developer tools](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/Tools_and_setup/What_are_browser_developer_tools) to inspect the page and find the right selectors for the significant section containing the terms you want to declare and parts to remove.
34
-
6. After you've properly chosen your selectors and structured your JSON file, you should test and validate your JSON file. To do this, you need to run `npx ota validate --services [service name]` from the root of the repository. This will run a validation on the declaration, highlighting any changes required.
35
-
7. If all tests pass, open a pull request to the main repository.
36
-
37
-
### Service ID
38
-
39
-
The service ID is exposed to developers. It should be easy to handle with scripts and other tools.
40
-
41
-
- Non-ASCII characters are not supported. Service IDs are derived from the service name by normalising it into ASCII.
- Punctuation is supported, except characters that have meaning at filesystem level (`:`, `/`, `\`). These are replaced with a dash (`-`).
47
-
-_Example: `Booking.com` → `Booking.com`_.
48
-
-_Example: `Yahoo!` → `Yahoo!`_.
49
-
-_Example: `re:start` → `re-start`_.
50
-
-_Example: `we://` → `we---`_.
51
-
- Capitals and spaces are supported. Casing and spacing are expected to reflect the official service name casing and spacing.
52
-
-_Example: `App Store` → `App Store`_.
53
-
-_Example: `DeviantArt` → `DeviantArt`_.
54
-
55
-
> If you have a hard time defining the service ID, check out the [practical guidelines to derive the ID from the service name](../guidelines/declaring#service-id), and feel free to mention your uncertainties in the pull request! We will help you improve the service ID if necessary 🙂
56
-
57
-
> More details on the ID and naming constraints and recommendations can be found in the relevant [decision record](https://github.com/OpenTermsArchive/engine/blob/main/decision-records/0001-service-name-and-id.md).
58
-
59
-
### Service declaration
60
-
61
-
Once you have the [service name](#service-name) and the [service ID](#service-id), create a JSON file in the `declarations` folder named after the ID of the service you want to add, with the following structure:
62
-
63
-
```json
64
-
{
65
-
"name": "<service name>",
66
-
"documents": {}
67
-
}
68
-
```
69
-
70
-
Within the `documents` JSON object, you will now declare terms.
71
-
72
-
- - -
73
-
74
-
## Declaring terms
75
-
76
-
Terms are declared in a service declaration file, under the `documents` property.
77
-
78
-
Most of the time, terms are written in only one source document (for example [Facebook Terms of Service](https://www.facebook.com/legal/terms)) but sometimes terms can be spread across multiple online source documents, and their combination constitutes the terms (for example [Facebook Community Guidelines](https://transparency.fb.com/policies/community-standards/)).
79
-
80
-
#### Source document
81
-
82
-
The way in which a source document is obtained is defined in a JSON object:
83
-
84
-
```json
85
-
{
86
-
"fetch": "The URL where the document can be found",
87
-
"executeClientScripts": "A boolean to execute client-side JavaScript loaded by the document before accessing the content, in case the DOM modifications are needed to access the content; defaults to false (fetch HTML only)",
88
-
"filter": "An array of service specific filter function names",
89
-
"remove": "A CSS selector, a range selector or an array of selectors that target the insignificant parts of the document that has to be removed. Useful to remove parts that are inside the selected parts",
90
-
"select": "A CSS selector, a range selector or an array of selectors that target the meaningful parts of the document, excluding elements such as headers, footers and navigation"
91
-
}
92
-
```
93
-
94
-
- For HTML files, `fetch` and `select` are mandatory.
95
-
- For PDF files, only `fetch` is mandatory.
96
-
97
-
Let’s start by defining these keys!
98
-
99
-
#### `fetch`
100
-
101
-
This property should simply contain the URL at which the terms you want to track can be downloaded. HTML and PDF files are supported.
102
-
103
-
When terms coexist in different languages and jurisdictions, please refer to the [scope of the collection](../#collections) to which you are contributing. This scope is usually defined in the README.
104
-
105
-
#### `select`
106
-
107
-
_This property is not needed for PDF documents._
108
-
109
-
Most of the time, contractual documents are exposed as web pages, with a header, a footer, navigation menus, possibly ads… We aim at tracking only the significant parts of the document. In order to achieve that, the `select` property allows to extract only those parts in the process of [converting from snapshot to version](https://opentermsarchive.org/#how-it-works).
110
-
111
-
The `select` value can be either a [CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors), a [range selector](#range-selectors) or an array of those.
112
-
113
-
##### CSS selectors
114
-
115
-
CSS selectors should be provided as a string. See the [specification](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) for how to write CSS selectors.
116
-
117
-
> For example, the following selector will select the content in the `<main>` tag of the HTML document:
118
-
>
119
-
> ```json
120
-
> "select": "main"
121
-
> ```
122
-
123
-
##### Range selectors
124
-
125
-
A range selector is defined with a _start_ and an _end_ CSS selector. It is also necessary to define if the range starts before or after the element targeted by the _start_ CSS selector and to define if it ends before or after the element targeted by the _end_ CSS selector.
126
-
127
-
To that end, a range selector is a JSON object containing two keys out of the four that are available: `startBefore`, `startAfter`, `endBefore` and `endAfter`.
128
-
129
-
```json
130
-
{
131
-
"start[Before|After]": "<CSS selector>",
132
-
"end[Before|After]": "<CSS selector>"
133
-
}
134
-
```
135
-
136
-
> For example, the following selector will select the content between the element targeted by the CSS selector `#privacy-eea`, including it, and the element targeted by the CSS selector `footer`, excluding it:
137
-
>
138
-
> ```json
139
-
> {
140
-
> "startBefore": "#privacy-eea",
141
-
> "endBefore": "footer"
142
-
> }
143
-
> ```
144
-
145
-
#### `remove`
146
-
147
-
_This property is optional._
148
-
149
-
Beyond [selecting a subset of a web page](#select), some documents will have non-significant parts in the middle of otherwise significant parts. For example, they can have “go to top” links or banner ads. These can be removed by listing [CSS selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors), [range selectors](#range-selectors) or an array of them under the `remove` property.
150
-
151
-
##### Example
152
-
153
-
Let's assume a web page contains the following content:
In some cases, the content of the document is only loaded (or is modified dynamically) by client scripts.
240
-
When set to `true`, this boolean property loads the page in a headless browser to load all assets and execute client scripts before trying to get the document contents.
241
-
242
-
Since the performance cost of this approach is high, it is set to `false` by default, relying on the HTML content only.
243
-
244
-
#### `filter`
245
-
246
-
_This property is optional._
247
-
248
-
Finally, some documents will need more complex filtering beyond simple element selection and removal, for example to remove noise (changes in textual content that are not meaningful to the terms of services). Such filters are declared as JavaScript functions that modify the downloaded web page through the [DOM API](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model).
249
-
250
-
Filters take the document DOM and the terms declaration as parameters and are:
251
-
252
-
-**in-place**: they modify the document structure and content directly;
253
-
-**idempotent**: they should return the same document structure and content even if run repeatedly on their own result.
254
-
255
-
Filters are loaded automatically from files named after the service they operate on. For example, filters for the Meetup service, which is declared in `declarations/Meetup.json`, are loaded from `declarations/Meetup.filters.js`.
Each filter is exposed as a named function export that takes a `document` parameter and behaves like the `document` object in a browser DOM. These functions can be `async`, but they will still run sequentially. The whole document declaration is passed as second parameter.
264
-
265
-
> The `document` parameter is actually a [JSDOM](https://github.com/jsdom/jsdom) document instance.
266
-
267
-
You can learn more about usual noise and ways to handle it [in the guidelines](../guidelines/declaring#usual-noise).
268
-
269
-
##### Example
270
-
271
-
Let's assume a service adds a unique `clickId` parameter in the query string of all link destinations. These parameters change on each page load, leading to recording noise in versions. Since links should still be recorded, it is not appropriate to use `remove` to remove the links entirely. Instead, a filter will manipulate the links destinations to remove the always-changing parameter. Concretely, the goal is to apply the following filter:
272
-
273
-
```diff
274
-
- Read the <a href="https://example.com/example-page?clickId=349A2033B&lang=en">list of our affiliates</a>.
275
-
+ Read the <a href="https://example.com/example-page?lang=en">list of our affiliates</a>.
276
-
```
277
-
278
-
The code below implements this filter:
279
-
280
-
```js
281
-
function removeTrackingIdsQueryParam(document) {
282
-
constQUERY_PARAM_TO_REMOVE='clickId';
283
-
284
-
document.querySelectorAll('a').forEach(link=> { // iterate over every link in the page
285
-
consturl=newURL(link.getAttribute('href'), document.location); // URL is part of the DOM API, see https://developer.mozilla.org/en-US/docs/Web/API/URL
286
-
constparams=newURLSearchParams(url.search); // URLSearchParams is part of the DOM API, see https://developer.mozilla.org/en-US/docs/Web/API/URLSearchParams
287
-
288
-
params.delete(QUERY_PARAM_TO_REMOVE); // we use the DOM API instead of RegExp because we can't know in advance in which order parameters will be written
289
-
url.search=params.toString(); // store the query string without the parameter
290
-
link.setAttribute('href', url.toString()); // write the destination URL without the parameter
291
-
});
292
-
}
293
-
```
294
-
295
-
##### Example usage of declaration parameter
296
-
297
-
The second parameter can be used to access the defined document URL or selector inside the filter.
298
-
299
-
Let's assume a service stores some of its legally-binding terms in images. To track these changes properly, images should be stored as part of the terms. By default, images are not stored since they significantly increase the document size. The filter below will store images inline in the terms, encoded in a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs). In order to download the images for conversion, the base URL of the web page is needed to resolve relative links. This information is obtained from the declaration.
In the case where terms are extracted from one single source document, they are declared by simply declaring that source document:
325
-
326
-
```json
327
-
…
328
-
"documents": {
329
-
"<terms type>": {
330
-
"fetch": "…",
331
-
"executeClientScripts": "…",
332
-
"filter": "…",
333
-
"remove": "…",
334
-
"select": "…"
335
-
}
336
-
}
337
-
…
338
-
```
339
-
340
-
#### Terms with multiple source documents
341
-
342
-
When the terms are spread across multiple source documents, they should be declared by declaring their combination:
343
-
344
-
```json
345
-
…
346
-
"documents": {
347
-
"<terms type>": {
348
-
"combine": [
349
-
{
350
-
"fetch": "…",
351
-
"executeClientScripts": "…",
352
-
"filter": "…",
353
-
"remove": "…",
354
-
"select": "…"
355
-
},
356
-
{
357
-
"fetch": "…",
358
-
"executeClientScripts": "…",
359
-
"filter": "…",
360
-
"remove": "…",
361
-
"select": "…"
362
-
}
363
-
]
364
-
}
365
-
}
366
-
…
367
-
```
368
-
369
-
If some parts of the source documents are repeated, they can be factorised. For example, it is common for the structure of HTML pages to be similar from page to page, so `select`, `remove` and `filter` would be the same. These elements can be shared instead of being duplicated:
370
-
371
-
```json
372
-
…
373
-
"documents": {
374
-
"<terms type>":
375
-
"executeClientScripts": "…",
376
-
"filter": "…",
377
-
"remove": "…",
378
-
"select": "…",
379
-
"combine": [
380
-
{
381
-
"fetch": "…",
382
-
},
383
-
{
384
-
"fetch": "…",
385
-
}
386
-
]
387
-
}
388
-
…
389
-
```
390
-
391
-
## Contributing new declarations
392
-
393
-
This is a step by step guide to help you add declarations to the [contrib-declaration](https://github.com/OpenTermsArchive/contrib-declarations) repository. This repository is dedicated for volunteer contribution of declarations to Open Terms Archive.
394
-
395
-
Having understood briefly how a declaration is structured in JSON format, we need to look at concrete steps on how you can add these JSON files to the repository. To add them, you need to:
396
-
397
23
1. Clone the [contrib-declaration](https://github.com/OpenTermsArchive/contrib-declarations) repository to your local machine.
398
24
2. Create a branch that describes your contribution e.g. `add-Open-Terms-Archive-ToS` or `add-firefox-privacy-policy`
399
25
3. Run `npm install`. This is to install all the dependencies including the Open Terms Archive engine which will allow you to test and validate your declaration to make sure it is ok.
0 commit comments