Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions content/api/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,50 +14,50 @@ In these commands:

## Tracking terms

{{< configOption name="ota track" description="Track the current terms of services according to provided declarations. The declarations, snapshots and versions paths are defined in the configuration." example="`npx ota track`" >}}
{{< configOption name="ota track" description="Track the current terms of services according to provided declarations. The declarations, snapshots and versions paths are defined in the configuration." example="`npx ota track`" />}}

> Note that the snapshots and versions will be recorded at the moment the command is executed, on top of the existing local history. If a shared history already exists and the goal is to add on top of it, that history has to be downloaded before executing that command.

{{< configOption name="ota track --help" description="Show help and available options for track command" example="`npx ota track --help`" >}}
{{< configOption name="ota track --help" description="Show help and available options for track command" example="`npx ota track --help`" />}}

{{< configOption name="ota track [--services <service_id>...]" description="Track terms of specific services only" example="`npx ota track --services \"Facebook\" \"LinkedIn\"`" >}}
{{< configOption name="ota track [--services <service_id>...]" description="Track terms of specific services only" example="`npx ota track --services \"Facebook\" \"LinkedIn\"`" />}}

{{< configOption name="ota track [--services <service_id>...] [--types <terms_type>...]" description="Track specific terms types of specific services only" example="`npx ota track --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"`" >}}
{{< configOption name="ota track [--services <service_id>...] [--types <terms_type>...]" description="Track specific terms types of specific services only" example="`npx ota track --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"`" />}}

{{< configOption name="ota track --schedule [--services <service_id>...] [--types <terms_type>...]" description="Track terms on the schedule defined in the configuration" example="`npx ota track --schedule`" >}}
{{< configOption name="ota track --schedule [--services <service_id>...] [--types <terms_type>...]" description="Track terms on the schedule defined in the configuration" example="`npx ota track --schedule`" />}}

## Validating declarations

{{< configOption name="ota validate [--services <service_id>...] [--types <terms_type>...]" description="Check that all declarations allow recording a snapshot and a version properly. If service IDs are provided, check only those services." example="`npx ota validate --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"`" >}}
{{< configOption name="ota validate [--services <service_id>...] [--types <terms_type>...]" description="Check that all declarations allow recording a snapshot and a version properly. If service IDs are provided, check only those services." example="`npx ota validate --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"`" />}}

{{< configOption name="ota validate --schema-only [--services <service_id>...] [--types <terms_type>...]" description="Check that all declarations are readable by the engine. Allows for a much faster check of declarations, but does not check that the terms are actually accessible." example="`npx ota validate --schema-only --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"`" >}}
{{< configOption name="ota validate --schema-only [--services <service_id>...] [--types <terms_type>...]" description="Check that all declarations are readable by the engine. Allows for a much faster check of declarations, but does not check that the terms are actually accessible." example="`npx ota validate --schema-only --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"`" />}}

{{< configOption name="ota validate --modified" description="Run ota validate only on files that have been modified in Git" example="`npx ota validate --modified`" >}}
{{< configOption name="ota validate --modified" description="Run ota validate only on files that have been modified in Git" example="`npx ota validate --modified`" />}}

## Linting declarations

{{< configOption name="ota lint [--services <service_id>...]" description="Test the format of declarations' normalisation." example="`npx ota lint --services \"Facebook\" \"LinkedIn\"`" >}}
{{< configOption name="ota lint [--services <service_id>...]" description="Test the format of declarations' normalisation." example="`npx ota lint --services \"Facebook\" \"LinkedIn\"`" />}}

{{< configOption name="ota lint --fix [--services <service_id>...]" description="Automatically correct formatting mistakes and ensure that all declarations are standardised" example="`npx ota lint --fix`" >}}
{{< configOption name="ota lint --fix [--services <service_id>...]" description="Automatically correct formatting mistakes and ensure that all declarations are standardised" example="`npx ota lint --fix`" />}}

{{< configOption name="ota lint --modified" description="Run ota lint only on files that have been modified in Git" example="`npx ota lint --modified`" >}}
{{< configOption name="ota lint --modified" description="Run ota lint only on files that have been modified in Git" example="`npx ota lint --modified`" />}}

## Publishing dataset

{{< configOption name="ota dataset [--file <filename>]" description="Export the versions dataset into a ZIP file and publish it to GitHub releases. The dataset title and the URL of the versions repository are defined in the configuration." example="`npx ota dataset --file dataset.zip`" >}}
{{< configOption name="ota dataset [--file <filename>]" description="Export the versions dataset into a ZIP file and publish it to GitHub releases. The dataset title and the URL of the versions repository are defined in the configuration." example="`npx ota dataset --file dataset.zip`" />}}

To export the dataset into a ZIP file and publish it on GitHub releases:

{{< configOption name="ota dataset --publish [--file <filename>]" description="Export and publish dataset to GitHub releases" example="`GITHUB_TOKEN=ghp_XXXXXXXXX npx ota dataset --publish`" >}}
{{< configOption name="ota dataset --publish [--file <filename>]" description="Export and publish dataset to GitHub releases" example="`GITHUB_TOKEN=ghp_XXXXXXXXX npx ota dataset --publish`" />}}

The `GITHUB_TOKEN` can also be defined in a [`.env` file]({{< relref "collections/reference/environment-variables" >}}).

To export, publish the dataset and remove the local copy that was created after it has been uploaded:

{{< configOption name="ota dataset --publish --remove-local-copy [--file <filename>]" description="Export, publish dataset and remove local copy after upload" example="`GITHUB_TOKEN=ghp_XXXXXXXXX npx ota dataset --publish --remove-local-copy`" >}}
{{< configOption name="ota dataset --publish --remove-local-copy [--file <filename>]" description="Export, publish dataset and remove local copy after upload" example="`GITHUB_TOKEN=ghp_XXXXXXXXX npx ota dataset --publish --remove-local-copy`" />}}

{{< configOption name="ota dataset --schedule [--file <filename>]" description="Schedule export, publishing and local copy removal" example="`GITHUB_TOKEN=ghp_XXXXXXXXX npx ota dataset --schedule --publish --remove-local-copy`" >}}
{{< configOption name="ota dataset --schedule [--file <filename>]" description="Schedule export, publishing and local copy removal" example="`GITHUB_TOKEN=ghp_XXXXXXXXX npx ota dataset --schedule --publish --remove-local-copy`" />}}

## Exposing the collection API

{{< configOption name="ota serve" description="Start the collection Web API server. The Web API will be available under `<http://localhost>:<port>/<basePath>/<apiVersion>/<resource>`. The server port and base path are defined in the configuration." example="`npx ota serve`" >}}
{{< configOption name="ota serve" description="Start the collection Web API server. The Web API will be available under `<http://localhost>:<port>/<basePath>/<apiVersion>/<resource>`. The server port and base path are defined in the configuration." example="`npx ota serve`" />}}
70 changes: 35 additions & 35 deletions content/collections/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@ As an example, see the [production configuration file](https://github.com/OpenTe
type="string"
description="Defines how often the engine should check for changes in terms. Uses standard cron syntax to set the schedule."
default="`30 */12 * * *` (runs every 12 hours at minute 30)"
>}}
/>}}

{{< configOption
name="collectionPath"
type="string"
description="Path to the collection's directory containing declarations directory and metadata file, relative to the engine execution location"
example="`../collections/demo-declarations`"
default="`./`"
>}}
/>}}

### Recorder

Expand All @@ -34,13 +34,13 @@ The recorder section manages how versions and snapshots of terms are stored, sup
name="recorder.versions.storage"
type="object"
description="Configuration for storing versions. Supports Git and MongoDB. See [Storage Repositories](#storage-repositories) for more information."
>}}
/>}}

{{< configOption
name="recorder.snapshots.storage"
type="object"
description="Configuration for storing snapshots. Supports Git and MongoDB. See [Storage Repositories](#storage-repositories) for more information."
>}}
/>}}

### Fetcher

Expand All @@ -51,21 +51,21 @@ The fetcher section configures how the engine retrieves documents from the web.
type="number"
description="Maximum wait time for elements to appear in a page (milliseconds)."
default="`10000`"
>}}
/>}}

{{< configOption
name="fetcher.navigationTimeout"
type="number"
description="Maximum wait time for a page to load (milliseconds)."
default="`30000`"
>}}
/>}}

{{< configOption
name="fetcher.language"
type="string"
description="Language code (ISO 639-1) for request headers."
default="`en`"
>}}
/>}}

### Notifier

Expand All @@ -76,14 +76,14 @@ The notifier section sets up how notifications are sent when new versions of ter
type="string"
description="SendInBlue contacts list ID of persons to notify on terms updates."
default="`850`"
>}}
/>}}

{{< configOption
name="notifier.sendInBlue.updateTemplateId"
type="string"
description="SendInBlue email template ID used for updates notifications."
default="`7`"
>}}
/>}}

### Logger

Expand All @@ -94,42 +94,42 @@ The logger section configures logging and error notification settings.
type="string"
description="SMTP server hostname."
default="`smtp-relay.sendinblue.com`"
>}}
/>}}

{{< configOption
name="logger.smtp.username"
type="string"
description="Username for SMTP server authentication."
default="`[email protected]`"
>}}
/>}}

{{< configOption
name="logger.sendMailOnError.to"
type="string"
description="Email address for error notifications."
example="`[email protected]`"
>}}
/>}}

{{< configOption
name="logger.sendMailOnError.from"
type="string"
description="Sender email address for error notifications."
example="`[email protected]`"
>}}
/>}}

{{< configOption
name="logger.sendMailOnError.sendWarnings"
type="boolean"
description="Set to true to also send email in case of warning."
default="`false`"
>}}
/>}}

{{< configOption
name="logger.timestampPrefix"
type="boolean"
description="Set to false to avoid duplicate timestamps if logs are managed by a process manager."
default="`true`"
>}}
/>}}

### Reporter

Expand All @@ -141,42 +141,42 @@ The reporter section manages how issues are reported when terms content is inacc
description="Type of reporter"
example="`github`"
allowedValues="`github`, `gitlab`"
>}}
/>}}

{{< configOption
name="reporter.repositories.declarations"
type="string"
description="Repository for creating issues."
example="`OpenTermsArchive/demo-declarations`"
>}}
/>}}

{{< configOption
name="reporter.repositories.versions"
type="string"
description="Repository for versions."
example="`OpenTermsArchive/demo-versions`"
>}}
/>}}

{{< configOption
name="reporter.repositories.snapshots"
type="string"
description="Repository for snapshots."
example="`OpenTermsArchive/demo-snapshots`"
>}}
/>}}

{{< configOption
name="reporter.baseURL"
type="string"
description="Base URL for GitLab (if applicable)."
example="`https://gitlab.example.com`"
>}}
/>}}

{{< configOption
name="reporter.apiBaseURL"
type="string"
description="API base URL for GitLab (if applicable)."
example="`https://api.gitlab.example.com`"
>}}
/>}}

### Dataset

Expand All @@ -187,21 +187,21 @@ The dataset section configures how datasets are published.
type="string"
description="Title of the dataset."
default="`sandbox`"
>}}
/>}}

{{< configOption
name="dataset.versionsRepositoryURL"
type="string"
description="Repository URL for dataset releases."
default="`https://github.com/OpenTermsArchive/sandbox`"
>}}
/>}}

{{< configOption
name="dataset.publishingSchedule"
type="string"
description="Cron expression for dataset publishing."
default="`30 8 * * MON` (runs every Monday at 8:30 AM)"
>}}
/>}}

### Collection API

Expand All @@ -213,15 +213,15 @@ The collection API section sets the parameters for the API server.
description="Port number for the API server."
example="`8080`"
required=true
>}}
/>}}

{{< configOption
name="collection-api.api.basePath"
type="string"
description="Base path for API endpoints."
example="`/collection-api`"
required=true
>}}
/>}}

---

Expand All @@ -235,7 +235,7 @@ The storage repositories section set the parameters for supported backends for s
description="Type of storage backend."
default="`git`"
allowedValues="`git`, `mongo`"
>}}
/>}}

### Git

Expand All @@ -246,35 +246,35 @@ The Git storage configuration allows to store versions in a Git repository.
type="string"
description="Path to the versions database directory."
default="`./data/versions`"
>}}
/>}}

{{< configOption
name="storage.git.publish"
type="boolean"
description="Boolean to push changes to the origin."
default="`false`"
>}}
/>}}

{{< configOption
name="storage.git.snapshotIdentiferTemplate"
type="string"
description="Template for snapshot ID reference. `%SNAPSHOT_ID` will be replaced with the actual snapshot ID."
default="`./data/snapshots/%SNAPSHOT_ID`"
>}}
/>}}

{{< configOption
name="storage.git.author.name"
type="string"
description="Author name for changes."
default="`Open Terms Archive Bot`"
>}}
/>}}

{{< configOption
name="storage.git.author.email"
type="string"
description="Author email for changes."
default="`[email protected]`"
>}}
/>}}

### MongoDB

Expand All @@ -285,18 +285,18 @@ The MongoDB storage configuration allows to store versions in a MongoDB database
type="string"
description="MongoDB connection URI."
default="`mongodb://127.0.0.1:27017`"
>}}
/>}}

{{< configOption
name="storage.mongo.database"
type="string"
description="Database name."
default="`open-terms-archive`"
>}}
/>}}

{{< configOption
name="storage.mongo.collection"
type="string"
description="Collection name."
default="`snapshots`"
>}}
/>}}
12 changes: 6 additions & 6 deletions content/collections/reference/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@ This reference documentation details all available environment variables that ca

### Engine

{{< configOption name="OTA_ENGINE_SMTP_PASSWORD" type="string" description="SMTP password for email error notifications." >}}
{{< configOption name="OTA_ENGINE_SMTP_PASSWORD" type="string" description="SMTP password for email error notifications." />}}

{{< configOption name="OTA_ENGINE_SENDINBLUE_API_KEY" type="string" description="API key for SendInBlue." >}}
{{< configOption name="OTA_ENGINE_SENDINBLUE_API_KEY" type="string" description="API key for SendInBlue." />}}

{{< configOption name="OTA_ENGINE_GITHUB_TOKEN" type="string" description="GitHub token for API access." >}}
{{< configOption name="OTA_ENGINE_GITHUB_TOKEN" type="string" description="GitHub token for API access." />}}

{{< configOption name="OTA_ENGINE_GITLAB_TOKEN" type="string" description="GitLab token for API access." >}}
{{< configOption name="OTA_ENGINE_GITLAB_TOKEN" type="string" description="GitLab token for API access." />}}

{{< configOption name="OTA_ENGINE_GITLAB_RELEASES_TOKEN" type="string" description="GitLab token for dataset releases." >}}
{{< configOption name="OTA_ENGINE_GITLAB_RELEASES_TOKEN" type="string" description="GitLab token for dataset releases." />}}

---

### Federation API

{{< configOption name="OTA_FEDERATION_API_SMTP_PASSWORD" type="string" description="SMTP password for email error notifications." >}}
{{< configOption name="OTA_FEDERATION_API_SMTP_PASSWORD" type="string" description="SMTP password for email error notifications." />}}
Loading