Skip to content

Commit 8ceec6b

Browse files
JohannesJanderJohannesJander
committed
First implementation of foreign key constraints across resources
1 parent 3018cc0 commit 8ceec6b

File tree

8 files changed

+170
-38
lines changed

8 files changed

+170
-38
lines changed

‎README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
# datapackage-java
22

3-
[![Build Status](https://travis-ci.org/frictionlessdata/datapackage-java.svg?branch=master)](https://travis-ci.org/frictionlessdata/datapackage-java)
4-
[![Coverage Status](https://coveralls.io/repos/github/frictionlessdata/datapackage-java/badge.svg?branch=master)](https://coveralls.io/github/frictionlessdata/datapackage-java?branch=master)
53
[![License](https://img.shields.io/github/license/frictionlessdata/datapackage-java.svg)](https://github.com/frictionlessdata/datapackage-java/blob/master/LICENSE)
64
[![Release](https://img.shields.io/jitpack/v/github/frictionlessdata/datapackage-java)](https://jitpack.io/#frictionlessdata/datapackage-java)
75
[![Codebase](https://img.shields.io/badge/codebase-github-brightgreen)](https://github.com/frictionlessdata/datapackage-java)
@@ -17,6 +15,13 @@ Please find releases on [Jitpack](https://jitpack.io/#frictionlessdata/datapacka
1715

1816
## Usage
1917

18+
- [Create a Data Package](#create_a_data_package) explains how to create a Data Package
19+
- [Iterate through Data](#iterate_through_data) explains how to iterate through data in Resources
20+
- [Edit a Data Package](#edit_a_data_package) explains how to add or remove Resources or properties to or from a Data Package
21+
- [Save to File](#save_to_file) explains how to save a Data Package to a file
22+
- [Working with Foreign Keys](docs/foreign-keys.md) explains how to set foreign key constraints in Data Packages
23+
- [Contributing](#contributing) contributions are welcome
24+
2025
### Create a Data Package
2126

2227
#### From JSONObject Object
@@ -208,6 +213,7 @@ dp.save("/destination/path/datapackage.zip")
208213

209214
Found a problem and would like to fix it? Have that great idea and would love to see it in the repository?
210215

216+
> [!NOTE]
211217
> Please open an issue before you start working.
212218
213219
It could save a lot of time for everyone and we are super happy to answer questions and help you along the way. Furthermore, feel free to join [frictionlessdata Gitter chat room](https://gitter.im/frictionlessdata/chat) and ask questions.

‎docs/foreign-keys.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Working with Foreign Keys
2+
3+
The library supports foreign keys described in the [Table Schema](http://specs.frictionlessdata.io/table-schema/#foreign-keys) specification. It means if your data package descriptor use `resources[].schema.foreignKeys` property for some resources a data integrity will be checked on reading operations.
4+
5+
Consider we have a data package:
6+
7+
```json
8+
{
9+
"name": "foreign-keys",
10+
"resources": [
11+
{
12+
"name": "teams",
13+
"data": [
14+
["id", "name", "city"],
15+
["1", "Arsenal", "London"],
16+
["2", "Real", "Madrid"],
17+
["3", "Bayern", "Munich"]
18+
],
19+
"schema": {
20+
"fields": [
21+
{
22+
"name": "id",
23+
"type": "integer"
24+
},
25+
{
26+
"name": "name",
27+
"type": "string"
28+
},
29+
{
30+
"name": "city",
31+
"type": "string"
32+
}
33+
],
34+
"foreignKeys": [
35+
{
36+
"fields": "city",
37+
"reference": {
38+
"resource": "cities",
39+
"fields": "name"
40+
}
41+
}
42+
]
43+
}
44+
},
45+
{
46+
"name": "cities",
47+
"data": [
48+
["name", "country"],
49+
["London", "England"],
50+
["Madrid", "Spain"]
51+
]
52+
}
53+
]
54+
}
55+
```
56+
57+
Let's check relations for a `teams` resource:
58+
59+
````java
60+
Package dp = new Package(DESCRIPTOR, Paths.get(""), true);
61+
Resource teams = dp.getResource("teams");
62+
DataPackageValidationException dpe
63+
= Assertions.assertThrows(DataPackageValidationException.class, () -> teams.checkRelations(dp));
64+
Assertions.assertEquals("Error reading data with relations: Foreign key validation failed: [city] -> [name]: 'Munich' not found in resource 'cities'.", dpe.getMessage());
65+
````
66+
As we could see, we can read the Datapackage, but if we call `teams.checkRelations(dp)`, there is a foreign key violation. That's because our lookup table `cities` doesn't have a city of `Munich` but we have a team from there. We need to fix it in `cities` resource:
67+
68+
````json
69+
{
70+
"name": "cities",
71+
"data": [
72+
["name", "country"],
73+
["London", "England"],
74+
["Madrid", "Spain"],
75+
["Munich", "Germany"]
76+
]
77+
}
78+
````
79+
80+
Now, calling `teams.checkRelations(dp)` will no longer throw an exception.

‎pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
<maven.compiler.source>${java.version}</maven.compiler.source>
2424
<maven.compiler.target>${java.version}</maven.compiler.target>
2525
<maven.compiler.compiler>${java.version}</maven.compiler.compiler>
26-
<tableschema-java-version>0.7.5</tableschema-java-version>
26+
<tableschema-java-version>0.8.0</tableschema-java-version>
2727
<junit.version>5.12.0</junit.version>
2828
<slf4j-simple.version>2.0.17</slf4j-simple.version>
2929
<apache-commons-collections4.version>4.4</apache-commons-collections4.version>

‎src/main/java/io/frictionlessdata/datapackage/Package.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,8 @@ public Package(Collection<Resource> resources) throws IOException {
9898
}
9999

100100
/**
101-
* Load from String representation of JSON object. To prevent file system traversal attacks
101+
* Load from String representation of JSON object. The resources of the package could be either inline JSON
102+
* or relative path references to files.To prevent file system traversal attacks
102103
* while loading Resources, the basePath must be explicitly set here, the `basePath`
103104
* variable cannot be null.
104105
*

‎src/main/java/io/frictionlessdata/datapackage/resource/AbstractResource.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -301,7 +301,7 @@ public void checkRelations(Package pkg) {
301301
}
302302
}
303303
if (!found) {
304-
throw new ForeignKeyException("Foreign key validation failed: " + fk.getForeignKey().getFieldNames() + " -> " + fk.getForeignKey().getReference().getFieldNames() + ": '" + fkVal + "' not found in resource '"+fk.getForeignKey().getReference().getResource()+"'");
304+
throw new ForeignKeyException("Foreign key validation failed: " + fk.getForeignKey().getFieldNames() + " -> " + fk.getForeignKey().getReference().getFieldNames() + ": '" + fkVal + "' not found in resource '"+fk.getForeignKey().getReference().getResource()+"'.");
305305
}
306306
}
307307
}

‎src/test/java/io/frictionlessdata/datapackage/DocumentationCases.java‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
package io.frictionlessdata.datapackage;
2+
3+
import io.frictionlessdata.datapackage.exceptions.DataPackageValidationException;
4+
import io.frictionlessdata.datapackage.resource.Resource;
5+
import io.frictionlessdata.tableschema.Table;
6+
import org.junit.jupiter.api.Assertions;
7+
import org.junit.jupiter.api.DisplayName;
8+
import org.junit.jupiter.api.Test;
9+
10+
import java.nio.file.Paths;
11+
12+
public class DocumentationCases {
13+
14+
@Test
15+
@DisplayName("Reading a Schema with a Foreign Key against non-matching data")
16+
void validateForeignKeyWithError() throws Exception{
17+
String DESCRIPTOR = "{\n" +
18+
" \"name\": \"foreign-keys\",\n" +
19+
" \"resources\": [\n" +
20+
" {\n" +
21+
" \"name\": \"teams\",\n" +
22+
" \"data\": [\n" +
23+
" [\"id\", \"name\", \"city\"],\n" +
24+
" [\"1\", \"Arsenal\", \"London\"],\n" +
25+
" [\"2\", \"Real\", \"Madrid\"],\n" +
26+
" [\"3\", \"Bayern\", \"Munich\"]\n" +
27+
" ],\n" +
28+
" \"schema\": {\n" +
29+
" \"fields\": [\n" +
30+
" {\n" +
31+
" \"name\": \"id\",\n" +
32+
" \"type\": \"integer\"\n" +
33+
" },\n" +
34+
" {\n" +
35+
" \"name\": \"name\",\n" +
36+
" \"type\": \"string\"\n" +
37+
" },\n" +
38+
" {\n" +
39+
" \"name\": \"city\",\n" +
40+
" \"type\": \"string\"\n" +
41+
" }\n" +
42+
" ],\n" +
43+
" \"foreignKeys\": [\n" +
44+
" {\n" +
45+
" \"fields\": \"city\",\n" +
46+
" \"reference\": {\n" +
47+
" \"resource\": \"cities\",\n" +
48+
" \"fields\": \"name\"\n" +
49+
" }\n" +
50+
" }\n" +
51+
" ]\n" +
52+
" }\n" +
53+
" },\n" +
54+
" {\n" +
55+
" \"name\": \"cities\",\n" +
56+
" \"data\": [\n" +
57+
" [\"name\", \"country\"],\n" +
58+
" [\"London\", \"England\"],\n" +
59+
" [\"Madrid\", \"Spain\"]\n" +
60+
" ]\n" +
61+
" }\n" +
62+
" ]\n" +
63+
"}";
64+
65+
Package dp = new Package(DESCRIPTOR, Paths.get(""), true);
66+
Resource teams = dp.getResource("teams");
67+
DataPackageValidationException dpe = Assertions.assertThrows(DataPackageValidationException.class, () -> teams.checkRelations(dp));
68+
Assertions.assertEquals("Error reading data with relations: Foreign key validation failed: [city] -> [name]: 'Munich' not found in resource 'cities'.", dpe.getMessage());
69+
}
70+
}

‎src/test/java/io/frictionlessdata/datapackage/ForeignKeysTest.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,6 @@ void testForeignKeysBadCase() throws Exception{
3434
() -> teams.checkRelations(pkg));
3535
Throwable cause = ex.getCause();
3636
Assertions.assertInstanceOf(ForeignKeyException.class, cause);
37-
Assertions.assertEquals("Foreign key validation failed: [city] -> [name]: 'Munich' not found in resource 'cities'", cause.getMessage());
37+
Assertions.assertEquals("Foreign key validation failed: [city] -> [name]: 'Munich' not found in resource 'cities'.", cause.getMessage());
3838
}
3939
}

‎src/test/resources/fixtures/datapackages/foreign_keys_invalid.json‎

Lines changed: 7 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -4,26 +4,10 @@
44
{
55
"name": "teams",
66
"data": [
7-
[
8-
"id",
9-
"name",
10-
"city"
11-
],
12-
[
13-
"1",
14-
"Arsenal",
15-
"London"
16-
],
17-
[
18-
"2",
19-
"Real",
20-
"Madrid"
21-
],
22-
[
23-
"3",
24-
"Bayern",
25-
"Munich"
26-
]
7+
["id", "name", "city"],
8+
["1", "Arsenal", "London"],
9+
["2", "Real", "Madrid"],
10+
["3", "Bayern", "Munich"]
2711
],
2812
"schema": {
2913
"fields": [
@@ -54,18 +38,9 @@
5438
{
5539
"name": "cities",
5640
"data": [
57-
[
58-
"name",
59-
"country"
60-
],
61-
[
62-
"London",
63-
"England"
64-
],
65-
[
66-
"Madrid",
67-
"Spain"
68-
]
41+
["name", "country"],
42+
["London", "England"],
43+
["Madrid", "Spain"]
6944
]
7045
}
7146
]

0 commit comments

Comments
 (0)