DataGraft API
Making data work for your apps
Security and authentication¶
The DataGraft API is restricted to authenticated application, using an access key and a secret. You can manage your access keys and secrets using the DataGraft API key manager.
HTTPS
The API is only accessible through HTTPS.
Authentication
The authentication is done using HTTP Basic authentication, and therefore the HTTP authorization header field. It consists of the text 'Basic ’ concatenated with the base64 encoding of the API key and the API secret separated by a colon.
Example in JavaScript:
>> request.headers.Authorization = 'Basic ' + btoa(key + ':' + secret);
=> "Basic a2V5OnNlY3JldA=="
CORS
The DataGraft API supports cross-origin requests without domains restrictions.
Model reference¶
The DataGraft API supports RDF/XML and JSON-LD.
Namespaces
-
dcat - http://www.w3.org/ns/dcat#
-
foaf - http://xmlns.com/foaf/0.1/
Dataset
Property | Range | Description |
---|---|---|
rdf:type | "dcat:Dataset" |
|
dct:title | Text | |
dct:description | Text | |
dct:issued | Date | |
dct:modified | Date | |
dcat:keyword | Text [] | List of keywords |
dct:publisher | Text | |
dcat:distribution | URI [] | List of the connected distributions |
dcat:public | "true" or "false" |
Distribution
Property | Range | Description |
---|---|---|
rdf:type | "dcat:Distribution" |
|
dct:title | Text | |
dct:description | Text | |
dct:issued | Date | |
dct:modified | Date | |
dcat:mediaType | Mimetype (Text) | |
dcat:accessURL | URI | The SPARQL endpoint URL for RDF distributions |
dcat:fileID | Text | UUID for the raw file generated by the system. |
dcat:fileName | Text | Original file name |
Transformation
Property | Range | Description |
---|---|---|
rdf:type | "dcat:Transformation" |
|
dct:title | Text | |
dct:description | Text | |
dct:issued | Date | |
dct:modified | Date | |
dcat:keyword | Text [] | List of keywords |
dct:publisher | Text | |
dcat:clojureDataID | Text | |
dcat:jsonDataID | Text | |
dcat:sourceData | URI | Reference to the input data Distribution object |
dcat:public | "true" or "false" |
|
dcat:transformationType | "pipe" or "graft" |
|
dcat:transformationCommand | Text | entry point (command/function) of the transformation. "my-pipe" for pipe and "my-graft " for graft are recommended. |
JSON-LD
JSON-LD or JavaScript Object Notation for Linked Data is the recommended way to consume and provide data. You can use the same software as traditional JSON. The @context
, @type
and @id
properties are JSON-LD specific.
Example:
{
"@context": {
"dcat": "http://www.w3.org/ns/dcat#",
"foaf": "http://xmlns.com/foaf/0.1/",
"dct": "http://purl.org/dc/terms/",
"xsd": "http://www.w3.org/2001/XMLSchema#"
},
"@type": "dcat:Catalog",
"@id": "http://dapaas.eu/users/dapaas/catalog",
"dct:publisher": "dapaas",
"dcat:record": [
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://dapaas.eu/users/15052/dataset/example-1",
"dct:issued": "2014-09-18",
"dct:modified": "2014-09-17",
"dct:publisher": "dapaas",
"dct:title": "Title for DS4 DaPaaS dataset UPDATED",
"dcat:public": "true"
},
...
]
}
Usage¶
Endpoint
https://api.datagraft.net/
Parameters
Unless otherwise stated, all the parameters are header parameters. It means they are sent in the HTTP headers.
For example, the q
search parameter of a catalog search is sent in the HTTP headers:
GET /catalog/datasets/search HTTP/1.1
Host: api.datagraft.net
Accept: application/ld+json
Accept-Encoding: gzip, deflate, sdch
Authorization: Basic a2V5OnNlY3JldA==
q: mysearch
Datasets Catalog¶
List All Datasets¶
List of datasets catalog records using the DCAT vocabulary in RDF or JSON-LD.
Use the showShared header parameter to include the public datasets.
List user's datasetsGET/catalog/datasets/catalog
- showShared
y|n
(optional)defaults to
n
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
showShared: n
200
Body
{
"@context": { ... }
"@type": "dcat:Catalog",
"dct:publisher": "dapaas",
"@id": "http://dapaas.eu/users/dapaas/catalog",
"dcat:record": [
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/dataset/1",
"dct:issued": "2014-09-16",
"dct:modified": "2014-09-17",
"dct:title": "My first DaPaaS dataset",
"dcat:public": "true",
"dct:publisher": "dapaas"
},
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/dataset/2",
"dct:issued": "2014-09-15",
"dct:modified": "2014-09-17",
"dct:title": "My second DaPaaS dataset",
"dcat:public": "false",
"dct:publisher": "dapaas"
}
]
}
Search Datasets¶
Text search on the datasets metadata.
Use the showShared header parameter to include the public datasets.
Search user's datasetsGET/catalog/datasets/search
- q
text
(required)The text query
- showShared
y|n
(optional)defaults to
n
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
showShared: n
q: DaPaaS
200
Body
{
"@context": { ... }
"@type": "dcat:Catalog",
"dct:publisher": "dapaas",
"@id": "http://dapaas.eu/users/dapaas/catalog",
"dcat:record": [
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/dataset/1",
"dct:issued": "2014-09-16",
"dct:modified": "2014-09-17",
"dct:title": "My first DaPaaS dataset",
"dcat:public": "true",
"dct:publisher": "dapaas"
},
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/dataset/2",
"dct:issued": "2014-09-15",
"dct:modified": "2014-09-17",
"dct:title": "My second DaPaaS dataset",
"dcat:public": "false",
"dct:publisher": "dapaas"
}
]
}
Dataset CRUD¶
Create DatasetPOST/catalog/datasets
If the description contains no @id
, the system will generate one from the title.
A successful creation returns the URI of the new dataset.
- Content-Type
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Content-Type: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
Body
{
'@context': {...},
'@type': 'dcat:Dataset',
'dct:title': 'My new dataset',
'dct:description': 'Example dataset for the documentation',
'dcat:public': 'false'
}
200
Body
{
"@id": "http://dapaas.eu/users/15052/dataset/my-new-dataset-1"
}
Read DatasetGET/catalog/datasets
Get the complete dataset description using the DCAT vocabulary in RDF or JSON-LD.
- dataset-id
URI
(required)URI of the dataset, taken from the catalog
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
dataset-id: http://dapaas.eu/users/15052/dataset/preview-1
200
Body
{
"@context": {...},
"@id": "http://dapaas.eu/users/15052/dataset/preview-1",
"@type": "dcat:Dataset",
"dcat:distribution": [
"http://dapaas.eu/users/15052/distribution/July+-+2005-2006.csv-1",
...],
"dcat:keyword": [],
"dct:description": "Dataset containing the previewed files from Grafterizer",
"dct:public": false,
"dct:publisher": "dapaas",
"dct:title": "Previewed files"
}
Update DatasetPUT/catalog/datasets
Note: The complete dataset should be included. It is not merged with the previous version.
- Content-Type
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Content-Type: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
Body
{
'@context': {...},
'@type': 'dcat:Dataset',
'@id': 'http://dapaas.eu/users/15052/dataset/my-new-dataset-1',
'dct:title': 'My new dataset',
'dct:description': 'My dataset now public',
'dcat:public': 'true'
}
200
Delete DatasetDELETE/catalog/datasets
When deleting a dataset, the system will also delete the attached distributions.
- dataset-id
URI
(required)URI of the dataset, taken from the catalog
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
dataset-id: http://dapaas.eu/users/15052/dataset/preview-1
200
Distribution CRUD¶
Create DistributionPOST/catalog/distributions
A successful creation returns the URI of the new dataset.
Attachments
meta: Distribution metadata as RDF or JSON-LD.
Use Content-Type
to determine the metadata-format. The metadata should contain the file meta data (content-type, size, name). See the Model reference for the details.
If the metadata doesn’t contain a distribution id, the system will generate one.
file: The raw file content.
If the distribution is not a RDF distribution, the file attachment is optional.
- dataset-id
URI
(required)URI of the dataset container
- Content-Type
mimetype
(required)multipart/mixed
ormultipart/form-data
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
Body
meta (application/ld+json): {
'@context': {...},
'@type': 'dcat:Distribution',
'dct:title': 'my-distribution',
'dct:description': 'Distribution example',
'dcat:fileName': 'source.csv',
'dcat:mediaType': 'text/csv'
},
file: <RAW>
200
Body
{
"@id": "http://dapaas.eu/users/15052/distribution/my-distribution-1"
}
Read DistributionGET/catalog/distributions
Get the complete distribution description.
- distrib-id
URI
(required)URI of the distribution, taken from the catalog
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
distrib-id: http://dapaas.eu/users/15052/distribution/my-distribution-1
200
Body
{
"@context": {...},
"@id": "http://dapaas.eu/users/15052/distribution/my-distribution-1",
"@type": "dcat:Distribution",
"dct:title": "my-distribution",
"dct:description": "Distribution example",
"dct:fileID": "159532fa-62ec-4543-a662-fc7a081181b3.bin",
"http://www.w3.org/ns/dcat#mediaType": "text/csv",
"http://www.w3.org/ns/dcat#fileName": "source.csv"
}
Update DistributionPUT/catalog/distributions
Attachments
meta: Distribution metadata as RDF or JSON-LD.
Use Content-Type
to determine the metadata-format.
file: The raw file content.
The file attachment is optional. If provided, it will overwrite the previous one. The metadata should also be provided to contain the updated file metadata (content-type, size, name).
- Content-Type
mimetype
(required)multipart/mixed
ormultipart/form-data
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
Body
meta (application/ld+json): {
'@context': {...},
'@type': 'dcat:Distribution',
'@id': 'http://dapaas.eu/users/15052/distribution/my-distribution-1',
'dct:title': 'my-distribution',
'dct:description': 'Distribution example',
'dcat:fileName': 'source.csv',
'dcat:mediaType': 'text/csv'
},
file: <RAW>
200
Delete DistributionDELETE/catalog/distributions
When deleting a distribution, the system will also release the attached RDF repositories.
- distrib-id
URI
(required)URI of the distribution
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
dataset-id: http://dapaas.eu/users/15052/dataset/preview-1
200
RDF Repositories¶
Querying data¶
The RDF repositories API are accessible at the address https://rdf.datagraft.net/
.
This endpoint exposes the OpenRDF Sesame API.
However the authentication is still required. Using the Authorization header as defined in the beginning of this document.
RDF repository¶
Provision RDF repositoryPUT/catalog/distributions/repository
Use /catalog/distributions/repository_async
to provision the repository asynchronously.
This service returns a repository URL.
- distrib-id
URI
(required)URI of the distribution
- repository-id
text
(optional)Identifier for the new repository (lowercased, alphanumerical)
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
distrib-id: http://dapaas.eu/users/15052/distribution/my-distribution-1
repository-id: my-new-repository
200
Body
{
"access-url": "<repository url>"
}
Release RDF repositoryDELETE/catalog/distributions/repository
- distrib-id
URI
(required)URI of the distribution
- repository-id
text
(required)Identifier for the new repository (lowercased, alphanumerical)
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
distrib-id: http://dapaas.eu/users/15052/distribution/my-distribution-1
repository-id: my-new-repository
200
Transformations Catalog¶
Transformations are usefull to clean and convert your data. The transformation main component consists of a Grafter script.
You can use Grafterizer to generate the transformations and the Clojure source code.
List all Transformations¶
List of transformations records using the DCAT vocabulary in RDF or JSON-LD.
Use the showShared header parameter to include the public transformations.
List user's transformationsGET/catalog/transformations/catalog
- showShared
y|n
(optional)defaults to
n
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
showShared: n
200
Body
{
"@context": { ... }
"@type": "dcat:Catalog",
"@id": "http://dapaas.eu/users/dapaas/catalog",
"dct:publisher": "dapaas",
"dcat:record": [
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/transformation/1",
"dct:issued": "2014-09-16",
"dct:modified": "2014-09-17",
"dct:title": "My first DaPaaS transformation",
"dcat:public: "true",
"dct:publisher": "dapaas"
},
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/transformation/2",
"dct:issued": "2014-09-15",
"dct:modified": "2014-09-17",
"dct:title": "My second DaPaaS transformation",
"dcat:public: "false",
"dct:publisher": "dapaas"
}
]
}
Search transformations¶
Text search on the transformations metadata.
Use the showShared header parameter to include the public transformations.
Search user's transformationsGET/catalog/transformations/search
- q
text
(required)The text query
- showShared
y|n
(optional)defaults to
n
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
showShared: n
q: DaPaaS
200
Body
{
"@context": { ... }
"@type": "dcat:Catalog",
"@id": "http://dapaas.eu/users/dapaas/catalog",
"dct:publisher": "dapaas",
"dcat:record": [
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/transformation/1",
"dct:issued": "2014-09-16",
"dct:modified": "2014-09-17",
"dct:title": "My first DaPaaS transformation",
"dcat:public: "true",
"dct:publisher": "dapaas"
},
{
"@type": "dcat:CatalogRecord",
"foaf:primaryTopic": "http://eu.dapaas/transformation/2",
"dct:issued": "2014-09-15",
"dct:modified": "2014-09-17",
"dct:title": "My second DaPaaS transformation",
"dcat:public: "false",
"dct:publisher": "dapaas"
}
]
}
Transformation CRUD¶
Create TransformationPOST/catalog/transformations
A successful creation returns the URI of the new transformation.
If the description contains no @id
, the system will generate one from the title.
Note: only new transformations are created with this method. Any subsequent calls with the same data generate an error response.
Attachments
meta: Transformation metadata as RDF or JSON-LD.
Use Content-Type
to determine the metadata-format.
If the metadata doesn’t contain a distribution id, the system will generate one.
tr-clojure: The code of the transformation as Grafter Clojure.
- Content-Type
mimetype
(required)multipart/mixed
ormultipart/form-data
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
Body
meta (application/ld+json): {
'@context': {...},
'@type': 'dcat:Transformation',
'dct:title': 'My new transformation',
'dct:description': 'Example transformation for the documentation',
'dcat:public': 'false',
'dct:modified': '2014-09-17',
'dcat:transformationType': 'pipe',
'dcat:transformationCommand': 'my-pipe'
},
tr-clojure: "<clojure source code>"
200
Body
{
"@id": "http://dapaas.eu/users/15052/transformation/my-new-transformation-1"
}
Read TransformationGET/catalog/transformations
Get the complete transformation description using the DCAT vocabulary in RDF or JSON-LD.
- transformation-id
URI
(required)URI of the transformation, taken from the catalog
- Accept
mimetype
(required)application/ld+json
orapplication/rdf+xml
Headers
Accept: application/ld+json
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
200
Body
{
"@context": {...},
"@id": "http://dapaas.eu/users/15052/transformation/my-new-transformation-1",
"@type": "dcat:Transformation",
"dct:public": false,
"dct:publisher": "dapaas",
"dct:title": "My new transformation",
"dcat:keyword": [],
"dct:description": "Example transformation for the documentation",
"dct:clojureDataID": "a959cdca-8a49-4a10-bba0-98dc59083246.clj",
"dct:jsonDataID": "2184a1bc-b0b7-450c-b839-9e74be065e77.json",
"dct:modified: "2015-07-15",
"http://www.w3.org/ns/dcat#transformationCommand": "my-pipe",
"http://www.w3.org/ns/dcat#transformationType": "pipe"
}
Update TransformationPUT/catalog/transformations
Attachments
meta: Optional transformation metadata as RDF or JSON-LD.
Use Content-Type
to determine the metadata-format.
Note: The complete transformation metadata should be included. It is not merged with the previous version.
tr-clojure: Optional code of the transformation as Grafter Clojure.
meta and tr-clojure can be managed separately.
- Content-Type
mimetype
(required)multipart/mixed
ormultipart/form-data
- transformation-id
URI
(required)URI of the transformation, taken from the catalog
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
Body
meta (application/ld+json): {
'@context': {...},
'@type': 'dcat:Transformation',
'dct:title': 'My new transformation',
'dct:description': 'Updated transformation',
'dcat:public': 'false',
'dct:modified': '2014-09-17',
'dcat:transformationType': 'pipe',
'dcat:transformationCommand': 'my-pipe'
},
tr-clojure: "<clojure source code>"
200
Delete TransformationDELETE/catalog/transformations
- transformation-id
URI
(required)URI of the transformation, taken from the catalog
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
200
Distribution Clojure code¶
The transformation code consists of clojure using the Grafter API.
Retrieve Clojure codeGET/catalog/distributions/code/clojure
- transformation-id
URI
(required)URI of the transformation
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
200
Delete Clojure codeDELETE/catalog/distributions/code/clojure
- transformation-id
URI
(required)URI of the transformation
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
200
Grafter Transformations¶
Performs Tabular transformation¶
Tabular transformationPOST/dapaas-services/grafter/transformation/pipe
Performs a PIPE transformation with Grafter and loads the result in a repository.
You can provide an input-distribution
URI or send an input-file
as file attachment.
- input-distribution
URI
(optional)URI of an existing distribution
- result-distribution
URI
(required)URI of an existing distribution
- result-file
text
(required)Result filename
- result-type
mimetype
(required)Result mimetype
text/csv
- transformation-id
URI
(required)URI of an existing distribution
Headers
Content-Type: input-file
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
result-distribution: http://dapaas.eu/users/15052/distribution/my-result-distribution-1
result-file: transformed-data.csv
result-type: text/csv
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
Body
input-file (text/csv): {
name,age,gender
…
}
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
input-distribution: http://dapaas.eu/users/15052/distribution/my-distribution-1
result-distribution: http://dapaas.eu/users/15052/distribution/my-result-distribution-2
result-file: transformed-data.csv
result-type: text/csv
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
200
Performs Graft Transformation¶
Graft TransformationPOST/dapaas-services/grafter/transformation/graft
Performs a GRAFT transformation with Grafter and loads the result in a repository.
You can provide an input-distribution
URI or send an input-file
as file attachment.
- input-distribution
URI
(optional)URI of an existing distribution
- result-distribution
URI
(required)URI of an existing distribution
- result-file
text
(required)Result filename
- result-type
mimetype
(required)Result mimetype
application/edn
- transformation-id
URI
(required)URI of an existing distribution
Headers
Content-Type: input-file
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
result-distribution: http://dapaas.eu/users/15052/distribution/my-result-distribution-1
result-file: transformed-data.nt
result-type: application/edn
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
Body
input-file (text/csv): {
name,age,gender
…
}
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
input-distribution: http://dapaas.eu/users/15052/distribution/my-distribution-1
result-distribution: http://dapaas.eu/users/15052/distribution/my-result-distribution-2
result-file: transformed-data.nt
result-type: application/edn
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
200
Preview Transformation¶
Preview TransformationPOST/dapaas-services/grafter/transformation/preview
Performs a data transformation with Grafter and returns directly the output without saving it.
Attachments
input-file: Optional input file (as an alternative to the input-distribution header).
transformation-code: The Clojure code (as an alternative to the transformation-id header).
- transformation-id
URI
(optional)URI of an existing distribution
- input-distribution
URI
(optional)URI of an existing distribution
- transformation-type
text
(optional)graft
orpipe
. Required when attaching the transformation-code.- command
text
(optional)The command to execute. Required when attaching the transformation-code.
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
input-distribution: http://dapaas.eu/users/15052/distribution/my-distribution-1
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-id: http://dapaas.eu/users/15052/transformation/my-new-transformation-1
Body
input-file (text/csv): {
person-uri,name,gender
…
}
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
input-distribution: http://dapaas.eu/users/15052/distribution/my-distribution-1
transformation-type: pipe
command: my-pipe
Body
transformation-code (text/plain): {
(defpipe my-pipe "Example pipeline" [data-file] (-> (read-dataset data-file :format :csv)))
}
Headers
Authorization: Basic YXBpa2V5OmFwaXBhc3M=
transformation-type: pipe
command: my-pipe
Body
input-file (text/csv): {
person-uri,name,gender
…
},
transformation-code (text/plain): {
(defpipe my-pipe "Example pipeline" [data-file] (-> (read-dataset data-file :format :csv)…))
}
200
Body
:name,:age:gender
…