EEA Sparql

EEA Sparql is a plone product for fetching data from Linked open data servers (sparql endpoints).

Introduction

It is simple to use, you only have to specify the sparql endpoint url and the sparql query. The results will be converted in an exhibit json, what can easily be reused (ex. by eea.daviz).

Main features

The main features are:

1. create sparql queries
2. create sparql bookmark folders, sparql queries are created automatically, and can be synchronized manually or automatically (with a cronjob). If a query is changed, a new version of the object is created, so older ones are not lost.
3. results are downloadable in various formats: JSON, Exhibit JSON, HTML, XML, XML with Schema, CSV, TSV

Installation

To install eea.sparql into the global Python environment (or a workingenv), using a traditional Zope 2 instance, you can do this:

• When you're reading this you have probably already run easy_install eea.sparql. Find out how to install setuptools (and EasyInstall) here:

• If you are using Zope 2.9 (not 2.10), get pythonproducts and install it via:

  python setup.py install --home /path/to/instance
  

  into your Zope instance.

• Create a file called eea.sparql-configure.zcml in the /path/to/instance/etc/package-includes directory. The file should only contain this:

Alternatively, if you are using zc.buildout and the plone.recipe.zope2instance recipe to manage your project, you can do this:

• Add eea.sparql to the list of eggs to install, e.g.:

  [buildout]
  eggs = eea.sparql
  

• Tell the plone.recipe.zope2instance recipe to install a ZCML slug:

  [instance]
  recipe = plone.recipe.zope2instance
  zcml = eea.sparql
  

You can skip the ZCML slug if you are going to explicitly include the package from another package's configure.zcml file.

You will also need a worker instance to be set up. This is required for the async update of the last working results.

• For both, normal and worker instances, the plone.app.async should be added in the EGG slug:

  eggs =
    ...
    plone.app.async
  

• For the normal instances, the plone.app.async-single_db_instance should be added in the ZCML slug:

  zcml =
    ...
    plone.app.async-single_db_instance
  

• For the worker instances, the plone.app.async-single_db_worker should be added in the ZCML slug:

  zcml =
    ...
    plone.app.async-single_db_worker
  

• Re-run buildout, e.g. with:

  $ ./bin/buildout
  

Dependecies

• Products.ZSPARQLMethod
• eea.versions
• plone.app.async
• eea.cache 7.0+ (optional)

Source code

Latest source code (Plone 4 compatible): - Plone Collective on Github - EEA on Github

Copyright and license

The Initial Owner of the Original Code is European Environment Agency (EEA). All Rights Reserved.

The EEA Sparql (the Original Code) is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

More details under docs/License.txt

Links

Funding

EEA - European Environment Agency (EU)

Changelog

7.4 - (2020-02-28)

• Change: use prop instead of attr when setting value on input tags for jquery 1.9+ compatibility [ichim-david [refs #105112]

7.3 - (2019-12-02)

• Bug fix: fixed height of sparql results when results have large text results [ichim-david refs #111794]

7.2 - (2019-08-05)

• Change: Fix MANIFEST.in file [valentinab25]

7.1 - (2019-08-01)

• Bug fix: fixed setup.py metadata [alecghica refs #108090]
• Bug fix: Avoid memory-leak generator ContentHistoryView while getting Sparql info on view [avoinea refs #108090]

7.0 - (2019-01-28)

• Jenkins: Add sonarqube step [avoinea refs #101552]

6.9 - (2018-12-11)

• Change: updated URLs pointing to eea.europa.eu with [alecghica refs #95849]
• Bug fix: fixed error display on running a SPARQL query [alecghica refs #100666]

6.8 - (2018-05-30)

• Bug fix: Restrict getBRefs only to related items [avoinea refs #95796]

6.7 - (2018-04-04)

• Bug fix: regression fix for bug introduced in previest egg release. Removed call method for Sparql browserView as we have code calling the sparql within eea.daviz and eea.googlecharts which breaks and the template itself calls the checking for the integrity of export data [ichim-david refs #94465]

6.6 - (2018-03-21)

• Upgrade step: added upgrade steps in order to set the exportWorks attribute for sparql objects
• Change: Added loggers, a check regarding the status of the html/csv/tsv export and also a warning message if the export fails [petchesi-iulian refs #89670]

6.5 - (2017-12-12)

• Change: Replace eeacms/zptlint with eeacms/plone-test:4 zptlint [avoinea refs #90415]

6.4 - (2017-11-07)

• Change: Remove Sphinx generated documentation [petchesi-iulian refs #88212]

6.3 - (2017-09-11)

• Bug fix: fixed download error when no result is received [alecghica refs #87576]

6.2 - (2017-08-16)

• Change: Added sparql export method formats (xml, xmlschema, json) to be stored and retrieved from cache [zoltan_andras refs #86464]

6.1 - (2017-07-03)

• Bug fix: Changed download.json to get data from cached results instead of querying the results everytime from [zoltan_andras refs #85891]

6.0 - (2017-06-26)

• Upgrade step: Call / and check zc.async job scheduled to run daily via Site Setup > Async Manager [avoinea refs #85379]
• Change: Move sparql_check_sendmail cron to zc.async [avoinea refs #85379]

5.9 - (2017-05-22)

• Change: Add dedicated async quota for Sparql jobs [avoinea refs #84915]
• Change: Remove hard-dependency on plone.app.async [avoinea refs #84915]
• Change: fixed PyLint warnings and errors [valipod refs #84949]

5.8 - (2017-05-15)

• Change: fixed PyLint warnings and errors [eduard-fironda refs #84949]

5.7 - (2017-04-24)

• Change: updated package information [eduard-fironda]

5.6 - (2016-10-03)

• Bug fix: Update cached results for a SPARQL query after modifying it. [chiridra refs #74166]
• Bug fix: fix broken control panel page template [tiberich]

5.5 - (2016-05-19)

• Bug fix: Fix pylint warnings [ichim-david refs #71940]

5.4 - (2016-05-04)

• Change: changed control panel entry name [alecghica refs #70987]
• Upgrade step: re-import control panel profile [alecghica refs #70987]

5.3 - (2016-03-01)

• Bug fix: take into consideration that sparql_results might miss the blob file [ichimdav refs #69841]
• Bug fix: avoid 'index out of range' error when updating sparql results [danielm-gh refs #69841]

5.2 - (2016-02-15)

• Feature: moved cached_result sparql cache to a blob field in order to decrease object size, especially useful in versioning and history diff [ichimdav refs #17334]
• Upgrade step: Within "Plone > Site setup > Add-ons" click on upgrade button available for eea.sparql with www8 service instance [ichimdav refs #17334]

5.1 - (2016-01-19)

• Feature: changed the default refresh rate for new Sparql objects from "Daily" to "Weekly" [danielm-gh refs #31723]
• Feature: improved the "SPARQL Schedule Status" control panel view - after the e-mail notification about the unqueued sparql queries, the stopped ones are restarted [danielm-gh refs #31723]

5.0 - (2016-01-04)

• Feature: improved the "SPARQL Schedule Status" control panel view - only the unscheduled sparqls are displayed, and they can be restarted all at once (not only individually). Also added a method which can be used to notify by e-mail the development team about unqueued sparql queries. [danielm-gh refs #31031]

4.9 - (2015-12-08)

• Bug fix: fixed pylint warnings. Fixed HISTORY.txt markup [ghicaale refs #30813]

4.8 - (2015-12-07)

• Feature: Add a Control Panel entry to show the async queue status of sparql queries [danielm-gh refs #30665]

4.7 - (2015-11-16)

• Bug fix: Fixed upgrade step 4400 for eea.sparql to start all async jobs for sparal methods [szabozo0 refs #30673]

4.6 - (2015-11-03)

• Bug fix: added profile dependency to Products.DataGridField [szabozo0]
• Bug fix: use 'code/pre' tags instead of 'blockquote' for sparql queries and errors [danielm-gh refs #29992]

4.5 - (2015-08-18)

• Change: Auto-include zcml within plone context in order to make this package work without having to add it within buildout zcml directive. [avoinea]

4.4 - (2015-07-06)

• Bug fix: Added upgrade step to restart the sparql sync jobs [szabozo0 refs #25448]

4.3 - (2015-05-15)

• Bug fix: Fixed a conflict between provenance info for sparql and arguments for sparql queries [szabozo0 refs #25341]

4.2 - (2015-05-05)

• Bug fix: updateLastWorkingResults now checks for Querytimeout exception in order to avoid crash of async_updateLastWorkingResults [ichimdav refs #25018]
• Bug fix: quote the query when user tries to download the dataset [szabozo0]
• Change: endpoint_url now uses the isURL validator in order to ensure that the endpoint url is a proper link [ichimdav refs #24130]
• Change: download links for datasets no longer carry a question mark if there are no arguments massed [ichimdav refs #18753]

4.1 - (2015-04-02)

• Change: Updated Multiple data source schema extender adapter to correlate with the changes in eea.daviz [olimpiurob refs #24121]

4.0 - (2015-03-17)

• Change: Switched to curl in the jenkins build install script [olimpiurob refs #22402]
• Change: Changed fetch url for jenkins build install script to the install script from the main EEA CPB repository [olimpiurob refs #22402]

3.9 - (2015-02-03)

• Feature: refined history messages for results [szabozo0 refs #22240]

3.8 - (2014-11-20)

• Bug fix: fixed "add new argument" CSS [ghicaale refs #20945]

3.7 - (2014-10-15)

• Feature: added a howto for building Sparqls with arguments [pasoviul refs #19537]
• Feature: changed dropdown values to labels insted of URIs [pasoviul refs #19537]
• Bug fix: fixed error on 'Test and preview query' [pasoviul refs #19537]
• Bug fix: fixed argument handling between sparq and ZSPARQL [pasoviul refs #19537]
• Change: updated sparqls to provide queries for arguments, besides names and types [pasoviul refs #19537]
• Bug fix: fixed documentation generated by Jenkins [ghicaale refs #19537]

3.6 - (2014-10-01)

• Feature: removed "Caching" action. We are now using the more complex "Cache" action registered by eea.cache [ghicaale refs #21143]
• Feature: added upgrade step to remove absolete "Caching" action [ghicaale refs #21143]

3.5 - (2014-09-15)

• Bug fix: Fixed html, csv, tsv and exhibit json exports [pasoviul refs #19537]

3.4 - (2014-08-27)

• Change: Cleanup memcache invalidation code as it's already handled by eea.cache (7.0+) [voineali refs #20678]

3.3 - (2014-01-21)

• Bug fix: removed wrongly added blockquotes within README.rst [ichim-david refs #18064]
• Feature: adding Sphinx-generated documentation in Page Template format [batradav refs #9502]

3.2 - (2013-12-10)

• Bug fix: fixed ReST markup of HISTORY.txt [ghicaale refs #17716]

3.1 - (2013-12-10)

• Change: removed versions macro; now using viewlet [simiamih refs #17321]

3.0 - (2013-11-20)

• Bug fix: Fixed syncing of sparql bookmark folders [szabozo0 refs #17260]

2.9 - (2013-08-08)

• Change: Update the latest working results with plone.app.async [szabozo0 #14826]

2.8 - (2013-07-08)

• Change: On preview use a LIMIT 10 to have faster response. When a new resultset is too large don't store it's text version for comparasion is it takes a lot of time, and blocks the instance [szabozo0 #14826]

2.7 - (2013-06-17)

• Feature: Changed data provenance to allow multiple data provenances [szabozo0 #9561]

2.6 - (2013-05-24)

• Bug fix: Updated for eea.versions v6.0 [szabozo0 refs #14627]

2.5 - (2013-05-20)

• Feature: Removed lovely.memcached dependency [voineali refs #14343]
• Bug fix: Fixed preview for sparql queries when arguments are missing [szabozo0 #14349]
• Bug fix: conform to latest eea.versions api changes [tiberich #14008]

2.4 - (2013-02-05)

• Feature: added information for contributors [ciobabog refs #13892]

• Upgrade step: Within "Plone > Site setup > Add-ons" click on upgrade button available for eea.sparql.

• Bug fix: Fixed tests for eea.sparql [szabozo0 refs #9424]

• Feature: Handling specific annotations for data values [voineali refs #9558]

• Bug fix: Fixed preview for queries returning empty fields [szabozo0 refs #9608]

• Feature: Added quick preview for sparql queries [szabozo0 refs #9608]

• Feature: Data persistence and versioning

  • Added versioning for sparql results

  - Possibility to select static & live queries [szabozo0 refs #9424]

2.3 - (2013-01-18)

• Bug fix: Fixed compatibility with jquery 1.7.2 [szabozo0]

2.2 - (2013-01-10)

• Change: moved the olderVersions to the portalMessage div that displays the old versions link for eea.sparql templates [ichimdav #10182]

2.1 - (2012-11-09)

• Feature: Added i18n translations [avoinea]

2.0 - (2012-10-08)

• Feature: Added Visualization data source provenance adapter for Sparql objects [voineali refs #5490]
• Bug fix: Updated converter for results containing exceptions [szabozo0 refs #5515]
• Change: Merged daviz-view.json and daviz-relateditems.json into one single json file called daviz.json [voineali refs #5449]
• Bug fix: Allow forcing Sparql column-types within daviz-view.json [voineali refs #5511]
• Bug fix: Fixed SPARQL bookmarks folder default view [szabozo0 #5512]
• Bug fix: don't break bookmarks view page when endpoint doesn't work [tiberich #5534]

1.8 - (2012-08-27)

• Feature: Show action "create visuallization" for anonymous. This makes it clear what functions are available and "call new users to action". [demarant]
• Bug fix: existing visualisation of sparql are now shown even though the image tile is not available (valentine.imagescales). Use grid style for table. [demarant fixes #5458]
• Bug fix: fixed issue with column names json coversion when ending with 'label'. Now it uses simile exhibit explicit type formatting. [demarant]

1.7 - (2012-08-10)

• Bug fix: Fixed columns order [szabozo0 refs #5137]
• Feature: Added shortcuts on sparql bookmarks for creating visualizations. [szabozo0 refs #5347]

1.6 - (2012-07-23)

• Bug fix: Fixed sparql bookmarks view [szabozo0 #5388]

1.5 - (2012-07-13)

• Bug fix: Fixed sparql table preview CSS when used as jQuery overlay [voineali]
• Bug Fix: Improved layout of sparql bookmarks folder. Focus on datasets. Do not show administrative actions for anonymous. Removed unnecessary code. Improved i18n and labels. [demarant refs #5151]
• Change: layout changes to the view page to focus on the dataset download, clarify formats and the linked data information. better labels and descriptions. [demarant fixes #5317]
• Bug Fix: Fixed i18n missing tags and duplicate keys in .pt files [prospchr refs #5162]

1.4 - (2012-06-13)

• Bug fix: fixed markup of HISTORY.txt file [ciobabog refs #5231]
• Feature: Add visualizations column on sparql folder listing view [voineali refs #5005]
• Bug fix: Include dependencies within zcml and profile metadata [voineali]

1.3 - (2012-04-26)

• Bug fix: Split @@sparql-download into multiple views (download.tsv, download.csv, etc) in order to easily cache and purge them with plone.app.caching [voineali #5074]
• Bug fix: Set default format to html on download view [szabozo0 #5089]

1.2 - (2012-04-11)

• Bug fix: Invalidate memcache and varnish for Sparql instances and related items [voineali #5074]
• Feature: Made timeout mandatory, and set default value to 10 seconds [szabozo0 #5079]
• Bug fix: Fixed synchronization problems for sparql bookmark folders [szabozo0 #5078]
• Bug fix: Remove comments from queries so sparql-query should not fail [szabozo0 #5073]
• Bug fix: Explicitly close connections with remote servers and serve files in small chuncks on sparql to JSON/XML export. [voineali #5073]
• Upgrade step: Manually remove apache quickfixes from frontend servers

1.1 - (2012-03-30)

• Change: Added default workflow mapping. [demarant #5002]
• Upgrade step: Within ZMI > Site setup > Import types. For workflow change if have old content to migrate: Plone > Types > change workflow:
  • Sparql -> one_state_workflow;
  • SparqlBookmarksFolder -> folder_workflow define the state mapping as you prefer.
• Upgrade step: Within "Plone > Site setup > Add-ons" click on upgrade button available for eea.sparql [voineali #4968]
• Feature: Added a table preview for sparql objects when used togheter with eea.relations and eea.facetednavigation [voineali #4968]
• Change: moved sparql.css in portal_css [szabozo0 #4968]
• Feature: added page for syncing all sparql bookmark folders [szabozo0 #5004]
• Change: added default icons for sparql and sparql bookmarks folder [demarant]
• Change: Improved table preview dissolve design and template headings [demarant]
• Bug fix: Fixed default view for sparql, so workflow dropdown is now displayed [szabozo0 #5002]
• Bug fix: Fixed format of column properties in exhibit json [szabozo0 #4992]

1.0 - (2012-03-05)

• Initial release [Zoltan Szabo]