soweego: link Wikidata to large catalogs¶
soweego is a pipeline that connects Wikidata to large-scale third-party catalogs.
soweego is the only system that makes statisticians, epidemiologists, historians, and computer scientists agree. Why? Because it performs record linkage, data matching, and entity resolution at the same time. Too easy, they all seem to be synonyms!
Oh, soweego also embeds Machine Learning and advocates for Linked Data.
Official Project Pages¶
soweego is made possible thanks to the Wikimedia Foundation:
Highlights¶
Run the whole pipeline, or
use the command line;
import
large catalogs into a SQL database;gather
live Wikidata datasets;connect
them to target catalogs via rule-based and supervised linkers;upload
links to Wikidata and Mix’n’match;synchronize
Wikidata to imported catalogs;enrich
Wikidata items with relevant statements.
Get Ready¶
Install Docker and Compose, then enter soweego:
$ git clone -b v1.1 https://github.com/Wikidata/soweego.git
$ cd soweego
$ ./docker/run.sh
Building soweego
...
root@70c9b4894a30:/app/soweego#
Now it’s too late to get out!
Run the Pipeline¶
Piece of cake:
:/app/soweego# python -m soweego run CATALOG
Pick CATALOG
from discogs
, imdb
, or musicbrainz
.
These steps are executed by default:
import the target catalog into a local database;
link Wikidata to the target with a supervised linker;
synchronize Wikidata to the target.
Results are in /app/shared/results
.
Use the Command Line¶
You can launch every single soweego action with CLI commands:
:/app/soweego# python -m soweego
Usage: soweego [OPTIONS] COMMAND [ARGS]...
Link Wikidata to large catalogs.
Options:
-l, --log-level <TEXT CHOICE>...
Module name followed by one of [DEBUG, INFO,
WARNING, ERROR, CRITICAL]. Multiple pairs
allowed.
--help Show this message and exit.
Commands:
importer Import target catalog dumps into a SQL database.
ingester Take soweego output into Wikidata items.
linker Link Wikidata items to target catalog identifiers.
run Launch the whole pipeline.
sync Sync Wikidata to target catalogs.
Just two things to remember:
you can always get
--help
;each command may have sub-commands.
Find all details in the CLI Documentation.
How-tos¶
CLI Documentation¶
API Documentation¶
Contribute¶
Note
the best way is to Import a new catalog.
Please also have a look here:
Experiments & notes¶
License¶
The source code is under the terms of the GNU General Public License, version 3.