The CrowdTruth Framework

Installs: 6

Dependents: 0

Suggesters: 0

Security: 0

Stars: 61

Watchers: 19

Forks: 17

Open Issues: 33



Latest Stable Version Build Status Code Coverage Scrutinizer Code Quality

The CrowdTruth Framework implements an approach to machine-human computing for collecting annotation data on text, images, sounds and videos. The approach is focussed specifically on collecting gold standard data for training and evaluation of cognitive computing systems. The original framework was inspired by the IBM Watson project for providing improved (multi-perspective) gold standard (medical) text annotation data for the training and evaluation of various IBM Watson components, such as Medical Relation Extraction, Medical Factor Extraction and Question-Answer passage alignment.

68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f656e2f652f65312f43726f7764666c6f7765722d6c6f676f2e706e67 687474703a2f2f746865636f6c6c656765696e766573746f72636f6d2e632e707265737363646e2e636f6d2f77702d636f6e74656e742f75706c6f6164732f323030392f31302f616d617a6f6e2d6d7475726b2d6c6f676f2e6a7067

The CrowdTruth framework supports the composition of CrowdTruth gathering workflows, where a sequence of micro-annotation tasks can be configured and sent out to a number of crowdsourcing platforms (e.g. CrowdFlower and Amazon Mechanical Turk) and applications (e.g. Expert annotation game Dr. Detective). The CrowdTruth framework has a special focus on micro-tasks for knowledge extraction in medical text (e.g. medical documents, from various sources such as Wikipedia articles or patient case reports). The main steps involved in the CrowdTruth workflow are:

  1. Exploring & processing of input data
  2. Collecting of annotation data
  3. Applying disagreement analytics on the results

These steps are realised in an automatic end-to-end workflow, that can support a continuous collection of high quality gold standard data with feedback loop to all steps of the process. Have a look at our presentations and papers for more details on the research.

Using CrowdTruth

Start using CrowdTruth right now, completely free, and explore all its possiblities. Follow the installation guide to get started, or check out our wiki for all documentation of the platform. We have some crowdsourcing templates ready for you to start with.