Welcome to Orchestra!¶

Install Dependencies¶

Orchestra requires Python 3 and Django version 1.8 or higher to run, so make sure you have them installed. We recommend setting up a virtual environment to isolate your Python dependencies, and we’re fond of virtualenvwrapper to make that process easier. Make sure to create your virual environment with Python 3 by passing --python=/path/to/bin/python3 if it isn’t your default development setup.

Orchestra requires a number of Python dependencies to run. You can install them by simply pulling down and installing our requirements.txt file:

wget https://raw.githubusercontent.com/unlimitedlabs/orchestra/stable/requirements.txt
pip install -r requirements.txt

Create a Django Project¶

Orchestra is a Django app, which means that it must be run within a Django project (for more details, read the Django tutorial on this topic). Start a project with django-admin startproject your_project, replacing your_project with your favorite project name (but don’t name it orchestra, which will conflict with our namespace). From here on out, this document will assume that you stuck with your_project, and you should replace it appropriately.

Install and Configure Orchestra¶

Next, let’s get Orchestra installed and running. To get the code, just install using pip: pip install orchestra.

Orchestra has a number of custom settings that require configuration before use. First, download the default Orchestra settings file and place it next to the project settings file:

wget https://raw.githubusercontent.com/unlimitedlabs/orchestra/stable/example_project/example_project/orchestra_settings.py
mv orchestra_settings.py your_project/your_project

Next, edit the orchestra_settings.py file:

• Add 'simple_workflow' to settings.ORCHESTRA_WORKFLOWS in the “General” section if you want to run the demo workflow (instructions below), and add 'journalism_workflow' if you want to run the journalism workflow.
• Adjust your email settings. By default, Orchestra will direct all messages to the console, but for a realistic registration workflow you’ll want to set up a real mail server that can actually send emails.
• Change settings like the ORCHESTRA_PROJECT_API_SECRET from 'CHANGEME' to more appropriate values.
• Optionally, add 3rd party credentials in the “3rd Party Integrations” section so that Orchestra can store files on Amazon S3, use Google Apps and Slack to help communicate with expert workers, and track usage in Google Analytics.

Then, at the bottom of your existing settings file (your_project/your_project/settings.py), import the Orchestra settings:

from .orchestra_settings import setup_orchestra
setup_orchestra(__name__)

You’ll also need to set up Orchestra’s URLs, so that Django knows where to route users when they view Orchestra in the browser. If you don’t have any URLs of your own yet, you can just download our barebones example file with wget https://raw.githubusercontent.com/unlimitedlabs/orchestra/stable/example_project/example_project/urls.py.

Alternatively, just make sure to add the following code inside the urlpatterns variable in your_project/your_project/urls.py:

# Admin Views
url(r'^orchestra/admin/',
    include(admin.site.urls)),

# Registration Views
# Eventually these will be auto-registered with the Orchestra URLs, but for
# now we need to add them separately.
url(r'^orchestra/accounts/',
    include('registration.backends.default.urls')),

# Optionally include these routes to enable user hijack functionality.
url(r'^orchestra/switch/', include('hijack.urls')),

# Logout then login is not available as a standard django
# registration route.
url(r'^orchestra/accounts/logout_then_login/$',
    auth_views.logout_then_login,
    name='logout_then_login'),

# Orchestra URLs
url(r'^orchestra/',
    include('orchestra.urls', namespace='orchestra')),

# Beanstalk Dispatch URLs
url(r'^beanstalk_dispatch/',
    include('beanstalk_dispatch.urls')),

Finally, you’ll need to get the database set up. Create your database with python manage.py migrate. You’ll also want to make sure you have loaded our example workflows and set up some user accounts to try them out. To load the workflows, run:

python manage.py loadworkflow <APP_LABEL> <WORKFLOW_VERSION>

Each of our example workflows provides a set of sample users already configured with proper certifications. To load them, run:

python manage.py loadworkflowsampledata <WORKFLOW_SLUG>/<WORKFLOW_VERSION>

If you would like to load all of the workflows, then run:

python manage.py loadallworkflows

The example workflows we currently release with Orchestra are:

• A simple demo workflow with one human and one machine step. Its app label is simple_workflow, its workflow slug is simple_workflow, and the latest version is v1.
• A more complicated journalism workflow. Its app label is journalism_workflow, its workflow slug is journalism, and the latest version is v1.

In addition, you can use the Orchestra admin (http://127.0.0.1:8000/orchestra/admin) to create new users and certifications of your own at any time once Orchestra is running. If you haven’t created an admin account for your Django project, you can load a sample one (username: admin, password: admin) with python manage.py loaddata demo_admin.

We provide the option to use the third-party package django-hijack to act on behalf of users. To enable this setting, ensure that the following setting is set HIJACK_ALLOW_GET_REQUESTS = True, in addition to including the urls referenced above.

Now Orchestra should be ready to go! If you’re confused about any of the above, check out our barebones example project.

Run Orchestra¶

Now that Orchestra is configured, all that remains is to fire it up! Run your Django project with python manage.py runserver (you’ll want to switch to something more robust in production, of course), and navigate to http://127.0.0.1:8000/orchestra/app in your favorite browser.

If you see the Orchestra sign-in page, your setup is working! If you loaded the simple workflow’s sample data above, logging in as its user (username demo, password demo) should show you a dashboard with no available tasks.

Run the Example Project Demo¶

To give you a feel for what it means to run an Orchestra workflow from end to end, we’ve included a very simple example workflow with two steps, one machine and one human. The machine step takes a URL and extracts a random image from the page. The human step asks an expert to rate how “awesome” the image is on a scale from one to five. If you’re interested in how we defined the workflow, take a look at the code, though we walk through a more interesting example in this documentation.

We’ve written an interactive script to walk through this simple workflow. To run it:

• Make sure you added simple_workflow to your ORCHESTRA_WORKFLOWS setting following the previous section.

• Make sure you loaded the workflow and its sample data following the previous section. This should have created a user with username demo and password demo.

• Run the interactive walkthrough:

  python manage.py interactive_simple_workflow_demo
  

The script will walk you through using the Orchestra Client API to create a new project based on the simple workflow, explaining which API calls to use, what their output looks like, and how machine steps interact with human steps and pass data back and forth.

If you don’t want to go to the trouble of running the script yourself, take a look at the transcript of expected output.

The workflow¶

The image above depicts our example workflow, which is composed of the following steps:

• An editor finds a good story and sends a reporter off to investigate.
• The reporter writes up a draft article.
• A more experienced reporter then reviews the article and suggests improvements.
• In parallel with the reporting step, a photographer captures photos for the story.
• A senior photographer reviews the photos and selects the best ones.
• The selected photos are resized and recolored for display across different media.
• Finally, a copy editor adds headlines and photo captions to complete the story.

To make things work in practice, there’s also a hidden machine step at the beginning of the workflow to set up some google documents and folders for article writing and image storage.

Running the workflow¶

Complete the steps¶

To navigate the workflow, first log in as the journalism-editor user and request a new task. The interface should look like the image below:

Fill out the high-level story idea and submit the task.

Next, log in as the journalism-reporter-1 worker, and you should now have a reporting task available. The interface looks like the image below–use the google doc to write your article.

When you submit, you’ll note that the task appears in the ‘Awaiting Review’ section. That’s your cue to log in as journalism-reporter-2 and review the work. Once you’re satisfied with it, accept it.

In parallel to logging in as a reporter, you can log in as journalism-photographer-1 and journalism-photographer-2 to take and review photographs relevant to the article. You should see an interface like the image below, which encourages you to add photos to a shared ‘Raw Photos’ folder. The interface should look like the below:

Once you’ve accepted the photography as journalism-photographer-2, the machine task to auto-process the photos should run. Our implementation simply makes any images in ‘Raw Photos’ greyscale, but you could imagine more complicated adjustments.

Finally, log in as journalism-copy-editor to give the article a headline and caption the photos. You should observe that your photos have been greyscaled as desired, as in the image below:

Once you submit the task, the workflow is done! You’ve successfully coordinated 6 expert workers and 2 machine tasks to tell a story.

python manage.py journalism_workflow_ctl --finish -p <PROJECT_ID>

{'articleDocument': 'https://docs.google.com/document/d/someid',
 'headline': 'Your Headline',
 'photos': [{'caption': 'Your Caption 1',
             'src': 'https://docs.google.com/uc?id=someid'},
            {'caption': 'Your Caption 2',
     'src': 'htps://docs.google.com/uc?id=someid2'},
     ...
    ]
 }

The code¶

All of the code used to create the new room workflow is located in our github repo. There are three main components to the code: The workflow definition, the interface implementations for the human steps, and the python code for the machine steps.

{
  "slug": "journalism",
  "name": "Journalism Workflow",
  "description": "Create polished newspaper articles from scratch.",
}

{
  "certifications": [
    {
      "slug": "editor",
      "name": "Editor",
      "description": "Trained in planning story ideas"
    },
    {
      "slug": "reporter",
      "name": "Reporter",
      "description": "Trained in researching and writing articles"
    },
    {
      "slug": "photographer",
      "name": "Photographer",
      "description": "Trained in taking photos for articles"
    },
    {
      "slug": "copy_editor",
      "name": "Copy Editor",
      "description": "Trained in assembling photos and text into article layout"
    }
  ]
}

{
  "sample_data_load_function": {
    "module": "journalism_workflow.load_sample_data",
    "name": "load"
  }
}

{
  "slug": "reporting",
  "name": "Reporting",
  "description": "Research and draft the article text",
  "is_human": true,
  "creation_depends_on": [
    "article_planning"
  ],
  "required_certifications": [
    "reporter"
  ],
  "review_policy": {
    "policy": "sampled_review",
    "rate": 1,
    "max_reviews": 1
  },
  "user_interface": {
    "angular_module": "journalism_workflow.v1.reporter",
    "angular_directive": "reporter",
    "javascript_includes": [
      "journalism_workflow/v1/reporter/js/modules.js",
      "journalism_workflow/v1/reporter/js/controllers.js",
      "journalism_workflow/v1/reporter/js/directives.js"
    ]
  }
}

{
  "slug": "photo_adjustment",
  "name": "Photo Adjustment",
  "description": "Automatically crop and rescale images",
  "is_human": false,
  "creation_depends_on": [
    "photography"
  ],
  "execution_function": {
    "path": "journalism_workflow.v1.adjust_photos.autoadjust_photos"
  }
}

{
  "angular_module": "journalism_workflow.v1.editor",
  "angular_directive": "editor",
  "javascript_includes": [
    "journalism_workflow/v1/editor/js/modules.js",
    "journalism_workflow/v1/editor/js/controllers.js",
    "journalism_workflow/v1/editor/js/directives.js"
  ],
  "stylesheet_includes": []
}

def my_machine_step(project_data, prerequisites):
  # implement machine-y goodness
  return { 'output_data_key': 'value' }

{
  "sample_data_load_function": {
    "path": "journalism_workflow.load_sample_data.load"
  }
}

from django.contrib.auth.models import User

def load(workflow_version):
  user = User.objects.update_or_create(
    username='test_user',
    defaults={
      'first_name': 'Test',
      'last_name': 'User',
  })
  user.set_password('test')

python manage.py loadworkflowsampledata <WORKFLOW_SLUG>/<WORKFLOW_VERSION>

Workflows¶

• The entire process above is called a workflow, comprised of five component steps.
• Two of these steps require review, where more experienced experts review the original work performed. Custom review policies (e.g., sampled or systematic review) for tasks can be easily created in Orchestra.
• The photo resizing step is a machine step, completed by automation rather than by experts.
• Each step emits a JSON blob with structured data generated by either humans or machines.
• Steps have access to data emitted by previous steps that they depend on. In the example, the copy editor has access to both the story and the resized photos.

Project Distribution¶

• Projects are a series of interconnected tasks. A project is an instance of a workflow; a task is an instance of a step.
  • An editor with a story about local elections would create an elections project, with tasks for a reporter/photographer/copy editor.
• Tasks are carried out by an expert or by a machine.
  • Photographers capture the story.
  • Machines resize and recolor the photos.
• Experts can come from anywhere, from a company’s employees to freelancers on platforms like Upwork.

Hierarchical Review¶

• Core experts do the initial work on a task.
• Reviewers provide feedback to other experts to make their work even better.
• The core expert submits the task when their work is complete.
• The reviewer can choose to accept the task, which is either selected for further review or marked as complete.
• They could also choose to return the task, requesting changes from and giving feedback to the worker they are reviewing.

Worker Certification¶

• Certifications allow experts to work on tasks they’re great at.
• Experts can work toward all sorts of certifications, picking up practice tasks to build experience.
  • Joseph is a solid reporter but needs a little more practice as a photographer—let’s give him some simple tasks so he can improve!
• Experts need additional certification to work in a reviewer role.
  • Amy has been reporting for quite some time and would be great at mentoring new reporters.

Life of a Task¶

Below are two images of the Orchestra dashboard, the launching point for expert workers. Click to see how tasks move differently across the dashboard for core workers and reviewers.

Core Expert¶

A core expert performs initial task work which will later be reviewed. The diagram below shows a task’s movement through the core worker’s dashboard.

Reviewer¶

A reviewer evaluates the core expert’s work and provides feedback. The diagram below shows a task’s movement through a reviewer’s dashboard.

Time Tracking¶

Workers can track time spent on a task using the timer/timecard feature. We provide both a timer widget that workers can turn on to track time, and a timecard page where workers can update and manually create time entries.

Timer¶

The navigation bar in the task dashboard and task pages has a timer icon.

If you click on the timer icon, a dropdown appears that allows the worker to toggle a timer to track their work time.

After the timer is started, the worker can add a description and specify the task they are working on. If the timer is started in a task interface, the task field is pre-populated.

When the timer is stopped, a time entry is automatically created for the amount of time worked. A worker can go to the timecard page to edit and manually add time entries.

Timecard¶

The timecard page by default shows a list of time entries for the current work week, starting on Monday, grouped by date. You can use the date range pickers at the top to change the entries shown.

The time to the right of the date shows the total time worked that day.

A time entry can be manually added (versus automatically added by stopping the timer) by clicking the plus icon next to each date.

Each time entry can be edited by clicking the pencil icon, or deleted by clicking the x icon.

If a time entry is missing a description or a task, the missing fields are highlighted and the total hours for the day is hidden.

Getting involved¶

First, if you find a bug in the code or documentation, check out our open issues and pull requests to see if we’re already aware of the problem. Also feel free to reach out to us on gitter to answer questions at any time, or subscribe to the Orchestra mailing list for longer conversations.

If you’ve uncovered something new, please create an issue describing the problem. If you’ve written code that fixes an issue, create a pull request (PR) so it’s easy for us to incorporate your changes.

Setting up for Development¶

We have a .editorconfig specified in the top level providing editor defaults for our code style. We recommend to install an `EditorConfig plugin<http://editorconfig.org/#download>`_ so your editor automatically adheres to our style :).

We recommend using a virtualenv to install the required packages in requirements.txt. In addition, we use Gulp as a frontend build system. To build the frontend resources you can run make gulp_build once npm is installed.

Development Workflow¶

Github provides a nice overview on how to create a pull request.

Some general rules to follow:

• Do your work in a fork of the Orchestra repo.

• Create a branch for each feature/bug in Orchestra that you’re working on. These branches are often called “feature” or “topic” branches.

• Use your feature branch in the pull request. Any changes that you push to your feature branch will automatically be shown in the pull request.

• Keep your pull requests as small as possible. Large pull requests are hard to review. Try to break up your changes into self-contained and incremental pull requests, if need be, and reference dependent pull requests, e.g. “This pull request builds on request #92. Please review #92 first.”

• Include unit tests with your pull request. We love tests and use CircleCI to check every pull request and commit. Check out our tests in orchestra/tests to see examples of how to write unit tests. Before submitting a PR, make sure that running make test from the root directory of the repository succeeds.

• Additionally, we try to maintain high code coverage. Aim for 100% for every new file you create!

• Once you submit a PR, you’ll get feedback on your code, sometimes asking for a few rounds of changes before your PR is accepted. After each round of comments, make changes accordingly, then squash all changes for that round into a single commit with a name like ‘changes round 0’.

• After your PR is accepted, you should squash all of your changes into a single commit before we can merge them into the main codebase.

• If your feature branch is not based off the latest master, you will be asked to rebase it before it is merged. This ensures that the commit history is linear, which makes the commit history easier to read.

• How do you rebase on to master, you ask? After syncing your fork against the Orchestra master, run:

  git checkout master
  git pull
  git checkout your-branch
  git rebase master
  

• How do you squash changes, you ask? Easy:

  git log
  <find the commit hash that happened immediately before your first commit>
  git reset --soft $THAT_COMMIT_HASH$
  git commit -am "A commit message that summarizes all of your changes."
  git push -f origin your-branch
  

• Remember to reference any issues that your pull request fixes in the commit message, for example ‘Fixes #12’. This will ensure that the issue is automatically closed when the PR is merged.

Quick Style Guide¶

We generally stick to PEP8 for our coding style, use spaces for indenting, and make sure to wrap lines at 79 characters.

We have a linter built in to our test infrastructure, so make test won’t succeed until the code is cleaned up. To run the linter standalone, just run make lint. Of course, sometimes you’ll write code that will never make the linter happy (for example, URL strings longer than 80 characters). In that case, just add a # noqa comment to the end of the line to tell the linter to ignore it. But use this sparingly!

When working on frontend resources, we use Gulp as a frontend build system. This means that after making any changes to frontend resources, you must run make gulp_build to include the modified resources. This moves resources to the build folder, compiling scss and linting your javascript.

For stylesheets we only compile scss files so if your file is at orchestra/common/static/common/scss/example.scss, to include it in an HTML file you should write {% static 'common/css/example.css' %}"> as the static file path.

When including angular templates, we wrap references to static files with the function $static(static_url_path). The $static function is defined in the base template, and for development simply returns the url it is given. The purpose is to decouple static file storage from the Django path, so if you host your static files on a CDN, you can simply override this function and put the appropriate urls.

Client API¶

Endpoints for communicating with Orchestra.

All requests must be signed using HTTP signatures:

from httpsig.requests_auth import HTTPSignatureAuth

auth = HTTPSignatureAuth(key_id=settings.ORCHESTRA_PROJECT_API_KEY,
                         secret=settings.ORCHESTRA_PROJECT_API_SECRET,
                         algorithm='hmac-sha256')
response = requests.get('https://www.example.com/orchestra/api/project/create_project', auth=auth)

POST /orchestra/api/project/create_project¶

Creates a project with the given data and returns its ID.

Query Parameters:

task_class – One of real or training to specify the task class type. workflow_slug – The slug corresponding to the desired project’s workflow. workflow_version_slug – The slug corresponding to the desired version of the workflow. description – A short description of the project. priority – An integer describing the priority of the project, with higher numbers describing a greater priority. project_data – Other miscellaneous data with which to initialize the project.

Example response:

{
  "project_id": 123,
}

POST /orchestra/api/project/project_information¶

Retrieve detailed information about a given project.

Query Parameters:
	project_id – The ID for the desired project.

Example response:

{
    "project": {
        "id": 123,
        "short_description": "Project Description",
        "priority": 10,
        "team_messages_url": "http://review.document.url",
        "task_class": 1,
        "project_data": {
            "sample_data_item": "sample_data_value_new"
        },
        "workflow_slug": "sample_workflow_slug",
        "workflow_version_slug": "v1",
        "start_datetime": "2015-09-23T20:16:02.667288Z"
    },
    "steps": [
        ["sample_step_slug", "Sample step description"]
    ],
    "tasks": {
        "sample_step_slug": {
            "id": 456,
            "project": 123,
            "status": "Processing",
            "step_slug": "sample_step_slug",
            "latest_data": {
              "sample_data_item": "sample_data_value_new"
            },
            "assignments": [
                {
                    "id": 558,
                    "iterations": [
                        {
                            "id": 92134,
                            "start_datetime": "2015-09-20T12:02:14.214553",
                            "end_datetime": "2015-09-23T20:16:15.821171",
                            "submitted_data": {
                                "sample_data_item": "sample_data_value_old",
                            },
                            "status": 'Requested Review'
                        }
                    ],
                    "worker": "sample_worker_username",
                    "task": 456,
                    "in_progress_task_data": {
                        "sample_data_item": "sample_data_value_new"
                    },
                    "status": "Processing",
                    "start_datetime": "2015-09-23T20:16:17.355291Z"
                }
            ]
        }
    }
}

GET /orchestra/api/project/workflow_types¶

Return all stored workflows and their versions.

Example response:

{
    "workflows": {
        "journalism": {
            "name": "Journalism Workflow",
            "versions": {
                "v1": {
                    "name": "Journalism Workflow Version 1",
                    "description": "Create polished newspaper articles from scratch."
                },
                "v2": {
                    "name": "Journalism Workflow Version 2",
                    "description": "Create polished newspaper articles from scratch."
                }
            }
        },
        "simple_workflow": {
            "name": "Simple Workflow",
            "versions": {
                "v1": {
                    "name": "Simple Workflow Version 1",
                    "description": "Crawl a web page for an image and rate it."
                }
            }
        }
    }
}

Models¶

Task Lifecycle¶