Metis Machine's Skafos

Machine Learning Delivered. A Machine Learning deployment platform built to unite Data Scientist, DevOps and Engineering.

Welcome to the Metis Machine documentation hub. You'll find comprehensive guides and documentation to help you start working with Metis Machine's Skafos platform as quickly as possible, as well as support if you get stuck. Fire it up!

Get Started    

A Job is a single step within a larger end-to-end machine learning pipeline (we call them Projects). Using Python, you can write code to perform a single task within a larger sequence. (coming soon - additional open source language support, such as R)

Examples of jobs in a ML pipeline:

  • Ingest data from an external source and use Skafos SDK to persist it to backend storage with the Data Engine.
  • Perform data processing, cleaning, and other ETL techniques at scale by leveraging the power of a Spark Cluster or a messaging Queue, automatically provisioned for you with AddOns, and simplified by the SDK.
  • Perform feature engineering, train a predictive model, and score unseen data.
  • Deliver model predictions automatically to a REST API living at a Data Endpoint.
  • Write data to an external source utilizing the Skafos SDK.

What’s in a Job?

A job is just a bit of code that performs a task.

Parts of a job:

  • Code that executes a particular task within a ML pipeline (my_job.py)
  • Run-time Configuration
  • Dependencies & Requirements
  • Usage of the Skafos SDK

While a job is live & running, you can view real-time Logs & Metrics associated with that job, and get notifications if something is wrong by creating a user-defined Alert. After each run, you'll be able to go back and reference historical information, including searchable logs, system metrics, and user-defined metrics on the Skafos Dashboard.

Configuration

In the Projects section, we discussed the centrallity of the metis.config.yml file, and how to organize jobs to run as a pipeline. When designing your pipeline, jobs can be:

  • Scheduled
  • Chained
  • Independently Organized
  • Parallelized
  • Scaled-up

Job Configuration Options

Below is a complete table of available job configuration options that can be included for each job in the project’s metis.config.yml file:

Option

Description

job_id **

UUID automatically generated for each new job you create. This can be done through the Skafos CLI.

language **

Programming language used in the job. Currently, Python is supported.

name **

User-defined name of the job.

entrypoint **

Command for Skafos to execute for this job. (Typically, this is the name of a script, e.g. main.py)

schedule

Cron-tab syntax for job scheduling.

dependencies

Job IDs that are required to be finished before the current job executes. Enables chaining.

parallelism

Integer representing how many instances of the job to run in parallel. This is particularly useful when consuming from a Skafos Queue.

resources

Number of CPUs and amount of Memory to give the job. CPUs (1-7), Memory (0-30G).

** required configuration options

Once you’ve created some jobs and included the appropriate configuration options, you are ready to deploy your Project. This is described in detail in the Deployments section.