A Job is a single step within a larger end-to-end machine learning pipeline (we call them Projects). Using Python, you can write code to perform a single task within a larger sequence. (coming soon - additional open source language support, such as R)
Examples of jobs in a ML pipeline:
- Ingest data from an external source and use Skafos SDK to persist it to backend storage with the Data Engine.
- Perform data processing, cleaning, and other ETL techniques at scale by leveraging the power of a Spark Cluster or a messaging Queue, automatically provisioned for you with AddOns, and simplified by the SDK.
- Perform feature engineering, train a predictive model, and score unseen data.
- Deliver model predictions automatically to a REST API living at a Data Endpoint.
- Write data to an external source utilizing the Skafos SDK.
A job is just a bit of code that performs a task.
Parts of a job:
- Code that executes a particular task within a ML pipeline (
- Run-time Configuration
- Dependencies & Requirements
- Usage of the Skafos SDK
While a job is live & running, you can view real-time Logs & Metrics associated with that job, and get notifications if something is wrong by creating a user-defined Alert. After each run, you'll be able to go back and reference historical information, including searchable logs, system metrics, and user-defined metrics on the Skafos Dashboard.
In the Projects section, we discussed the centrallity of the
metis.config.yml file, and how to organize jobs to run as a pipeline. When designing your pipeline, jobs can be:
- Independently Organized
Below is a complete table of available job configuration options that can be included for each job in the project’s
UUID automatically generated for each new job you create. This can be done through the Skafos CLI.
Programming language used in the job. Currently, Python is supported.
User-defined name of the job.
Command for Skafos to execute for this job. (Typically, this is the name of a script, e.g. main.py)
Cron-tab syntax for job scheduling.
Job IDs that are required to be finished before the current job executes. Enables chaining.
Integer representing how many instances of the job to run in parallel. This is particularly useful when consuming from a Skafos Queue.
Number of CPUs and amount of Memory to give the job. CPUs (1-7), Memory (0-30G).