Metis Machine's Skafos

Machine Learning Delivered. A Machine Learning deployment platform for Engineers and Data Scientist.

Welcome to the Metis Machine documentation hub. You'll find comprehensive guides and documentation to help you start working with Metis Machine's Skafos platform as quickly as possible, as well as support if you get stuck. Fire it up!

Get Started    

Predict the Price of Bitcoin in 10 Minutes

A tutorial on how to implement a Skafos platform template to predict the price of Bitcoin, a crypto currency asset, in under 10 minutes

This tutorial will teach you how to use a Skafos neural network template to predict the price of Bitcoin*. It should take about 10 minutes to get the model up and running. Of course, it will take longer than that for the data model to actually run on the platform due to default resource constraints.

* Disclaimer!

Obviously, this model is intended to get you started working with neural networks on the Skafos platform. DO NOT use this example model to "predict" the price of Bitcoin, or any other cryptocurrency asset. Using only past observations and Google trend data leaves lots of room for improvement! See full legal disclaimer below.

Requirements

Before you get started, there are some requirements which are outside of the scope of this article:

  • Skafos Account - Click here to sign up for Early Access
  • MacOS (Linux support coming soon!)
  • Homebrew - run the following command to install. You will be prompted to install Xcode as well - please do.
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Git - brew install git
  • GitHub Account
  • cURL - brew install curl
  • Quandl api key to use this template out of the box. Otherwise, user can provide their own coin data.

Obtain a Quandl API Key - 3 minutes

Since you'll need a Quandl API key, navigate to the link above and click "LOG IN." If you have an account, log in here. If you don't, you'll notice a "CREATE ACCOUNT" link on the bottom of the login page.

Once you've logged into Quandl, click your icon on the top right of the screen, "ACCOUNT SETTINGS", then "API KEY" on the menu. Your API key will be displayed. Copy it for this tutorial.

Account Settings

Account Settings

Copy Quandl API Key

Copy Quandl API Key

1. Skafos Command Line Setup

Fire up your favorite shell and install Skafos using homebrew.

brew install skafos

You can see what commands are available by typing: skafos --help. The command we care about is the templates command. You'll want to update your local template cache and if this is the first time you've run a skafos command, you'll be prompted to sign in with the email and password of your Skafos account.

Skafos Command Line

Check out the Command Line docs here for more on each command

skafos templates --update

Let's search for a Bitcoin template.

skafos templates --search bitcoin

Which should return something like this:

➜ bitcoinRNN (0.1.0) 
  Base template for a crypto currency price prediction model using deep learning.
  url: https://github.com/metismachine/bitcoinRNN.template

There may be other crypto templates listed, but this is the one we'll be using.

Now use that template to initialize your project.

skafos init my-bitcoin-predictor --template bitcoinRNN

This will create a directory called my-bitcoin-predictor within your current directory. If you view the contents of the my-bitcoin-predictor directory, you'll see the following files:

README.md
main.py
metis.config.yml
bitcoinRNN.ipynb
requirements.txt

File Content Review

You don't need to mess with any of the files for this tutorial to start predicting the price of Bitcoin, but here's a quick rundown of what's in those files for when you're ready to tweak them or play around:

  • README.md: Just some general and specific information about the new project.

  • main.py: This is the default entrypoint for the project when it gets run as a job. Our template has kindly auto-populated this file with the code necessary to predict the price of Bitcoin on a daily basis.

  • metis.config.yml: This holds the configuration data Skafos uses to know how and when to run the job. Also, the resources it should allocate to the job.

  • bitcoinRNN.ipynb: This is a Jupyter notebook that you can run from our dashboard to get the same results to the job run in a more interactive way. This is a great way to play around with the code to tweak it to your particular needs, as well as visualize the output and view the data in a more immediate way. You can then export the code changes into your project for deployment. For more on the Dashboard check out these [docs](doc:admin-dashboard

  • requirements.txt: This holds all of the requirements necessary to run your job on Skafos. You may notice the notebook may have some !pip install <some lib> statements. This is so that the Jupyter notebook you launch from the dashboard has all the necessary dependencies as well, though many of the standard ML libraries are provided by default.

Lets now use the skafos CLI to set the Quandl API key provided in your Quandl Account Settings:

skafos env QUANDL_KEY --set <API KEY>

2. Skafos GitHub App Setup

Now it's time to add our project to a GitHub repository. First thing to do is create a new GitHub repo.You'll want to do this first because you'll be adding the Skafos GitHub app to that GitHub repo. GitHub has some comprehensive docs on how to create a new repo in their web UI, which can be found here: https://help.github.com/articles/create-a-repo/.

Then, you'll head to https://github.com/apps/skafos to install the Skafos GitHub app. This is Skafos' link to your project and code and how it's able to find what to run. Just click Install. Not to worry, the GitHub app only requires read access to your repo and metadata which includes being notified when events happen, like when commits are pushed.

Skafos GitHub app Install

Skafos GitHub app Install

The Skafos GitHub app is installed on your user or organization, but you can (and should) specify only select repos on which to install. This works on private repositories as well, meaning it's not required to make your repo public in order to use the Skafos platform.

Select the repo you just added (it must exist) and click `Install` again to complete the process.

Select the repo you just added (it must exist) and click Install again to complete the process.

But what's it doing?

The Skafos GitHub app will be listening for commits pushed to the master branch of your project's repo. When commits are pushed to master, Skafos will create an instance of the project that lives in your newly created GitHub repo. That instance of your project is called a job, which lives only as long as your code takes to run, then it goes away. You can also schedule a project to run on a given schedule, in which case a new job will be generated on the given schedule.

Now you can run a few git commands in your terminal to link your local project to the newly created GitHub repo. Make sure you're doing this from the root directory of the project you created.

cd my-bitcoin-predictor
git init
git add .
git commit -m "My very first Skafos bitcoin predictor commit"
git remote add origin https://github.com/<user or org name>/my-bitcoin-predictor.git
git push origin master

And that's it! You're done! You are now predicting the price of cryptocurrencies on the Skafos platform.

But how do I know what's happening and more importantly where's my data, you may ask.

Great question! Let's look at some logs of the job being run:

3. Logs & Results

skafos logs --tail

The --tail flag will give you continuous output of the job's progress. You can also specify -n <number> to load a specific number of log lines and these can be used together. You're likely to see output like this:

➜ Connecting to task logs...
➜ 	2018-01-24T20:56:50.600106Z Received User Task
<elided messages>
➜ 21:01:56 epoch 0 loss: 0.3236694931983948\n
➜ 21:02:01 epoch 10 loss: 0.2749073803424835\n
➜ 21:02:05 epoch 20 loss: 0.25935661792755127\n
➜ 21:02:09 epoch 30 loss: 0.2570462226867676\n
<elided messages>
➜ 21:05:29 epoch 490 loss: 0.24802275002002716\n
➜ The RNN predicts the closing price for: \n
➜ 2018-01-24 to be 11006.293620583383 $\n
➜ Saving to the data engine.\n
➜ 2018-01-24 21:05:33,555 - skafossdk.data_engine - INFO - Saving Data ...\n
➜ 2018-01-24 21:05:33,555 - skafossdk.data_engine - INFO - Sending msg...\n
➜ Done.\n
➜ 2018-01-24 21:05:34,555 - skafossdk.monitor - INFO - Monitor Connection Closed\n
➜ 2018-01-24 21:05:34,556 - skafossdk.data_engine - INFO - DataEngine Connection Closed\n
➜ No more messages

Once the job has finished you can get a preview of your results with the following command:

skafos fetch --table crypto_predictions
➜ 
Fetching records from crypto_predictions

➜ {"coin": "bitcoin", "date": 2147501203, "date_updated": 1516817054000, "price_prediction": 10953.34765625}
➜ {"coin": "bitcoin", "date": 2147501203, "date_updated": 1516827933000, "price_prediction": 11006.2939453125}
➜ CURL Request:
➜ curl -D - -H "X-API-Token: <api token elided>" -H "Accept: application/json" http://api.metismachine.io/v1/data/<project ID elided>/crypto_predictions

The fetch command also provides a cURL command you can run in order to preview the data (10 rows by default). There are a couple of params when retrieving the results, both of which makes it simpler to write a script to retrieve all of the results:

  • per_page: which when given a number returns that number of records
  • cursor: value returned by the cURL command to get the next set of records

And that's actually it! Congratulations, you predicted the price of Bitcoin!

What's Next?

That's up to you! You've seen all of the basic steps necessary to run something on the Skafos platform. Now all you need is to create a few models or data ingests of your own. Make sure to search the template library when you're looking to create a new project as there may be a template that stubs out everything you need to solve a particular problem.

If you have an idea or feature request, please don't hesitate to upvote it or add it to our list of ideas

Legal Disclaimer

NOTHING IN THIS SITE CONSTITUTES OR SHOULD BE CONSTRUED AS PROVIDING FINANCIAL, INVESTMENT, BROKERAGE, TAX, ACCOUNTING, OR LEGAL ADVICE.

General. You understand and acknowledge that there is a very high degree of risk involved in trading securities and, in particular, crypto assets.

THE MODEL IS PROVIDED "AS IS." WE AND OUR AFFILIATES AND LICENSORS MAKE NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE REGARDING THE MODEL, THE SERVICE OR THE THIRD PARTY CONTENT, INCLUDING ANY WARRANTY THAT THE MODEL, THE SERVICE OR THIRD PARTY CONTENT WILL PRODUCE RESULTS OR PROFITS OR WILL ACHIEVE ANY PARTICULAR PURPOSE, WILL BE UNINTERRUPTED, ERROR FREE OR FREE OF HARMFUL COMPONENTS, OR THAT ANY CONTENT, INCLUDING YOUR CONTENT OR THE THIRD PARTY CONTENT, WILL BE SECURE OR NOT OTHERWISE LOST OR DAMAGED. EXCEPT TO THE EXTENT PROHIBITED BY LAW, WE AND OUR AFFILIATES AND LICENSORS DISCLAIM ALL WARRANTIES, INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE, NON- INFRINGEMENT, OR QUIET ENJOYMENT, AND ANY WARRANTIES ARISING OUT OF ANY COURSE OF DEALING OR USAGE OF TRADE.

Financial and Other.

NEITHER METIS MACHINE LLC., INDIVIDUALLY, NOR ITS PLATFORM INCLUDING, BUT NOT LIMTED TO SKAFOS, WEBSITE, NOR ANY OF ITS OFFICERS, EMPLOYEES, REPRESENTATIVES, AGENTS, INDEPENDENT CONTRACTORS, OR SUBCONTRACTORS ARE, IN SUCH CAPACITIES, LICENSED FINANCIAL ADVISORS, REGISTERED INVESTMENT ADVISORS, OR REGISTERED BROKER-DEALERS . METISMACHINE.com, SKAFOS AND THIS MODEL ARE NOT INTENDED AS SECURITIES OR CRYPTO ASSET BROKERAGE, INVESTMENT, TAX, ACCOUNTING OR LEGAL ADVICE.

METIS MACHINE, LLC., INCLUDING ITS OFFICERS, DIRECTORS, OWNERS, EMPLOYEES, AGENTS, OR AFFILIATES MAKES NO REPRESENTATIONS OR WARRANTIES ABOUT THE ACCURACY OR COMPLETENESS OF THE INFORMATION PROVIDED IN CONNECTION WITH THE SERVICE. BY YOUR CONTINUED USE OF THE MODEL, YOU AGREE THAT NO INFORMATION PROVIDED BY METIS MACHINE, LLC, OR ANY OFFICER, DIRECTOR, OWNER, EMPLOYEE, AGENT, OR AFFILIATE THEREOF, IS OR WILL BE RELIED UPON AS SECURITIES BROKERAGE, INVESTMENT, TAX, ACCOUNTING OR LEGAL ADVICE, IS AN OFFER OR SOLICITATION OF AN OFFER TO SELL OR BUY, OR IS AN ENDORSEMENT, RECOMMENDATION OR SPONSORSHIP OF ANY COMPANY, SECURITY OR FUND OR AS ADVICE TAILORED TO YOUR OR ANY MEMBER'S PARTICULAR SITUATION. ANY URL LINKS OR COMPANY NAMES OR TICKER SYMBOLS ARE OFFERED AS A MATTER OF CONVENIENCE AND NOTHING CONTAINED IN THIS AGREEMENT OR IN THE SERVICE CONSTITUTES A SOLICITATION, RECOMMENDATION, PROMOTION, ENDORSEMENT, OR OFFER BY METIS MACHINE, LLC OR ANY OFFICER, DIRECTOR, OWNER, EMPLOYEE, AGENT, OR AFFILIATE, OF ANY PARTICULAR SECURITY, TRANSACTION, OR INVESTMENT.

METIS MACHINE, LLC, OR ANY OFFICER, DIRECTOR, OWNER, EMPLOYEE, AGENT, OR AFFILIATE, MAY HAVE POSITIONS IN THE SECURITIES AND ASSETS MENTIONED AND MAY ENTER AND EXIT SUCH POSITIONS AT ANY TIME AND FOR ANY REASON. THE RISK OF LOSS IN TRADING SECURITIES, OPTIONS, FUTURES, FOREX, AND COMMODITIES CAN BE SUBSTANTIAL AND ARE NOT SUITABLE FOR ALL INVESTORS. YOU MUST CONSIDER ALL RELEVANT RISK FACTORS, INCLUDING YOUR OWN PERSONAL FINANCIAL SITUATION, BEFORE TRADING. YOU ASSUME THE RISK OF ANY AND ALL FINANCIAL INVESTMENTS YOU MAKE. METIS MACHINE, LLC., AND ITS OFFICERS, DIRECTORS, OWNERS, EMPLOYEES, AGENTS, OR AFFILIATES ARE NOT RESPONSIBLE FOR ANY FINANCIAL INVESTMENTS YOU MAKE.

RESULTS VARY AND PAST PERFORMANCE IS NOT INDICATIVE OF FUTURE RETURNS.

Predict the Price of Bitcoin in 10 Minutes

A tutorial on how to implement a Skafos platform template to predict the price of Bitcoin, a crypto currency asset, in under 10 minutes