Quickstart

Let's just start monitoring models with Superwise ASAP.
This guide will walk you through the creation and configuration of a new model into the platform and give you model observability in 5 minutes.

📘

Before you begin

1. Make sure you have an active Superwise account. If you don't, signup for free at portal.superwise.ai
2. Create your Authentication keys
3. Install our SDK

👍

End-to-End Notebook example

The following quickstart guide is also available in a runnable notebook, which can be found here

1. Register a new model and its first version

To start working with Superwise SDK, first, import the package and initialize the Superwise object.

import os
from superwise import Superwise

os.environ['SUPERWISE_CLIENT_ID'] = 'REPLACE_WITH_YOUR_CLIENT'
os.environ['SUPERWISE_SECRET'] = 'REPLACE_WITH_YOUR_SECRET'

sw = Superwise()

Next, register a new model by defining its name and description.

from superwise.models.model import Model

diamond_model = Model(
    name="Diamond Model",
    description="Regression model which predict the diamond price"
)

diamond_model = sw.model.create(diamond_model)
print(f"New task Created - {diamond_model.id}")

Now that we have a new model in place, we can start and log its first version.
To create a new version, you should provide a baseline dataset (e.g., the version training dataset) so the platform will automatically infer the version schema and baseline distributions that your production data will be compared to.
The baseline dataset may contain schema with different entity roles included (visit our data entity guide to see the complete role list).

import requests
from io import StringIO
import pandas as pd 

url = 'https://gitlab.com/superwise.ai-public/integration/-/raw/main/getting_started/data/baseline.csv?inline=false'
baseline_data = pd.read_csv(StringIO(requests.get(url).text))
baseline_data.head()

🚧

File size limitation

The baseline data file should be up to 100MB. if your file is more extensive, make sure you split it

Now it's time for version summary - sw.data_entity.summarise

In order to finish connecting a version, you need to supply the collection of entities that belong to the version and their summary. The summary will include the data type and role of each entity, the importance of each feature (entities that are not features will get the value 0 automatically), and the statistical summary on each entity.

This summary will be used later on to create the appropriate metrics per entity and enable comparison between the baseline statistical behavior and production.

To automate the process of creating version summaries, the Superwise SDK requires only two parameters:

• Baseline dataset
• Role definition for each entity that is not a "Feature"

from superwise.resources.superwise_enums import DataEntityRole

entities_collection = sw.data_entity.summarise(
    data=baseline_data,
    specific_roles = {
      'id': DataEntityRole.ID,
      'ts': DataEntityRole.TIMESTAMP,
      'prediction': DataEntityRole.PREDICTION_VALUE,
      'price': DataEntityRole.LABEL
    }
)

You can now activate your version based on the schema you just created in the previous step.

from superwise.models.version import Version

new_version = Version(
    model_id=diamond_model.id,
    name="1.0.0",
    data_entities=entities_collection,
)

new_version = sw.version.create(new_version)
sw.version.activate(new_version.id)

Now you're good to go and can start logging production data.

📘

Register a model

To read more about advanced options on how to register a new model and log versions visit our Connecting guides section

2. Create Policy

configure and set monitoring policies to automatically detect important issues that might impact your organization or teams.

To create a policy using the SDK, your first step is to create a new email notification channel.

notification = sw.notification.create_email_notification(name="my_email",emaill="[email protected]")
print(notification.get_properties())

Now, use the SDK to retrieve all the policy templates available on the platform

all_templates = sw.policy.get_policy_templates()
print(all_templates)

After you find a template that meets your needs, add a policy based on that template:

from superwise.resources.superwise_enums import NotifyUpon, ScheduleCron
  
notification = sw.notification.get('my_email')
sw.policy.create_policy_from_template(diamond_model.id, 'Label shift', NotifyUpon.detection_and_resolution, notification,
                                      ScheduleCron.EVERY_DAY_AT_6AM)

3. Log production data

Use the SDK to create new transactions containing production data you want to log and monitor.
The transaction will contain the model id, the version it refers to, and the records we want to log. The records format should be aligned with the version schema definition.

url = 'https://gitlab.com/superwise.ai-public/integration/-/raw/main/getting_started/data/ongoing_predictions.csv?inline=false'
ongoing_predictions = pd.read_csv(StringIO(requests.get(url).text))
ongoing_predictions.head()

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

ongoing_predictions_chuncks = chunks(ongoing_predictions.to_dict(orient='records'), 1000)

transaction_ids = list()
for ongoing_predictions_chunck in ongoing_predictions_chuncks:
    transaction_id = sw.transaction.log_records(
        model_id=diamond_model.id, 
        version_id=new_version.id, 
        records=ongoing_predictions_chunck
    )
    transaction_ids.append(transaction_id)
    print(transaction_id)

📘

Production data logging

To further review the different possible methods and formats you can log your production data, visit our Connecting guides section

See, that was simple. ✌️

Now what?

• Investigate your model's metrics in the Trends screen
• Set up segments
• Create a monitoring policy to automate anomaly detection