7. Log ground truth / labels

Labels (also known as ground truths) are data relevant only for supervised use cases. Supervised models rely on labels for historical datasets, but in ongoing production operations, labels usually arrive much later and sometimes are not even available on an ongoing basis.

• Fast feedback delay - In cases like Real-Time Bidding where the model tries to predict the chances of a user clicking on an ad. This usually happens a few seconds after the ad appeared.

• Long feedback delay - In cases where the model is trying to predict an event that may only be known in the far future. For example, Predicting User Life-Time-Value (let's assume that the User LTV is based on the user's annual revenue potential).

• Partial feedback - Let's take a fraud detection model that predicts the possibility of a given money transaction being discovered as a fraudulent transaction. Once we have a model in production and it predicts that a specific transaction will be discovered as fraudulent, most probably the business will decide to reject the transaction, in such cases, we will not know if the transaction was indeed a fraudulent one or legit.

To support the various cases, Superwise offers a variety of options to send the label information into the platform:

Together with the predictions
In case you have a very fast feedback cycle you can send the labels alongside the prediction data.
It uses the same APIs so any mode that was available for sending predictions (stream or batch) is also applicable here.

from superwise import Superwise

sw = Superwise()

predictions_and_labels = predictions_and_labels.to_dict(orient='records')

transaction_id = sw.transaction.log_records(
  model_id=diamond_model.id,
  version_id=new_version.id,
  records=predictions_and_labels,
  metadata = {"KEY": "VALUE"} # OPTIONAL
)

Separately from the predictions
If your feedback delay is too long in order to send the predictions and labels together you can send them in a separate flow. The same unified logging API that was used for predictions data will also be used here, therefore you can do it in stream or batch similarly to what was described when sending predictions data.

Below is an example of how to send labels separately from predictions in a stream mode. Please notice that in order to compute performance metrics, each label should contain the record id, which should be the identifier that connects the prediction and the label.

from superwise import Superwise

sw = Superwise()

labels = labels.to_dict(orient='records')

transaction_id = sw.transaction.log_records(
  model_id=diamond_model.id,
  records=labels,
  metadata = {"KEY": "VALUE"} # OPTIONAL
)

A label DataFrame example:

📘

No version id required

Labels or ground truths reflect an event that we tried to predict. They are not related to the specific model implementation or versioning. Therefore when reporting only labels, you don't need to supply the model version id column.