Custom-made mannequin monitoring for close to real-time batch inference with Amazon SageMaker

Actual-world purposes differ in inference necessities for his or her synthetic intelligence and machine studying (AI/ML) options to optimize efficiency and cut back prices. Examples embrace monetary methods processing transaction information streams, suggestion engines processing consumer exercise information, and laptop imaginative and prescient fashions processing video frames. In these eventualities, custom-made mannequin monitoring for close to real-time batch inference with Amazon SageMaker is crucial, ensuring the standard of predictions is repeatedly monitored and any deviations are promptly detected.

On this submit, we current a framework to customise using Amazon SageMaker Mannequin Monitor for dealing with multi-payload inference requests for close to real-time inference eventualities. SageMaker Mannequin Monitor screens the standard of SageMaker ML fashions in manufacturing. Early and proactive detection of deviations in mannequin high quality lets you take corrective actions, comparable to retraining fashions, auditing upstream methods, or fixing high quality points with out having to observe fashions manually or construct further tooling. SageMaker Mannequin Monitor supplies monitoring capabilities for information high quality, mannequin high quality, bias drift in a mannequin’s predictions, and drift in characteristic attribution. SageMaker Mannequin Monitor adapts effectively to widespread AI/ML use instances and supplies superior capabilities given edge case necessities comparable to monitoring customized metrics, dealing with floor reality information, or processing inference information seize.

You may deploy your ML mannequin to SageMaker internet hosting providers and get a SageMaker endpoint for real-time inference. Your consumer purposes invoke this endpoint to get inferences from the mannequin. To scale back the variety of invocations and meet customized enterprise targets, AI/ML builders can customise inference code to ship a number of inference information in a single payload to the endpoint for close to real-time mannequin predictions. Relatively than utilizing a SageMaker Mannequin Monitoring schedule with native configurations, a SageMaker Mannequin Monitor Convey Your Personal Container (BYOC) strategy meets these customized necessities. Though this superior BYOC subject can seem overwhelming to AI/ML builders, with the suitable framework, there may be alternative to speed up SageMaker Mannequin Monitor BYOC growth for custom-made mannequin monitoring necessities.

On this submit, we offer a BYOC framework with SageMaker Mannequin Monitor to allow custom-made payload dealing with (comparable to multi-payload requests) from SageMaker endpoint information seize, use floor reality information, and output customized enterprise metrics for mannequin high quality.

Overview of answer

SageMaker Mannequin Monitor makes use of a SageMaker pre-built picture utilizing Spark Deequ, which accelerates the utilization of mannequin monitoring. Utilizing this pre-built picture often turns into problematic when customization is required. For instance, the pre-built picture requires one inference payload per inference invocation (request to a SageMaker endpoint). Nonetheless, in case you’re sending a number of payloads in a single invocation to scale back the variety of invocations and establishing mannequin monitoring with SageMaker Mannequin Monitor, then you will have to discover further capabilities inside SageMaker Mannequin Monitor.

A preprocessor script is a functionality of SageMaker Mannequin Monitor to preprocess SageMaker endpoint information seize earlier than creating metrics for mannequin high quality. Nonetheless, even with a preprocessor script, you continue to face a mismatch within the designed habits of SageMaker Mannequin Monitor, which expects one inference payload per request.

Given these necessities, we create the BYOC framework proven within the following diagram. On this instance, we exhibit establishing a SageMaker Mannequin Monitor job for monitoring mannequin high quality.

The workflow contains the next steps:

1.  Earlier than and after coaching an AI/ML mannequin, an AI/ML developer creates baseline and validation information that’s used downstream for monitoring mannequin high quality. For instance, customers can save the accuracy rating of a mannequin, or create customized metrics, to validate mannequin high quality.
2. An AI/ML developer creates a SageMaker endpoint together with customized inference scripts. Information seize should be enabled for the SageMaker endpoint to save lots of real-time inference information to Amazon Easy Storage Service (Amazon S3) and help downstream SageMaker Mannequin Monitor.
3. A consumer or utility sends a request together with a number of inference payloads. When you have a big quantity of inference information, SageMaker batch remodel could also be an appropriate choice to your use case.
4. The SageMaker endpoint (which incorporates the customized inference code to preprocesses the multi-payload request) passes the inference information to the ML mannequin, postprocesses the predictions, and sends a response to the consumer or utility. The data pertaining to the request and response is saved in Amazon S3.
5. Unbiased of calling the SageMaker endpoint, the consumer or utility generates floor reality for the predictions returned by the SageMaker endpoint.
6. A buyer picture (BYOC) is pushed to Amazon Elastic Container Registry (Amazon ECR) that accommodates code to carry out the next actions:
   • Learn enter and output contracts required for SageMaker Mannequin Monitor.
   • Learn floor reality information.
   • Optionally, learn any baseline constraint or validation information (comparable to accuracy rating threshold).
   • Course of information seize saved in Amazon S3 from the SageMaker endpoint.
   • Evaluate real-time information with floor reality and create mannequin high quality metrics.
   • Publish metrics to Amazon CloudWatch Logs and output a mannequin high quality report.
7. The AI/ML developer creates a SageMaker Mannequin Monitor schedule and units the customized picture (BYOC) because the referable picture URI.

This submit makes use of code supplied within the following GitHub repo to exhibit the answer. The method contains the next steps:

1. Prepare a multi-classification XGBoost mannequin utilizing the general public forest protection dataset.
2. Create an inference script for the SageMaker endpoint for customized inference logic.
3. Create a SageMaker endpoint with information seize enabled.
4. Create a constraint file that accommodates metrics used to find out if mannequin high quality alerts needs to be generated.
5. Create a customized Docker picture for SageMaker Mannequin Monitor by utilizing the SageMaker Docker Construct CLI and push it to Amazon ECR.
6. Create a SageMaker Mannequin Monitor schedule with the BYOC picture.
7. View the customized mannequin high quality report generated by the SageMaker Mannequin Monitor job.

Stipulations

To observe together with this walkthrough, be sure you have the next stipulations:

Prepare the mannequin

Within the SageMaker Studio atmosphere, launch a SageMaker coaching job to coach a multi-classification mannequin and output mannequin artifacts to Amazon S3:

from sagemaker.xgboost.estimator import XGBoost
from sagemaker.estimator import Estimator

hyperparameters = {
    "max_depth": 5,
    "eta": 0.36,
    "gamma": 2.88,
    "min_child_weight": 9.89,
    "subsample": 0.77,
    "goal": "multi:softprob",
    "num_class": 7,
    "num_round": 50
}

xgb_estimator = XGBoost(
    entry_point="./src/prepare.py",
    hyperparameters=hyperparameters,
    function=function,
    instance_count=1,
    instance_type="ml.m5.2xlarge",
    framework_version="1.5-1",
    output_path=f's3://{bucket}/{prefix_name}/fashions'
)

xgb_estimator.match(
    {
        "prepare": train_data_path,
        "validation": validation_data_path
    },
    wait=True,
    logs=True
)

Create Inference Code

Earlier than you deploy the SageMaker endpoint, create an inference script (inference.py) that accommodates a operate to preprocess the request with a number of payloads, invoke the mannequin, and postprocess outcomes.

For output_fn, a payload index is created for every inference report discovered within the request. This lets you merge floor reality information with information seize throughout the SageMaker Mannequin Monitor job.

See the next code:

def input_fn(input_data, content_type):
    """Take request information and de-serializes the info into an object for prediction.
        When an InvokeEndpoint operation is made in opposition to an Endpoint working SageMaker mannequin server,
        the mannequin server receives two items of data:
            - The request Content material-Kind, for instance "utility/json"
            - The request information, which is at most 5 MB (5 * 1024 * 1024 bytes) in dimension.
    Args:
        input_data (obj): the request information.
        content_type (str): the request Content material-Kind.
    Returns:
        (obj): information prepared for prediction. For XGBoost, this defaults to DMatrix.
    """
    
    if content_type == "utility/json":
        request_json = json.masses(input_data)
        prediction_df = pd.DataFrame.from_dict(request_json)
        return xgb.DMatrix(prediction_df)
    else:
        increase ValueError


def predict_fn(input_data, mannequin):
    """A predict_fn for XGBooost Framework. Calls a mannequin on information deserialized in input_fn.
    Args:
        input_data: enter information (DMatrix) for prediction deserialized by input_fn
        mannequin: XGBoost mannequin loaded in reminiscence by model_fn
    Returns: a prediction
    """
    output = mannequin.predict(input_data, validate_features=True)
    return output


def output_fn(prediction, settle for):
    """Perform accountable to serialize the prediction for the response.
    Args:
        prediction (obj): prediction returned by predict_fn .
        settle for (str): settle for content-type anticipated by the consumer.
    Returns: JSON output
    """
    
    if settle for == "utility/json":
        prediction_labels = np.argmax(prediction, axis=1)
        prediction_scores = np.max(prediction, axis=1)
        output_returns = [
            {
                "payload_index": int(index), 
                "label": int(label), 
                "score": float(score)} for label, score, index in zip(
                prediction_labels, prediction_scores, range(len(prediction_labels))
            )
        ]
        return employee.Response(encoders.encode(output_returns, settle for), mimetype=settle for)
    
    else:
        increase ValueError

Deploy the SageMaker endpoint

Now that you’ve created the inference script, you’ll be able to create the SageMaker endpoint:

from sagemaker.model_monitor import DataCaptureConfig

predictor = xgb_estimator.deploy(
    instance_type="ml.m5.giant",
    initial_instance_count=1,
    wait=True,
    data_capture_config=DataCaptureConfig(
        enable_capture=True,
        sampling_percentage=100,
        destination_s3_uri=f"s3://{bucket}/{prefix_name}/model-monitor/data-capture"
    ),
    source_dir="./src",
    entry_point="inference.py"
)

Create constraints for mannequin high quality monitoring

In mannequin high quality monitoring, that you must evaluate your metric generated from floor reality and information seize with a pre-specified threshold. On this instance, we use the accuracy worth of the educated mannequin on the check set as a threshold. If the newly computed accuracy metric (generated utilizing floor reality and information seize) is decrease than this threshold, a violation report can be generated and the metrics can be printed to CloudWatch.

See the next code:

constraints_dict = {
    "accuracy":{
        "threshold": accuracy_value
    }
}
    

# Serializing json
json_object = json.dumps(constraints_dict, indent=4)
 
# Writing to pattern.json
with open("constraints.json", "w") as outfile:
    outfile.write(json_object)

This contraints.json file is written to Amazon S3 and would be the enter for the processing job for the SageMaker Mannequin Monitor job downstream.

Publish the BYOC picture to Amazon ECR

Create a script named model_quality_monitoring.py to carry out the next capabilities:

• Learn atmosphere variables and any arguments handed to the SageMaker Mannequin Monitor job
• Learn SageMaker endpoint information seize and constraint metadata configured with the SageMaker Mannequin Monitor job
• Learn floor reality information from Amazon S3 utilizing the AWS SDK for pandas
• Create accuracy metrics with information seize and floor reality
• Create metrics and violation reviews given constraint violations
• Publish metrics to CloudWatch if violations are current

This script serves because the entry level for the SageMaker Mannequin Monitor job. With a customized picture, the entry level script must be specified within the Docker picture, as proven within the following code. This manner, when the SageMaker Mannequin Monitor job initiates, the desired script is run. The sm-mm-mqm-byoc:1.0 picture URI is handed to the image_uri argument whenever you outline the SageMaker Mannequin Monitor job downstream.

FROM 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:1.2-1-cpu-py3

RUN python3 -m pip set up awswrangler

ENV PYTHONUNBUFFERED=TRUE

ADD ./src/model_quality_monitoring.py /

ENTRYPOINT ["python3", "/model_quality_monitoring.py"]

The customized BYOC picture is pushed to Amazon ECR utilizing the SageMaker Docker Construct CLI:

sm-docker construct . --file ./docker/Dockerfile --repository sm-mm-mqm-byoc:1.0

Create a SageMaker Mannequin Monitor schedule

Subsequent, you employ the Amazon SageMaker Python SDK to create a mannequin monitoring schedule. You may outline the BYOC ECR picture created within the earlier part because the image_uri parameter.

You may customise the atmosphere variables and arguments handed to the SageMaker Processing job when SageMaker Mannequin Monitor runs the mannequin high quality monitoring job. On this instance, the bottom reality Amazon S3 URI path is handed as an atmosphere variable and is used throughout the SageMaker Processing job:

sm_mm_mqm = ModelMonitor(
    function=function, 
    image_uri=f"{account_id}.dkr.ecr.us-east-1.amazonaws.com/sm-mm-mqm-byoc:1.0", 
    instance_count=1, 
    instance_type="ml.m5.xlarge", 
    base_job_name="sm-mm-mqm-byoc",
    sagemaker_session=sess,
    env={
        "ground_truth_s3_uri_path": f"s3://{bucket}/{prefix_name}/model-monitor/mqm/ground_truth/{predictor.endpoint_name}"
    }
)

Earlier than you create the schedule, specify the endpoint title, the Amazon S3 URI output location you wish to ship violation reviews to, the statistics and constraints metadata recordsdata (if relevant), and any customized arguments you wish to go to your entry script inside your BYOC SageMaker Processing job. On this instance, the argument –-create-violation-tests is handed, which creates a mock violation for demonstration functions. SageMaker Mannequin Monitor accepts the remainder of the parameters and interprets them into atmosphere variables, which you need to use inside your customized monitoring job.

sm_mm_mqm.create_monitoring_schedule(
    endpoint_input=predictor.endpoint_name,
    output=MonitoringOutput(
        supply="/decide/ml/processing/output",
        vacation spot=f"s3://{bucket}/{prefix_name}/model-monitor/mqm/reviews"
    ),
    statistics=f"s3://{bucket}/{prefix_name}/model-monitor/mqm/baseline-data/statistics.json",
    constraints=f"s3://{bucket}/{prefix_name}/model-monitor/mqm/baseline-data/constraints.json",
    monitor_schedule_name="sm-mm-byoc-batch-inf-schedule",
    schedule_cron_expression=CronExpressionGenerator().hourly(),
    arguments=[
        "--create-violation-tests"
    ]
)

Assessment the entry level script model_quallity_monitoring.py to higher perceive the way to use customized arguments and atmosphere variables supplied by the SageMaker Mannequin Monitor job.

Observe the SageMaker Mannequin Monitor job output

Now that the SageMaker Mannequin Monitor useful resource is created, the SageMaker endpoint is invoked.

On this instance, a request is supplied that features a listing of two payloads by which we wish to accumulate predictions:

sm_runtime = boto3.consumer("sagemaker-runtime")

response = sm_runtime.invoke_endpoint(
    EndpointName=predictor.endpoint_name,
    ContentType="utility/json",
    Settle for="utility/json",
    Physique=test_records,
    InferenceId="0"
)

InferenceId is handed as an argument to the invoke_endpoint methodology. This ID is used downstream when merging the bottom reality information to the real-time SageMaker endpoint information seize. On this instance, we wish to accumulate floor reality with the next construction.

This makes it easier when merging the bottom reality information with real-time information throughout the SageMaker Mannequin Monitor customized job.

As a result of we set the CRON schedule for the SageMaker Mannequin Monitor job to an hourly schedule, we will view the outcomes on the finish of the hour. In SageMaker Studio Traditional, by navigating the SageMaker endpoint particulars web page, you’ll be able to select the Monitoring job historical past tab to view standing reviews of the SageMaker Mannequin Monitor job.

If a difficulty is discovered, you’ll be able to select the monitoring job title to evaluate the report.

On this instance, the customized mannequin monitoring metric created within the BYOC flagged an accuracy rating violation of -1 (this was performed purposely for demonstration with the argument --create-violation-tests).

This provides you the power to observe mannequin high quality violations to your customized SageMaker Mannequin Monitor job throughout the SageMaker Studio console. If you wish to invoke CloudWatch alarms based mostly on printed CloudWatch metrics, it’s essential to create these CloudWatch metrics together with your BYOC job. You may evaluate how that is performed throughout the monitor_quality_monitoring.py script. For automated alerts for mannequin monitoring, creating an Amazon Easy Notification Service (Amazon SNS) subject is really useful, which electronic mail consumer teams will subscribe to for alerts on a given CloudWatch metric alarm.

Clear up

To keep away from incurring future fees, delete all sources associated to the SageMaker Mannequin Monitor schedule by finishing the next steps:

1. Delete information seize and any floor reality information:

   ! aws s3 rm s3://{bucket}/{prefix_name}/model-monitor/data-capture/{predictor.endpoint_name} --recursive
   ! aws s3 rm s3://{bucket}/{prefix_name}/model-monitor/mqm/ground_truth/{predictor.endpoint_name} --recursive

2. Delete the monitoring schedule:

   sm_mm_mqm.delete_monitoring_schedule()

3. Delete the SageMaker mannequin and SageMaker endpoint:

   predictor.delete_model()
   predictor.delete_endpoint()

Conclusion

Customized enterprise or technical necessities for a SageMaker endpoint ceaselessly have an effect on downstream efforts in mannequin monitoring. On this submit, we supplied a framework that lets you customise SageMaker Mannequin Monitor jobs (on this case, for monitoring mannequin high quality) to deal with the use case of passing a number of inference payloads to a SageMaker endpoint.

Discover the supplied GitHub repository to implement this custom-made mannequin monitoring framework with SageMaker Mannequin Monitor. You need to use this framework as a place to begin to observe your customized metrics or deal with different distinctive necessities for mannequin high quality monitoring in your AI/ML purposes.

In regards to the Authors

Joe King is a Sr. Information Scientist at AWS, bringing a breadth of knowledge science, ML engineering, MLOps, and AI/ML architecting to assist companies create scalable options on AWS.

Ajay Raghunathan is a Machine Studying Engineer at AWS. His present work focuses on architecting and implementing ML options at scale. He’s a expertise fanatic and a builder with a core space of curiosity in AI/ML, information analytics, serverless, and DevOps. Exterior of labor, he enjoys spending time with household, touring, and enjoying soccer.

Raju Patil is a Sr. Information Scientist with AWS Skilled Providers. He architects, builds, and deploys AI/ML options to assist AWS prospects throughout totally different verticals overcome enterprise challenges in quite a lot of AI/ML use instances.