---
title: Deploy models on AWS EKS
description: Deploy and monitor DataRobot models on AWS Elastic Kubernetes Service (EKS).

---

# Deploy models on AWS EKS {: #deploy-models-on-aws-eks }

With DataRobot MLOps, you can deploy DataRobot models into your own AWS Elastic Kubernetes Service (EKS) clusters and still have the advantages of using the [monitoring](monitor/index) and [governance](governance/index) capabilities of MLOps. 

These exportable DataRobot models are known as [Portable Prediction Servers (PPSs)](glossary/index#portable-prediction-server-pps). The models are embedded into Docker containers, providing flexibility and portability, and making them suitable for container orchestration tools such as Kubernetes.

The following steps show how to deploy a DataRobot model on EKS.

## Before you start {: #before-you-start }

!!! info "Availability information"
    To deploy a DataRobot model on EKS, you need to export a model package which requires the "Enable MMM model package export" flag. Contact your DataRobot representative for more information about enabling this feature.
    
Before deploying to Amazon EKS, you need to create an EKS cluster.  There are two approaches to spin up the cluster: 

* Using the **eksctl** tool (CLI for Amazon EKS). This is the simplest and fastest way to create an EKS cluster.
   
* Using AWS Management Console. This method provides more fine-grained tuning (for example, IAM role and VPC creation).

This topic shows how to install using the **eksctl** tool. See [Getting started with Amazon EKS – eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html){ target=_blank } for detailed instructions.

If any of the tools are already installed and configured, you can skip the corresponding steps. 

### Install and configure AWS and EKS {: #install-and-configure-aws-and-eks }

Follow the steps described below to install and configure Amazon Web Services CLI, EKS, and Kubernetes CLI. 

1. Install the AWS CLI, version 2.  

    `aws --version`

2. Configure your AWS CLI credentials.

3. Install **eksctl**. 

    `eksctl version`

4. Install and configure **kubectl** (CLI for Kubernetes clusters). 

    `kubectl version --short --client`

## Deploy models to EKS {: #deploy-models-to-eks }

Deploying DataRobot models on a Kubernetes infrastructure consists of three main activities:

* Preparing and pushing the Docker container with the MLOps package to the container registry
* Creating the external deployment in DataRobot
* Creating the Kubernetes cluster

### Configure and run the PPS Docker image {: #configure-and-run-the-pps-docker-image }

To complete the following steps, you need to first generate your model and [create an MLOps model package](reg-create).

DataRobot provides a UI to help you configure the Portable Prediction Server and create the Docker image. Follow these steps:

1. [Configure the Portable Prediction Server (PPS)](portable-pps#configure-the-portable-prediction-server).
2. [Obtain the PPS Docker image](portable-pps#obtain-the-pps-docker-image).
3. [Load the image to Docker](portable-pps#load-the-image-to-docker).
4. [Download the model package](portable-pps#deployment-download).
5. [Run your Docker image](portable-pps#running-modes).
6. [Monitor your model](portable-pps#monitoring).
7. [Create an external deployment](deploy-external-model). When you create the deployment, make note of the MLOps model ID and the MLOps deployment ID. You’re going to need these IDs when you deploy the MLOps package to Kubernetes.

### Push the Docker image to Amazon ECR {: #push-the-docker-image-to-amazon-ecr }

You need to upload the container image to Amazon Elastic Container Registry (ECR) so that your Amazon EKS cluster can download and run it.

1. Configure the Docker CLI tool to authenticate to Elastic Container Registry:

    ```bash
    aws ecr get-login-password --region us-east-1 | docker login --username XXX --password-stdin 00000000000.xxx.ecr.us-east-1.amazonaws.com
    ```

2. Push the Docker image you just built to ECR:

    ```bash
    docker push 00000000000.xxx.ecr.us-east-1.amazonaws.com/house-regression-model:latest
    ```

### Create an Amazon EKS cluster {: #create-an-amazon-eks-cluster }

Now that the Docker image is stored in ECR and the external deployment is created, you can spin up an Amazon EKS cluster. The EKS cluster needs VPC with either of the following:

* Two public subnets and two private subnets
* A VPC with three public subnets

The Amazon EKS requires subnets in at least two Availability Zones. A VPC with public and private subnets is recommended so that Kubernetes can create public load balancers in the public subnets to control traffic to the pods that run on nodes in private subnets.

To create the Amazon EKS cluster:

1. Optionally, create or choose two public and two private subnets in your VPC. Make sure that “Auto-assign public IPv4 address” is enabled for the public subnets.

    !!! note
        The **eksctl** tool creates all necessary subnets behind the scenes if you don’t provide the corresponding `--vpc-private-subnets` and `--vpc-public-subnets` parameters.

2. Create the cluster:

    ```bash
    eksctl create cluster \
    --name house-regression \
    --vpc-private-subnets=subnet-xxxxxxx,subnet-xxxxxxx \
    --vpc-public-subnets=subnet-xxxxxxx,subnet-xxxxxxx \
    --ssh-access \
    --ssh-public-key my-public-key.pub \
    --managed
    ```
    
   ** Notes**
    
    * Usage of the `--managed` parameter enables [Amazon EKS-managed nodegroups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html){ target=_blank }. This feature automates the provisioning and lifecycle management of nodes (EC2 instances) for Amazon EKS Kubernetes clusters. You can provision optimized groups of nodes for their clusters. EKS will keep their nodes up-to-date with the latest Kubernetes and host OS versions. The **eksctl** tool makes it possible to choose the specific size and instance type family via command line flags or config files.

    * Although `--ssh-public-key` is optional, it is highly recommended that you specify it when you create your node group with a cluster. This option enables SSH access to the nodes in your managed node group. Enabling SSH access allows you to connect to your instances and gather diagnostic information if there are issues. You cannot enable remote access after the node group is created.

    Cluster provisioning usually takes between 10 and 15 minutes and results in the following:  

    ![](images/int_damdmoa_large_009.png)

3. When your cluster is ready, test that your kubectl configuration is correct:

    ```bash
    kubectl get svc
    ```

    ![](images/int_damdmoa_large_020.png)

### Deploy the MLOps package to Kubernetes {: #deploy-the-mlops-package-to-kubernetes }

To deploy the MLOps package to Kubernetes:

1. Create a Kubernetes namespace, for example:

    ```bash
    kubectl create namespace house-regression-namespace
    ```

2. Save the following contents to a `yaml` file on your local machine (in this case, `house-regression-service.yaml`), replacing the values for your project. Provide the values of **image**, **DataRobot API token**, **model ID**, and **deployment ID**. (You should have saved the IDs when you [created the external deployment in MLOps](deploy-external-model).)

    ```yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: house-regression-service
      namespace: house-regression-namespace
      labels:
        app: house-regression-app
    spec:
      selector:
        app: house-regression-app
      ports:
        - protocol: TCP
          port: 80
          targetPort: 8080
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: house-regression-deployment
      namespace: house-regression-namespace
      labels:
        app: house-regression-app
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: house-regression-app
      template:
        metadata:
          labels:
            app: house-regression-app
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                    - amd64
          containers:
          - name: house-regression-model
            image: <your_aws_account_endpoint>.ecr.us-east-1.amazonaws.com/house-regression-model:latest
            env:
            - name: PORTABLE_PREDICTION_API_WORKERS_NUMBER
              value: "2"
            - name: PORTABLE_PREDICTION_API_MONITORING_ACTIVE
              value: "True"
            - name: PORTABLE_PREDICTION_API_MONITORING_SETTINGS
              value: output_type=spooler_type=filesystem;directory=/tmp;max_files=50;file_max_size=10240000;model_id=<your mlops_model_id_obtained_at_step_5>;deployment_id=<your mlops_deployment_id_obtained_at_step_5>
            - name: MONITORING_AGENT
              value: "True"
            - name: MONITORING_AGENT_DATAROBOT_APP_URL
              value: https://app.datarobot.com/
            - name: MONITORING_AGENT_DATAROBOT_APP_TOKEN
              value: <your_datarobot_api_token>
            ports:
            - containerPort: 80
    ```

3. Create a Kubernetes service and deployment:

    ```bash
    kubectl apply -f house-regression-service.yaml
    ```

4. View all resources that exist in the namespace:

    ```bash
    kubectl get all -n house-regression-namespace
    ```

    ![](images/int_damdmoa_large_018.png)

### Set up horizontal pod autoscaling {: #set-up-horizontal-pod-autoscaling }

The Kubernetes Horizontal Pod Autoscaler automatically scales the number of pods in a deployment, replication controller, or replica set based on that resource's CPU utilization. This can help your applications scale up to meet increased demand or scale back when resources are not needed, thus freeing up your nodes for other applications. When you set a target CPU utilization percentage, the Horizontal Pod Autoscaler scales your application up or back to try to meet that target.

1. Create a Horizontal Pod Autoscaler resource for the php-apache deployment.

    ```bash
    kubectl autoscale deployment house-regression-deployment -n house-regression-namespace --cpu-percent=80 --min=1 --max=5
    ```

2. View all resources that exist in the namespace.

    ```bash
      kubectl get all -n house-regression-namespace
    ```

    Horizontal Pod Autoscaler appears in the resources list.

    ![](images/int_damdmoa_large_012.png)

### Expose your model to the world (load balancing) {: #expose-your-model-to-the-world-load-balancing }

Amazon EKS supports the Network Load Balancer and the Classic Load Balancer for pods running on Amazon EC2 instance nodes through the Kubernetes service of type LoadBalancer.


1. Tag the public subnets in your VPC so that Kubernetes knows to use only those subnets for external load balancers instead of choosing a public subnet in each Availability Zone (in lexicographical order by subnet ID):

    ```
    kubernetes.io/role/elb = 1
    ```

2. Tag the private subnets in the following way so that Kubernetes knows it can use the subnets for internal load balancers.

    ```
    kubernetes.io/role/internal-elb = 1
    ```

    !!! info "Important"
        If you use an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets created by the template are tagged when they're created as explained [here](https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html){ target=_blank }.

3. Use the kubectl `expose` command to generate a Kubernetes service for the deployment.

    ```bash
    kubectl expose deployment house-regression-deployment -n house-regression-namespace --name=house-regression-external --type=LoadBalancer --port 80 --target-port 8080
    ```
    
    *  `--port` is the port number configured on the Load Balancer.
    *  `--target-port` is the port number that the deployment container is listening on.

2. Run the following command to get the service details:

    ```bash
    kubectl get service -n house-regression-namespace
    ```

    ![](images/int_damdmoa_large_010.png)

3. Copy the `EXTERNAL_IP` address.

4. Score your model using the `EXTERNAL_IP` address:

    ```bash
    curl -X POST http://<EXTERNAL_IP>/predictions -H "Content-Type: text/csv" --data-binary @kaggle_house_test_dataset.csv
    ```

    ![](images/int_damdmoa_large_003.png)
    
5. Check the service health of [the external deployment you created](deploy-external-model). The scoring request is now included in the statistics.

    ![](images/int_damdmoa_large_008.png)

### Clean up {: #clean-up }

1. Remove the sample service, deployment, pods, and namespace:

    ```bash
    kubectl delete namespace house-regression-namespace
    ```

2. Delete the cluster:

    ```bash
    eksctl delete cluster --name house-regression
    ```

    ![](images/int_damdmoa_large_004.png)
    
## Wrap-up {: #wrap-up }
    
This walkthrough described how to deploy and monitor DataRobot models on the Amazon EKS platform via a Portable Prediction Server (PPS). A PPS is embedded into Docker containers alongside the MLOps agents, making it possible to acquire the principal IT (service health, number of requests, etc.) and ML (accuracy, data drift etc.) metrics in the cloud and monitor them on the centralized DataRobot MLOps dashboard.

Using DataRobot PPSs allows you to avoid vendor lock-in and easily migrate between cloud environments or deploy models across them simultaneously.
