---
title: Evaluate experiments
description: Describes how to filter the Leaderboard and use visualization tools to evaluate time series models in the DataRobot Workbench interface.
section_name: Time series
maturity: public-preview
---

# Evaluate experiments {: #evaluate-experiments }

Once you start modeling, Workbench begins to construct your model Leaderboard, a list of models ranked by performance, to help with quick model evaluation. The Leaderboard provides a summary, including scoring information, for each model built in an experiment. From the Leaderboard, you can click a model to access visualizations for further exploration. These tools can help you determine what to do in your next experiment.

After Workbench completes [Quick mode](model-data#modeling-modes-explained){ target=_blank }, the most accurate model is selected and trained on 100% of the data. That model is marked with the [**Prepared for Deployment**](model-rec-process){ target=_blank } badge.

![](images/wb-ts-eval-1.png)

## Manage the Leaderboard {: #manage-the-leaderboard }

There are several controls available, described in the following sections, for navigating the Leaderboard and exploring models:

* [View experiment info](#view-experiment-info)
* [Filter models](#filter-models)
* [Sort models by](#sort-models-by)
* [Controls](#actions)

### View experiment info {: #view-experiment-info }

!!! info "Availability information"
    The **Data** and **Feature lists** tabs are Preview options that are on by default.

    <b>Feature flag:</b> Enable Data and Feature Lists tabs in Workbench

Click **View experiment info** to access the following tabs:

Tab | Description
--- | -----------
[Setup](#setup-tab) | Provides summary of information about the experiment.
[Original data](#data-tab) | Displays the data provided for model training, filterable by the contents of a selected feature list.
[Derived modeling data](#data-tab) | Displays the data used for model training, after the feature derivation process was applied, filterable by the contents of any applicable feature list.
[Feature lists](#feature-lists-tab) | Displays feature lists associated with the dataset used in the experiment; all lists are comprised exclusively of derived modeling data.
[Blueprint repository](ml-experiment-add#blueprint-repository) | Provides access to additional blueprints for training.

![](images/wb-ts-eval-2.png)


#### Setup tab {: #setup-tab }

The **Setup** tab reports the parameters used to build the models on this Leaderboard.

![](images/wb-ts-eval-14.png)


Field | Reports...
----- | -----------
Created | A time stamp indicating the creation of the experiment as well as the user who initiated the model run.
Dataset | The name, number of features, and number of rows in the modeling dataset. This is the same information available from the [data preview](wb-data-tab) page.
Target | The feature selected as the basis for predictions, the resulting experiment type, and the [optimization metric](opt-metric){ target=_blank } used to define how to score the experiment's models. You can [change the metric](#sort-models-by) the Leaderboard is sorted by, but the metric displayed in the summary is the one used for the build.
[Partitioning](ts-experiment-create#change-default-settings){ target=_blank } | Details of the date/time partitioning done for the experiment, including the ordering feature, backtest setting, and sampling method. It also provides a [backtest summary and partitioning log](#partitioning-details), described in more detail below.
[Time series modeling](ts-experiment-create#time-series-modeling){ target=_blank } | Details of the time series modeling setup including ordering, series, excluded, and known in advance features, as well as window settings and events calendar information.
Additional settings | Advanced settings that were configured from the [**Additional settings**](ts-experiment-create#configure-additional-settings){ target=_blank } tab.


#### Partitioning details {: #partitioning-details }

The partitioning section's backtest summary and derivation log provide insight into how DataRobot handled the data in preparation for modeling.

![](images/wb-ts-eval-15.png)

* Click **View backtests** for a visual summary of the observations and testing partitions used to build the model. You cannot change the partitions from this window, but you can make modifications to [retrain the model](ts-experiment-add) from the **Model Overview**.

	![](images/wb-ts-eval-16.png)

* Click **View partitioning log** to see (and optionally export) a record of how DataRobot created backtests, assigned data based on window settings, and applied other setup criteria.

	![](images/wb-ts-eval-17.png)


#### Original data tab {: #original-data-tab }

The **Original data** tab provides [summary analytics](model-ref#data-summary-information) of the data used in the experiment setup. To [view exploratory data insights](wb-data-tab), click the **dataset preview** link.

![](images/wb-ts-eval-3.png)

By default, the display includes all features in the dataset prior to feature derivation. You can view analytics for features specific to a single feature list by enabling **Filter by feature list** and selecting a list.

![](images/wb-exp-eval-32.png)

When viewing the list, you can click on the arrow or three dots next to a column name to change the sort order.

![](images/wb-ts-eval-4.png)

#### Derived modeling data tab {: #derived-modeling-data-tab }

The **Derived modeling data** tab is populated after the [feature derivation process](feature-eng){ target=_blank } runs. It shows basic data analytics as well as information that is not available in the **Original data** tab. Use the **Filter by feature list** toggle, described in the **Original data** section, to update the display. Values update to the selected list, as applicable.

![](images/wb-ts-eval-18.png)


| | Element | Description |
|---|---|---|
| <div class="table-label">1</div> | Window setting summaries | Summarizes the [FDW and FW](ts-experiment-create#customize-window-settings){ target=_blank } used in  model training.
| <div class="table-label">2</div> | Basic counts | Reports the number of features and rows in the selected list.
| <div class="table-label">3</div> | View more | Shows additional [time series-specific experiment setup](ts-experiment-create#set-additional-optional-features){ target=_blank } settings.
| <div class="table-label">4</div> | Preview derivation log | Shows the first 100 lines of a log that records the [feature derivation and reduction process](ts-create-data#review-data-and-new-features){ target=_blank } and provides an option to download the full log.
| <div class="table-label">5</div> | [Importance score](model-ref#importance-score){ target=_blank } | Indicates the degree to which a feature is correlated with the target.


#### Feature lists tab {: #feature-lists-tab }

DataRobot automatically constructs time-aware features based on the characteristics of the data. Multiple periodicities can result in several possibilities when constructing the features and, in some cases, it is better to not transform the target by differencing. The choice that yields the optimal accuracy often depends on the data.

After constructing time series features for the data, DataRobot automatically creates multiple [feature lists](ts-feature-lists){ target=_blank }. (Feature lists control the subset of features that DataRobot uses to build models.) Then, at project start, DataRobot automatically runs blueprints using several feature lists, selecting the list that best suits the model type.

Click the **Feature lists** tab to view all feature lists associated with the experiment. The display shows both DataRobot's automatically created time series feature lists and any [custom](wb-data-tab#create-a-feature-list){ target=_blank } feature lists that were created prior to model training.

![](images/wb-ts-eval-5.png)

The following actions are available for feature lists:

Action | Description
------ | -----------
View features | Displays insights for a feature list. This selection opens the **Data** tab with the filter set to the selected list.
Edit name and description | Provides a dialog to change the list name. You cannot change these values for a DataRobot-created list.
Download | Downloads the features contained in that list as a CSV file.
Rerun modeling | Opens the **Rerun modeling** modal to allow selecting a new feature list and restarting Autopilot.  


#### Blueprints repository tab {: #blueprints-repository-tab }

{% include 'includes/blueprint-repo.md' %}


### Filter models {: #filter-models }

Filtering makes viewing and focusing on relevant models easier. Click **Filter models** to set the criteria for the models that Workbench displays on the Leaderboard. The choices available for each filter are dependent on the experiment and/or model type&mdash;they were used in at least one Leaderboard model&mdash;and will potentially change as models are added to the experiment. For example:

![](images/wb-exp-eval-3a.png)


Filter | Displays models that...
----- | -------------
Labeled models | Have been assigned the listed tag, either [starred models](leaderboard-ref#tag-and-filter-models){ target=_blank } or models [recommended for deployment](model-rec-process#prepare-a-model-for-deployment){ target=_blank }.
Feature list | Were built with the selected feature list.
Training period | Were trained on the selected training period, either a specific duration or [start/end dates](ts-leaderboard#setting-the-start-and-end-dates){ target=_blank }.
Model family | Are part of the selected [model family](wb-model-compare#model-family-selections){ target=_blank }.

### Sort models by {: #sort-models-by }

By default, the Leaderboard sorts models based on the score of the validation partition, using the selected [optimization metric](opt-metric){ target=_blank }. You can, however, control the basis of the display parameter when evaluating models.

Note that although Workbench built the experiment using the most appropriate metric for your data, it computes many applicable metrics on each of the models. After the build completes, you can redisplay the Leaderboard listing based on a different metric. It doesn't change any values within the models; it simply reorders the model listing based on model performance in this alternate metric.

![](images/wb-ts-eval-6.png)


See the page on [optimization metrics](opt-metric){ target=_blank } for detailed information on each.

### Controls {: #controls }

Workbench provides simple, quick shorthand controls:

Icon | Action
---- | ------
![](images/icon-wb-rerun.png)| Reruns Quick mode with a different feature list. If you select a feature list that has already run, Workbench replaces the deleted models or make no changes.
![](images/icon-wb-duplicate.png) | [Duplicates the experiment](manage-projects#duplicate-a-project){ target=_blank } with an option to reuse just the dataset, or the dataset and settings.
![](images/icon-wb-delete.png) | Deletes the experiment and its models. If the experiment is being used by an application, you cannot delete it.
![](images/icon-wb-close.png) | Slides the Leaderboard panel closed to make additional room for, for example, viewing insights.


## Model insights  {: #model-insights }

Model insights help to interpret, explain, and validate what drives a model’s predictions. Available insights are dependent on experiment type, but may include the insights listed in the table below. Availability of [sliced insights](sliced-insights){ target=_blank } is also model-dependent.

!!! info "Availability information"
	* Sliced insights in Workbench are off by default. Contact your DataRobot representative or administrator for information on enabling the feature.  **Feature flag:** Slices in Workbench
	* Feature Effects is _off_ by default. Contact your DataRobot representative or administrator for information on enabling it. **Feature flag:** Slices in Workbench


Insight | Description | Problem type | Sliced insights?
------- | ----------- | ------------ | ----------------
[Accuracy Over Time](#accuracy-over-time) | Visualizes how predictions change over time. | Time-aware
[Blueprint](#blueprint) | Provides a graphical representation of data preprocessing and parameter settings. | All |
[Feature Effects](#feature-effects) | Conveys how changes to the value of each feature change model predictions | All |✔
[Feature Impact](#feature-impact) | Shows which features are driving model decisions. | All | ✔
[Forecasting Accuracy](#forecasting-accuracy) | Depicts how well a model predicts at each forecast distance in the experiment's forecast window. | Time series |
[Forecast vs Actual](#forecast-vs-actual) | Predicts multiple values for each point in time (forecast distances). | Time series |
[Lift Chart](#lift-chart) | Depicts how well a model segments the target population and how capable it is of predicting the target.| All |  ✔
[Period Accuracy](#period-accuracy) | Shows model performance over periods within the training dataset. | Time-aware |  
[ROC Curve](#roc-curve) | Provides tools for exploring classification, performance, and statistics related to a model. | Classification | ✔
[Series Insights](#series-insights) | Provides series-specific information for multiseries experiments. | Time series |
[Stability](#stability) | Provides a summary of how well a model performs on different backtests. | Time-aware |

To see a model's insights, click on the model in the left-pane Leaderboard.

### Accuracy Over Time  {: #accuracy-over-time }

For time-aware experiments, [Accuracy Over Time](aot){ target=_blank } helps to visualize how predictions change over time. By default, the view shows predicted and actual vs. time values for the training and validation data of the most recent (first) backtest. This is the backtest model DataRobot uses to deploy and make predictions. (In other words, the model used to generate the error metric for the validation set.) For time series experiments, you can control the series (if applicable) and forecast distance used in the display. Note that series-based experiments are sometimes [compute-on-demand](aot#display-by-series){ target=_blank }, depending on projected space and memory requirements.

![](images/wb-ts-eval-13.png)

The visualization also has a time-aware [Residuals](aot#interpret-the-residuals-chart){ target=_blank } tab that plots the difference between actual and predicted values. It helps to visualize whether there is an unexplained trend in your data that the model did not account for and how the model errors change over time. For time series experiments, you can additionally set the forecast distance used in the display.


![](images/wb-exp-eval-19.png)

### Blueprint {: #blueprint }

Blueprints are ML pipelines containing preprocessing steps (tasks), modeling algorithms, and post-processing steps that go into building a model. The [Blueprint](blueprints){ target=_blank } tab provides a graphical representation of the blueprint, showing each step. Click on any task in the blueprint to see more detail, including more complete model documentation (by clicking **DataRobot Model Docs** from inside the blueprint’s task).

![](images/wb-ts-eval-7.png)


Additionally, you can access the [blueprint repository](ml-experiment-add#blueprint-repository) from the **Blueprint** tab:

![](images/wb-exp-eval-22.png)

### Feature Effects {: #feature-effects }

The [Feature Effects](feature-effects){ target=_blank } insight shows the effect of changes in the value of each feature on model predictions&mdash;how does a model "understand" the relationship between each feature and the target? It is an on-demand feature, dependent on the [Feature Impact](feature-impact){ target=_blank } calculation, which is prompted for when first opening the visualization. The insight is communicated in terms of [partial dependence](feature-effects#partial-dependence-logic){ target=_blank }, an illustration of how changing a feature's value, while keeping all other features as they were, impacts a model's predictions.

![](images/wb-exp-eval-29.png)

### Feature Impact {: #feature-impact }

{% include 'includes/feature-impact-include.md' %}

You can view impact for the original features:

![](images/wb-ts-eval-8.png)

And [impact for derived modeling data](feature-impact#feature-impact-with-time-series-permutation-only){ target=_blank }:

![](images/wb-ts-eval-9.png)

### Forecasting Accuracy {: #forecasting-accuracy }

The [Forecasting Accuracy](forecast-acc){ target=_blank } tab provides a visual indicator of how well a model predicts at each forecast distance in the experiment's forecast window. It is available for all time series experiments (both single series and multiseries). Use it to help determine, for example, how much harder it is to accurately forecast four days out as opposed to two days out. The chart depicts how accuracy changes as you move further into the future.

![](images/wb-ts-eval-10.png)


### Forecast vs Actual {: #forecast-vs-actual }

[Forecast vs. Actual](fore-act){ target=_blank } allows you to compare how different predictions behave from different forecast points to different times in the future. Use the chart to help answer what, for your needs, is the best distance to predict. Forecasting out only one day may provide the best results, but it may not be the most actionable for your business. Forecasting the next three days out, however, may provide relatively good accuracy and give your business time to react to the information provided. If the experiment included calendar data, those events are displayed on this chart, providing insight into the effects of those events. Note that series-based experiments are sometimes [compute-on-demand](aot#display-by-series){ target=_blank }, depending on projected space and memory requirements.

![](images/wb-ts-eval-11.png)


### Lift Chart {: #lift-chart }

{% include 'includes/lift-chart-include.md' %}

![](images/wb-exp-eval-17.png)

### Period Accuracy {: #period-accuracy }

[Period Accuracy](ts-period-accuracy) gives you the ability to specify which are the more important periods within your training dataset, which DataRobot can then provide aggregate accuracy metrics for and surface those results on the Leaderboard.

Period Accuracy lets you define periods within your dataset and then compare their metric scores against the metric score of the model as a whole. In other words, you can specify which are the more important periods within your training dataset, and DataRobot can then provide aggregate accuracy metrics for that period and surface those results on the Leaderboard. Periods are defined in a separate CSV file that identifies which rows to group based on the experiment’s data/time feature. Once uploaded, and with the insight calculated, DataRobot provides a table of period-based results and an “over time” histogram for each period.

![](images/period-accuracy-wb1.png)

### ROC Curve {: #roc-curve }

For classification experiments, the ROC Curve tab provides the following tools for exploring classification, performance, and statistics related to a selected model at any point on the probability scale:

* An [ROC Curve](roc-curve){ target=_blank }
* [Cumulative charts](cumulative-charts){ target=_blank }
* A [confusion matrix](confusion-matrix){ target=_blank }
* A [payoff matrix/profit curve](profit-curve){ target=_blank }
* [Metrics](metrics){ target=_blank }

![](images/wb-exp-eval-6.png)

### Series Insights {: #series-insights }

The [Series Insights](series-insights-multi){ target=_blank } tab for multiseries experiments provides series-specific information in both charted and tabular format:

![](images/wb-ts-eval-12.png)

To speed processing, Series Insights visualizations are initially computed for the first 1000 series (sorted by ID). You can, however, **Compute accuracy scores** for the remaining series data. Use the **Plot distribution** controls and binning to change the display.


### Stability {: #stability }

The [Stability](stability) tab provides an at-a-glance summary of how well a model performs on different backtests. It helps to measure performance and gives an indication of how long a model can be in production (how long it is "stable") before needing retraining. The values in the chart represent the validation scores for each backtest and the holdout.

![](images/wb-exp-eval-20.png)

{% include 'includes/wb-compliance-experiment.md' %}

## Manage experiments {: #manage-experiments }

At any point after models have been built, you can manage an individual experiment from within its Use Case. Click on the three dots to the right of the experiment name to delete it. To share the experiment, use the Use Case [**Manage members**](wb-build-usecase#share){ target=_blank } tool to share the experiment and other associated assets.

![](images/wb-exp-15.png)

## What's next? {: #whats-next }

After selecting a model, you can, from within the experiment:

* [Add models to experiments](ts-experiment-add){ target=_blank }.
* [Make predictions](wb-predict){ target=_blank }.
* [Create No-Code AI Apps](wb-apps/index){ target=_blank }.
* [Generate a compliance report](#compliance-documentation).
