---
title: Create experiments
description: Describes how to create and manage time-aware experiments in the DataRobot Workbench interface.
section_name: Time-aware
maturity: public-preview
---

# Create experiments {: #create-experiments }

There are two AI experimentation "types" available in Workbench:

* _Time-aware modeling_, described on this page, models using _time-relevant data_ to make row-by-row predictions, time series forecasts, or current value predictions ["nowcasts"](nowcasting).

* _Predictive modeling_, described [here](ml-experiment-create){ target=_blank }, makes row-by-row predictions based on your data.

!!! note
	There is extensive material available about the fundamentals of [time aware](whatis-time) modeling. While the instructions largely represent the workflow as applied in DataRobot Classic, the reference material describing the [framework](ts-framework), [feature derivation process](feature-eng), and more are still applicable.


Experiments are the individual "projects" within a [Use Case](wb-build-usecase). They allow you to vary data, targets, and modeling settings to find the optimal models to solve your business problem. Within each experiment, you have access to its Leaderboard and [model insights](ts-experiment-evaluate#insights), as well as [experiment summary information](ts-experiment-evaluate#view-experiment-info).

!!! info "Public Preview"
	Time series modeling is on by default.

	<b>Feature flag: Enable Workbench for Time Series Projects

{% include 'includes/wb-experiment-create-basic.md' %}

DataRobot automatically creates new feature lists after the feature derivation process. Once modeling completes, you can [train new models](#change-feature-list-post-modeling) using the time-aware lists. Learn more about feature lists and the data tab [here](wb-data-tab#create-a-feature-list){ target=_blank }.

{% include 'includes/wb-add-experiment-add-data.md' %}

{% include 'includes/wb-select-target.md' %}

After the target is defined, Workbench displays a histogram providing information about the target feature's distribution and, in the right panel, a summary of the experiment settings. From here, you can build models with the default settings for [predictive modeling](ml-experiment-create).

If DataRobot detected a column with time features ([variable type "Date"](file-types#special-column-detection){ target=_blank }) in the dataset, as reported in the **Experiment summary**, you can [build time-aware models](#enable-time-aware-modeling).

![](images/wb-ts-exp-10.png)

## Customize basic settings {: #customize-basic-settings }

Prior to enabling time-aware modeling, you can change several of the basic modeling settings. These options are common to both predictive and time-aware modeling.

{% include 'includes/wb-customize-basic-experiment-1.md' %}

After changing any or all of the settings described, click **Next** to customize more [advanced settings](#configure-additional-settings) and [enable time-aware modeling](#enable-time-aware-modeling).  

### Change modeling mode {: #change-modeling-mode }

By default, DataRobot builds experiments using Quick Autopilot; however, you can change the modeling mode to train specific blueprints or all applicable repository blueprints.

![](images/wb-exp-32.png)

The following table describes each of the modeling modes:

|  Modeling mode | Description |
|-------------|-------------|
|  Quick (default)  | Using a sample size of 64%, Quick Autopilot runs a [subset](model-ref#specifics-of-quick-autopilot) of models, based on the specified target feature and performance metric, to provide a base set of models that build and provide insights quickly.|
|  Manual | Manual mode gives you full control over which blueprints to execute. After [EDA2](eda-explained) completes, DataRobot redirects you to the [blueprint repository](ml-experiment-evaluate#blueprints-repository-tab) where you can select one or more blueprints for training. |

{% include 'includes/wb-customize-basic-experiment-2.md' %}

You can also [change the selected list](ts-experiment-add#train-on-new-settings) on a per-model basis once the experiment finishes building.
	
### Configure additional settings {: #configure-additional-settings }

Choose the **Additional settings** tab to set more advanced modeling capabilities. Note that the **Time series modeling** tab will be available or greyed out depending on whether DataRobot found any date/time features in the dataset.

![](images/wb-exp-33.png)

Configure the following, as required by your business use case.

* [Monotonic feature constraints](#monotonic-feature-constraints)
* [Weight](#weight)

!!! note
    You can complete the [time-aware configuration](#enable-time-aware-modeling) or the additional settings in either order.

{% include 'includes/wb-config-additional-settings.md' %}

## Enable time-aware modeling {: #enable-time-aware-modeling }

Time-aware modeling is based on **Date/time** as the partitioning method. There are two types of time-aware modeling:

* Simple **Date/time** partitioning, which assigns rows to backtests chronologically.
* [Time series modeling](#5-continue-time-series-configuration) for forecasting multiple future values of the target.

The workflow for both begins with partitioning. If you want forecasts instead of predictions, or want to take advantage of DataRobot's automated [time series feature engineering](feature-eng), you will continue setup on the **Time series modeling** tab.

``` mermaid
graph TB
  A(Start time-aware modeling) --> B[Enable date/time partitioning];
  B --> C[Set ordering feature];
  C --> D[Set backtest partitions];
  D --> E[Set sampling];
  E --> F{Predictions or forecasts?} --> | Predictions|G[Start modeling];
  F --> |Forecasts|H[Enable time series modeling]
  H --> I[Continue experiment setup]
  I --> J[Start modeling]

```


??? info "Date and date range representation"
    DataRobot uses <em>date points</em>  to represent dates and date ranges within the data, applying the following principles:

    * All date points adhere to ISO 8601, UTC (e.g., 2016-05-12T12:15:02+00:00), an internationally accepted way to represent dates and times, with some small variation in the duration format. Specifically, there is no support for ISO weeks (e.g., P5W).
    * Models are trained on data between two ISO dates. DataRobot displays these dates as a date range, but inclusion decisions and all key boundaries are expressed as date points. When you specify a date, DataRobot includes start dates and excludes end dates.
    * Once changes are made to formats using the date partitioning column, DataRobot converts all charts, selectors, etc. to this format for the project.
    * When changing partition year/month/day settings, the month and year values rebalance to fit the larger class (for example, 24 months becomes two years) when possible. However, because DataRobot cannot account for leap years or days in a month as it relates to your data, it cannot convert days into the larger container.

### 1. Enable date/time partitioning {: #1-enable-date-time-partitioning }

To enable time-aware modeling, first set the partitioning method to **Date/time**. To do this do one of the following:

* Open the **Data partitioning** tab. 
* Click on **Partitioning** in the experiment summary panel, which opens the **Data partitioning** tab.

From the **Data partitioning** tab, select **Date/time** as the partitioning method:

![](images/wb-ts-exp-15.png)

### 2. Set ordering feature {: #2-set-ordering-feature }

After setting the partitioning method to date/time, set the _ordering feature_&mdash;the primary date/time feature DataRobot uses for modeling. 

!!! note
	All other settings can be changed or left at the default. The **Experiment summary**, in the right-hand panel, updates as setup continues.

Select the ordering feature. If only one qualifying feature is detected, the field is autofilled. If multiple features are available, click in the box to display a list of all qualifying features. If a feature is not listed, it was not detected as type `date` and cannot be used. 

Once set, DataRobot:

* Detects and reports the date and/or time format ([standard GLIBC strings](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior){ target=_blank }) for the selected feature:

	![](images/wb-exp-13d.png)

* Computes and then loads a feature-over-time histogram of the ordering feature. Note that if your dataset qualifies for [multiseries modeling](#continue-time-series-configuration), this histogram represents the average of the time feature values across all series plotted against the target feature.

	![](images/wb-ts-exp-13.png)

### 3. Set backtest partitions {: #3-set-backtest-partitions }

Once the ordering feature is set, backtest configuration becomes available. Backtests are the time-aware equivalent of cross-validation, but based on time periods or durations instead of random rows. DataRobot sets defaults based on the characteristics of the dataset and can generally be left as-is&mdash;they will result in robust models. 

![](images/wb-exp-13b.png)

Use the links in the table below to change the default settings:

|  &nbsp; | Field	| Description |
|---------|--------|--------------|
| <div class="table-label">1</div> | Backtests | Sets the [number](#change-number-of-backtests) and [duration](#change-partition-durations) of backtesting partitions. | 
| <div class="table-label">2</div> | [Row assignment](#row-assignment) | Sets how rows are assigned to backtests and the sampling method. |
| <div class="table-label">3</div> | Partitioning log / [Reset](#reset-defaults)| Provides a downloadable log that reports on partition creation and provides a reset link. |
| <div class="table-label">4</div> | [Time series modeling](#5-continue-time-series-configuration) settings | Opens the **Time series modeling**tab to enable forecasting and access additional options. |

??? tip "Learn more"
	The following sections provide detailed explanations of each concept described in this configuration:
	
	* [Understanding backtesting](otv#understanding-backtests){ target=_blank }
	* [Understanding gaps](otv#understanding-gaps){ target=_blank }
	* [Interpret the backtest visualization](otv#interpret-the-display){ target=_blank }

#### Change number of backtests {: #change-number-of-backtests }

First, change the number of backtests if desired. 

![](images/wb-backtests-1.png)

The default number of backtests is dependent on the project parameters, but you can configure up to 20. Before setting the number of backtests, use the histogram to validate that the training and validation sets of each fold will have sufficient data to train a model. 

??? tip "Backtest requirements"
	* For OTV, backtests require at least 20 rows in each validation and holdout fold and at least 100 rows in each training fold. If you set a number of backtests that results in any of the partitions not meeting those criteria, DataRobot only runs the number of backtests that do meet the minimums (and marks the display with an asterisk).
	* For time series, backtests require at least 4 rows in validation and holdout and at least 20 rows in the training fold. If you set a number of backtests that results in any of the partitions not meeting those criteria, the project could fail. See the [time series partitioning reference](ts-customization) for more information.

#### Change partition durations {: #change-partition-durations }

Next, configure the backtesting partitions. If you don't modify any settings, DataRobot disperses rows to backtests equally. However, you can customize backtest gap, training, validation, and holdout data either:

* To apply globally to all backtests in the experiment, use the dropdowns.
* To [apply changes to individual backtests](#set-backtest-partitions-individually), click the bars below the visualization. Individual settings override global settings. Once you modify settings for an individual backtest, any changes to the global settings are not applied to the edited backtest.

As you add backtests to the experiment configuration, the period of training data used shortens. The validation and gap remain set to the duration set in the dropdowns (unless changed individually per backtest).

Review the default partition settings and click to make changes if needed:

![](images/wb-backtests-2.png)

The following provides an overview of the application of each partition, but review the linked material for more detail.

Partition | Description
--------- | -----------
[Default validation duration](otv#set-the-validation-length){ target=_blank } | Sets the size of the partition used for testing&mdash;data that is not part of the training set that is used to evaluate a model’s performance.
[Default gap duration](otv#set-the-gap-length) | Sets spaces in time, representing gaps between model training and model deployment. Initially set to zero, DataRobot does not process a gap in testing. When set, DataRobot excludes the data that falls in the gap from use in training or evaluation of the model.

Note how the changes are reflected in the testing representation:

![](images/wb-backtests-3.png)

#### Set backtest partitions individually {: #set-backtest-partitions-individually }

Regardless of which partition you are setting (training, validation, or gap) elements of the editing screens function the same ([holdout](#change-holdout) is a bit different). To change an individual backtest's duration, first, hover on the color band to see specifics of the specific duration settings: 

![](images/wb-backtests-4.png)

Then, to modify the duration for an individual backtest, click in the backtest to open the inputs:

![](images/wb-backtests-5.png)

Backtests are based on either *start and end dates* or *start date and duration*. Gaps&mdash;toggle **Add gap between partitions** to on to enabled&mdash;are derived from the date or duration settings. That is, the gap is created by leaving time steps between the training end and validation start (which, for no gap are the same).

![](images/wb-backtests-6.png)

To customize based on *start and end dates*:

* With a gap, set the start and duration for both training and validation.
* Without a gap, set the training start, training duration, and validation duration.

To customize based on *start date and duration*:

* With a gap, set the training and validation start and end.
* Without a gap, set the training start, training end, validation start, and validation end.

In all cases, DataRobot verifies entries and reports any required changes:

![](images/wb-backtests-8.png)

Then, reports valid, set time windows under the input boxes:

![](images/wb-backtests-7.png)

Click **Save changes** when configuration is complete. 

Once you have made changes to a data element, DataRobot adds an **EDITED** label to the backtest. Use the [**Reset to defaults**](#reset-defaults) link to manually reset durations or number of backtests to the default settings.

#### Change holdout {: #change-holdout }

By default, DataRobot creates a holdout fold for training models in your project. 

??? tip "Notes on changing holdout"

	* Generally speaking, although very experiment-dependent, holdout is roughly 10% of the total duration with possible rounding to some "natural" time frame like _n_ weeks, _n_ months. (This is dependent on whether it is simple time-aware or time series.) View the partitioning log for a description of DataRobot's calculations.
	* You can only set holdout in the holdout backtest, you cannot change the training data size in that backtest. DataRobot automatically configures the training partition of the holdout backtest.
	* If, during the initial partitioning detection, the backtest configuration of the ordering (date/time) feature, series ID, or target results in insufficient rows to cover both validation and holdout, DataRobot automatically disables holdout. If other partitioning settings are changed (validation or gap duration, start/end dates, etc.), holdout is not affected unless manually disabled.

To modify the holdout length, click the holdout backtest to open the inputs and enter new values: 

* To customize based on *start and end dates*, enter holdout start and end dates.
* To customize based on *start date and duration* enter holdout start and duration.

Note that the training time span and gap settings are configured automatically and cannot be changed on the holdout backtest:

![](images/wb-backtests-9.png)

!!! note
	[In some cases](ts-date-time#partition-without-holdout), however, you may want to create a project without a holdout set. To do so, uncheck the **Add holdout** box. Any insights that provide an option to switch between validation and holdout will not show the holdout option.

### 4. Set sampling {: #4-set-sampling }

After completing backtests, set the [row assignment](#row-assignment) and sampling method.

#### Row assignment {: #row-assignment }

By default, DataRobot ensures that each backtest has the same _duration_, either the default or the values set from the dropdown(s) or via the [bars in the visualization](#change-backtest-partitions). If you want the backtest to use the same number of _rows_, instead of the same length of time, use the **Equal rows per backtest** toggle:

!!! note
	Time series projects also have an option to set row assignment (number of rows or duration) for the training data that is used during feature engineering. Configure this setting in the [training window format](ts-customization#duration-and-row-count) section.

* When **Equal rows per backtest** is checked (which sets the partitions to row-based assignment), only the training end date is applicable.

* For time series experiments, when **Equal rows per backtest** is checked, the dates displayed are informative only (that is, they are approximate) and they include padding that is set by the feature derivation and forecast point windows.

#### Sampling method {: #sampling-method }

Once you have selected the mechanism/mode for assigning data to backtests, select the sampling method, either **Random** or **Latest**, to set _how_ to assign rows from the dataset.

Setting the sampling method is particularly useful if a dataset is not distributed equally over time. For example, if data is skewed to the most recent date, the results of using 50% of random rows versus 50% of the latest will be quite different. By selecting the data more precisely, you have more control over the data that DataRobot trains on.

#### Reset defaults {: #reset-defaults }

When you make and save changes to any of the backtest settings, the backtest is marked with a badge (EDITED). Use the **Reset to defaults** link to manually reset durations or number of backtests to the default settings. Note that for time series experiments, this action does not reset any window settings.

#### Predictions or forecasts? {: predictions-or-forecasts }

For predictive modeling, when you're satisfied with the modeling settings, click **Start modeling** to begin the Quick mode Autopilot modeling process.  

For forecasting, if your experiment is eligible click **Go to time series modeling settings** to open the **Time series modeling** tab and continue the configuration.

### 5. Continue time series configuration {: #5-continue-time-series-configuration }

Time series modeling provides an additional set of options, such as identifying series IDs (as applicable), initiating the creation of a derived feature set for modeling, and other particulars of forecasting.

To create experiments that launch time-aware modeling, you can select:

*  **Go to time series modeling settings** from the **Date/time** partitioning setup page for time relevant data.

	![](images/wb-ts-exp-16.png)

* **Time series modeling** in the **Experiment summary** panel.

	![](images/wb-ts-exp-17.png)

Either option opens the settings to the **Time series modeling** tab. Settings are inherited from the **Data partitioning** tab (ordering feature and backtests). See and complete, as needed:

* [Enable date/time partitioning](#1-enable-date-time-partitioning)
* [Set ordering feature](#2-set-ordering-feature)
* [Set backtest partitions](#3-set-backtest-partitions)
* [Set sampling](#4-set-sampling)

The following additional settings are available to configure your time series experiment:

* [Select an ordering feature](#select-an-ordering-feature)
* [Set the series ID](#set-series-id) (for multiseries projects)
* [Customize window settings](#customize-window-settings)
* [Set additional optional features](#set-additional-optional-features)

![](images/wb-ts-exp-11.png)

#### Set series ID {: #set-series-id }

If duplicate time stamps are detected in the data, DataRobot provides options for configuring [multiseries modeling](multiseries). Multiseries modeling allows you to model datasets that contain duplicate timestamps by handling them as multiple, individual time-series datasets. Select a series identifier to indicate which series each row belongs to.

![](images/wb-ts-exp-14.png)


#### Customize window settings {: #customize-window-settings }

DataRobot provides default window settings, the Feature Derivation Window (FDW) and Forecast Window (FW), based on the characteristics of the dataset. These settings determine how DataRobot derives features for the modeling dataset by defining the [basic framework](ts-framework#basic-time-series-framework){ target=_blank } used for the feature derivation process. They can generally be left as-is.

![](images/wb-ts-exp-18.png)

The table below briefly describes the elements of the window setting section of the screen:

!!! important
	If you do decide to modify these values, see the [detailed guidance](ts-customization#set-window-values){ target=_blank } for the meaning and implication of each window.

| | Option | Description |
|---|---|---|
| <div class="table-label">1</div> | [Feature Derivation Window](glossary/index#feature-derivation-window){ target=_blank } (FDW) |  Configures the periods of data that DataRobot uses to [derive features](ts-create-data){ target=_blank } for the modeling dataset.
| <div class="table-label">2</div> | [Exclude listed features from derivation](ts-adv-opt#exclude-features-from-derivation){ target=_blank } | Excludes specified features from automated time-based feature engineering (for example, if you have extracted your own time-oriented features and do not want further derivation performed on them). Toggle the option on and select features from the dropdown.
| <div class="table-label">3</div> | [Forecast Window](glossary/index#forecast-window){ target=_blank } | Sets the time range of forecasts that the model outputs after the [forecast point](glossary/index#forecast-point){ target=_blank }.
| <div class="table-label">4</div> | Windows summary | Provides a graphical representation of the window settings. Any changes to window values are immediately reflected in the visual.


#### Set additional optional features {: #set-additional-optional-features }

Two additional optional experiment settings are available:

![](images/wb-ts-exp-19.png)

* [**Set features that are known in advance**](ts-adv-opt#set-known-in-advance-ka){ target=_blank }
* [**Include events calendar**](ts-adv-opt#calendar-files){ target=_blank }

Use **Set features that are known in advance** to exclude features for which you know their value at modeling time. When a feature is identified with this option, DataRobot will not create lags when deriving modeling data. By informing DataRobot that some variables are known in advance and providing them at prediction time, forecast accuracy is significantly improved. If a feature is flagged as known, however, you must provide its future value at prediction time or predictions will fail. To use this option, toggle it on and select features from the dropdown.

Use **Include events calendar** to upload or generate an event file that specifies dates or events that require additional attention. DataRobot will use the file to automatically create features based on the listed events. You can choose a local file or one stored in the data registry. Or, click **Generate calendar** to let DataRobot generate a file of events based on a selected region.

![](images/wb-ts-exp-20.png)


### 6. Start forecast modeling {: #6-start-forecast-modeling }

After you are satisfied with the modeling settings (which are summarized in the **Experiment summary**), simply click **Start modeling**. When the process begins, DataRobot analyzes the target and creates time-based features to use for modeling.

## What's next? {: #whats-next }

After you start modeling, DataRobot populates the Leaderboard with models as they complete. You can:

* Begin [model evaluation](ts-experiment-evaluate) on any available model.
* Use the [**View experiment info**](ts-experiment-evaluate#view-experiment-info){ target=_blank } option to view a variety of information about the model.
* Display derived modeling data, which is the data that was used to build the model.

![](images/wb-ts-exp-24.png)

See the following sections for more information on derived modeling data:

* [Date duration features](ts-leaderboard#build-time-aware-models).
* [Time series feature derivation process](feature-eng).
