Manage a run with the SDK#

Runs can be managed through the Python API. Below outlines how to work with runs, including creation, following, getting, stopping, and deleting runs. Before getting started, familiarize yourself with Run Lifecycle.

Creating a run#

mcli.api.runs.create_run(run, *, timeout=10, future=False)[source]

Launch a run in the MosaicML platform

The provided run must contain enough information to fully detail the run

Parameters

run – A fully-configured run to launch. The run will be queued and persisted in the run database.
timeout – Time, in seconds, in which the call should complete. If the run creation takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future – Return the output as a Future. If True, the call to create_run will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the :type Run: output, use return_value.result() with an optional timeout argument.

Returns

A Run that includes the launched run details and the run status

Runs can programmatically be created, giving you flexibility to define custom workflows or create similar runs in quick succession. create_run() will takes a RunConfig object, which is a fully-configured run ready to launch. The method will launch the run and then return a Run object, which includes the RunConfig data in Run.config but also data received at the time the run was launched.

The `RunConfig` object#

The RunConfig object holds configuration data needed to launch a run. This is the underlying python data structure MCLI uses, so before beginning make sure to familiarize yourself with the Run schema.

class mcli.api.runs.RunConfig(name=None, parent_name=None, image=None, gpu_type=None, gpu_num=None, cpus=None, cluster=None, scheduling=<factory>, compute=<factory>, parameters=<factory>, integrations=<factory>, env_variables=<factory>, metadata=<factory>, command='', dependent_deployment=<factory>, _suppress_deprecation_warnings=False)[source]

A run configuration for the MosaicML platform

Values in here are not yet validated and some required values may be missing. On attempting to create the run, a bad config will raise a MapiException with a 400 status code.

Required args:

name (str): User-defined name of the run
image (str): Docker image (e.g. mosaicml/composer)
command (str): Command to use when a run starts
compute (ComputeConfig or Dict[str, Any]): Compute configuration. Typically
a subset of the following fields will be required:
- cluster (str): Name of cluster to use
- instance (str): Name of instance to use
- gpu_type (str): Name of gpu type to use
- gpus (int): Number of GPUs to use
- cpus (int): Number of CPUs to use
- nodes (int): Number of nodes to use
See mcli get clusters for a list of available clusters and instances

Optional args:

parameters (Dict[str, Any]): Parameters to mount into the environment
scheduling (SchedulingConfig or Dict[str, Any]): Scheduling configuration
- priority (str): Priority of the run (default auto with options low and lowest`)
- preemptible (bool): Whether the run is preemptible (default False)
- retry_on_system_failure (bool): Whether the run should be retried on system failure (default False)
- max_retries (int): Maximum number of retries (default 0)
- max_duration (float): Maximum duration of the run in hours (default None)
  Run will be automatically stopped after this duration has elapsed.
integrations (List[Dict[str, Any]]): List of integrations. See integration documentation for more details:
https://docs.mosaicml.com/projects/mcli/en/latest/resources/integrations/index.html
env_variables (Dict[str, str]): Dictionary of environment variables to set in the run
- key (str): Name of the environment variable
- value (str): Value of the environment variable
metadata (Dict[str, Any]): Arbitrary metadata to attach to the run

There are two ways to initialize a RunConfig object that can be used to config and create a run. The first is by referencing a YAML file, equivalent to the file argument MCLI:

from mcli.api.runs import RunConfig, create_run

run_config = RunConfig.from_file('hello_world.yaml')
created_run = create_run(run_config)

Alternatively, you can instantiate the RunConfig object directly in python:

from mcli.api.runs import RunConfig, create_run

cluster = "<your-cluster>"
run_config = RunConfig(
    name='hello-world',
    image='bash',
    command='echo "Hello World!" && sleep 60',
    compute={'gpu_type': 'none', 'cluster': cluster}
)
created_run = create_run(run_config)

These can also be used in combination, for example loading a base configuration file and modifying select fields:

from mcli.api.runs import RunConfig, create_run

special_config = RunConfig.from_file('base_config.yaml')
special_config.gpus = 8
created_run = create_run(special_config)

Changing parameters for parameter sweeps

If you are trying to kick off a bunch of runs with similar configurations and different training parameters, make sure you copy the parameters (and any other dict field) instead of modifying them directly

import copy

config = RunConfig.from_file('base_config.yaml')

params = { ... }
for lr in (0.1, 0.01, 0.001):
    new_params = copy.deepcopy(params)
    new_params['optimizers']['sgd']['lr'] = lr
    config.parameters = new_params
    created_run = create_run(config)

The `Run` object#

Created runs will be returned as a Run object in create_run(). This object can be used as input to any subsequent run function, for example you can start a run and then immediately start following it:

created_run = create_run(config)
for line in follow_run_logs(created_run):
    print(line)

class mcli.api.runs.Run(run_uid, name, status, created_at, updated_at, created_by, priority, preemptible, retry_on_system_failure, cluster, gpus, gpu_type, cpus, node_count, latest_resumption, is_deleted, run_type, max_retries=None, reason=None, nodes=<factory>, submitted_config=None, metadata=None, last_resumption_id=None, resumptions=<factory>, events=<factory>, lifecycle=<factory>, image=None, max_duration=None, _required_properties=('id', 'name', 'status', 'createdAt', 'updatedAt', 'reason', 'createdByEmail', 'priority', 'preemptible', 'retryOnSystemFailure', 'resumptions', 'isDeleted', 'runType'))[source]

A run that has been launched on the MosaicML platform

Parameters

run_uid (str) – Unique identifier for the run
name (str) – User-defined name of the run
status (RunStatus) – Status of the run at a moment in time
created_at (datetime) – Date and time when the run was created
updated_at (datetime) – Date and time when the run was last updated
created_by (str) – Email of the user who created the run
priority (str) – Priority of the run; defaults to auto but can be updated to low or lowest
preemptible (bool) – Whether the run can be stopped and re-queued by higher priority jobs
retry_on_system_failure (bool) – Whether the run should be retried on system failure
cluster (str) – Cluster the run is running on
gpus (int) – Number of GPUs the run is using
gpu_type (str) – Type of GPU the run is using
cpus (int) – Number of CPUs the run is using
node_count (int) – Number of nodes the run is using
latest_resumption (Resumption) – Latest resumption of the run
max_retries (Optional[int]) – Maximum number of times the run can be retried
reason (Optional[str]) – Reason the run was stopped
nodes (List[:class:`~mcli.api.model.run.Node]`) – Nodes the run is using
submitted_config (Optional[:class:`~mcli.models.run_config.RunConfig]`) – Submitted run configuration
metadata (Optional[Dict[str, Any]]) – Metadata associated with the run
last_resumption_id (Optional[str]) – ID of the last resumption of the run
resumptions (List[:class:`~mcli.api.model.run.Resumption]`) – Resumptions of the run
lifecycle (List[:class:`~mcli.api.model.run.RunLifecycle]`) – Lifecycle of the run
image (Optional[str]) – Image the run is using

property started_at

The time the run was first started

If there are multiple resumptions, this will be the earliest start time Started At will be None if the first resumption has not been started

Returns: The time the run was first started

property completed_at

The time the run was completed

If there are multiple resumptions, this will be the last end time Completed At will be None if the last resumption has not been completed

Returns: The time the run was last completed

property display_name

The name of the run to display in the CLI

Returns: The name of the run

property cumulative_pending_time

Cumulative time spent in the PENDING state

Returns: The cumulative time (seconds)

property cumulative_running_time

Cumulative time spent in the RUNNING state

Returns: The cumulative time (seconds)

property resumption_count

Number of times the run has been resumed

Returns: The number of times the run has been resumed

clone(name=None, image=None, cluster=None, instance=None, nodes=None, gpu_type=None, gpus=None, priority=None, preemptible=None, max_retries=None, max_duration=None)[source]

Submits a new run with the same configuration as this run

Parameters

name (str) – Override the name of the run
image (str) – Override the image of the run
cluster (str) – Override the cluster of the run
instance (str) – Override the instance of the run
nodes (int) – Override the number of nodes of the run
gpu_type (str) – Override the GPU type of the run
gpus (int) – Override the number of GPUs of the run
priority (str) – Override the default priority of the run from auto to low or lowest
preemptible (bool) – Override whether the run can be stopped and re-queued by higher priority jobs
max_retries (int) – Override the max number of times the run can be retried
max_duration (float) – Override the max duration (in hours) that a run can run for

Returns

New :class:`~mcli.api.model.run.Run` object

refresh()[source]

Refreshes the data on the run object

Returns: Refreshed :class:`~mcli.api.model.run.Run` object

stop()[source]

Stops the run

Returns: Stopped :class:`~mcli.api.model.run.Run` object

delete()[source]

Deletes the run

Returns: Deleted :class:`~mcli.api.model.run.Run` object

update(preemptible=None, priority=None, max_retries=None, retry_on_system_failure=None, max_duration=None)[source]

Updates the run’s data

Parameters

preemptible (bool) – Update whether the run can be stopped and re-queued by higher priority jobs; default is False
priority (str) – Update the default priority of the run from auto to low or lowest
max_retries (int) – Update the max number of times the run can be retried; default is 0
retry_on_system_failure (bool) – Update whether the run should be retried on system failure (i.e. a node failure); default is False

Returns

Updated :class:`~mcli.api.model.run.Run` object

update_metadata(metadata)[source]

Updates the run’s metadata

Parameters: metadata (Dict[str, Any]) – The metadata to update the run with. This will be merged with the existing metadata. Keys not specified in this dictionary will not be modified.
Returns: Updated :class:`~mcli.api.model.run.Run` object

Observing a run#

Getting a run’s logs#

There are two functions for fetching run logs:

get_run_logs(): Gets currently available logs for any run. Ideal for completed runs or checking progress of an active run
follow_run_logs(): Follows logs line-by-line for any run. Ideal for monitoring active runs in real time or a condition is reached (see also wait_for_run_status())

mcli.api.runs.get_run_logs(run, rank=None, *, node_rank=None, local_gpu_rank=None, global_gpu_rank=None, timeout=None, future=False, failed=False, resumption=None, tail=None, container=None)[source]

Get the current logs for an active or completed run

Get the current logs for an active or completed run in the MosaicML platform. This returns the full logs as a str, as they exist at the time the request is made. If you want to follow the logs for an active run line-by-line, use follow_run_logs().

Parameters

run (str | Run) – The run to get logs for. If a name is provided, the remaining required run details will be queried with get_runs().
rank (Optional[int]) – [DEPRECATED, Use node_rank instead] Node rank of a run to get logs for. Defaults to the lowest available rank. This will usually be rank 0 unless something has gone wrong.
node_rank (Optional[int]) – Specifies the node rank within a multi-node run to fetch logs for. Defaults to lowest available rank. Indexing starts from 0.
local_gpu_rank (Optional[int]) – Specifies the GPU rank on the specified node to fetch logs for. Cannot be used with global_gpu_rank. Indexing starts from 0. Note: GPU rank logs are only available for runs using Composer and/or LLM Foundry and MAIN container logs.
global_gpu_rank (Optional[int]) –
Specifies the global GPU rank to fetch logs for. Cannot be used with node_rank and local_gpu_rank. Indexing starts from 0. Note: GPU rank logs are only available for runs using Composer and/or LLM Foundry and MAIN container logs.
timeout (Optional[float]) – Time, in seconds, in which the call should complete. If the the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future (bool) – Return the output as a Future . If True, the call to get_run_logs() will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the log text, use return_value.result() with an optional timeout argument.
failed (bool) – Return the logs of the first failed rank for the provided resumption if True. False by default.
resumption (Optional[int]) – Resumption (0-indexed) of a run to get logs for. Defaults to the last resumption
tail (Optional[int]) – Number of chars to read from the end of the log. Defaults to reading the entire log.
container (Optional[str]) – Container name of a run to get logs for. Defaults to the MAIN container.

Returns

If future is False – The full log text for a run at the time of the request as a str
Otherwise – A Future for the log text

mcli.api.runs.follow_run_logs(run, rank=None, *, node_rank=None, local_gpu_rank=None, global_gpu_rank=None, timeout=None, future=False, resumption=None, tail=None, container=None)[source]

Follow the logs for an active or completed run in the MosaicML platform

This returns a generator of individual log lines, line-by-line, and will wait until new lines are produced if the run is still active.

Parameters

run (str | Run) – The run to get logs for. If a name is provided, the remaining required run details will be queried with get_runs().
rank (Optional[int]) – Node rank of a run to get logs for. Defaults to the lowest available rank. This will usually be rank 0 unless something has gone wrong.
timeout (Optional[float]) – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored. A run may take some time to generate logs, so you likely do not want to set a timeout.
future (bool) – Return the output as a Future . If True, the call to follow_run_logs() will return immediately and the request will be processed in the background. The generator returned by the ~concurrent.futures.Future will yield a ~concurrent.futures.Future for each new log string returned from the cloud. This takes precedence over the timeout argument. To get the generator, use return_value.result() with an optional timeout argument and log_future.result() for each new log string.
resumption (Optional[int]) – Resumption (0-indexed) of a run to get logs for. Defaults to the last resumption
tail (Optional[int]) – Number of chars to read from the end of the log. Defaults to reading the entire log.
container (Optional[str]) – Container name of a run to get logs for. Defaults to the MAIN container.

Returns

If future is False – A line-by-line Generator of the logs for a run
Otherwise – A Future of a line-by-line generator of the logs for a run

Monitoring a run throughout its lifecycle#

mcli.api.runs.wait_for_run_status(run, status, timeout=None, future=False)[source]

Wait for a launched run to reach a specific status

Parameters

run (str | Run) – The run whose status should be watched. This can be provided using the run’s name or an existing Run object.
status (str | RunStatus) – Status to wait for. This can be any valid RunStatus value. If the status is short-lived, or the run terminates, it is possible the run will reach a LATER status than the one requested. If the run never reaches this state (e.g. it stops early or the wait times out), then an error will be raised. See exception details below.
timeout (Optional[float]) – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future (bool) – Return the output as a Future. If True, the call to wait_for_run_status() will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the Run output, use return_value.result() with an optional timeout argument.

Raises

MAPIException – Raised if the run does not exist or there is an issue connecting to the MAPI service.
RunStatusNotReached – Raised in the event that the watch closes before the run reaches the desired status. If this happens, the connection to MAPI may have dropped, so try again.
TimeoutError – Raised if the run did not reach the correct status in the specified time

Returns

If future is False – A Run object once it has reached the requested status

Otherwise:: A Future for the run. This will not resolve until the run reaches the requested status

The `RunStatus` object#

The RunStatus object is attached to each Run object and reflects the most recent status the run has been observed with.

class mcli.api.runs.RunStatus(value)[source]

Possible statuses of a run

PENDING = 'PENDING': The run has been submitted and is waiting to be scheduled

QUEUED = 'QUEUED': The run is awaiting execution

STARTING = 'STARTING': The run is starting up and preparing to run

RUNNING = 'RUNNING': The run is actively running

TERMINATING = 'TERMINATING': The run is in the process of being terminated

COMPLETED = 'COMPLETED': The run has finished without any errors

STOPPED = 'STOPPED': The run has stopped

FAILED = 'FAILED': The run has failed due to an issue at runtime

UNKNOWN = 'UNKNOWN': A valid run status cannot be found

before(other, inclusive=False)[source]

Returns True if this state usually comes “before” the other

Parameters

other – Another RunStatus
inclusive – If True, equality evaluates to True. Default False.

Returns

If this state is “before” the other

Example

>>> RunStatus.RUNNING.before(RunStatus.COMPLETED)
True
>>> RunStatus.PENDING.before(RunStatus.RUNNING)
True

after(other, inclusive=False)[source]

Returns True if this state usually comes “after” the other

Parameters

other – Another RunStatus
inclusive – If True, equality evaluates to True. Default False.

Returns

If this state is “after” the other

Example

>>> RunStatus.COMPLETED.after(RunStatus.RUNNING)
True
>>> RunStatus.RUNNING.after(RunStatus.PENDING)
True

is_terminal()[source]

Returns True if this state is terminal

Returns: If this state is terminal

Example

>>> RunStatus.RUNNING.is_terminal()
False
>>> RunStatus.COMPLETED.is_terminal()
True

classmethod from_string(run_status)[source]

Convert a string to a valid RunStatus Enum

If the run status string is not recognized, will return RunStatus.UNKNOWN instead of raising a KeyError

Listing runs#

All runs that you have launched and have not deleted can be accessed using the get_runs() function. Optional filters allow you to specify a subset of runs to list by name, cluster, gpu type, gpu number, or status.

mcli.api.runs.get_runs(runs=None, *, cluster_names=None, before=None, after=None, gpu_types=None, gpu_nums=None, statuses=None, timeout=10, future=False, user_emails=None, run_types=None, include_details=False, include_deleted=False, ended_before=None, ended_after=None, limit=100)[source]

List runs that have been launched in the MosaicML platform

The returned list will contain all of the details stored about the requested runs.

Parameters

runs – List of runs on which to get information
cluster_names – List of cluster names to filter runs. This can be a list of str or :type Cluster: objects. Only runs submitted to these clusters will be returned.
before – Only runs created strictly before this time will be returned. This can be a str in ISO 8601 format(e.g 2023-03-31T12:23:04.34+05:30) or a datetime object.
after – Only runs created at or after this time will be returned. This can be a str in ISO 8601 format(e.g 2023-03-31T12:23:04.34+05:30) or a datetime object.
gpu_types – List of gpu types to filter runs. This can be a list of str or :type GPUType: enums. Only runs scheduled on these GPUs will be returned.
gpu_nums – List of gpu counts to filter runs. Only runs scheduled on this number of GPUs will be returned.
statuses – List of run statuses to filter runs. This can be a list of str or :type RunStatus: enums. Only runs currently in these phases will be returned.
user_emails – List of user emails to filter runs. Only runs submitted by these users will be returned. By default, will return runs submitted by the current user. Requires shared runs or admin permission
run_types – List of run types to filter runs - ‘INTERACTIVE’: Runs created with the mcli interactive command - ‘HERO_RUN’: Runs created with is_hero_run in the metadata - ‘TRAINING’: All other runs
timeout – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future – Return the output as a Future. If True, the call to get_runs will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the list of runs, use return_value.result() with an optional timeout argument.
include_details – If true, will fetch detailed information like run input for each run.
include_deleted – If true, will include deleted runs in the response.
ended_before – Only runs ended strictly before this time will be returned.
ended_after – Only runs ended at or after this time will be returned.
limit – Maximum number of runs to return. If None, the latest 100 runs will be returned.

Raises

MAPIException – If connecting to MAPI, raised when a MAPI communication error occurs

Updating runs#

mcli.api.runs.update_run(run, update_run_data=None, *, preemptible=None, priority=None, max_retries=None, retry_on_system_failure=None, timeout=10, future=False, max_duration=None)[source]

Update a run’s data in the MosaicML platform.

Any values that are not specified will not be modified.

Parameters

run (Optional[str | ``:class:`~mcli.api.model.run.Run` ``]) – A run or run name to update. Using Run objects is most efficient. See the note below.
update_run_data (Dict[str, Any]) – DEPRECATED: Use the individual named-arguments instead. The data to update the run with. This can include preemptible, priority, maxRetries, and retryOnSystemFailure
preemptible (bool) – Update whether the run can be stopped and re-queued by higher priority jobs; default is False
priority (str) – Update the default priority of the run from auto to low or lowest
max_retries (int) – Update the max number of times the run can be retried; default is 0
retry_on_system_failure (bool) – Update whether the run should be retried on system failure (i.e. a node failure); default is False
timeout (Optional[float]) – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
max_duration – Update the max time that a run can run for (in hours).
future (bool) – Return the output as a Future. If True, the call to update_run() will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the list of Run output, use return_value.result() with an optional timeout argument.

Raises

MAPIException – Raised if updating the requested run failed

Returns

If future is False – Updated Run object
Otherwise – A Future for the list

Stopping runs#

mcli.api.runs.stop_run(run, *, reason=None, timeout=10, future=False)[source]

Stop a run

Stop a run currently running in the MosaicML platform.

Parameters

run (Optional[str | ``:class:`~mcli.api.model.run.Run` ``]) – A run or run name to stop. Using Run objects is most efficient. See the note below.
reason (Optional[str]) – A reason for stopping the run
timeout (Optional[float]) – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future (bool) – Return the output as a Future. If True, the call to stop_run() will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the list of Run output, use return_value.result() with an optional timeout argument.

Raises

MAPIException – Raised if stopping the requested runs failed A successfully stopped run will have the status `RunStatus.STOPPED`

Returns

If future is False – Stopped Run object
Otherwise – A Future for the object

mcli.api.runs.stop_runs(runs, *, reason=None, timeout=10, future=False)[source]

Stop a list of runs

Stop a list of runs currently running in the MosaicML platform.

Parameters

runs (Optional[List[str] | List[Run ]]) – A list of runs or run names to stop. Using Run objects is most efficient. See the note below.
reason (Optional[str]) – A reason for stopping the run
timeout (Optional[float]) – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future (bool) – Return the output as a Future. If True, the call to stop_runs() will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the list of Run output, use return_value.result() with an optional timeout argument.

Raises

MAPIException – Raised if stopping any of the requested runs failed. All successfully stopped runs will have the status `RunStatus.STOPPED`. You can freely retry any stopped and unstopped runs if this error is raised due to a connection issue.

Returns

If future is False – A list of stopped Run objects
Otherwise – A Future for the list

Deleting runs#

To delete runs, you must supply the run names or Run object. To delete a set of runs, you can use the output of get_runs() or even define your own filters directly:

# delete a run by name
delete_run('delete-this-run')

# delete failed runs on cluster xyz using 1 or 2 GPUs
failed_runs = get_runs(statuses=['FAILED'], cluster_names=['xyz'], gpu_nums=[1, 2])
delete_runs(failed_runs)

# delete completed runs older than a month with name pattern
completed = get_runs(statuses=['COMPLETED'])
ref_date = dt.datetime.now() - dt.timedelta(days=30)
old_runs = [r for r in completed if 'experiment1' in r.name and r.created_at < ref_date ]
delete_runs(old_runs)

mcli.api.runs.delete_run(run, *, timeout=10, future=False)[source]

Delete a run in the MosaicML platform

If a run is currently running, it will first be stopped.

Parameters

run – A run to delete
timeout – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future – Return the output as a Future. If True, the call to delete_run will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the :type Run: output, use return_value.result() with an optional timeout argument.

Returns

A – type Run: for the run that was deleted

mcli.api.runs.delete_runs(runs, *, timeout=10, future=False)[source]

Delete a list of runs in the MosaicML platform

Any runs that are currently running will first be stopped.

Parameters

runs – A list of runs or run names to delete
timeout – Time, in seconds, in which the call should complete. If the call takes too long, a TimeoutError will be raised. If future is True, this value will be ignored.
future – Return the output as a Future. If True, the call to delete_run will return immediately and the request will be processed in the background. This takes precedence over the timeout argument. To get the :type Run: output, use return_value.result() with an optional timeout argument.

Returns

A list of – type Run: for the runs that were deleted

Manage a run with the SDK#

Creating a run#

The RunConfig object#

The Run object#

Observing a run#

Getting a run’s logs#

Monitoring a run throughout its lifecycle#

The RunStatus object#

Listing runs#

Updating runs#

Stopping runs#

Deleting runs#

The `RunConfig` object#

The `Run` object#

The `RunStatus` object#