☁️ Train ResNet-50 on AWS#
Composer is a PyTorch library that accelerates training for deep learning models while improving quality at significantly lower cost. Composer makes it possible to train ResNet-50 on the ImageNet dataset to the standard 76.6% top-1 accuracy in 27 minutes on an AWS EC2 instance for a mere $15. In this tutorial we’ll teach you how simple it is to do this yourself!
The starting point for this tutorial are the training recipes we present in our Mosaic ResNet blog post. We’ll walk through:
Launching an AWS EC2 instance capable of running GPU training
Configuring your AWS EC2 instance to run Composer with our pre-built Docker images
Running Composer training using the ResNet-50 Mild recipe introduced in our blog post
AWS account with permissions to:
Create/manage EC2 instances, EBS volumes
Create/manage Security Groups, Key Pairs (alternatively, IT admin provided)
AWS quota to create Accelerated Computing EC2 instances
We use a p4d.24xlarge instance in this tutorial. However these steps should run on any P-type EC2 instance.
Download the latest Imagenet Dataset
Due to the challenges associated with distributing ImageNet, we assume users to provide their own version of this dataset for the purpose of this tutorial.
The image comes pre-configured with the following dependencies:
Mosaic ResNet Training recipes
Composer Version: 0.9.0
PyTorch Version: 1.11.0
CUDA Version: 11.3
Python Version: 3.9
Ubuntu Version: 20.04
Launching an AWS EC2 Instance#
First let’s create an EC2 instance that we can run GPU training on.
Login to your AWS account and open the Management Console
For the purposes of this material, we will configure and launch a new
p4d.24xlargeinstance. On your
EC2 Dashboardclick the
Name your instance and select an AMI, Instance type, Key pair and Network settings. The following settings were used for this tutorial, customize as required depending on your AWS setup and IT requirements:
Amazon Machine Image (AMI):
Deep Learning AMI GPU PyTorch 1.12.0 (Amazon Linux 2)
Create key pair(make sure to note where you save the private key)
Key pair name:
Key pair type:
Private key format:
Network settings: Use defaults
Configuring your AWS EC2 instance#
Next we will connect to our newly launched
p4d.24xlarge instance, perform some basic system configuration to optimize our runtime environment and setup our dataset area.
Navigate back to the
Instancespage in your AWS console. Click on the running instance you just launched and in the Details pane, copy the instance’s
Public IPv4 DNSaddress. You will need this value to connect to the instance.
Using the private key you downloaded during the launch configuration and the instance’s public DNS address, connect to the system using SSH:
ssh -i <path_to_private_key> [email protected]<public_dns_address>
ssh -i ~/composer_demo.pem [email protected]
Now let’s create a
datasetsarea to place the ImageNet data as follows:
sudo mkdir -p /datasets/ImageNet sudo chmod -R 777 /datasets
(Optional )If the EC2 instance you selected comes direct attached Instance Store Volumes, it can be mounted as follows:
sudo mkfs -t xfs /dev/nvme1n1 sudo mkdir ImageNet sudo mount /dev/nvme1n1 /ImageNet sudo chmod 777 ImageNet/
Instance Store Volumes (ISV) generally have better performance than EBS volumes since they are directly attached to the instance, at the expense of persistence. Thus Instance Store Volumes are ephemeral and any data stored on these volumes will be inaccessible after the instance is powered off.
Regardless of whether you choose to use an EBS volume or Instance Store Volume to host your dataset, the ImageNet data can be copied to the
/datasets/Imagenetfolder. In our example, the directory tree under
/datasetslooks as follows:
[[email protected] /]$ find ./datasets/ -maxdepth 2 ./datasets/ ./datasets/imagenet_files.tar ./datasets/ImageNet ./datasets/ImageNet/train ./datasets/ImageNet/val
Once you populate the dataset area, you’ll be ready to start training!
Train ResNet-50 on ImageNet#
Now that we have launched an EC2 instance, configured the runtime and populated the dataset area, we are ready to kick off training.
Pull and run the
mosaicml/pytorch_vision:resnet50_recipesDocker image. The image contains everything required to train including: pre-installed Composer, package dependencies, training entrypoint and recipe configuration files.
docker run -it -v /datasets:/datasets --gpus all --shm-size 1g mosaicml/pytorch_vision:resnet50_recipes
The default shared memory size of a Docker container is typically too small for larger datasets. In this example, increasing the shared memory size to 1GB is usually sufficient.
Run ResNet-50 Training using the Mild recipe!
composer train.py -f recipes/resnet50_mild.yaml --scale_schedule_ratio 0.36 \ --train_dataset.imagenet.ffcv_dir /datasets/ImageNet/ffcv \ --val_dataset.imagenet.ffcv_dir /datasets/ImageNet/ffcv
The ResNet-50 Mild and Medium recipes utilize the very efficient and high performing [FFCV dataloader](https://ffcv.io/), requiring the raw ImageNet data to be processed into FFCV format. Composer can automatically perform this step for you prior to launching the training run, simply append the following command line arguments to the training command above: –train_dataset.imagenet.datadir /datasets/ImageNet/ –val_dataset.imagenet.datadir /datasets/ImageNet/ –train_dataset.imagenet.ffcv_write_dataset –val_dataset.imagenet.ffcv_write_dataset` The first two arguments simply specify the area of the raw ImageNet training and validation data, respectively. The second two arguments enable dataset conversion if the expected FFCV formatted files do not exist.
To perform this conversion manually, please follow the instructions detailed in the [README](https://github.com/mosaicml/benchmarks/tree/main/blogs/resnet#running-a-recipe) in our [benchmarks repository](https://github.com/mosaicml/benchmarks/tree/main/blogs/resnet), which contains all the code associated with our original blog post.
We’ve performed various sweeps on AWS EC2 instances to understand the efficiency frontier across time, accuracy and cost as shown below.
The recipe explored in this tutorial should result in a model trained to a Top-1 accuracy of 76.6% in about 27 minutes for a total cost of $14.77.
You can explore the results of our other ResNet-50 runs on AWS in Explorer, our tool for exploring efficiency frontiers for different models and datasets with different speed-up techniques across various clouds.
Check out our GitHub repository for the latest information on Composer
Check out Composer + FFCV: Faster Together blog post for more information on how FFCV and Composer work together
Reproduce our record setting MLPerf ResNet-50 benchmark! Note, you will require access to the
p4de.24xlarge(in preview) EC2 instances which contain the Nvidia A100 80GB GPUs. Please see the MLPerf Training Results v2.0 GitHub Repository for additional details.
Try training on your models and datasets using Composer!