CoreWeave Object Storage#

CoreWeave uses an s3-compatabile storage system. This allows developers to CRUD from their CoreWeave blob stores with AWS CLIs and SDKs. In practice, a CoreWeave integration feels very much like an s3 integration.

First, follow these instructions to create a token configuration file. The file should look like this:

[default]
access_key = <your_coreweave_access_key>
secret_key = <your_coreweave_secret_key>
# The region for the host_bucket and host_base must be the same.
host_base = object.lga1.coreweave.com # instead of lga1 it could also be ord1 or las1.
host_bucket = %(bucket)s.object.lga1.coreweave.com
check_ssl_certificate = True
check_ssl_hostname = True

Take your access_key and secret_key from your token configuration file to create a credentials file in ~/.coreweave/credentials. The credentials file should look like this:

[default]
aws_access_key_id=<your_coreweave_access_key>
aws_secret_access_key=<your_coreweave_secret_key>

Create an empty config file in ~/.coreweave/config as:

[default]

Next, create an environment variable for the endpoint url using the host_base from your CoreWeave token configuration file:

mcli create secret env S3_ENDPOINT_URL='https://object.lga1.coreweave.com' #  insead of lga1 it could also be ord1 or las1.

Now we can treat these credentials as if they are for aws s3. Run the following command:

> mcli create secret s3
? What would you like to name this secret? my-coreweave-credentials
? Where is your S3 config file located? ~/.coreweave/config
? Where is your S3 credentials file located? ~/.coreweave/credentials
✔  Created secret: my-coreweave-credentials

The values for each of these queries can be also passed as arguments using the --name, --config-file and --credentials-file arguments, respectively.

Once you’ve created an S3 secret, we mount these secrets inside all of your runs and export two environment variables:

$AWS_CONFIG_FILE: Path to your config file
$AWS_SHARED_CREDENTIALS_FILE: Path to your credentials file

A library like boto3 uses these environment variables by default to discover your s3 credentials:

import boto3
import os

# boto3 automatically pulls from $AWS_CONFIG_FILE and $AWS_SHARED_CREDENTIALS_FILE
s3 = boto3.client('s3', endpoint_url=os.environ['S3_ENDPOINT_URL'])

🙌