CoreWeave Object Storage#
CoreWeave uses an s3-compatabile storage system. This allows developers to CRUD from their CoreWeave blob stores with AWS CLIs and SDKs. In practice, a CoreWeave integration feels very much like an s3 integration.
First, follow these instructions to create a token configuration file. The file should look like this:
[default] access_key = <your_coreweave_access_key> secret_key = <your_coreweave_secret_key> # The region for the host_bucket and host_base must be the same. host_base = object.lga1.coreweave.com # instead of lga1 it could also be ord1 or las1. host_bucket = %(bucket)s.object.lga1.coreweave.com check_ssl_certificate = True check_ssl_hostname = True
secret_key from your token configuration file to create a credentials file in
The credentials file should look like this:
[default] aws_access_key_id=<your_coreweave_access_key> aws_secret_access_key=<your_coreweave_secret_key>
Create an empty config file in
Next, create an environment variable for the endpoint url using the
host_base from your CoreWeave token configuration file:
mcli create secret env S3_ENDPOINT_URL='https://object.lga1.coreweave.com' # insead of lga1 it could also be ord1 or las1.
Now we can treat these credentials as if they are for aws s3. Run the following command:
> mcli create secret s3 ? What would you like to name this secret? my-coreweave-credentials ? Where is your S3 config file located? ~/.coreweave/config ? Where is your S3 credentials file located? ~/.coreweave/credentials ✔ Created secret: my-coreweave-credentials
The values for each of these queries can be also passed as arguments using the
--credentials-file arguments, respectively.
Once you’ve created an S3 secret, we mount these secrets inside all of your runs and export two environment variables:
$AWS_CONFIG_FILE: Path to your config file
$AWS_SHARED_CREDENTIALS_FILE: Path to your credentials file
A library like boto3 uses these environment variables by default to discover your s3 credentials:
import boto3 import os # boto3 automatically pulls from $AWS_CONFIG_FILE and $AWS_SHARED_CREDENTIALS_FILE s3 = boto3.client('s3', endpoint_url=os.environ['S3_ENDPOINT_URL'])