Databricks

To set up your Databricks credentials within the MCloud environment, use the databricks secret:

mcli create secret databricks

This secret requires providing your workspace URL and Databricks Personal Access Token (PAT) as host and token, respectively.

host: Your Databricks workspace URL—more information on workspace URLs can be found in this documentation.

token: Generate a Databricks PAT following the instructions here.

Instead of using the interactive prompt above, all the settings can also be provided as flags in one command:

mcli create secret databricks --host my-host --token my-token

Testing your Credentials

To test your Databricks credentials, create a databricks_secret_check.yaml with the following run config:

name: databricks-secret-check
image: python
compute:
  gpus: 0
integrations:
  - integration_type: pip_packages
    packages:
      - databricks-sdk==0.28.0
command: |
  cat << EOF > /tmp/test.py
  from databricks.sdk import WorkspaceClient
  w = WorkspaceClient()
  print(w.current_user.me())
  EOF
  python /tmp/test.py
  echo "Done"

mcli run -f databricks_secret_check.yaml --follow

If all goes well, you should see a Databricks SDK User object and Done printed at the bottom of the output.

# pip installation output
# ...
User(<fields>)
Done