Set up and use the the deployment scripts locally¶
While deploys generally go through our GitHub Actions workflow, sometimes you need to deploy from your laptop - primarily when testing changes on staging or actively fixing an outage. This isn’t ideal, but this is where we are now.
Our deployment scripts live in the
of this repository, and can deploy one or more hubs to our clusters.
Setting up local environment¶
Create a virtual environment for use with this repository
python3 -m venv . source bin/activate
You can also use
condaif you prefer instead.
Install python packages required by our deployment script
pip install -r requirements.txt
Install the Google Cloud SDK so our scripts can authenticate to Google Cloud Platform for access to our secret keys. Note that we use Google Cloud Platform (with
sopssecure secrets stored in this repository - so you would need the google cloud sdk regardless of the final location of the hub itself.
Authenticate to Google Cloud Platform! First, you should run
gcloud auth loginand follow the prompts - this authenticates your
gcloudcommandline tool. Next, you should run
gcloud auth application-default login, and follow the prompts - this authenticates you to any tools that want to authenticate to Google Cloud on your behalf - including our deploy scripts! See Application Default Credentials for more information.
sopsfor secret management. We use
sopswith Google KMS key to automatically encrypt/decrypt files when needed. So access to the KMS key grants access to the secrets. Setting up the application default login in step 4 means sops can use that to decrypt the secrets when necessary. The key currently in use is in the
two-eye-two-seeGCP project, so you must already have access to it to decrypt the files
You can test it by running
sops config/secrets.yaml, and checking if your
$EDITORpops up with the decrypted contents of the secret file.
Doing a deploy¶
Make the config change you want to deploy.
Deploy just a single hub:
python3 deployer deploy <cluster-name> <hub-name>
The script will look for a hub named
<hub-name>in the cluster config defined at
You can deploy to all hubs on a given cluster by omitting the hub name.
python3 deployer deploy <cluster-name>
You should mostly use the
staging hub in the
2i2c cluster for testing.
Hub health test¶
After each deployment, the deployer will run health checks on the hub before proceeding. If an entire cluster is being deployed, one failing health check will stop all further deployments.
There are two types of test notebooks are ran on the hub, based on it’s type:
dask specific notebooks - are ran on the the daskhub hubs
simple notebooks - are ran on all the other types of hubs
This test, creates a new hub user called
deployment-service-check, starts a
server for them, and runs the test notebooks. It checks that the notebook
runs to completion and the output cells have the expected value.
If the health check passes, then the user’s server is stopped and deleted.
If the health check fails, then their user server will be left running, but
it will get deleted together with the user in the next iteration.