GraceDB and Hopskotch deployment

Helm repositories

The Helm charts for installing GraceDB and the Hopskotch server are defined in this repository. Default values of both charts allow for a sandboxed deployment on minikube.

The first step is to add the necessary repositories. To add the GraceDB repository:

user> helm repo add --username <username> --password <token> --force-update \
    gracedb-helm \
    https://git.ligo.org/api/v4/projects/15655/packages/helm/stable

Where:

• <username> is your marie.curie username

• <token> is a read_api scoped personal access token.

The Traefik proxy is a dependency of the GraceDB chart.

Attention

Traefik is already installed on the LDAS fluxuser machines, so it's not necessary to add the corresponding repository. It's also installed by default in K3s, unless you specify differently. To check if Treafik is installed in your cluster:

user> kubectl get pods -A | grep traefik
kube-system   helm-install-traefik-5jdcx                         0/1     Completed   1                59d
kube-system   helm-install-traefik-crd-zkqcn                     0/1     Completed   0                59d
kube-system   svclb-traefik-30234bcf-2n9cg                       2/2     Running     0                59d
kube-system   traefik-d7c9c5778-p4fff                            1/1     Running     0                59d

If Traefik is not already installed in your Kubernetes cluster, add its Helm repository:

user> helm repo add traefik https://traefik.github.io/charts

Chart installation

Hopskotch server

Run the command below to install the Hopskotch Helm chart. The chart automatically creates all the necessary Kafka topics via a Kubernetes Job.

K3sminikube

user> helm  upgrade --install --set storageClassName=local-path \
    hopskotch gracedb-helm/hopskotch 

user> helm  upgrade --install \
   hopskotch gracedb-helm/hopskotch 

To check that Pods are properly running execute the command:

user> kubectl get pods 
NAME                                               READY   STATUS    RESTARTS        AGE
 hopskotch-6898b47c8f-487pd                        1/1     Running   0               1m
 hopskotch-create-topics-bnlcw                     1/1     Running   0               1m

Wait for the Pod that creates topics to complete before proceeding with the following steps.

Tip

You can enter the Hopskotch Pod and execute kcat commands to inspect available topics:

user> kubectl exec -it <hopskotch-pod-name> -- bash

For example, to verify that all topics have been created, from inside the Pod run:

[root@hopskotch tmp]# kcat -b localhost -L

or you can listen to a specific topic:

[root@hopskotch tmp]# kcat -b localhost -t default.burst_cwb_allsky

GraceDB

Run the command below to install the GraceDB Helm chart. This step will also install an istance of Memcached and cert-manager as dependencies.

K3sminikube

Attention

The step below is slightly different in the case of the LDAS fluxuser machines.

On your local K3s instance run:

user> helm  upgrade --install \
    --set storageClassName=local-path \
    --set traefik.service.spec.clusterIP="" \
    gracedb gracedb-helm/gracedb

if Traefik is already installed in your cluster, add the flag --set traefik.install=false to the command above.

On the LDAS fluxuser machines run:

user> helm  upgrade --install \
    --set storageClassName=local-path \
    --set traefik.install=false \
    --set publicName="gracedb-dev<X>.ldas.cit" \
    gracedb gracedb-helm/gracedb

where X is the ID of your personal VM.

user> helm  upgrade --install \
    gracedb gracedb-helm/gracedb

To check that Pods are properly running execute the command:

user> kubectl get pods 
NAME                                               READY   STATUS    RESTARTS        AGE
gracedb-0                                          1/1     Running   0               1m
gracedb-cert-manager-7b4f8687f8-ct9s9              1/1     Running   0               1m
gracedb-cert-manager-cainjector-7cb7f656db-7l9bq   1/1     Running   0               1m
gracedb-cert-manager-webhook-6d57cc7bb6-dh7xq      1/1     Running   0               1m
gracedb-memcached-5655cd7dd-tpm6z                  1/1     Running   0               1m
gracedb-postgres-0                                 1/1     Running   0               1m
hopskotch-6898b47c8f-w6v9m                         1/1     Running   0               1m

Run your own GraceDB image

The GraceDB version to be used can be specified adding the following flag to the installation command:

--set gracedb.image="containers.ligo.org/computing/gracedb/server:gracedb-x.x.x"

You can also specify your custom image, either hosted on a public repository or local.

Tip

In order to make your local image available to the minikube environment, please refer to this documentation. For instance, you could build it directly into the minikube cluster by using the following command:

user> minikube image build -t "gracedb-custom/development:mytag"

Access the local GraceDB instance

Warning

If you get a Bad Gateway (502) error when connecting to your GraceDB instance, make sure that Firewalld is turned off:

admin> systemctl disable firewalld --now

The GraceDB endpoint will be made available on the local network. If you are working on a remote machine, it might be convenient to configure a SOCKS proxy to access the URL from your local browser, avoiding X11 forwarding.

Tip

To configure a SOCKS proxy to access a GraceDB instance running on the LDAS fluxuser machines:

• open a tunnel to one of the public login machines: ssh -D 8080 marie.curie@ldas-pcdev14.ligo.caltech.edu
• on Firefox go to: Preferences -> Network Settings and configure a manual proxy configuration with 127.0.0.1 as SOCKS host and 8080 as port. If Firefox is not your primary browser, we recommend using it as a secondary option with a SOCKS proxy always enabled. This way, you won't need to adjust the configuration every time you want to access a remote host through your browser.

GraceDB is now available at the following URL:

https://gracedb-dev<X>.ldas.cit

as specified with the publicName variable during the installation prcedure.

If you are working on your own resources, the GraceDB instance will have the following default name: gracedb.default.svc.cluster.local.

Since this name is resolved only by the Kubernetes internal DNS, you need to make it available on your local machine by adding the following line to the /etc/hosts file:

127.0.0.1 gracedb gracedb.default.svc.cluster.local

Additionally, only in the minikube case, run in a separate terminal:

minikube

user> minikube tunnel

Your personal GraceDB instance shoud now be available at:

https://gracedb.default.svc.cluster.local

If you are running on a remote machine, you can configure a SOCKS proxy as explained above, opening the tunnel towards your remote machine.

Get the custom Certification Authority certificate

Your sandboxed GraceDB deployment uses a self-signed server certificate automatically created by cert-manager during the installation. The Certification Authirity (CA) certificate used to sign the server certificate can be retrieved in the following way:

Attention

The script requires python3 with the certifi package installed. On the LDAS fluxuser machines you can enable Python 3 via Conda as explained in the Preliminary steps paragraph.

user> ./utility/get-gracedb-ca.sh default

This will create two new files in the local directory:

• ca.crt is the CA certificate to be imported in your web browser (see these instructions for Firefox).
• cacerts.pem is the CA bundle to be used by the requests Python package when querying the GraceDB API. This can be achieved by exporting the following environment variable: REQUESTS_CA_BUNDLE=cacerts.pem.

User account configuration

In order to add your LIGO user as admin user of the newly created GraceDB instance, run the following script:

user> ALL_PERMS=True ./utility/add-account.sh <marie.curie>

where marie.curie is your LIGO username, you will be prompted for your user account password.

In order to grant full privileges to your user on the GraceDB instance, access the admin interface at the URL:

https://gracedb.default.svc.cluster.local/admin/ 

or:

https://gracedb-dev<X>.ldas.cit/admin

Where the default login credentials are:

• username: admin
• password: mypassword.

Navigate to AUTHENTICATION AND AUTHORIZATION->Users and search for your marie.curie@ligo.org account. In the Permissions section, choose all the available groups and permissions and Save.

Testing

To test the installation you can generate some event using the Mock Event Generator (MEG). Instructions to deploy MEG on your Kubernetes cluster can be found here.

To inspect the GraceDB logs while creating events, use the following command:

user> kubectl logs -f gracedb-0 | grep -v heartbeat

Uninstall

To uninstall the Helm charts, run:

user> helm uninstall hopskotch
user> helm uninstall gracedb

To manually delete the certificate Secrets, run:

user> kubectl delete secrets gracedb-ca gracedb-cert-manager-webhook-ca gracedb-cert-tls

To manually delete persistent volumes (this step will completely reset the database), run:

user> kubectl delete pvc db-data-gracedb-0 postgres-persistent-storage-gracedb-postgres-0