EXSUM: Ray on Minikube

I was asked about how to setup Kubernetes and deploy Ray on it. The last time I did this was before this was officially supported – I just manually built a container with Ray and deployed it as master on K8s master and workers on K8s workers. However, over the last 3 years or so a lot has happened and we can simply use off-the-shelf Helm charts now. Here is a minimal example. Since I like the idea of having online IDEs I have also included a command to run Eclipse Che. I should write a dedicated post about why online IDEs that natively support container development are compelling.

Minikube

Setup kubectl, see docs:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(<kubectl.sha256) kubectl" | sha256sum --check
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Install Helm, see docs:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Setup minikube, see docs:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start --addons=ingress --memory=8192
kubectl version

Note: It is very likely that just minikube start would not work.

Minikube with GPU Acceleration

Setup minikube as above.

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start --addons=ingress --memory=8192
kubectl version
kubectl get pod
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/nvidia-device-plugin.yml

Ray

See docs.

Deploy

cd ${RAY_DIR}/deploy/charts
helm -n ray install example-cluster --create-namespace ./ray

Run Test Job

$ kubectl -n ray get pods
NAME                                    READY   STATUS      RESTARTS   AGE
example-cluster-ray-head-type-lsbwn     1/1     Running     0          28m
example-cluster-ray-worker-type-bltlj   1/1     Running     0          22m
example-cluster-ray-worker-type-srwtl   1/1     Running     0          28m
ray-test-job-5gwx8                      0/1     Completed   0          22m

Now run test job directly on head:

$ kubectl -n ray cp ray/doc/kubernetes/example_scripts/run_on_head.py example-cluster-ray-head-type-lsbwn:/home/ray
$ kubectl -n ray exec example-cluster-ray-head-type-lsbwn -- python /home/ray/run_on_head.py

Addendum: IBM-Specific Ray Deployments

There is an article on how to use Ray with CodeEngine. There was also a demo which combines OpenDataHub and Ray about 2 years ago. Also, CodeFlare leverages Ray and is maybe the most stable way to use Ray in an IBM-affine way. [Not speaking for IBM, of course.]

Update: KubeRay for Ray 2.0+

With the release of Ray 2.0 the preferred way to run Ray applications on K8s is KubeRay.


Optional Addendum: Istio, KNative, Che and Camel K

While they have nothing to do with Ray, additional projects like Istio, KNative, Che and Camel K are oftentimes useful when working with a Ray K8s deployment. I will briefly describe how to set them up as well.

Eclipse Che

Install Che into Minikube:

bash <(curl -sL  https://www.eclipse.org/che/chectl/)
chectl --version
chectl server:deploy --platform minikube

To update chectl use chectl update.

Set certificate

Documentation.

cp /tmp/cheCA.crt ~/Downloads/

Chrome -> chrome://settings/security -> Advanced -> Privacy and security -> Manage Certificates -> Authorities Import cheCA.crt, “Trust this certificate for identifying websites”, OK, reload.

Go to https://che-eclipse-che.192.168.49.2.nip.io login with admin/admin, change password.

Istio

sudo mkdir -p /opt/istio && cd $_
sudo chown -R ${USER} .
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.9.3
export PATH=$PWD/bin:$PATH
istioctl install

KNative

KNative requires Istio.

See docs. Pick release.

kubectl apply -f https://github.com/knative/operator/releases/download/v0.22.1/operator.yaml
kubectl logs -f deploy/knative-operator
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
 name: knative-serving
---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
EOF
watch -n 5 kubectl get deployment -n knative-serving
kubectl get KnativeServing knative-serving -n knative-serving

Camel K

ReleasesExamples

cd ~/Downloads/
wget https://github.com/apache/camel-k/releases/download/v1.3.2/camel-k-client-1.3.2-linux-64bit.tar.gz
mkdir ~/Downloads/tmp_camel_k && tar -xf camel-k-client-1.3.2-linux-64bit.tar.gz -C ~/Downloads/tmp_camel_k
sudo cp ~/Downloads/tmp_camel_k/kamel /usr/local/bin/
kamel version
rm -rf ~/Downloads/tmp_camel_k/
cd -
kubectl create namespace camel-basic
kubectl config set-context --current --namespace=camel-basic
minikube addons enable registry
kamel install
0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply