EXSUM: Ray on Minikube
I was asked about how to setup Kubernetes and deploy Ray on it. The last time I did this was before this was officially supported – I just manually built a container with Ray and deployed it as master on K8s master and workers on K8s workers. However, over the last 3 years or so a lot has happened and we can simply use off-the-shelf Helm charts now. Here is a minimal example. Since I like the idea of having online IDEs I have also included a command to run Eclipse Che. I should write a dedicated post about why online IDEs that natively support container development are compelling.
Minikube
Setup kubectl, see docs:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(<kubectl.sha256) kubectl" | sha256sum --check
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
Install Helm, see docs:
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
Setup minikube, see docs:
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start --addons=ingress --memory=8192
kubectl version
Note: It is very likely that just minikube start would not work.
Minikube with GPU Acceleration
Setup minikube as above.
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start --addons=ingress --memory=8192
kubectl version
kubectl get pod
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/nvidia-device-plugin.yml
Ray
See docs.
Deploy
cd ${RAY_DIR}/deploy/charts
helm -n ray install example-cluster --create-namespace ./ray
Run Test Job
$ kubectl -n ray get pods
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-type-lsbwn 1/1 Running 0 28m
example-cluster-ray-worker-type-bltlj 1/1 Running 0 22m
example-cluster-ray-worker-type-srwtl 1/1 Running 0 28m
ray-test-job-5gwx8 0/1 Completed 0 22m
Now run test job directly on head:
$ kubectl -n ray cp ray/doc/kubernetes/example_scripts/run_on_head.py example-cluster-ray-head-type-lsbwn:/home/ray
$ kubectl -n ray exec example-cluster-ray-head-type-lsbwn -- python /home/ray/run_on_head.py
Addendum: IBM-Specific Ray Deployments
There is an article on how to use Ray with CodeEngine. There was also a demo which combines OpenDataHub and Ray about 2 years ago. Also, CodeFlare leverages Ray and is maybe the most stable way to use Ray in an IBM-affine way. [Not speaking for IBM, of course.]
Update: KubeRay for Ray 2.0+
With the release of Ray 2.0 the preferred way to run Ray applications on K8s is KubeRay.
Optional Addendum: Istio, KNative, Che and Camel K
While they have nothing to do with Ray, additional projects like Istio, KNative, Che and Camel K are oftentimes useful when working with a Ray K8s deployment. I will briefly describe how to set them up as well.
Eclipse Che
Install Che into Minikube:
bash <(curl -sL https://www.eclipse.org/che/chectl/)
chectl --version
chectl server:deploy --platform minikube
To update chectl use chectl update.
Set certificate
cp /tmp/cheCA.crt ~/Downloads/
Chrome -> chrome://settings/security -> Advanced -> Privacy and security -> Manage Certificates -> Authorities Import cheCA.crt, “Trust this certificate for identifying websites”, OK, reload.
Go to https://che-eclipse-che.192.168.49.2.nip.io login with admin/admin, change password.
Istio
sudo mkdir -p /opt/istio && cd $_
sudo chown -R ${USER} .
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.9.3
export PATH=$PWD/bin:$PATH
istioctl install
KNative
KNative requires Istio.
kubectl apply -f https://github.com/knative/operator/releases/download/v0.22.1/operator.yaml
kubectl logs -f deploy/knative-operator
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: knative-serving
---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
name: knative-serving
namespace: knative-serving
EOF
watch -n 5 kubectl get deployment -n knative-serving
kubectl get KnativeServing knative-serving -n knative-serving
Camel K
cd ~/Downloads/
wget https://github.com/apache/camel-k/releases/download/v1.3.2/camel-k-client-1.3.2-linux-64bit.tar.gz
mkdir ~/Downloads/tmp_camel_k && tar -xf camel-k-client-1.3.2-linux-64bit.tar.gz -C ~/Downloads/tmp_camel_k
sudo cp ~/Downloads/tmp_camel_k/kamel /usr/local/bin/
kamel version
rm -rf ~/Downloads/tmp_camel_k/
cd -
kubectl create namespace camel-basic
kubectl config set-context --current --namespace=camel-basic
minikube addons enable registry
kamel install
Leave a Reply
Want to join the discussion?Feel free to contribute!