{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7300096f-f240-4102-be8a-07ee60bd390f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **Overview**\n",
    "\n",
    "This notebook demonstrates the process of deploying a **VLLM (Very Large Language Model)** using **Ray Serve**. Follow these steps to set up and deploy your VLLM model:\n",
    "\n",
    "1. **Prepare the Application**: Ensure your Ray Serve application is correctly set up in the `ray_serve_vllm_example.py` file. This includes defining the VLLM model and making sure it is properly referenced during creation.\n",
    "\n",
    "2. **Build the Ray Serve Application**: Use the `serve build` command to generate a configuration file for your deployment. This configuration file will specify how your VLLM application should be deployed.\n",
    "\n",
    "3. **Configuring Runtime Options**: Two actions need to be taken here:\n",
    "    - Due to current limitations with Ray Serve’s `--working-dir` option, we use a workaround to upload the working directory to Google Cloud Storage (GCS) and include it in the deployment configuration.\n",
    "    - `vllm` package needs to be installed under `pip:packages`. Please refer to Readme file to see the details.\n",
    "\n",
    "4. **Deploy the Model**: Deploy the VLLM model using the generated configuration file. This step will start the deployment process and make your model available for serving.\n",
    "\n",
    "5. **Send Sample Requests**: After deployment, send sample requests to the deployed VLLM model to test its functionality and ensure it is working as expected.\n",
    "\n",
    "6. **Shutdown the Deployment**: Once you are done with testing, terminate the deployment to free up resources.\n",
    "\n",
    "**Note**: This notebook is tailored for deploying VLLM models and includes steps specific to such deployments using Ray Serve. Follow each step carefully and check the outputs for any warnings or errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9e60cc86-a3e1-4c58-937b-9d231d3a8438",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING 08-01 17:53:02 _custom_ops.py:14] Failed to import from vllm._C with ImportError('libcuda.so.1: cannot open shared object file: No such file or directory')\n",
      "2024-08-01 17:53:02,482\tINFO scripts.py:848 -- The auto-generated application names default to `app1`, `app2`, ... etc. Rename as necessary.\n",
      "\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Building Ray Serve app\n",
    "# !serve build <module_name>:<app_name> -o <config_file_name>.yaml\n",
    "# This will generate config file\n",
    "!serve build --app-dir \"./\" ray_serve_vllm_example:ray_serve_vllm_deployment -o ray_vllm_deployment_config.yaml\n",
    "#Ignore Failed to import WARNING"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c3ebe7-9635-4c67-9b14-888e2a53dac4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Attention!\n",
    "Following cell is a workaround. Currently, serve deploy does not support --working-dir directly. Please see https://github.com/ray-project/ray/issues/29354\n",
    "\n",
    "Suggested way to provide files from NB side to Ray cluster as below:\n",
    "\n",
    "Create connection with JobSubmissionClient with working dir option but without entrypoint.\n",
    "JobSubmissionClient will upload working_dir to GCS and print the URI.\n",
    "Specify the above mentioned URI in config file as below example:\n",
    "\n",
    "    runtime_env:\n",
    "        working_dir: \"gcs://_ray_pkg_fef565b457f470d9.zip\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "d2e97496-6a04-4303-b4e6-ae2126675730",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-10-16 13:55:17,346\tINFO dashboard_sdk.py:338 -- Uploading package gcs://_ray_pkg_db274df5fea1cead.zip.\n",
      "2024-10-16 13:55:17,347\tINFO packaging.py:530 -- Creating a file package for local directory './'.\n"
     ]
    }
   ],
   "source": [
    "# Workaround!\n",
    "# This is to upload the working dir to GCS\n",
    "# Once the URI is ready, please modify config dir before deployment\n",
    "import ray\n",
    "from ray.job_submission import JobSubmissionClient\n",
    "\n",
    "ray_head_ip = \"kuberay-head-svc.kuberay.svc.cluster.local\"\n",
    "ray_head_port = 8265\n",
    "ray_address = f\"http://{ray_head_ip}:{ray_head_port}\"\n",
    "client = JobSubmissionClient(ray_address)\n",
    "\n",
    "job_id = client.submit_job(\n",
    "    entrypoint=\"\",\n",
    "    runtime_env={\n",
    "        \"working_dir\": \"./\",\n",
    "    }\n",
    ")\n",
    "\n",
    "# We do not need this connection    \n",
    "ray.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "86ba3672-df64-4264-9a8d-80d8175eb58a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-10-16 13:55:37,079\tINFO scripts.py:243 -- Deploying from config file: 'ray_vllm_deployment_config.yaml'.\n",
      "2024-10-16 13:55:40,984\tSUCC scripts.py:350 -- \u001b[32m\n",
      "Sent deploy request successfully.\n",
      " * Use `serve status` to check applications' statuses.\n",
      " * Use `serve config` to see the current application config(s).\n",
      "\u001b[39m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!serve deploy --address \"http://kuberay-head-svc.kuberay.svc.cluster.local:8265\" ray_vllm_deployment_config.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dee6a45b-6b81-4119-89ae-595b5ddd001e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def send_sample_request():\n",
    "    import requests\n",
    "\n",
    "    prompt = \"How do I cook rice?\"\n",
    "    sample_input = {\"prompt\": prompt, \"stream\": True}\n",
    "    output = requests.post(\"http://kuberay-head-svc.kuberay.svc.cluster.local:8000/\", json=sample_input)\n",
    "    for line in output.iter_lines():\n",
    "        print(line.decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4109973d-aafe-46c0-b6e1-084865fcd5df",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"text\": \"\\n\"}\n",
      "{\"text\": \"What\"}\n",
      "{\"text\": \" kind\"}\n",
      "{\"text\": \" of\"}\n",
      "{\"text\": \" rice\"}\n",
      "{\"text\": \"?\"}\n",
      "{\"text\": \" Green\"}\n",
      "{\"text\": \" tea\"}\n",
      "{\"text\": \",\"}\n",
      "{\"text\": \" milk\"}\n",
      "{\"text\": \" rice\"}\n",
      "{\"text\": \"...\"}\n",
      "{\"text\": \"\\n\"}\n",
      "{\"text\": \"K\"}\n",
      "{\"text\": \"orean\"}\n",
      "{\"text\": \" rice\"}\n"
     ]
    }
   ],
   "source": [
    "send_sample_request()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "441e2bcf-3ef0-4e49-ad01-e1b1ed6e866b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-10-16 13:49:33,694\tSUCC scripts.py:747 -- \u001b[32mSent shutdown request; applications will be deleted asynchronously.\u001b[39m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Terminating the deployment\n",
    "!serve shutdown --address \"http://kuberay-head-svc.kuberay.svc.cluster.local:8265\" -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7309505e-02ad-41b7-afa5-ffa706b49eb3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Ray",
   "language": "python",
   "name": "ray"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}