{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y7wSVnqQ7xZB"
   },
   "source": [
    "# MLFlow Bike Sharing Use Case\n",
    "\n",
    "---\n",
    "\n",
    "This notebook demonstrates an example of dataset preprocessing, model training and evaluation, model tuning via MLflow tracking, finding best trained model and finally deploying the model using KServe.\n",
    "\n",
    "---\n",
    "- **Dateset:** Bike Sharing Dataset: http://archive.ics.uci.edu/ml/machine-learning-databases/00275/Bike-Sharing-Dataset.zip\n",
    "- **Goal:** predict `rented_bikes` (count per hour) based on weather and time information.\n",
    "\n",
    "\n",
    "**References:**\n",
    "- https://docs.databricks.com/_static/notebooks/gbt-regression.html\n",
    "- https://www.kaggle.com/pratsiuk/mlflow-experiment-automation-top-9\n",
    "- https://mlflow.org/docs/latest/tracking.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlflow\n",
    "experiment_name = 'bike-sharing-exp'\n",
    "\n",
    "mlflow.set_experiment(experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6dR-JRDBngFJ"
   },
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip3 install --proxy <PROXY> pydotplus graphviz seaborn\n",
    "!pip3 install pydotplus graphviz seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "5C1LuP7Oodd8",
    "outputId": "6a47110e-5994-4d05-e380-58540915e624"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import json, datetime\n",
    "import mlflow\n",
    "import mlflow.sklearn\n",
    "from mlflow import log_metric, log_param, log_artifact\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import KFold, cross_val_score, train_test_split\n",
    "from sklearn.inspection import permutation_importance\n",
    "from mlflow.models.signature import infer_signature\n",
    "from sklearn import tree\n",
    "\n",
    "from pydotplus import graph_from_dot_data\n",
    "import graphviz\n",
    "from IPython.display import Image\n",
    "\n",
    "import itertools, os\n",
    "\n",
    "plt.style.use(\"fivethirtyeight\")\n",
    "pd.plotting.register_matplotlib_converters()\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "if os.path.exists(\"model_artifacts\"):\n",
    "    os.system(\"rm -rf model_artifacts\")\n",
    "os.mkdir(\"model_artifacts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TwKZC40S-e0R"
   },
   "source": [
    "## Import Data\n",
    "\n",
    "Dataset and explanation:\n",
    "http://archive.ics.uci.edu/ml/datasets/Bike+Sharing+Dataset\n",
    "\n",
    "- Input file: `bike-sharing.csv` - contains bike sharing counts aggregated on hourly basis. \n",
    "- Size: 17379 hours / rows\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 276
    },
    "id": "1SZF_ZgD-gez",
    "outputId": "2dad51e5-5194-44af-c06c-5481c81d7639"
   },
   "outputs": [],
   "source": [
    "# Dataset is already available in github repository if not you can download and extract csv files as well.\n",
    "#!wget -e use_proxy=yes -e http_proxy=http://web-proxy.corp.hpecorp.net:8080 -nc \"http://archive.ics.uci.edu/ml/machine-learning-databases/00275/Bike-Sharing-Dataset.zip\"\n",
    "#!unzip -o \"Bike-Sharing-Dataset.zip\"\n",
    "#!rm -rf \"Bike-Sharing-Dataset.zip\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 407
    },
    "id": "mFGzYdKCCNiK",
    "outputId": "8783bf81-d46a-4958-d2dc-59a324868a64"
   },
   "outputs": [],
   "source": [
    "# load input data into pandas dataframe\n",
    "bike_sharing = pd.read_csv(\"bike-sharing.csv\")\n",
    "bike_sharing        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fQk3RQt2FB8x"
   },
   "source": [
    "## Data preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 407
    },
    "id": "vyS5Ru5aE5Y7",
    "outputId": "5d0b2528-9664-437d-8e3f-61119fdaad5e"
   },
   "outputs": [],
   "source": [
    "# remove unused columns\n",
    "bike_sharing.drop(columns=[\"instant\", \"dteday\", \"registered\", \"casual\"], inplace=True)\n",
    "\n",
    "# use better names\n",
    "bike_sharing.rename(\n",
    "    columns={\n",
    "        \"yr\": \"year\",\n",
    "        \"mnth\": \"month\",\n",
    "        \"hr\": \"hour_of_day\",\n",
    "        \"holiday\": \"is_holiday\",\n",
    "        \"workingday\": \"is_workingday\",\n",
    "        \"weathersit\": \"weather_situation\",\n",
    "        \"temp\": \"temperature\",\n",
    "        \"atemp\": \"feels_like_temperature\",\n",
    "        \"hum\": \"humidity\",\n",
    "        \"cnt\": \"rented_bikes\",\n",
    "    },\n",
    "    inplace=True,\n",
    ")\n",
    "\n",
    "# show samples\n",
    "\n",
    "\n",
    "cols = bike_sharing.select_dtypes(exclude=['float64']).columns\n",
    "\n",
    "for i in ['season', 'year', 'month', 'hour_of_day', 'is_holiday', 'weekday',\n",
    "       'is_workingday', 'weather_situation', 'rented_bikes']:\n",
    "    bike_sharing[i] = bike_sharing[i].astype('float64')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "40MGTHbNFKTP"
   },
   "source": [
    "### Data Visualization "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 639
    },
    "id": "bNZOegwGHzUR",
    "outputId": "45a00d75-c019-4c39-96c4-08e3995fb381"
   },
   "outputs": [],
   "source": [
    "hour_of_day_agg = bike_sharing.groupby([\"hour_of_day\"])[\"rented_bikes\"].sum()\n",
    "\n",
    "hour_of_day_agg.plot(\n",
    "    kind=\"line\", \n",
    "    title=\"Total rented bikes by hour of day\",\n",
    "    xticks=hour_of_day_agg.index,\n",
    "    figsize=(15, 10),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZMg_JKoUKq9j"
   },
   "source": [
    "## Prepare training and test data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "ZwtDgaZ9Ktie",
    "outputId": "4971f3d6-5e99-4583-acb6-4ef73892a633"
   },
   "outputs": [],
   "source": [
    "# Split the dataset randomly into 70% for training and 30% for testing.\n",
    "X = bike_sharing.drop(\"rented_bikes\", axis=1)\n",
    "y = bike_sharing.rented_bikes\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, test_size=0.3, random_state=42)\n",
    "\n",
    "print(f\"Training samples: {X_train.size}\")\n",
    "print(f\"Test samples: {X_test.size}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HN0w6zFJSb87"
   },
   "source": [
    "## Evaluation Metrics\n",
    "\n",
    "Create evaluation methods to be used in training stage (next step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eC1wzz_T_tSA"
   },
   "source": [
    "## Root Mean Square Error (RMSE)\n",
    "\n",
    "References: \n",
    "- https://medium.com/@xaviergeerinck/artificial-intelligence-how-to-measure-performance-accuracy-precision-recall-f1-roc-rmse-611d10e4caac\n",
    "- https://www.kaggle.com/residentmario/model-fit-metrics#Root-mean-squared-error-(RMSE)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MhPcLCteQy6j"
   },
   "outputs": [],
   "source": [
    "def rmse(y, y_pred):\n",
    "    return np.sqrt(mean_squared_error(y, y_pred))\n",
    "\n",
    "\n",
    "def rmse_score(y, y_pred):\n",
    "    score = rmse(y, y_pred)\n",
    "    print(\"RMSE score: {:.4f}\".format(score))\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fZ3nr3D_AE85"
   },
   "source": [
    "## Cross-Validation RMSLE score\n",
    "\n",
    "cross-validation combines (averages) measures of fitness in prediction to derive a more accurate estimate of model prediction performance.\n",
    "\n",
    "Background: \n",
    "- https://en.wikipedia.org/wiki/Cross-validation_(statistics)\n",
    "- https://www.kaggle.com/carlolepelaars/understanding-the-metric-rmsle\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9H9CZAP2ASe6"
   },
   "outputs": [],
   "source": [
    "def rmsle_cv(model, X_train, y_train):\n",
    "    kf = KFold(n_splits=3, shuffle=True, random_state=42).get_n_splits(X_train.values)\n",
    "    # Evaluate a score by cross-validation\n",
    "    rmse = np.sqrt(-cross_val_score(model, X_train.values, y_train, scoring=\"neg_mean_squared_error\", cv=kf))\n",
    "    return rmse\n",
    "\n",
    "\n",
    "def rmse_cv_score(model, X_train, y_train):\n",
    "    score = rmsle_cv(model, X_train, y_train)\n",
    "    print(\"Cross-Validation RMSE score: {:.4f} (std = {:.4f})\".format(score.mean(), score.std()))\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ad0mABWEarsA"
   },
   "source": [
    "## Feature Importance\n",
    "\n",
    "Background: https://medium.com/bigdatarepublic/feature-importance-whats-in-a-name-79532e59eea3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OZ7kzjbOWae8"
   },
   "outputs": [],
   "source": [
    "def model_feature_importance(model):\n",
    "    feature_importance = pd.DataFrame(\n",
    "        model.feature_importances_,\n",
    "        index=X_train.columns,\n",
    "        columns=[\"Importance\"],\n",
    "    )\n",
    "\n",
    "    # sort by importance\n",
    "    feature_importance.sort_values(by=\"Importance\", ascending=False, inplace=True)\n",
    "\n",
    "    # plot\n",
    "    plt.figure(figsize=(12, 8))\n",
    "    sns.barplot(\n",
    "        data=feature_importance.reset_index(),\n",
    "        y=\"index\",\n",
    "        x=\"Importance\",\n",
    "    ).set_title(\"Feature Importance\")\n",
    "    # save image\n",
    "    plt.savefig(\"model_artifacts/feature_importance.png\", bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rYfCxPo8w-Gn"
   },
   "source": [
    "## Permutation Importance\n",
    "\n",
    "Background: https://www.kaggle.com/dansbecker/permutation-importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b_vzVVbGcS6M"
   },
   "outputs": [],
   "source": [
    "def model_permutation_importance(model):\n",
    "    p_importance = permutation_importance(model, X_test, y_test, random_state=42, n_jobs=-1)\n",
    "\n",
    "    # sort by importance\n",
    "    sorted_idx = p_importance.importances_mean.argsort()[::-1]\n",
    "    p_importance = pd.DataFrame(\n",
    "        data=p_importance.importances[sorted_idx].T,\n",
    "        columns=X_train.columns[sorted_idx]\n",
    "    )\n",
    "\n",
    "    # plot\n",
    "    plt.figure(figsize=(12, 8))\n",
    "    sns.barplot(\n",
    "        data=p_importance,\n",
    "        orient=\"h\"\n",
    "    ).set_title(\"Permutation Importance\")\n",
    "\n",
    "    # save image\n",
    "    plt.savefig(\"model_artifacts/permutation_importance.png\", bbox_inches=\"tight\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VtQGsSNU8hWc"
   },
   "source": [
    "## Decision Tree Visualization\n",
    "\n",
    "Reference: https://towardsdatascience.com/visualizing-decision-trees-with-python-scikit-learn-graphviz-matplotlib-1c50b4aa68dc \n",
    "\n",
    "\n",
    "TODO: plot all trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rxKIpaE-g-b1"
   },
   "outputs": [],
   "source": [
    "def model_tree_visualization(model):\n",
    "    # generate visualization\n",
    "    tree_dot_data = tree.export_graphviz(\n",
    "        decision_tree=model.estimators_[0, 0],  # Get the first tree,\n",
    "        label=\"all\",\n",
    "        feature_names=X_train.columns,\n",
    "        filled=True,\n",
    "        rounded=True,\n",
    "        proportion=True,\n",
    "        impurity=False,\n",
    "        precision=1,\n",
    "    )\n",
    "\n",
    "    # save image\n",
    "    graph_from_dot_data(tree_dot_data).write_png(\"model_artifacts/Decision_Tree_Visualization.png\")\n",
    "\n",
    "    # show tree\n",
    "    return graphviz.Source(tree_dot_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "warpAv8RFSOI"
   },
   "source": [
    "# MLflow Tracking\n",
    "\n",
    "Reference: https://www.mlflow.org/docs/latest/cli.html#mlflow-ui\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1kU8w1TNGG4Y"
   },
   "source": [
    "## MLflow Logger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kyQRcKslAwv-"
   },
   "outputs": [],
   "source": [
    "# Track params and metrics\n",
    "def log_mlflow_run(model, signature):\n",
    "    # Auto-logging for scikit-learn estimators\n",
    "    # mlflow.sklearn.autolog()\n",
    "\n",
    "    # log estimator_name name\n",
    "    name = model.__class__.__name__\n",
    "    mlflow.set_tag(\"estimator_name\", name)\n",
    "\n",
    "    # log input features\n",
    "    mlflow.set_tag(\"features\", str(X_train.columns.values.tolist()))\n",
    "\n",
    "    # Log tracked parameters only\n",
    "    mlflow.log_params({key: model.get_params()[key] for key in parameters})\n",
    "\n",
    "    mlflow.log_metrics({\n",
    "        'RMSE_CV': score_cv.mean(),\n",
    "        'RMSE': score,\n",
    "    })\n",
    "\n",
    "    # log training loss\n",
    "    for s in model.train_score_:\n",
    "        mlflow.log_metric(\"Train Loss\", s)\n",
    "\n",
    "    # Save model to artifacts\n",
    "    mlflow.sklearn.log_model(model, \"model\", signature=signature)\n",
    "\n",
    "    # log charts\n",
    "    mlflow.log_artifacts(\"model_artifacts\")\n",
    "\n",
    "    # misc\n",
    "    # Log all model parameters\n",
    "    # mlflow.log_params(model.get_params())\n",
    "    mlflow.log_param(\"Training size\", X_test.size) \n",
    "    mlflow.log_param(\"Test size\", y_test.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7xRa6vYWMed1"
   },
   "source": [
    "# Model Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mDAdPTeDTjr1"
   },
   "source": [
    "## Model Type & Method\n",
    "\n",
    "For this example,\n",
    "- Approache: Decision tree (Supervised learning)\n",
    "- Tree type: Regression tree\n",
    "- Technique/ensemble method: Gradient boosting\n",
    "\n",
    "**All put together we get:** [GBRT (Gradient Boosted Regression Tree)](https://orbi.uliege.be/bitstream/2268/163521/1/slides.pdf)\n",
    "\n",
    "Background:\n",
    "- Choosing a model: https://scikit-learn.org/stable/tutorial/machine_learning_map\n",
    "- Machine Learning Models Explained\n",
    ": https://docs.paperspace.com/machine-learning/wiki/machine-learning-models-explained\n",
    "- Gradient Boosted Regression Trees: https://orbi.uliege.be/bitstream/2268/163521/1/slides.pdf\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OSbcPvkBThXV"
   },
   "outputs": [],
   "source": [
    "# GBRT (Gradient Boosted Regression Tree) scikit-learn implementation \n",
    "model_class = GradientBoostingRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l7BYFTSRzLk2"
   },
   "source": [
    "## Model Hyper-parameters "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1Mu88JOkMiJF"
   },
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    \"learning_rate\": [0.1, 0.05, 0.01],\n",
    "    \"max_depth\": [4, 5, 6],\n",
    "    # \"verbose\": True,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LnUDX2p2j9p_"
   },
   "source": [
    "## Tuning the hyper-parameters: Grid search\n",
    "\n",
    "- Simple but inefficient\n",
    "- more advanced tuning techniques: https://research.fb.com/efficient-tuning-of-online-systems-using-bayesian-optimization/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CybsVlgCw6n9"
   },
   "outputs": [],
   "source": [
    "# generate parameters combinations\n",
    "params_keys = parameters.keys()\n",
    "params_values = [\n",
    "    parameters[key] if isinstance(parameters[key], list) else [parameters[key]]\n",
    "    for key in params_keys\n",
    "]\n",
    "runs_parameters = [\n",
    "    dict(zip(params_keys, combination)) for combination in itertools.product(*params_values)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u23-Tpn_0X7d"
   },
   "source": [
    "## Training runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 639
    },
    "id": "Le6sa7jjg37v",
    "outputId": "7e8c3e45-e75a-45ce-8157-38b097d623cc"
   },
   "outputs": [],
   "source": [
    "# training loop\n",
    "for i, run_parameters in enumerate(runs_parameters):\n",
    "    print(f\"Run {i}: {run_parameters}\")\n",
    "\n",
    "    # mlflow: stop active runs if any\n",
    "    if mlflow.active_run():\n",
    "        mlflow.end_run()\n",
    "    # mlflow:track run\n",
    "    mlflow.start_run(run_name=f\"Run {i}\")\n",
    "\n",
    "    # create model instance\n",
    "    model = model_class(**run_parameters)\n",
    "\n",
    "    # train\n",
    "    model.fit(X_train, y_train)\n",
    "\n",
    "    # get evaluations scores\n",
    "    score = rmse_score(y_test, model.predict(X_test))\n",
    "    score_cv = rmse_cv_score(model, X_train, y_train)\n",
    "    \n",
    "    # generate charts\n",
    "    model_feature_importance(model)\n",
    "    plt.close()\n",
    "    model_permutation_importance(model)\n",
    "    plt.close()\n",
    "    # model_tree_visualization(model)\n",
    "\n",
    "    # get model signature\n",
    "    signature = infer_signature(model_input=X_train, model_output=model.predict(X_train))\n",
    "\n",
    "    # mlflow: log metrics\n",
    "    log_mlflow_run(model, signature)\n",
    "\n",
    "    # mlflow: end tracking\n",
    "    mlflow.end_run()\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tOHX6U3ABTSE"
   },
   "source": [
    "## Best Model Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "I5jKy850zKtS"
   },
   "outputs": [],
   "source": [
    "best_run_df = mlflow.search_runs(order_by=['metrics.RMSE_CV ASC'], max_results=1)\n",
    "if len(best_run_df.index) == 0:\n",
    "    raise Exception(f\"Found no runs for experiment '{experiment_name}'\")\n",
    "\n",
    "best_run = mlflow.get_run(best_run_df.at[0, 'run_id'])\n",
    "best_model_uri = f\"{best_run.info.artifact_uri}/model\"\n",
    "with open('best-model-uri.txt','w+') as f:\n",
    "    f.write(best_model_uri)\n",
    "best_model = mlflow.sklearn.load_model(best_model_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104
    },
    "id": "wHVM74A--4-C",
    "outputId": "b28470ff-aa99-4f19-c124-d4b9c3a88233"
   },
   "outputs": [],
   "source": [
    "# print best run info\n",
    "print(\"Best run info:\")\n",
    "print(f\"Run id: {best_run.info.run_id}\")\n",
    "print(f\"Run parameters: {best_run.data.params}\")\n",
    "print(\"Run score: RMSE_CV = {:.4f}\\n\\n\".format(best_run.data.metrics['RMSE_CV']))\n",
    "print(f\"Run model URI: {best_model_uri}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 513
    },
    "id": "WmjSO3vhCP7u",
    "outputId": "52e2c4ac-4aeb-44b8-d65d-7d2de8122fb8"
   },
   "outputs": [],
   "source": [
    "model_feature_importance(best_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 496
    },
    "id": "LQRJKFuJCSBZ",
    "outputId": "3a60cd14-f402-4304-ee8c-7f3647be4721"
   },
   "outputs": [],
   "source": [
    "model_permutation_importance(best_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 815
    },
    "id": "fR2F0ex7CS4I",
    "outputId": "ff25041b-c91e-4372-ab49-514e350bc709"
   },
   "outputs": [],
   "source": [
    "# model_tree_visualization(best_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UDhu91aa8vuw"
   },
   "source": [
    "## Test the Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 407
    },
    "id": "EiQwrb7TK40n",
    "outputId": "709d749f-bc0d-4b68-c2c4-8c1a0197eca6"
   },
   "outputs": [],
   "source": [
    "test_predictions = X_test.copy()\n",
    "# real output (rented_bikes) from test dataset\n",
    "test_predictions[\"rented_bikes\"] = y_test\n",
    "\n",
    "# add \"predicted_rented_bikes\" from test dataset\n",
    "test_predictions[\"predicted_rented_bikes\"] = best_model.predict(X_test).astype(int)\n",
    "\n",
    "# show results\n",
    "test_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 911
    },
    "id": "SwfQEr_NGlDa",
    "outputId": "e153d67b-b13f-4b13-bbe9-542eed7442a8"
   },
   "outputs": [],
   "source": [
    "# plot truth vs prediction values\n",
    "test_predictions.plot(\n",
    "    kind=\"scatter\",\n",
    "    x=\"rented_bikes\",\n",
    "    y=\"predicted_rented_bikes\",\n",
    "    title=\"Rented bikes vs predicted rented bikes\",\n",
    "    figsize=(15, 15),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Role Based Access Control\n",
    "\n",
    "By default, users recieve `MANAGE` permissions if they create an object, `NO_PERMISSIONS` otherwise. A full breakdown of all roles and their access is described [here](https://mlflow.org/docs/latest/auth/index.html#permissions)\n",
    "\n",
    "To share experiments/models, MLFlow provides an `AuthServiceCLient` implementing CRUD functionality for `experiment_permission` and `model_permission` objects. `AuthServiceClient` is documented [here](https://mlflow.org/docs/latest/auth/python-api.html#mlflow.server.auth.client.AuthServiceClient)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip3 install --proxy <PROXY> mlflow[auth]\n",
    "!pip3 install mlflow[auth]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlflow.server.auth.client import AuthServiceClient\n",
    "\n",
    "user = #\" USERNAME\"\n",
    "permission = #\"READ\", \"EDIT\", \"MANAGE\", \"NO_PERMISSIONS\"\n",
    "exp_id = mlflow.get_experiment_by_name(experiment_name).experiment_id\n",
    "\n",
    "client = AuthServiceClient(mlflow.get_tracking_uri())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating Permission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "permission = \"READ\"\n",
    "exp_permission = client.create_experiment_permission(exp_id, user, permission)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Modifying Permission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "permission = \"EDIT\"\n",
    "\n",
    "exp_permission = client.update_experiment_permission(exp_id, user, permission)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "permission = \"NO_PERMISSIONS\"\n",
    "\n",
    "exp_permission = client.update_experiment_permission(exp_id, user, permission)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Delete Permissions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_permission = client.delete_experiment_permission(exp_id, user)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyMUyEIXKPIvKiU5I2T//pwx",
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "MLflow-example-notebook.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}