HyperGBM-FAQ.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1：HyperGBM 特征预处理环节中，如何进行数据清洗？缺失值处理机制是什么?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### A1：\n",
    "HyperGBM 在特征预处理环节中，数据清洗操作如下：特殊空值符处理、列类型校正、常量列清理、重复列清理、删除标签列为空的样本、非法值替换、ID列清理、共线性特征清理。\n",
    "** **\n",
    "缺失值在模型优化过程中，在搜索空间中，对于类别型变量（Categorical）和数值型变量（Numeric）分别进行操作，处理机制如下：\n",
    "#####  1）类别型变量：\n",
    "#####     1> Sample_pipeline: NaN-> Imputer(‘constant’) -> SafeOrdinalEncoder \n",
    "#####    2> Complex_pipeline: NaN -> Imputer(Choice(['constant','most_frequent'])) -> ModuleChoice([LabelEncoder, SafeOneHotEncoder + TruncatedSVD])\n",
    "#####  2) 数值型变量：\n",
    "#####     1> Sample_pipeline: NaN -> Imputer(‘mean’) -> StandardScaler\n",
    "#####     2> Complex_pipeline: NaN ->Imputer(Choice(['mean', 'median', 'constant', 'most_frequent'])) -> ModuleChoice([LogStandardScaler, StandardScaler, MinMaxScaler, MaxAbsScaler, RobustScaler])\n",
    "** **\n",
    "Note：缺失值处理可参考代码：hypergbm -> sklearn -> sklearn_ops.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q2： HyperGBM 目前已经具备了哪些算法？是否兼容并内置了聚类、时间序列等算法?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### A2:\n",
    "HyperGBM 目前支持XGBoost、CatBoost、LightGBM、HistGridientBoosting等算法。\n",
    "1) HyperGBM是以各类GBM为依托，端到端地建模的自动化机器学习算法，支持二分类、多分类以及回归任务。\n",
    "2) 其中对于时间序列任务，在某些情况转换为时间序列分类，预测任务时，HyperGBM可支持，性能看表现。\n",
    "3) 目前HyperGBM是监督学习模型，而聚类是无监督模型，二者的建模理念不同，当下不支持聚类任务。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q3: HyperGBM 做了哪些特征工程？主要价值体现是什么？ 在自动化流程里面是通用的吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### A3：\n",
    "HyperGBM 所做特征工程工程包括：\n",
    "   1> 特征衍生、特征降维;\n",
    "   2> 特殊空值符处理、列类型校正、常量列清理、重复列清理、删除标签列为空的样本、非法值替换、id列清理、共线性特征清理；\n",
    "   3> SimpleImputer, SafeOrdinalEncoder, SafeOneHotEncoder, TruncatedSVD, StandardScaler, MinMaxScaler, MaxAbsScaler, RobustScaler, LogStandardScaler.\n",
    "   其中：1）通过特征衍生，可以为模型提供更加丰富的信息提出有用的特征；2）通过数据降维可以压缩掉冗余特征并缓解高维问题引发的维度灾难的问题；\n",
    " 3）通过各种数据清洗以及缺失值填充操作可以增强模型对数据的易读性和一致性，减小数据的方差，优化数据的分布, 从而提高所建模型的鲁棒性和泛化能力。\n",
    "** **\n",
    " 价值体现：1）防止用户导入的数据不符合模型要求而导致失败；2）节省用户的时间，提供端到端的建模能力.\n",
    "** **\n",
    " 在自动化流程中是通用的，有利于提升建模的效率。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q4: HyperGBM 算法第3方算法快速接入吗？算法接入规范是怎样的？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### A4：\n",
    "支持。定义一个符合hypergbm接口的estimator，然后将其定义到搜索空间中。\n",
    "例如，接入DeepForest："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DeepForestEstimator(HyperEstimator):\n",
    "    def __init__(self):\n",
    "        ...\n",
    "\n",
    "def search_space():\n",
    "    space = HyperSpace()\n",
    "    with space.as_default():\n",
    "        input = HyperInput(name='input1')\n",
    "        scaler_choice = ModuleChoice(\n",
    "            [\n",
    "                StandardScaler(name=f'numeric_standard_scaler'),\n",
    "                MinMaxScaler(name=f'numeric_minmax_scaler')\n",
    "            ], name=f'numeric_or_scaler'\n",
    "        )\n",
    "        num_pipeline = Pipeline([scaler_choice], name'numeric_pipeline',\n",
    "                               columns=column_number_exclude_timedelta)(input)\n",
    "        union_pipeline = DataFramePapper(default=None, input_df=True, df_out=True)([num_pipeline])\n",
    "        deepforest_est = DeepForestEstimator(fit_keargs={})\n",
    "        ModuleChoice([deepforest_est], name='estimator_options')(union_pipeline)\n",
    "        space.set_inputs(input)\n",
    "    return space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q5: HyperGBM 为什么选择 Dask 而不是 Spark？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### A5：\n",
    "数据科学家通常使用Pandas，Scikit-Learn，Numpy和其他Python生态系统之类的工具来分析其个人计算机上的数据。Dask提供了一种以最少的重写就可以更 \n",
    "朴素地扩展Pandas，Scikit-Learn和Numpy工作流的方法。它与这些工具可以很好地集成在一起，因此可以复制其大多数API并在内部使用其数据结构。此外，Dask与这些库共同开\n",
    "发，以确保它们一致地发展，从而最大程度地减少了从本地笔记本电脑过渡到多核工作站，再过渡到分布式集群时的摩擦。熟悉Pandas / Scikit-Learn / Numpy的科学家将立即\n",
    "熟悉其Dask等价物，并将其大部分直觉转移到可扩展的上下文中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}