Python multi process.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python多任务处理(多进程篇)\n",
    "作者: tushushu  \n",
    "项目地址: https://github.com/tushushu/flying-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多进程处理CPU密集型任务\n",
    "CPU密集型任务的特点是要进行大量的计算，消耗CPU资源，比如计算圆周率、对视频进行高清解码等等，全靠CPU的运算能力。一个线程执行CPU密集型任务的时候，CPU处于忙碌状态，运行1000个字节码之后GIL会被释放给其他线程，加上切换线程的时间有可能会比串行代码更慢。在Python多任务处理(多线程篇)，我们试图用多线程执行CPU密集型任务，然而并没有性能上的提升。现在我们试一下用多进程来处理CPU密集型任务。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 建立进程池"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU核数为8个!\n"
     ]
    }
   ],
   "source": [
    "from concurrent.futures import ProcessPoolExecutor\n",
    "from time import sleep, time\n",
    "import os\n",
    "print(\"CPU核数为%s个!\" % os.cpu_count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Worker数量\n",
    "N = 8\n",
    "# 建立进程池\n",
    "pool = ProcessPoolExecutor(max_workers=N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 定义一个CPU密集型函数\n",
    "该函数会对[1, x]之间的整数进行求和。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cpu_bound_func(x):\n",
    "    tot = 0\n",
    "    a = 1\n",
    "    while a <= x:\n",
    "        tot += x\n",
    "        a += 1\n",
    "    print(\"Finish sum from 1 to %d!\" % x)\n",
    "    return tot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 使用串行的方式处理\n",
    "遍历一个列表的所有元素，执行func函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_array(arr):\n",
    "    for x in arr:\n",
    "        cpu_bound_func(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 使用多进程处理\n",
    "通过线程池的map方法，可以将同一个函数作用在列表中的所有元素上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fast_process_array(arr):\n",
    "    for x in pool.map(cpu_bound_func, arr):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 计算函数运行时间\n",
    "- 串行版本的运行时间5.7秒\n",
    "- 多进程版本的运行时间1.6秒"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def time_it(fn, *args):\n",
    "    start = time()\n",
    "    fn(*args)\n",
    "    print(\"%s版本的运行时间为 %.5f 秒!\" % (fn.__name__, time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "Finish sum from 1 to 10000000!\n",
      "process_array版本的运行时间为 5.74394 秒!\n"
     ]
    }
   ],
   "source": [
    "time_it(process_array, [10**7 for _ in range(8)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fast_process_array版本的运行时间为 1.62266 秒!\n"
     ]
    }
   ],
   "source": [
    "time_it(fast_process_array, [10**7 for _ in range(8)])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}