experiences-making-cpu-bound-tasks-run-much-faste.json

{
  "alias": "video/1071/experiences-making-cpu-bound-tasks-run-much-faste",
  "category": "EuroPython 2011",
  "copyright_text": "Standard YouTube License",
  "description": "UPDATE - post-event I've created a `49 page PDF write-\nup <http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1%20-from-my-4-hour-tutorial-at-europython-2011/>`__\nwhich summarises the 4 hour tutorial\n\nAs a long-time R&D consultant I'm often working to make slow,\nexperimental code run faster for tasks like physics simulation, flood\nmodeling and natural language processing. Python allows a smooth\nprogression from rough-and-ready (but slow) algorithms through to finely\ntuned tasks that efficiently use as much CPU power as you can bring to\nbear. Speed-ups of 10-500\\* can be expected for the Mandelbrot code\nwe'll use.\n\nIn this talk I'll cover a set of libraries that make CPU-bound tasks run\nmuch faster. We'll begin with a look at profiling using RunSnakeRun and\nline\\_profiler to identify our bottleneck. We'll take a look at slow\nalgorithms in Python and how they can run faster using numpy and\nnumexpr.\n\nNext we'll cover the use of multiprocessing to utilise multiple CPU\ncores along with Cython or ShedSkin to easily use C code in a friendly\nPython wrapper. Multiprocessing on a quad-core system can often provide\na 4\\* speed-up for the right tasks. Next parallelpython will let us run\nour code on a network of machines.\n\nFinally we'll look at pyCUDA to utilise an NVIDIA GPU. CUDA can give the\nbest improvements for mathematical problems (over 100\\* on the right\ntasks) but works on a narrower set of problems.\n\nHow it'll work: The tutorial will be hands on, you'll be converting\nexample files from normal Python to faster variants using the tools\nbelow. All of it is optional, you'll get the most benefit by having\neverything installed. We'll work in groups and open discussion is\nencouraged.\n\nNOTE - you are expected to have all these tools installed *before* the\ntutorial (if you don't, you might find it hard to follow what's going\non!).\n\nI'll be using Python 2.7.1 on a Macbook (Snow Leopard). All of these\ntools run on Windows and Linux, as long as your versions are fairly\nrecent everything should run just fine.\n\nMy versions (roughly ordered by importance):\n\n-  Python 2.7.1\n-  RunSnakeRun 2.0.1b6 (with wxPython 2.8.12.0 Unicode)\n-  line\\_profiler (1.0b2)\n-  Cython 0.14.1\n-  ShedSkin 0.7.1\n-  numpy 1.5.1\n-  numexpr 1.4.2\n-  ParallelPython 1.6.1\n-  pyCUDA HEAD from git as of 14th June 2011 (with CUDA 4.0 drivers)\n-  PyPy 1.5\n\nSome background reading:\n\n-  http://ianozsvald.com/2010/07/14/22937-faster-python-math-using-pycuda/\n-  http://ianozsvald.com/2008/11/17/making-python-math-196-faster-with-shedskin/\n\n",
  "duration": null,
  "id": 1071,
  "language": "eng",
  "quality_notes": "",
  "recorded": "2011-07-18",
  "related_urls": [
    "http://ianozsvald.com/2008/11/17/making-python-math-196-faster-with-shedskin/",
    "http://ianozsvald.com/2010/07/14/22937-faster-python-math-using-pycuda/",
    "http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1%20-from-my-4-hour-tutorial-at-europython-2011/"
  ],
  "slug": "experiences-making-cpu-bound-tasks-run-much-faste",
  "speakers": [
    "Ian Ozsvald"
  ],
  "summary": "[EuroPython 2011] Ian Ozsvald - 22 June 2011 in \"Training Pizza\nMargherita \"\n",
  "tags": [
    "cython",
    "git",
    "multiprocessing",
    "network",
    "numpy",
    "nvidia",
    "profiling",
    "pycuda",
    "runsnakerun",
    "tutorial",
    "windows",
    "wxpython"
  ],
  "thumbnail_url": "http://i.ytimg.com/vi/F7NOKSP_vlQ/hqdefault.jpg",
  "title": "Experiences making CPU-bound tasks run much faster",
  "videos": [
    {
      "length": 0,
      "type": "youtube",
      "url": "http://www.youtube.com/watch?v=F7NOKSP_vlQ"
    }
  ]
}