python-without-filesystem.json

{
  "alias": "video/1332/python-without-filesystem",
  "category": "EuroPython 2012",
  "copyright_text": "Standard YouTube License",
  "description": "Recent cloud architecture are heavily based on key-value storage\ndatabases rather than distributed filesystems. While the data itself is\nusually stored in scalable database, the source code is mainly kept on\nthe filesystem. However, having the source code on filesystem raises a\nlot of issues for cloud applications and more generally for\ntransactional web applications supposed to be deployed on a large number\nof servers: Everytime the source code is modified, it has to be\nsynchronized on all the nodes somehow and the application server must be\nrestarted, therefore implying downtime or inconsistency. Usually, the\ndeveloper cannot directly modify the source code as this task is left to\nthe administrator. This talk will show how source code, including\nmodules and their classes, can be migrated away from filesystem, thanks\nto import hooks, meta classes, reflection and metaprogramming provided\nby Python, and the different challenges involved: Technical: modules and\nclasses must be loaded and reset while carefully considering concurrency\nand database issues and ensuring that the system will stay usable\nwhatever happens and without breaking existing class instances. Also,\ndevelopers must be efficient with the new system, including convenient\nsource code edition and meaningful backtraces. Performance: having\ndynamic modules and classes involved a performance tradeoff which should\nbe kept to the minimum. Security: as the source code is dynamically\nloaded from the database, extra care must be taken care of compared to\nfilesystem source code where the source code is never supposed to change\nonce the applications has been started. In order to give practical\nexamples of how this migration can be achieved, two examples will be\nstudied and explain thoroughly, based on the experience I have acquired\nthrough the implementation for ERP5 I have worked on during the past\nyear: First, through a basic WSGI application based on a lightweight web\napplication framework to explain the basics behind having Python source\ncode available in a transactional key-value storage database and more\nprecisely how modules and their classes can be dynamically loaded and\nreset on-demand upon modifications while keeping good performances. This\nusually involved specific features from the database point of view which\nwill also be discussed. Secondly, through ERP5 which is mainly known as\nan open source ERP based on Zope and thus ZODB, and as such it provides\nfeatures an ERP typically provides, but it is also used for a wide range\nof other purposes. Therefore, this will give a more real-world example\non how this migration has been performed on a complex web application\nwhere 90% of the code is going to be moved away from filesystem into the\nZODB.\n",
  "duration": null,
  "id": 1332,
  "language": "eng",
  "quality_notes": "",
  "recorded": "2012-07-04",
  "slug": "python-without-filesystem",
  "speakers": [
    "Arnaud Fontaine"
  ],
  "summary": "[EuroPython 2012] Arnaud Fontaine - 3 JULY 2012 in \"Track Lasagne\"\n",
  "tags": [],
  "thumbnail_url": "http://i.ytimg.com/vi/xWY0UQn3dcA/hqdefault.jpg",
  "title": "Python without filesystem",
  "videos": [
    {
      "length": 0,
      "type": "youtube",
      "url": "http://www.youtube.com/watch?v=xWY0UQn3dcA"
    }
  ]
}