From b31661a639043177418cc53c022f84e5d3712ac7 Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 15 Jul 2026 22:25:11 +0100 Subject: [PATCH 1/3] Convert categorical operations from pandas to Polars --- categorical-data.ipynb | 247 ++++++++++++------ workflow-style.ipynb | 558 ++++++++++++++++++++--------------------- 2 files changed, 444 insertions(+), 361 deletions(-) diff --git a/categorical-data.ipynb b/categorical-data.ipynb index ab6ba30..c391411 100644 --- a/categorical-data.ipynb +++ b/categorical-data.ipynb @@ -2,25 +2,21 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "cell_000", "metadata": {}, "source": [ "# Categorical Data {#sec-categorical-data}\n", "\n", "## Introduction\n", "\n", - "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **pandas** [documentation](https://pandas.pydata.org/).\n" + "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **polars** [documentation](https://docs.pola.rs/api/python/stable/reference/index.html).\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "51a55374", - "metadata": { - "tags": [ - "remove-cell" - ] - }, + "id": "cell_001", + "metadata": {}, "outputs": [], "source": [ "# remove cell\n", @@ -34,248 +30,265 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "cell_002", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package.\n" ] }, { "cell_type": "markdown", - "id": "8757f369", + "id": "cell_003", "metadata": {}, "source": [ "## The Category Datatype\n", "\n", - "Everything in Python has a type, even the data in **pandas** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical`. There are some benefits to using categorical variables (where appropriate):\n", + "Everything in Python has a type, even the data in **polars** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical` and `Enum`. There are some benefits to using categorical variables (where appropriate):\n", "\n", "- they can keep track even when elements of the category isn't present, which can sometimes be as interesting as when they are (imagine you find no-one from a particular school goes to university)\n", "- they can use vastly less of your computer's memory than encoding the same information in other ways\n", "- they can be used efficiently with modelling packages, where they will be recognised as potential 'dummy variables', or with plotting packages, which will treat them as discrete values\n", - "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\")\n", + "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\") using pl.Enum\n", + "\n", + "All values of categorical data for a **polars** column are either in the given categories or take the value `None`.\n", + "\n", + "Polars has two categorical types: \n", "\n", - "All values of categorical data for a **pandas** column are either in the given categories or take the value `np.nan`. \n", + "- pl.Categorical: Unordered categories (like pandas category)\n", + "\n", + "- pl.Enum: Ordered categories with a fixed set of values (similar to pandas ordered categorical)\n", "\n", "## Creating Categorical Data\n", "\n", - "Let's create a categorical column of data:" + "Let's create a categorical column of data:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "535ef959", + "id": "cell_004", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", + "df = pl.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", "\n", - "df[\"A\"] = df[\"A\"].astype(\"category\")\n", + "df = df.with_columns(pl.col(\"A\").cast(pl.Categorical))\n", "df[\"A\"]" ] }, { "cell_type": "markdown", - "id": "62db4dec", + "id": "cell_005", "metadata": {}, "source": [ - "Notice that we get some additional information at the bottom of the shown series: we get told that not only is this a categorical column type, but it has three values 'a', 'b', and 'c'.\n", + "Notice that polars tracks the categorical type, though the display is more compact than pandas.\n", "\n", - "You can also use special functions, such as `pd.cut()`, to groups data into discrete bins. Here's an example where specify the labels for the categories directly:\n" + "You can also use special expression methods, such as pl.col().cut(), to group data into discrete bins. Here's an example where we specify the labels for each category directly.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "358c83bb", + "id": "cell_006", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", + "df = pl.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", "labels = [f\"{i} - {i+9}\" for i in range(0, 100, 10)]\n", - "df[\"group\"] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)\n", + "df = df.with_columns(\n", + " pl.col(\"value\")\n", + " .cut(breaks=list(range(10, 100, 10)), labels=labels, left_closed=True)\n", + " .alias(\"group\")\n", + ")\n", "df.head()" ] }, { "cell_type": "markdown", - "id": "87bc56b6", + "id": "cell_007", "metadata": {}, "source": [ "In the example above, the `group` column is of categorical type.\n", "\n", - "Another way to create a categorical variable is directly using the `pd.Categorical()` function:" + "Another way to create a categorical variable is directly using the `pl.Categorical` or `pl.Enum` types:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "fb389105", + "id": "cell_008", "metadata": {}, "outputs": [], "source": [ - "raw_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"], categories=[\"b\", \"c\", \"d\"]\n", + "raw_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"b\", \"c\", \"d\"]), strict=False\n", ")\n", "raw_cat" ] }, { "cell_type": "markdown", - "id": "ebfdf815", + "id": "cell_009", "metadata": {}, "source": [ - "We can then enter this into a data frame:" + "We can then enter this into a data frame:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "0497fc16", + "id": "cell_010", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(raw_cat, columns=[\"cat_type\"])\n", + "df = pl.DataFrame({\"cat_type\": raw_cat})\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "76dc9dd6", + "id": "cell_011", "metadata": {}, "source": [ - "Note that NaNs appear for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values." + "Note that `None` appears for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values.\n" ] }, { "cell_type": "markdown", - "id": "aa76a795", + "id": "cell_012", "metadata": {}, "source": [ - "You can also create ordered categories:" + "You can also create ordered categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f7520d3d", + "id": "cell_013", "metadata": {}, "outputs": [], "source": [ - "ordered_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"],\n", - " categories=[\"a\", \"b\", \"c\", \"d\"],\n", - " ordered=True,\n", + "ordered_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"a\", \"b\", \"c\", \"d\"])\n", ")\n", "ordered_cat" ] }, { "cell_type": "markdown", - "id": "258cc6f5", + "id": "cell_014", "metadata": {}, "source": [ "## Working with Categories\n", "\n", - "Categorical data has a `categories` and a `ordered` property; these list the possible values and whether the ordering matters or not respectively. These properties are exposed as `.cat.categories` and `.cat.ordered`. If you don’t manually specify categories and ordering, they are inferred from the passed arguments.\n", + "Categorical data in polars has categories and ordering properties. For `pl.Enum`, the categories are fixed and ordered. For `pl.Categorical`, categories are not ordered and can be more flexible.\n", "\n", - "Let's see some examples:" + "Let's see some examples:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "2caba354", + "id": "cell_015", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.categories" + "df[\"cat_type\"].dtype.categories" ] }, { "cell_type": "code", "execution_count": null, - "id": "5f1fe093", + "id": "cell_016", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.ordered" + "isinstance(df[\"cat_type\"].dtype, pl.Enum)" ] }, { "cell_type": "markdown", - "id": "694c07b0", + "id": "cell_017", "metadata": {}, "source": [ - "If categorical data is ordered (ie `.cat.ordered == True`), then the order of the categories has a meaning and certain operations are possible: you can sort values (with `.sort_values`), and apply `.min` and `.max`." + "If categorical data is ordered (ie `pl.Enum`), then the order of the categories has a meaning and certain operations are possible: you can sort values, and apply `.min()` and `.max()`.\n" ] }, { "cell_type": "markdown", - "id": "0a6d7a90", + "id": "cell_018", "metadata": {}, "source": [ "### Renaming Categories\n", "\n", - "Renaming categories is done via the `rename_categories()` method (which works with a list or a dictionary)." + "Renaming categories in polars requires re-casting with new categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "097171b8", + "id": "cell_019", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.rename_categories([\"alpha\", \"beta\", \"gamma\"])" + "df = df.with_columns(\n", + " pl.col(\"cat_type\")\n", + " .cast(pl.String)\n", + " .replace({\"b\": \"alpha\", \"c\": \"beta\", \"d\": \"gamma\"})\n", + " .cast(pl.Enum([\"alpha\", \"beta\", \"gamma\"]))\n", + ")" ] }, { "cell_type": "markdown", - "id": "3de7fd23", + "id": "cell_020", "metadata": {}, "source": [ - "Quite often, you'll run into a situation where you want to add a category. You can do this with `.add_categories()`:" + "To add categories, you need to re-cast with an extended Enum:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4ae6df38", + "id": "cell_021", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.add_categories([\"delta\"])\n", + "df = df.with_columns(\n", + " pl.col(\"cat_type\").cast(pl.Enum([\"alpha\", \"beta\", \"gamma\", \"delta\"]))\n", + ")\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "ba525491", + "id": "cell_022", "metadata": {}, "source": [ - "Similarly, there is a `.remove_categories()` function and a `.remove_unused_categories()` function. `.set_categories` adds and removes categories in one fell swoop. One of the nice properties of set categories is that Remember that you need to do `df[\"columnname\"].cat` before calling any cat(egory) functions though." + "Because Enum categories are fixed in Polars, you cannot remove categories directly. Instead, ensure that the data contains only the desired values, then re-cast the column to a new Enum containing just those categories.\n" ] }, { "cell_type": "markdown", - "id": "76b38a2a", + "id": "cell_023", "metadata": {}, "source": [ "## Operations on Categories\n", "\n", - "As noted, ordered categories will already undergo some operations. But there are some that work on any set of categories. Perhaps the most useful is `value_counts()`" + "`Enum` columns behave like any other Polars column, but they also retain the complete set of allowed categories. This means Polars remembers categories even if no rows currently contain them.\n", + "\n", + "For example, suppose we define an `Enum` with five possible categories, but only use four of them:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "19f5bdda", + "id": "cell_024", "metadata": {}, "outputs": [], "source": [ @@ -284,42 +297,112 @@ }, { "cell_type": "markdown", - "id": "a36db155", + "id": "cell_025", + "metadata": {}, + "source": [ + "Notice that \"delta\" is not shown because value_counts() only reports values that actually occur in the data.\n", + "\n", + "However, the Enum still knows that \"delta\" is a valid category.\n", + "\n", + "You can inspect the complete set of categories stored in the Enum:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_026", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"cat_type\"].dtype.categories" + ] + }, + { + "cell_type": "markdown", + "id": "cell_027", + "metadata": {}, + "source": [ + "If you want a frequency table that includes unused categories, combine the stored category list with the observed counts:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_028", + "metadata": {}, + "outputs": [], + "source": [ + "all_categories = pl.DataFrame(\n", + " {\"cat_type\": df[\"cat_type\"].dtype.categories.cast(df[\"cat_type\"].dtype)}\n", + ")\n", + "all_categories.join(df[\"cat_type\"].value_counts(), on=\"cat_type\", how=\"left\").fill_null(\n", + " 0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "cell_029", "metadata": {}, "source": [ - "Note that even though 'delta' doesn't appear at all, it gets a count (of zero). This tracking of missing values can be quite handy.\n", + "This demonstrates one advantage of `Enum`: the complete set of allowed categories is retained, even when some categories are not present in the data.\n", "\n", - "`mode()` is another one:" + "You can also inspect the datatype of the column:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f52d5d0d", + "id": "cell_030", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].mode()" + "df[\"cat_type\"].dtype" ] }, { "cell_type": "markdown", - "id": "d28f34d8", + "id": "cell_031", "metadata": {}, "source": [ - "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example," + "You can compute the most frequent category:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4d43e94d", + "id": "cell_032", "metadata": {}, "outputs": [], "source": [ - "time_df = pd.DataFrame(\n", - " pd.Series(pd.date_range(\"2015/05/01\", periods=5, freq=\"M\"), dtype=\"category\"),\n", - " columns=[\"datetime\"],\n", + "df[\"cat_type\"].drop_nulls().mode()" + ] + }, + { + "cell_type": "markdown", + "id": "cell_033", + "metadata": {}, + "source": [ + "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_034", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "time_df = pl.DataFrame(\n", + " {\n", + " \"datetime\": pl.date_range(\n", + " datetime.date(2015, 5, 31), datetime.date(2015, 9, 30), \"1mo\", eager=True\n", + " )\n", + " .cast(pl.String)\n", + " .cast(pl.Categorical)\n", + " }\n", ")\n", "time_df" ] @@ -327,29 +410,29 @@ { "cell_type": "code", "execution_count": null, - "id": "db697f86", + "id": "cell_035", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].dt.month" + "time_df[\"datetime\"].cast(pl.String).str.to_date().dt.month()" ] }, { "cell_type": "markdown", - "id": "c202f0d1", + "id": "cell_036", "metadata": {}, "source": [ - "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.cat.codes` to get unique codes for each value." + "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.to_physical()` to get unique codes for each value.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "13e7bc66", + "id": "cell_037", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].cat.codes" + "time_df[\"datetime\"].to_physical()" ] } ], diff --git a/workflow-style.ipynb b/workflow-style.ipynb index 3de114b..ac02375 100644 --- a/workflow-style.ipynb +++ b/workflow-style.ipynb @@ -1,280 +1,280 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "95f0a171", - "metadata": {}, - "source": [ - "# Workflow: Style {#sec-workflow-style}\n", - "\n", - "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", - "\n", - "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", - "\n", - "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", - "\n", - "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" - ] - }, - { - "cell_type": "markdown", - "id": "c8111c37", - "metadata": {}, - "source": [ - "## Names\n", - "\n", - "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", - "\n", - "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", - "\n", - "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", - "\n", - "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", - "\n", - "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", - "\n", - "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", - "\n", - "In summary:\n", - "\n", - "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", - "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", - "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", - "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", - "\n", - " ```python\n", - " import random\n", - "\n", - " # This is bad\n", - " def roll():\n", - " return random.randint(0, 36) # magic number!\n", - "\n", - " # This is good\n", - " MAX_INT_VALUE = 36\n", - "\n", - " def roll():\n", - " return random.randint(0, MAX_INT_VALUE)\n", - " ```\n", - "\n", - "- use verbs for function names, eg `get_regression()`\n", - "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", - "- variable names should be snake_case and all lowercase, eg `first_name`\n", - "- class names should be CamelCase, eg `MyClass`\n", - "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", - "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", - "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", - "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" - ] - }, - { - "cell_type": "markdown", - "id": "b552c344", - "metadata": {}, - "source": [ - "## Whitespace\n", - "\n", - "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", - "\n", - "```python\n", - "this_is_a_var = 5\n", - "```\n", - "\n", - "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", - "\n", - "```python\n", - "list_var = [1, 2, 3, 4]\n", - "```\n", - "\n", - "rather than\n", - "\n", - "```python\n", - "list_var = [1,2,3,4]\n", - "# or\n", - "list_var = [1 , 2 , 3 , 4]\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "54048af1", - "metadata": {}, - "source": [ - "## Code Comments\n", - "\n", - "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", - "\n", - "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", - "\n", - "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", - "\n", - "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", - "\n", - "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", - "\n", - "```python\n", - "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", - " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", - " Args:\n", - " df (pl.DataFrame): Input dataframe\n", - " Returns:\n", - " pl.DataFrame: Dataframe of mean value of each numeric column.\n", - " \"\"\"\n", - " df_mean = df.select(S.numeric()).mean()\n", - " return df_mean\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "861f3102", - "metadata": {}, - "source": [ - "## Line width and line continuation\n", - "\n", - "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", - "\n", - "```python\n", - "def function(input_one, input_two,\n", - " input_three, input_four):\n", - " result = (input_one,\n", - " + input_two,\n", - " + input_three,\n", - " + input_four)\n", - " return result\n", - "```\n", - "\n", - "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f0f5bb37", - "metadata": {}, - "outputs": [], - "source": [ - "import polars as pl\n", - "\n", - "df = pl.DataFrame(\n", - " data={\n", - " \"col0\": [0, 0, 0, 0],\n", - " \"col1\": [0, 0, 0, 0],\n", - " \"col2\": [0, 0, 0, 0],\n", - " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", - " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", - " },\n", - ")\n", - "\n", - "\n", - "# Chaining inside parentheses works\n", - "\n", - "results = df.group_by([\"col3\", \"col4\"]).agg(\n", - " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", - ")\n", - "\n", - "results" - ] - }, - { - "cell_type": "markdown", - "id": "1d6f3bf8", - "metadata": {}, - "source": [ - "And this is what _not_ to do:\n", - "\n", - "```python\n", - "results = df\n", - " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", - "\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "d016d530", - "metadata": {}, - "source": [ - "## Principles of Clean Code\n", - "\n", - "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", - "\n", - "### Do not repeat yourself (DRY)\n", - "\n", - "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", - "\n", - "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", - "\n", - "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", - "\n", - "::: {.callout-tip title=\"Tip\"}\n", - "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", - ":::\n", - "\n", - "### KISS (Keep It Simple, Stupid)\n", - "\n", - "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", - "\n", - "### SoC (Separation of Concerns) / Make it Modular\n", - "\n", - "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", - "\n", - "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", - "\n", - "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", - "\n", - "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", - "\n", - "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", - "\n", - "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", - "\n", - "```python\n", - "# This is bad\n", - "def transform(text, uppercase):\n", - " if uppercase:\n", - " return text.upper()\n", - " else:\n", - " return text.lower()\n", - "\n", - "# This is good\n", - "def uppercase(text):\n", - " return text.upper()\n", - "\n", - "def lowercase(text):\n", - " return text.lower()\n", - "```\n" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, - "jupytext": { - "cell_metadata_filter": "-all", - "encoding": "# -*- coding: utf-8 -*-", - "formats": "md:myst", - "main_language": "python" - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - }, - "toc-showtags": true - }, - "nbformat": 4, - "nbformat_minor": 5 -} + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Workflow: Style {#sec-workflow-style}\n", + "\n", + "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", + "\n", + "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", + "\n", + "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", + "\n", + "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## Names\n", + "\n", + "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", + "\n", + "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", + "\n", + "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", + "\n", + "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", + "\n", + "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", + "\n", + "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", + "\n", + "In summary:\n", + "\n", + "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", + "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", + "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", + "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", + "\n", + " ```python\n", + " import random\n", + "\n", + " # This is bad\n", + " def roll():\n", + " return random.randint(0, 36) # magic number!\n", + "\n", + " # This is good\n", + " MAX_INT_VALUE = 36\n", + "\n", + " def roll():\n", + " return random.randint(0, MAX_INT_VALUE)\n", + " ```\n", + "\n", + "- use verbs for function names, eg `get_regression()`\n", + "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", + "- variable names should be snake_case and all lowercase, eg `first_name`\n", + "- class names should be CamelCase, eg `MyClass`\n", + "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", + "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", + "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", + "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" + ] + }, + { + "cell_type": "markdown", + "id": "2", + "metadata": {}, + "source": [ + "## Whitespace\n", + "\n", + "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", + "\n", + "```python\n", + "this_is_a_var = 5\n", + "```\n", + "\n", + "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", + "\n", + "```python\n", + "list_var = [1, 2, 3, 4]\n", + "```\n", + "\n", + "rather than\n", + "\n", + "```python\n", + "list_var = [1,2,3,4]\n", + "# or\n", + "list_var = [1 , 2 , 3 , 4]\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## Code Comments\n", + "\n", + "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", + "\n", + "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", + "\n", + "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", + "\n", + "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", + "\n", + "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", + "\n", + "```python\n", + "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", + " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", + " Args:\n", + " df (pl.DataFrame): Input dataframe\n", + " Returns:\n", + " pl.DataFrame: Dataframe of mean value of each numeric column.\n", + " \"\"\"\n", + " df_mean = df.select(S.numeric()).mean()\n", + " return df_mean\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "4", + "metadata": {}, + "source": [ + "## Line width and line continuation\n", + "\n", + "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", + "\n", + "```python\n", + "def function(input_one, input_two,\n", + " input_three, input_four):\n", + " result = (input_one,\n", + " + input_two,\n", + " + input_three,\n", + " + input_four)\n", + " return result\n", + "```\n", + "\n", + "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5", + "metadata": {}, + "outputs": [], + "source": [ + "import polars as pl\n", + "\n", + "df = pl.DataFrame(\n", + " data={\n", + " \"col0\": [0, 0, 0, 0],\n", + " \"col1\": [0, 0, 0, 0],\n", + " \"col2\": [0, 0, 0, 0],\n", + " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", + " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", + " },\n", + ")\n", + "\n", + "\n", + "# Chaining inside parentheses works\n", + "\n", + "results = df.group_by([\"col3\", \"col4\"]).agg(\n", + " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", + ")\n", + "\n", + "results" + ] + }, + { + "cell_type": "markdown", + "id": "6", + "metadata": {}, + "source": [ + "And this is what _not_ to do:\n", + "\n", + "```python\n", + "results = df\n", + " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", + "\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## Principles of Clean Code\n", + "\n", + "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", + "\n", + "### Do not repeat yourself (DRY)\n", + "\n", + "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", + "\n", + "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", + "\n", + "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", + "\n", + "::: {.callout-tip title=\"Tip\"}\n", + "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", + ":::\n", + "\n", + "### KISS (Keep It Simple, Stupid)\n", + "\n", + "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", + "\n", + "### SoC (Separation of Concerns) / Make it Modular\n", + "\n", + "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", + "\n", + "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", + "\n", + "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", + "\n", + "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", + "\n", + "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", + "\n", + "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", + "\n", + "```python\n", + "# This is bad\n", + "def transform(text, uppercase):\n", + " if uppercase:\n", + " return text.upper()\n", + " else:\n", + " return text.lower()\n", + "\n", + "# This is good\n", + "def uppercase(text):\n", + " return text.upper()\n", + "\n", + "def lowercase(text):\n", + " return text.lower()\n", + "```\n" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" + }, + "jupytext": { + "cell_metadata_filter": "-all", + "encoding": "# -*- coding: utf-8 -*-", + "formats": "md:myst", + "main_language": "python" + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + }, + "toc-showtags": true + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file From 179dd052455c7ecd362b60a9152e19154d57a41f Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 16 Jul 2026 21:38:24 +0100 Subject: [PATCH 2/3] fix(datetime): add time_zone='UTC' to to_datetime() to resolve Polars parsing error --- data-transform.ipynb | 2 +- workflow-style.ipynb | 558 +++++++++++++++++++++---------------------- 2 files changed, 280 insertions(+), 280 deletions(-) diff --git a/data-transform.ipynb b/data-transform.ipynb index 8dfc2c1..ccc75f0 100644 --- a/data-transform.ipynb +++ b/data-transform.ipynb @@ -164,7 +164,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights.with_columns(pl.col(\"time_hour\").str.to_datetime())" + "flights.with_columns(pl.col(\"time_hour\").str.to_datetime(time_zone=\"UTC\"))" ] }, { diff --git a/workflow-style.ipynb b/workflow-style.ipynb index ac02375..98b5469 100644 --- a/workflow-style.ipynb +++ b/workflow-style.ipynb @@ -1,280 +1,280 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "0", - "metadata": {}, - "source": [ - "# Workflow: Style {#sec-workflow-style}\n", - "\n", - "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", - "\n", - "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", - "\n", - "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", - "\n", - "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" - ] - }, - { - "cell_type": "markdown", - "id": "1", - "metadata": {}, - "source": [ - "## Names\n", - "\n", - "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", - "\n", - "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", - "\n", - "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", - "\n", - "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", - "\n", - "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", - "\n", - "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", - "\n", - "In summary:\n", - "\n", - "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", - "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", - "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", - "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", - "\n", - " ```python\n", - " import random\n", - "\n", - " # This is bad\n", - " def roll():\n", - " return random.randint(0, 36) # magic number!\n", - "\n", - " # This is good\n", - " MAX_INT_VALUE = 36\n", - "\n", - " def roll():\n", - " return random.randint(0, MAX_INT_VALUE)\n", - " ```\n", - "\n", - "- use verbs for function names, eg `get_regression()`\n", - "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", - "- variable names should be snake_case and all lowercase, eg `first_name`\n", - "- class names should be CamelCase, eg `MyClass`\n", - "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", - "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", - "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", - "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" - ] - }, - { - "cell_type": "markdown", - "id": "2", - "metadata": {}, - "source": [ - "## Whitespace\n", - "\n", - "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", - "\n", - "```python\n", - "this_is_a_var = 5\n", - "```\n", - "\n", - "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", - "\n", - "```python\n", - "list_var = [1, 2, 3, 4]\n", - "```\n", - "\n", - "rather than\n", - "\n", - "```python\n", - "list_var = [1,2,3,4]\n", - "# or\n", - "list_var = [1 , 2 , 3 , 4]\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "3", - "metadata": {}, - "source": [ - "## Code Comments\n", - "\n", - "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", - "\n", - "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", - "\n", - "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", - "\n", - "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", - "\n", - "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", - "\n", - "```python\n", - "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", - " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", - " Args:\n", - " df (pl.DataFrame): Input dataframe\n", - " Returns:\n", - " pl.DataFrame: Dataframe of mean value of each numeric column.\n", - " \"\"\"\n", - " df_mean = df.select(S.numeric()).mean()\n", - " return df_mean\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "4", - "metadata": {}, - "source": [ - "## Line width and line continuation\n", - "\n", - "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", - "\n", - "```python\n", - "def function(input_one, input_two,\n", - " input_three, input_four):\n", - " result = (input_one,\n", - " + input_two,\n", - " + input_three,\n", - " + input_four)\n", - " return result\n", - "```\n", - "\n", - "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5", - "metadata": {}, - "outputs": [], - "source": [ - "import polars as pl\n", - "\n", - "df = pl.DataFrame(\n", - " data={\n", - " \"col0\": [0, 0, 0, 0],\n", - " \"col1\": [0, 0, 0, 0],\n", - " \"col2\": [0, 0, 0, 0],\n", - " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", - " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", - " },\n", - ")\n", - "\n", - "\n", - "# Chaining inside parentheses works\n", - "\n", - "results = df.group_by([\"col3\", \"col4\"]).agg(\n", - " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", - ")\n", - "\n", - "results" - ] - }, - { - "cell_type": "markdown", - "id": "6", - "metadata": {}, - "source": [ - "And this is what _not_ to do:\n", - "\n", - "```python\n", - "results = df\n", - " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", - "\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "7", - "metadata": {}, - "source": [ - "## Principles of Clean Code\n", - "\n", - "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", - "\n", - "### Do not repeat yourself (DRY)\n", - "\n", - "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", - "\n", - "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", - "\n", - "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", - "\n", - "::: {.callout-tip title=\"Tip\"}\n", - "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", - ":::\n", - "\n", - "### KISS (Keep It Simple, Stupid)\n", - "\n", - "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", - "\n", - "### SoC (Separation of Concerns) / Make it Modular\n", - "\n", - "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", - "\n", - "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", - "\n", - "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", - "\n", - "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", - "\n", - "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", - "\n", - "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", - "\n", - "```python\n", - "# This is bad\n", - "def transform(text, uppercase):\n", - " if uppercase:\n", - " return text.upper()\n", - " else:\n", - " return text.lower()\n", - "\n", - "# This is good\n", - "def uppercase(text):\n", - " return text.upper()\n", - "\n", - "def lowercase(text):\n", - " return text.lower()\n", - "```\n" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, - "jupytext": { - "cell_metadata_filter": "-all", - "encoding": "# -*- coding: utf-8 -*-", - "formats": "md:myst", - "main_language": "python" - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - }, - "toc-showtags": true - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Workflow: Style {#sec-workflow-style}\n", + "\n", + "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", + "\n", + "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", + "\n", + "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", + "\n", + "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## Names\n", + "\n", + "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", + "\n", + "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", + "\n", + "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", + "\n", + "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", + "\n", + "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", + "\n", + "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", + "\n", + "In summary:\n", + "\n", + "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", + "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", + "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", + "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", + "\n", + " ```python\n", + " import random\n", + "\n", + " # This is bad\n", + " def roll():\n", + " return random.randint(0, 36) # magic number!\n", + "\n", + " # This is good\n", + " MAX_INT_VALUE = 36\n", + "\n", + " def roll():\n", + " return random.randint(0, MAX_INT_VALUE)\n", + " ```\n", + "\n", + "- use verbs for function names, eg `get_regression()`\n", + "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", + "- variable names should be snake_case and all lowercase, eg `first_name`\n", + "- class names should be CamelCase, eg `MyClass`\n", + "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", + "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", + "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", + "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" + ] + }, + { + "cell_type": "markdown", + "id": "2", + "metadata": {}, + "source": [ + "## Whitespace\n", + "\n", + "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", + "\n", + "```python\n", + "this_is_a_var = 5\n", + "```\n", + "\n", + "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", + "\n", + "```python\n", + "list_var = [1, 2, 3, 4]\n", + "```\n", + "\n", + "rather than\n", + "\n", + "```python\n", + "list_var = [1,2,3,4]\n", + "# or\n", + "list_var = [1 , 2 , 3 , 4]\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## Code Comments\n", + "\n", + "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", + "\n", + "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", + "\n", + "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", + "\n", + "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", + "\n", + "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", + "\n", + "```python\n", + "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", + " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", + " Args:\n", + " df (pl.DataFrame): Input dataframe\n", + " Returns:\n", + " pl.DataFrame: Dataframe of mean value of each numeric column.\n", + " \"\"\"\n", + " df_mean = df.select(S.numeric()).mean()\n", + " return df_mean\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "4", + "metadata": {}, + "source": [ + "## Line width and line continuation\n", + "\n", + "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", + "\n", + "```python\n", + "def function(input_one, input_two,\n", + " input_three, input_four):\n", + " result = (input_one,\n", + " + input_two,\n", + " + input_three,\n", + " + input_four)\n", + " return result\n", + "```\n", + "\n", + "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5", + "metadata": {}, + "outputs": [], + "source": [ + "import polars as pl\n", + "\n", + "df = pl.DataFrame(\n", + " data={\n", + " \"col0\": [0, 0, 0, 0],\n", + " \"col1\": [0, 0, 0, 0],\n", + " \"col2\": [0, 0, 0, 0],\n", + " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", + " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", + " },\n", + ")\n", + "\n", + "\n", + "# Chaining inside parentheses works\n", + "\n", + "results = df.group_by([\"col3\", \"col4\"]).agg(\n", + " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", + ")\n", + "\n", + "results" + ] + }, + { + "cell_type": "markdown", + "id": "6", + "metadata": {}, + "source": [ + "And this is what _not_ to do:\n", + "\n", + "```python\n", + "results = df\n", + " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", + "\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## Principles of Clean Code\n", + "\n", + "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", + "\n", + "### Do not repeat yourself (DRY)\n", + "\n", + "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", + "\n", + "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", + "\n", + "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", + "\n", + "::: {.callout-tip title=\"Tip\"}\n", + "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", + ":::\n", + "\n", + "### KISS (Keep It Simple, Stupid)\n", + "\n", + "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", + "\n", + "### SoC (Separation of Concerns) / Make it Modular\n", + "\n", + "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", + "\n", + "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", + "\n", + "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", + "\n", + "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", + "\n", + "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", + "\n", + "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", + "\n", + "```python\n", + "# This is bad\n", + "def transform(text, uppercase):\n", + " if uppercase:\n", + " return text.upper()\n", + " else:\n", + " return text.lower()\n", + "\n", + "# This is good\n", + "def uppercase(text):\n", + " return text.upper()\n", + "\n", + "def lowercase(text):\n", + " return text.lower()\n", + "```\n" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" + }, + "jupytext": { + "cell_metadata_filter": "-all", + "encoding": "# -*- coding: utf-8 -*-", + "formats": "md:myst", + "main_language": "python" + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + }, + "toc-showtags": true + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 630e0677e0bd06e9281e2b4dfc369b53bdeb0254 Mon Sep 17 00:00:00 2001 From: Uchenna Ugoh Date: Thu, 16 Jul 2026 22:28:18 +0100 Subject: [PATCH 3/3] Updated Categorical-data notebook with minor edits --- categorical-data.ipynb | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/categorical-data.ipynb b/categorical-data.ipynb index c391411..123f03d 100644 --- a/categorical-data.ipynb +++ b/categorical-data.ipynb @@ -52,9 +52,9 @@ "- they can be used efficiently with modelling packages, where they will be recognised as potential 'dummy variables', or with plotting packages, which will treat them as discrete values\n", "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\") using pl.Enum\n", "\n", - "All values of categorical data for a **polars** column are either in the given categories or take the value `None`.\n", + "All values of categorical data for a **polars** column are either in the given categories or take the value `null`.\n", "\n", - "Polars has two categorical types: \n", + "Polars has two categorical types:\n", "\n", "- pl.Categorical: Unordered categories (like pandas category)\n", "\n", @@ -135,6 +135,14 @@ "cell_type": "markdown", "id": "cell_009", "metadata": {}, + "source": [ + "`pl.Enum` is preferred when the specific categories are known upfront. `pl.Categorical` infers categories as it reads the dataframe column.\n" + ] + }, + { + "cell_type": "markdown", + "id": "5decf775", + "metadata": {}, "source": [ "We can then enter this into a data frame:\n" ] @@ -155,7 +163,7 @@ "id": "cell_011", "metadata": {}, "source": [ - "Note that `None` appears for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values.\n" + "Note that `null` appears for any value that _isn't_ in the categories we specified—you can find more on this in @sec-missing-values.\n" ] }, { @@ -383,7 +391,7 @@ "id": "cell_033", "metadata": {}, "source": [ - "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example:\n" + "And if your categorical column happens to consist of _elements_ that can undergo operations, those same operations will still work. For example:\n" ] }, { @@ -437,9 +445,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -447,7 +452,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "python4ds (3.12.12.final.0)", "language": "python", "name": "python3" }, @@ -461,7 +466,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.13" + "version": "3.12.12" }, "toc-showtags": true },