diff --git a/categorical-data.ipynb b/categorical-data.ipynb index ab6ba30..123f03d 100644 --- a/categorical-data.ipynb +++ b/categorical-data.ipynb @@ -2,25 +2,21 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "cell_000", "metadata": {}, "source": [ "# Categorical Data {#sec-categorical-data}\n", "\n", "## Introduction\n", "\n", - "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **pandas** [documentation](https://pandas.pydata.org/).\n" + "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **polars** [documentation](https://docs.pola.rs/api/python/stable/reference/index.html).\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "51a55374", - "metadata": { - "tags": [ - "remove-cell" - ] - }, + "id": "cell_001", + "metadata": {}, "outputs": [], "source": [ "# remove cell\n", @@ -34,248 +30,273 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "cell_002", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package.\n" ] }, { "cell_type": "markdown", - "id": "8757f369", + "id": "cell_003", "metadata": {}, "source": [ "## The Category Datatype\n", "\n", - "Everything in Python has a type, even the data in **pandas** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical`. There are some benefits to using categorical variables (where appropriate):\n", + "Everything in Python has a type, even the data in **polars** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical` and `Enum`. There are some benefits to using categorical variables (where appropriate):\n", "\n", "- they can keep track even when elements of the category isn't present, which can sometimes be as interesting as when they are (imagine you find no-one from a particular school goes to university)\n", "- they can use vastly less of your computer's memory than encoding the same information in other ways\n", "- they can be used efficiently with modelling packages, where they will be recognised as potential 'dummy variables', or with plotting packages, which will treat them as discrete values\n", - "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\")\n", + "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\") using pl.Enum\n", + "\n", + "All values of categorical data for a **polars** column are either in the given categories or take the value `null`.\n", + "\n", + "Polars has two categorical types:\n", "\n", - "All values of categorical data for a **pandas** column are either in the given categories or take the value `np.nan`. \n", + "- pl.Categorical: Unordered categories (like pandas category)\n", + "\n", + "- pl.Enum: Ordered categories with a fixed set of values (similar to pandas ordered categorical)\n", "\n", "## Creating Categorical Data\n", "\n", - "Let's create a categorical column of data:" + "Let's create a categorical column of data:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "535ef959", + "id": "cell_004", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", + "df = pl.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", "\n", - "df[\"A\"] = df[\"A\"].astype(\"category\")\n", + "df = df.with_columns(pl.col(\"A\").cast(pl.Categorical))\n", "df[\"A\"]" ] }, { "cell_type": "markdown", - "id": "62db4dec", + "id": "cell_005", "metadata": {}, "source": [ - "Notice that we get some additional information at the bottom of the shown series: we get told that not only is this a categorical column type, but it has three values 'a', 'b', and 'c'.\n", + "Notice that polars tracks the categorical type, though the display is more compact than pandas.\n", "\n", - "You can also use special functions, such as `pd.cut()`, to groups data into discrete bins. Here's an example where specify the labels for the categories directly:\n" + "You can also use special expression methods, such as pl.col().cut(), to group data into discrete bins. Here's an example where we specify the labels for each category directly.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "358c83bb", + "id": "cell_006", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", + "df = pl.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", "labels = [f\"{i} - {i+9}\" for i in range(0, 100, 10)]\n", - "df[\"group\"] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)\n", + "df = df.with_columns(\n", + " pl.col(\"value\")\n", + " .cut(breaks=list(range(10, 100, 10)), labels=labels, left_closed=True)\n", + " .alias(\"group\")\n", + ")\n", "df.head()" ] }, { "cell_type": "markdown", - "id": "87bc56b6", + "id": "cell_007", "metadata": {}, "source": [ "In the example above, the `group` column is of categorical type.\n", "\n", - "Another way to create a categorical variable is directly using the `pd.Categorical()` function:" + "Another way to create a categorical variable is directly using the `pl.Categorical` or `pl.Enum` types:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "fb389105", + "id": "cell_008", "metadata": {}, "outputs": [], "source": [ - "raw_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"], categories=[\"b\", \"c\", \"d\"]\n", + "raw_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"b\", \"c\", \"d\"]), strict=False\n", ")\n", "raw_cat" ] }, { "cell_type": "markdown", - "id": "ebfdf815", + "id": "cell_009", + "metadata": {}, + "source": [ + "`pl.Enum` is preferred when the specific categories are known upfront. `pl.Categorical` infers categories as it reads the dataframe column.\n" + ] + }, + { + "cell_type": "markdown", + "id": "5decf775", "metadata": {}, "source": [ - "We can then enter this into a data frame:" + "We can then enter this into a data frame:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "0497fc16", + "id": "cell_010", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(raw_cat, columns=[\"cat_type\"])\n", + "df = pl.DataFrame({\"cat_type\": raw_cat})\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "76dc9dd6", + "id": "cell_011", "metadata": {}, "source": [ - "Note that NaNs appear for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values." + "Note that `null` appears for any value that _isn't_ in the categories we specified—you can find more on this in @sec-missing-values.\n" ] }, { "cell_type": "markdown", - "id": "aa76a795", + "id": "cell_012", "metadata": {}, "source": [ - "You can also create ordered categories:" + "You can also create ordered categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f7520d3d", + "id": "cell_013", "metadata": {}, "outputs": [], "source": [ - "ordered_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"],\n", - " categories=[\"a\", \"b\", \"c\", \"d\"],\n", - " ordered=True,\n", + "ordered_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"a\", \"b\", \"c\", \"d\"])\n", ")\n", "ordered_cat" ] }, { "cell_type": "markdown", - "id": "258cc6f5", + "id": "cell_014", "metadata": {}, "source": [ "## Working with Categories\n", "\n", - "Categorical data has a `categories` and a `ordered` property; these list the possible values and whether the ordering matters or not respectively. These properties are exposed as `.cat.categories` and `.cat.ordered`. If you don’t manually specify categories and ordering, they are inferred from the passed arguments.\n", + "Categorical data in polars has categories and ordering properties. For `pl.Enum`, the categories are fixed and ordered. For `pl.Categorical`, categories are not ordered and can be more flexible.\n", "\n", - "Let's see some examples:" + "Let's see some examples:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "2caba354", + "id": "cell_015", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.categories" + "df[\"cat_type\"].dtype.categories" ] }, { "cell_type": "code", "execution_count": null, - "id": "5f1fe093", + "id": "cell_016", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.ordered" + "isinstance(df[\"cat_type\"].dtype, pl.Enum)" ] }, { "cell_type": "markdown", - "id": "694c07b0", + "id": "cell_017", "metadata": {}, "source": [ - "If categorical data is ordered (ie `.cat.ordered == True`), then the order of the categories has a meaning and certain operations are possible: you can sort values (with `.sort_values`), and apply `.min` and `.max`." + "If categorical data is ordered (ie `pl.Enum`), then the order of the categories has a meaning and certain operations are possible: you can sort values, and apply `.min()` and `.max()`.\n" ] }, { "cell_type": "markdown", - "id": "0a6d7a90", + "id": "cell_018", "metadata": {}, "source": [ "### Renaming Categories\n", "\n", - "Renaming categories is done via the `rename_categories()` method (which works with a list or a dictionary)." + "Renaming categories in polars requires re-casting with new categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "097171b8", + "id": "cell_019", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.rename_categories([\"alpha\", \"beta\", \"gamma\"])" + "df = df.with_columns(\n", + " pl.col(\"cat_type\")\n", + " .cast(pl.String)\n", + " .replace({\"b\": \"alpha\", \"c\": \"beta\", \"d\": \"gamma\"})\n", + " .cast(pl.Enum([\"alpha\", \"beta\", \"gamma\"]))\n", + ")" ] }, { "cell_type": "markdown", - "id": "3de7fd23", + "id": "cell_020", "metadata": {}, "source": [ - "Quite often, you'll run into a situation where you want to add a category. You can do this with `.add_categories()`:" + "To add categories, you need to re-cast with an extended Enum:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4ae6df38", + "id": "cell_021", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.add_categories([\"delta\"])\n", + "df = df.with_columns(\n", + " pl.col(\"cat_type\").cast(pl.Enum([\"alpha\", \"beta\", \"gamma\", \"delta\"]))\n", + ")\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "ba525491", + "id": "cell_022", "metadata": {}, "source": [ - "Similarly, there is a `.remove_categories()` function and a `.remove_unused_categories()` function. `.set_categories` adds and removes categories in one fell swoop. One of the nice properties of set categories is that Remember that you need to do `df[\"columnname\"].cat` before calling any cat(egory) functions though." + "Because Enum categories are fixed in Polars, you cannot remove categories directly. Instead, ensure that the data contains only the desired values, then re-cast the column to a new Enum containing just those categories.\n" ] }, { "cell_type": "markdown", - "id": "76b38a2a", + "id": "cell_023", "metadata": {}, "source": [ "## Operations on Categories\n", "\n", - "As noted, ordered categories will already undergo some operations. But there are some that work on any set of categories. Perhaps the most useful is `value_counts()`" + "`Enum` columns behave like any other Polars column, but they also retain the complete set of allowed categories. This means Polars remembers categories even if no rows currently contain them.\n", + "\n", + "For example, suppose we define an `Enum` with five possible categories, but only use four of them:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "19f5bdda", + "id": "cell_024", "metadata": {}, "outputs": [], "source": [ @@ -284,42 +305,112 @@ }, { "cell_type": "markdown", - "id": "a36db155", + "id": "cell_025", "metadata": {}, "source": [ - "Note that even though 'delta' doesn't appear at all, it gets a count (of zero). This tracking of missing values can be quite handy.\n", + "Notice that \"delta\" is not shown because value_counts() only reports values that actually occur in the data.\n", "\n", - "`mode()` is another one:" + "However, the Enum still knows that \"delta\" is a valid category.\n", + "\n", + "You can inspect the complete set of categories stored in the Enum:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f52d5d0d", + "id": "cell_026", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].mode()" + "df[\"cat_type\"].dtype.categories" ] }, { "cell_type": "markdown", - "id": "d28f34d8", + "id": "cell_027", "metadata": {}, "source": [ - "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example," + "If you want a frequency table that includes unused categories, combine the stored category list with the observed counts:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4d43e94d", + "id": "cell_028", "metadata": {}, "outputs": [], "source": [ - "time_df = pd.DataFrame(\n", - " pd.Series(pd.date_range(\"2015/05/01\", periods=5, freq=\"M\"), dtype=\"category\"),\n", - " columns=[\"datetime\"],\n", + "all_categories = pl.DataFrame(\n", + " {\"cat_type\": df[\"cat_type\"].dtype.categories.cast(df[\"cat_type\"].dtype)}\n", + ")\n", + "all_categories.join(df[\"cat_type\"].value_counts(), on=\"cat_type\", how=\"left\").fill_null(\n", + " 0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "cell_029", + "metadata": {}, + "source": [ + "This demonstrates one advantage of `Enum`: the complete set of allowed categories is retained, even when some categories are not present in the data.\n", + "\n", + "You can also inspect the datatype of the column:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_030", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"cat_type\"].dtype" + ] + }, + { + "cell_type": "markdown", + "id": "cell_031", + "metadata": {}, + "source": [ + "You can compute the most frequent category:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_032", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"cat_type\"].drop_nulls().mode()" + ] + }, + { + "cell_type": "markdown", + "id": "cell_033", + "metadata": {}, + "source": [ + "And if your categorical column happens to consist of _elements_ that can undergo operations, those same operations will still work. For example:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_034", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "time_df = pl.DataFrame(\n", + " {\n", + " \"datetime\": pl.date_range(\n", + " datetime.date(2015, 5, 31), datetime.date(2015, 9, 30), \"1mo\", eager=True\n", + " )\n", + " .cast(pl.String)\n", + " .cast(pl.Categorical)\n", + " }\n", ")\n", "time_df" ] @@ -327,36 +418,33 @@ { "cell_type": "code", "execution_count": null, - "id": "db697f86", + "id": "cell_035", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].dt.month" + "time_df[\"datetime\"].cast(pl.String).str.to_date().dt.month()" ] }, { "cell_type": "markdown", - "id": "c202f0d1", + "id": "cell_036", "metadata": {}, "source": [ - "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.cat.codes` to get unique codes for each value." + "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.to_physical()` to get unique codes for each value.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "13e7bc66", + "id": "cell_037", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].cat.codes" + "time_df[\"datetime\"].to_physical()" ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -364,7 +452,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "python4ds (3.12.12.final.0)", "language": "python", "name": "python3" }, @@ -378,7 +466,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.13" + "version": "3.12.12" }, "toc-showtags": true }, diff --git a/data-transform.ipynb b/data-transform.ipynb index 8dfc2c1..ccc75f0 100644 --- a/data-transform.ipynb +++ b/data-transform.ipynb @@ -164,7 +164,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights.with_columns(pl.col(\"time_hour\").str.to_datetime())" + "flights.with_columns(pl.col(\"time_hour\").str.to_datetime(time_zone=\"UTC\"))" ] }, { diff --git a/workflow-style.ipynb b/workflow-style.ipynb index 3de114b..98b5469 100644 --- a/workflow-style.ipynb +++ b/workflow-style.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Workflow: Style {#sec-workflow-style}\n", @@ -18,7 +18,7 @@ }, { "cell_type": "markdown", - "id": "c8111c37", + "id": "1", "metadata": {}, "source": [ "## Names\n", @@ -68,7 +68,7 @@ }, { "cell_type": "markdown", - "id": "b552c344", + "id": "2", "metadata": {}, "source": [ "## Whitespace\n", @@ -96,7 +96,7 @@ }, { "cell_type": "markdown", - "id": "54048af1", + "id": "3", "metadata": {}, "source": [ "## Code Comments\n", @@ -126,7 +126,7 @@ }, { "cell_type": "markdown", - "id": "861f3102", + "id": "4", "metadata": {}, "source": [ "## Line width and line continuation\n", @@ -149,7 +149,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f0f5bb37", + "id": "5", "metadata": {}, "outputs": [], "source": [ @@ -177,7 +177,7 @@ }, { "cell_type": "markdown", - "id": "1d6f3bf8", + "id": "6", "metadata": {}, "source": [ "And this is what _not_ to do:\n", @@ -191,7 +191,7 @@ }, { "cell_type": "markdown", - "id": "d016d530", + "id": "7", "metadata": {}, "source": [ "## Principles of Clean Code\n",