diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index 071a96b..5841632 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Exploratory Data Analysis {#sec-exploratory-data-analysis}\n", @@ -21,7 +21,7 @@ "\n", "### Prerequisites\n", "\n", - "For doing EDA, we'll use the **pandas**, **skimpy**, and **pandas-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", + "For doing EDA, we'll use the **polars**, **skimpy**, and **polars-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", "\n", "As ever, we begin by loading these packages that we'll use:" ] @@ -29,14 +29,13 @@ { "cell_type": "code", "execution_count": null, - "id": "a3377aa6", + "id": "1", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "from lets_plot import *\n", "from lets_plot.mapping import as_discrete\n", - "from pandas.api.types import CategoricalDtype\n", "from skimpy import skim\n", "\n", "LetsPlot.setup_html()" @@ -45,7 +44,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "2", "metadata": { "tags": [ "remove-cell" @@ -56,14 +55,13 @@ "import matplotlib.pyplot as plt\n", "import matplotlib_inline.backend_inline\n", "\n", - "# Plot settings\n", "plt.style.use(\"https://github.com/aeturrell/python4DS/raw/main/plot_style.txt\")\n", "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")" ] }, { "cell_type": "markdown", - "id": "e4ddb863", + "id": "3", "metadata": {}, "source": [ "## Questions\n", @@ -90,7 +88,7 @@ }, { "cell_type": "markdown", - "id": "6adeff41", + "id": "4", "metadata": {}, "source": [ "## Variation\n", @@ -103,27 +101,30 @@ { "cell_type": "code", "execution_count": null, - "id": "069caa7c", + "id": "5", "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", + "diamonds = pl.read_csv(\n", " \"https://github.com/mwaskom/seaborn-data/raw/master/diamonds.csv\"\n", ")\n", - "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n", - " CategoricalDtype(\n", - " categories=[\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"], ordered=True\n", - " )\n", - ")\n", - "diamonds[\"color\"] = diamonds[\"color\"].astype(\n", - " CategoricalDtype(categories=[\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"], ordered=True)\n", + "\n", + "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", + "diamonds_color_order = [\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"]\n", + "\n", + "diamonds = diamonds.with_columns(\n", + " [\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", + " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", + " ]\n", ")\n", + "\n", "diamonds.head()" ] }, { "cell_type": "markdown", - "id": "01f5979c", + "id": "6", "metadata": {}, "source": [ "Since `\"carat\"` is a numerical variable, we can use a histogram:" @@ -132,7 +133,7 @@ { "cell_type": "code", "execution_count": null, - "id": "97900f58", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -141,7 +142,7 @@ }, { "cell_type": "markdown", - "id": "2307ba7c", + "id": "8", "metadata": {}, "source": [ "Now that you can visualize variation, what should you look for in your plots?\n", @@ -173,18 +174,18 @@ { "cell_type": "code", "execution_count": null, - "id": "20d75550", + "id": "9", "metadata": {}, "outputs": [], "source": [ - "smaller_diamonds = diamonds.query(\"carat < 3\").copy()\n", + "smaller_diamonds = diamonds.filter(pl.col(\"carat\") < 3)\n", "\n", "(ggplot(smaller_diamonds, aes(x=\"carat\")) + geom_histogram(binwidth=0.01))" ] }, { "cell_type": "markdown", - "id": "ba20a0e2", + "id": "10", "metadata": {}, "source": [ "This histogram suggests several interesting questions:\n", @@ -211,7 +212,7 @@ }, { "cell_type": "markdown", - "id": "0626d35a", + "id": "11", "metadata": {}, "source": [ "### Unusual values\n", @@ -226,7 +227,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d9d7e995", + "id": "12", "metadata": {}, "outputs": [], "source": [ @@ -235,7 +236,7 @@ }, { "cell_type": "markdown", - "id": "05bcf733", + "id": "13", "metadata": {}, "source": [ "There are so many observations in the common bins that the rare bins are very short, making it very difficult to see them (although maybe if you stare intently at 0 you'll spot something).\n", @@ -245,7 +246,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ea8f8bf3", + "id": "14", "metadata": {}, "outputs": [], "source": [ @@ -258,7 +259,7 @@ }, { "cell_type": "markdown", - "id": "ba2b2c79", + "id": "15", "metadata": {}, "source": [ "`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis.\n", @@ -270,17 +271,19 @@ { "cell_type": "code", "execution_count": null, - "id": "e81ffb55", + "id": "16", "metadata": {}, "outputs": [], "source": [ - "unusual = diamonds.query(\"y < 3 or y > 20\").loc[:, [\"x\", \"y\", \"z\", \"price\"]]\n", + "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select(\n", + " [\"x\", \"y\", \"z\", \"price\"]\n", + ")\n", "unusual" ] }, { "cell_type": "markdown", - "id": "c9321f36", + "id": "17", "metadata": {}, "source": [ "The `\"y\"` variable measures one of the three dimensions of these diamonds, in mm.\n", @@ -297,7 +300,7 @@ }, { "cell_type": "markdown", - "id": "142d21d7", + "id": "18", "metadata": {}, "source": [ "### Exercises\n", @@ -321,7 +324,7 @@ }, { "cell_type": "markdown", - "id": "c51e8d6f", + "id": "19", "metadata": {}, "source": [ "## Unusual Values\n", @@ -345,18 +348,16 @@ { "cell_type": "code", "execution_count": null, - "id": "ecf345a7", + "id": "20", "metadata": {}, "outputs": [], "source": [ - "diamonds2 = diamonds.copy()\n", - "condition = (diamonds2[\"y\"] < 3) | (diamonds2[\"y\"] > 20)\n", - "diamonds2.loc[condition, \"y\"] = pd.NA" + "diamonds2 = diamonds.filter(~((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)))" ] }, { "cell_type": "markdown", - "id": "d26d922e", + "id": "21", "metadata": {}, "source": [ "It's not obvious where you should plot missing values, so **lets-plot** doesn't include them in the plot:" @@ -365,7 +366,7 @@ { "cell_type": "code", "execution_count": null, - "id": "15a43255", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -374,7 +375,7 @@ }, { "cell_type": "markdown", - "id": "e8ae854b", + "id": "23", "metadata": {}, "source": [ "Other times you want to understand what makes observations with missing values different to observations with recorded values.\n", @@ -386,29 +387,40 @@ { "cell_type": "code", "execution_count": null, - "id": "0a4ea922", + "id": "24", "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", + "import polars as pl\n", + "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights = pd.read_csv(url)\n", + "flights_pd = pd.read_csv(url)\n", + "flights = pl.from_pandas(flights_pd)\n", "flights.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "6849f4d9", + "id": "25", "metadata": {}, "outputs": [], "source": [ - "flights2 = flights.assign(\n", - " cancelled=lambda x: pd.isna(x[\"dep_time\"]),\n", - " sched_hour=lambda x: x[\"sched_dep_time\"] // 100,\n", - " sched_min=lambda x: x[\"sched_dep_time\"] % 100,\n", - " sched_dep_time=lambda x: x[\"sched_hour\"] + x[\"sched_min\"] / 60,\n", + "from lets_plot import *\n", + "\n", + "flights2 = flights.with_columns(\n", + " [\n", + " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", + " (\n", + " (pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60\n", + " ).alias(\"sched_dep_time\"),\n", + " ]\n", ")\n", "\n", + "\n", "(\n", " ggplot(flights2, aes(x=\"sched_dep_time\"))\n", " + geom_freqpoly(aes(color=\"cancelled\"), binwidth=1 / 4)\n", @@ -417,7 +429,7 @@ }, { "cell_type": "markdown", - "id": "b97e453a", + "id": "26", "metadata": {}, "source": [ "However this plot isn't great because there are many more non-cancelled flights than cancelled flights.\n", @@ -436,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "f63d0b9f", + "id": "27", "metadata": {}, "source": [ "## Covariation\n", @@ -453,7 +465,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e1719d8f", + "id": "28", "metadata": {}, "outputs": [], "source": [ @@ -465,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "20387b96", + "id": "29", "metadata": {}, "source": [ "The default appearance of `geom_freqpoly()` is not that useful here because the height, determined by the overall count, differs so much across cuts, making it hard to see the differences in the shapes of their distributions.\n", @@ -477,7 +489,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9388e24b", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -489,7 +501,7 @@ }, { "cell_type": "markdown", - "id": "157f63dd", + "id": "31", "metadata": {}, "source": [ "There’s something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that’s because density plots are a little hard to interpret - there’s a lot going on in this plot.\n", @@ -500,7 +512,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a3f333a6", + "id": "32", "metadata": {}, "outputs": [], "source": [ @@ -509,7 +521,7 @@ }, { "cell_type": "markdown", - "id": "b9d9ef00", + "id": "33", "metadata": {}, "source": [ "We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counter-intuitive finding that better quality diamonds are typically cheaper! In the exercises, you’ll be challenged to figure out why.\n", @@ -522,21 +534,20 @@ { "cell_type": "code", "execution_count": null, - "id": "6949db81", + "id": "34", "metadata": {}, "outputs": [], "source": [ - "mpg = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n", - ")\n", - "mpg[\"class\"] = mpg[\"class\"].astype(\"category\")\n", + "mpg = pl.read_csv(\"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\")\n", + "\n", + "mpg = mpg.with_columns(pl.col(\"class\").cast(pl.Categorical))\n", "\n", "(ggplot(mpg, aes(x=\"class\", y=\"hwy\")) + geom_boxplot())" ] }, { "cell_type": "markdown", - "id": "871aaf1c", + "id": "35", "metadata": {}, "source": [ "To make the trend easier to see, we can reorder class based on the median value of `\"hwy\"`:" @@ -545,7 +556,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a5b1ed09", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -554,7 +565,7 @@ }, { "cell_type": "markdown", - "id": "dde59236", + "id": "37", "metadata": {}, "source": [ "If you have long variable names, geom_boxplot() will work better if you flip it 90°. You can do that by adding `coord_flip()`." @@ -563,7 +574,7 @@ { "cell_type": "code", "execution_count": null, - "id": "920a4268", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -576,7 +587,7 @@ }, { "cell_type": "markdown", - "id": "299635df", + "id": "39", "metadata": {}, "source": [ "#### Exercises\n", @@ -598,7 +609,7 @@ }, { "cell_type": "markdown", - "id": "6e6a0b46", + "id": "40", "metadata": {}, "source": [ "### Two categorical variables\n", @@ -609,20 +620,24 @@ { "cell_type": "code", "execution_count": null, - "id": "68d330d2", + "id": "41", "metadata": {}, "outputs": [], "source": [ - "ct_cut_color = pd.melt(\n", - " pd.crosstab(diamonds[\"cut\"], diamonds[\"color\"]).reset_index(),\n", - " id_vars=[\"cut\"],\n", - " value_vars=diamonds[\"color\"].unique(),\n", + "ct_cut_color = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", + ")\n", + "\n", + "ct_cut_color_long = ct_cut_color.melt(\n", + " id_vars=\"cut\", variable_name=\"color\", value_name=\"value\"\n", ")" ] }, { "cell_type": "markdown", - "id": "c2ebc0c6", + "id": "42", "metadata": {}, "source": [ "Followed by visualising it with `geom_tile()`:" @@ -631,16 +646,16 @@ { "cell_type": "code", "execution_count": null, - "id": "e858cd22", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "(ggplot(ct_cut_color, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" + "(ggplot(ct_cut_color_long, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" ] }, { "cell_type": "markdown", - "id": "8ccd5c1b", + "id": "44", "metadata": {}, "source": [ "### Exercises\n", @@ -654,7 +669,7 @@ }, { "cell_type": "markdown", - "id": "b2f19afc", + "id": "45", "metadata": {}, "source": [ "### Two numerical variables\n", @@ -668,7 +683,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2afe2535", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -677,7 +692,7 @@ }, { "cell_type": "markdown", - "id": "9d8034f2", + "id": "47", "metadata": {}, "source": [ "(In this section we'll use the `smaller_diamonds` dataset to stay focused on the bulk of the diamonds that are smaller than 3 carats)\n", @@ -689,7 +704,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b55707a9", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -698,7 +713,7 @@ }, { "cell_type": "markdown", - "id": "351c22bd", + "id": "49", "metadata": {}, "source": [ "But using transparency can be challenging for very large datasets. In that case, we recommend a *binscatter*, or binned scatterplot. A binned scatterplot divides the conditioning variable, `\"carat\"` in our example, into equally sized bins or quantiles, and then plots the conditional mean of the dependent variable, `\"price\"` in our example, within each bin. Bin scatters often come with confidence intervals too. A good bin scatter package in Python is [**binsreg**](https://nppackages.github.io/binsreg/). However, bin scatters are an advanced topic, and we won't cover them here." @@ -706,7 +721,7 @@ }, { "cell_type": "markdown", - "id": "14158b48", + "id": "50", "metadata": {}, "source": [ "## **pandas** built-in tools for EDA\n", @@ -721,7 +736,7 @@ { "cell_type": "code", "execution_count": null, - "id": "13079065", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -730,7 +745,7 @@ }, { "cell_type": "markdown", - "id": "db57dbb7", + "id": "52", "metadata": {}, "source": [ "Although helpful, that sure is hard to read! We can improve this by using the `round()` method too:\n" @@ -739,17 +754,18 @@ { "cell_type": "code", "execution_count": null, - "id": "b4144440", + "id": "53", "metadata": {}, "outputs": [], "source": [ - "sum_table = diamonds.describe().round(1)\n", + "sum_table = diamonds.describe().with_columns(pl.selectors.numeric().round(1))\n", + "\n", "sum_table" ] }, { "cell_type": "markdown", - "id": "8063b8f5", + "id": "54", "metadata": {}, "source": [ "Published summary statistics tables often list one variable per row, and if your data frame has many variables, `describe()` can quickly get too wide to read easily. You can transpose it using the `T` property (or the `transpose()` method):" @@ -758,17 +774,19 @@ { "cell_type": "code", "execution_count": null, - "id": "cd2f8772", + "id": "55", "metadata": {}, "outputs": [], "source": [ - "sum_table = sum_table.T\n", - "sum_table" + "sum_table_long = sum_table.melt(\n", + " id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\"\n", + ")\n", + "sum_table_long" ] }, { "cell_type": "markdown", - "id": "e67104d3", + "id": "56", "metadata": {}, "source": [ "Of course, the stats provided in this pre-built table are not very customised. So what do we do to get the table that we actually want? Well, the answer is to draw on the contents of the previous data chapters, particularly the introduction to data analysis. Groupbys, merges, aggregations: use all of them to produce the EDA table that you want.\n", @@ -781,24 +799,25 @@ { "cell_type": "code", "execution_count": null, - "id": "5afcacbc", + "id": "57", "metadata": {}, "outputs": [], "source": [ - "(\n", - " diamonds.groupby([\"cut\", \"color\"])[\"price\"]\n", - " .mean()\n", - " .unstack()\n", - " .apply(lambda x: x / 1e3)\n", - " .fillna(\"-\")\n", - " .style.format(precision=2)\n", - " .set_caption(\"Sale price (thousands)\")\n", - ")" + "import polars.selectors as cs\n", + "\n", + "price_summary = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.col(\"price\").mean())\n", + " .pivot(index=\"cut\", on=\"color\", values=\"price\")\n", + " .with_columns(cs.numeric().round(2))\n", + ")\n", + "\n", + "price_summary" ] }, { "cell_type": "markdown", - "id": "d16e0fc1", + "id": "58", "metadata": {}, "source": [ "Although a neater one than we've seen, this is still a drab table of numbers. The eye is not immediately drawn to it!\n", @@ -811,16 +830,26 @@ { "cell_type": "code", "execution_count": null, - "id": "21e65189", + "id": "59", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.background_gradient(cmap=\"plasma\")" + "ct_cut_color = (\n", + " diamonds.group_by([\"color\", \"cut\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(\n", + " index=\"color\",\n", + " on=\"cut\",\n", + " values=\"count\",\n", + " aggregate_function=\"sum\",\n", + " )\n", + " .with_columns(cs.numeric().fill_null(0))\n", + ")" ] }, { "cell_type": "markdown", - "id": "b7c6aa71", + "id": "60", "metadata": {}, "source": [ "By default, `background_gradient()` highlights each number relative to the others in its column; you can highlight by row using `axis=1` or relative to all table values using `axis=0`. And of course `plasma` is just one of [many available colormaps](https://matplotlib.org/stable/tutorials/colors/colormaps.html)!\n", @@ -837,38 +866,35 @@ { "cell_type": "code", "execution_count": null, - "id": "bb0162ba", + "id": "61", "metadata": {}, "outputs": [], "source": [ - "(\n", - " pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"])\n", - " .style.format(precision=0)\n", - " .bar(color=\"#d65f5f\")\n", - ")" + "(ct_cut_color.to_pandas().style.format(precision=0).bar(color=\"#d65f5f\"))" ] }, { "cell_type": "markdown", - "id": "e36d4795", + "id": "62", "metadata": {}, "source": [ - "Use `.hightlight_max()`, and similar commands, to show important entries:" + "Use `max_counts()`, and similar commands, to show important entries:" ] }, { "cell_type": "code", "execution_count": null, - "id": "5d19072c", + "id": "63", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.highlight_max().format(\"{:.0f}\")" + "max_counts = ct_cut_color.select(pl.exclude(\"color\").max())\n", + "max_counts" ] }, { "cell_type": "markdown", - "id": "63075da4", + "id": "64", "metadata": {}, "source": [ "You can find a full set of styling commands [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Styling)." @@ -876,10 +902,10 @@ }, { "cell_type": "markdown", - "id": "d8b6d225", + "id": "65", "metadata": {}, "source": [ - "### Exploratory Plotting with **pandas**\n", + "### Exploratory Plotting with **polars**\n", "\n", "**pandas** has some built-in plotting options to help you look at data quickly. These can be accessed via `.plot.*` or `.plot()`, depending on the context. Let's make a quick `.plot()` using a dataset on taxis." ] @@ -887,48 +913,50 @@ { "cell_type": "code", "execution_count": null, - "id": "b479d5b1", + "id": "66", "metadata": {}, "outputs": [], "source": [ - "taxis = pd.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", - "# turn the pickup time column into a datetime\n", - "taxis[\"pickup\"] = pd.to_datetime(taxis[\"pickup\"])\n", - "# set some other columns types\n", - "taxis = taxis.astype(\n", - " {\n", - " \"dropoff\": \"datetime64[ns]\",\n", - " \"pickup\": \"datetime64[ns]\",\n", - " \"color\": \"category\",\n", - " \"payment\": \"category\",\n", - " \"pickup_zone\": \"string\",\n", - " \"dropoff_zone\": \"string\",\n", - " \"pickup_borough\": \"category\",\n", - " \"dropoff_borough\": \"category\",\n", - " }\n", + "taxis = pl.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", + "\n", + "taxis = taxis.with_columns(\n", + " pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\")\n", + ")\n", + "\n", + "taxis = taxis.with_columns(\n", + " [\n", + " pl.col(\"color\").cast(pl.Categorical),\n", + " pl.col(\"payment\").cast(pl.Categorical),\n", + " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", + " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", + " ]\n", ")\n", + "\n", "taxis.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "ee971c9c", + "id": "67", "metadata": {}, "outputs": [], "source": [ - "taxis.info()" + "taxis.describe()" ] }, { "cell_type": "code", "execution_count": null, - "id": "2015b1dc", + "id": "68", "metadata": {}, "outputs": [], "source": [ + "taxis_pd = taxis.to_pandas()\n", + "taxis_pd[\"pickup\"] = pd.to_datetime(taxis_pd[\"pickup\"])\n", + "\n", "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[\"total\"]\n", " .mean()\n", " .plot(\n", @@ -936,12 +964,13 @@ " xlabel=\"\",\n", " ylabel=\"Fare (USD)\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "b9c0f990", + "id": "69", "metadata": {}, "source": [ "Again, if you can get the data in the right shape, you can plot it. The same function works with multiple lines\n" @@ -950,12 +979,12 @@ { "cell_type": "code", "execution_count": null, - "id": "51e86185", + "id": "70", "metadata": {}, "outputs": [], "source": [ "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[[\"fare\", \"tip\", \"tolls\"]]\n", " .mean()\n", " .plot(\n", @@ -964,12 +993,13 @@ " xlabel=\"\",\n", " ylabel=\"USD\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "cc227b81", + "id": "71", "metadata": {}, "source": [ "Now let's see some of the other quick `.plot.*` options." @@ -977,7 +1007,7 @@ }, { "cell_type": "markdown", - "id": "1c4278ac", + "id": "72", "metadata": {}, "source": [ "A bar chart (use `barh` for horizontal orientation; `rot` sets rotation of labels):" @@ -986,16 +1016,31 @@ { "cell_type": "code", "execution_count": null, - "id": "79ceca92", + "id": "73", "metadata": {}, "outputs": [], "source": [ - "taxis.value_counts(\"payment\").sort_index().plot.bar(title=\"Counts\", rot=0);" + "payment_counts = (\n", + " taxis.filter(pl.col(\"payment\").is_not_null())\n", + " .group_by(\"payment\")\n", + " .agg(pl.len().alias(\"count\"))\n", + " .sort(\"payment\")\n", + ")\n", + "\n", + "payment_counts.to_pandas().plot.bar(\n", + " x=\"payment\",\n", + " y=\"count\",\n", + " legend=False,\n", + " rot=0,\n", + " title=\"Counts\",\n", + ")\n", + "\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "4e6f1cc2", + "id": "74", "metadata": {}, "source": [ "This next one, uses `.plot.hist()` to create a histogram." @@ -1004,16 +1049,17 @@ { "cell_type": "code", "execution_count": null, - "id": "5efc5817", + "id": "75", "metadata": {}, "outputs": [], "source": [ - "taxis[\"tip\"].plot.hist(bins=30, title=\"Tip\");" + "(taxis_pd[\"tip\"].plot.hist(bins=30, title=\"Tip\"))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "3f15bfa7", + "id": "76", "metadata": {}, "source": [ "Boxplot:" @@ -1022,16 +1068,17 @@ { "cell_type": "code", "execution_count": null, - "id": "0b735d15", + "id": "77", "metadata": {}, "outputs": [], "source": [ - "(taxis[[\"fare\", \"tolls\", \"tip\"]].plot.box());" + "(taxis_pd[[\"fare\", \"tolls\", \"tip\"]].plot.box())\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "67017bef", + "id": "78", "metadata": {}, "source": [ "Scatter plot:" @@ -1040,16 +1087,17 @@ { "cell_type": "code", "execution_count": null, - "id": "66adada2", + "id": "79", "metadata": {}, "outputs": [], "source": [ - "taxis.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None));" + "(taxis_pd.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None)))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "9766d2c4", + "id": "80", "metadata": {}, "source": [ "## Other tools for EDA\n", @@ -1059,7 +1107,7 @@ }, { "cell_type": "markdown", - "id": "813ec70a", + "id": "81", "metadata": {}, "source": [ "### **skimpy** for summary statistics\n", @@ -1072,7 +1120,7 @@ { "cell_type": "code", "execution_count": null, - "id": "32796b5f", + "id": "82", "metadata": {}, "outputs": [], "source": [ @@ -1081,7 +1129,7 @@ }, { "cell_type": "markdown", - "id": "f2810c9e", + "id": "83", "metadata": {}, "source": [ "## Summary\n", @@ -1095,9 +1143,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -1105,7 +1150,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" },