Hot Take: There Are Only Four Data Visualization PatternsーThe Rest Are Just Fancy Pansies

Here me out. Data visualization is one of my biggest hobbies. In fact, back in 2019, I reproduced all 60+ plots from the Financial Times Visual Vocabulary using ggplot’s Python port plotnine. Just because I can. However, one thing I noticed since moving to Big Retail is that people really go out of their ways to prioritize substance over flair. We are famous for writing documents over slides; but I have never heard someone mentioned that the most common data visualization instrument is a table. I believe this is because if you are given time to carefully read and understand (10-30 minutes doc read vs 1 minute per slide), you can comprehend what even the most simplistic visualization like a table is trying to tell you. My premise is that if our goal is to tell a meaningful story with data, to a reasonable human being given reasonable time to consume, there are only four data visualization patterns you need: line plots, bar charts, histograms, and scatter plots.

featured_image

Let us reuse the transaction-level UCI Online Retail dataset with category column we added using LLM in the sales prediction post. Imagine you are a consultant the UK shop owner hired to have a look at their business during a time period. What are the key visualizations you need to understand the data and make critical suggestions for improvement?

TL;DR

• Line plots display changes over time.

• Bar charts compare numerical values among categorical variables.

• Histograms show how a numerical variable is distributed; box/violin/swarm plots are basically putting histograms next to one another.

• Scatter plots demonstrate relationship between two numerical variables; heatmaps do the same for two discrete variables.

from plotnine import *
from mizani.labels import *
from mizani.formatters import *

import seaborn as sns
import plotly.express as px
import matplotlib.pyplot as plt

import pandas as pd
import numpy as np
from tqdm.auto import tqdm
import datetime

from ucimlrepo import fetch_ucirepo 

def string_to_yearmon(date):
    date = date.split()
    date = date[0].split('/') + date[1].split(':')
    date = date[2] + '-' + date[0].zfill(2)
    return date

def string_to_date(date):
    date = date.split()
    date = date[0].split('/') + date[1].split(':')
    date = date[2] + '-' + date[0].zfill(2) + '-' + date[1].zfill(2) #+ ' ' + date[3].zfill(2) + ':' + date[4].zfill(2)
    return date

def date_to_dow(date_string):
  dt_object = datetime.datetime.strptime(date_string, '%Y-%m-%d')
  dow_number = dt_object.strftime('%w') 
  dow_name = dt_object.strftime('%A')
  return f"{dow_number}_{dow_name}"

online_retail = fetch_ucirepo(id=352) 
transaction_df = online_retail['data']['original']
original_nb = transaction_df.shape[0]

#create yearmon for train-valid split
transaction_df['yearmon'] = transaction_df.InvoiceDate.map(string_to_yearmon)
transaction_df['invoice_date'] = transaction_df.InvoiceDate.map(string_to_date)
transaction_df['dow'] = transaction_df.invoice_date.map(date_to_dow)

#get rid of transactions without cid
transaction_df = transaction_df[~transaction_df.CustomerID.isna()].reset_index(drop=True)
has_cid_nb = transaction_df.shape[0]

#fill in unknown descriptions
transaction_df.Description = transaction_df.Description.fillna('UNKNOWN')

#convert customer id to string
transaction_df['CustomerID'] = transaction_df['CustomerID'].map(lambda x: str(int(x)))

#simplify by filtering unit price and quantity to be non-zero (get rid of discounts, cancellations, etc)
transaction_df = transaction_df[(transaction_df.UnitPrice>0)&\
                                (transaction_df.Quantity>0)].reset_index(drop=True)
has_sales_nb = transaction_df.shape[0]

#add sales
transaction_df['Sales'] = transaction_df.UnitPrice * transaction_df.Quantity

#clean description
transaction_df['Description'] = transaction_df['Description'].map(lambda x: x.replace('.','').strip())

#attach category
product_description_category = pd.read_csv('../../data/sales_prediction/product_description_category.csv', sep='|')
transaction_category_df = transaction_df.merge(product_description_category,left_on='Description', right_on='product_description', how='left')
#fill in unknown if no category
transaction_category_df['category'] = transaction_category_df['category'].fillna('Unknown')

#focus on UK store; we have incomplete data for 2011-12 so we let's remove it
df = transaction_category_df[(transaction_category_df.Country=='United Kingdom')&
   (transaction_category_df.yearmon<'2011-12')]\
  .drop(['Country','Description','InvoiceDate'], axis=1)\
  .reset_index(drop=True)

#reshuffle columns
df = df[[
  'invoice_date',
  'InvoiceNo',
  'CustomerID',
  'StockCode',
  'product_description',
  'category',
  'UnitPrice',
  'Quantity',
  'Sales',
  'yearmon',
  'dow'
]]

df.tail(5)

	invoice_date	InvoiceNo	CustomerID	StockCode	product_description	category	UnitPrice	Quantity	Sales	yearmon	dow
340668	2011-11-30	579885	15444	85034C	3 ROSE MORRIS BOXED CANDLES	Home Decor	1.25	4	5.00	2011-11	3_Wednesday
340669	2011-11-30	579885	15444	21742	LARGE ROUND WICKER PLATTER	Kitchen and Dining	5.95	2	11.90	2011-11	3_Wednesday
340670	2011-11-30	579885	15444	23084	RABBIT NIGHT LIGHT	Home Decor	2.08	6	12.48	2011-11	3_Wednesday
340671	2011-11-30	579885	15444	21257	VICTORIAN SEWING BOX MEDIUM	Stationary and Gifts	7.95	1	7.95	2011-11	3_Wednesday
340672	2011-11-30	579885	15444	21259	VICTORIAN SEWING BOX SMALL	Home Decor	5.95	1	5.95	2011-11	3_Wednesday

Before the exploration, we need to know the scope of the dataset. Since the dataset only contains one year’s worth of transaction records, we will not be able to perform any analysis regarding conversions (since we do not have data on those who visited but DID NOT make a purchase), profitability (since costs are not available), and year-on-year growth (since we only have one year of data).

print(f'''
Time period: {df.invoice_date.min()} to {df.invoice_date.max()}
Number of unique customers: {df.CustomerID.nunique()}
Number of unique products: {df.StockCode.nunique()}
Number of unique categories: {df.category.nunique()}
Number of unique transactions: {df.InvoiceNo.nunique()}
Total Sales: {df.Sales.sum()}
''')

Time period: 2010-12-01 to 2011-11-30
Number of unique customers: 3886
Number of unique products: 3636
Number of unique categories: 10
Number of unique transactions: 15940
Total Sales: 6872399.784

With that out of the way, let us get exploring.

Business at a Glance

The first thing we want to look at is how the 6.9M GBP total sales is spread out during the year. We see the typical year-end peak season in retail. Even without year-on-year data, by comparing 2010-12 and 2011-11, we can see that the current year’s peak is likely 2x the previous year’s. Business seems to be going well overall.

agg = df.groupby('yearmon').Sales.sum().reset_index().sort_values('yearmon')
g = (ggplot(agg, aes(x='yearmon',y='Sales', group=1)) 
  + geom_point(color='darkred') + geom_line(color='darkred') 
  + xlab('Month') + ylab('Sales') 
  + scale_y_continuous(breaks = [i*100_000 for i in range(11)], 
    limits=(0, None), labels=label_comma())
  + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g 

Looking at the same line plot by category, we can see the shop focuses on Home Decor and Kitchen and Dining. During peak season, gift-worthy categories namely Seasonal and Holiday, Personal Care and Wellness, and Toys and Games made substantial jump in terms of sales.

agg = df.groupby(['category','yearmon']).Sales.sum().reset_index()
g = (ggplot(agg, 
            aes(x='yearmon',y='Sales', color='category', group='category')) 
  + geom_point() + geom_line() 
  + xlab('Month') + ylab('Sales') 
  + scale_y_continuous(limits=(0, None),
    breaks = [i*50_000 for i in range(7)], 
    labels=label_comma())
  + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g 

The seasonality of each category is even more transparent when we look at the faceted line plots. For instance, while most categories follow the peak-at-year-end pattern, we notice Outdoor and Garden had its peak during the spring months and Kitchen and Dining has stable sales throughout the year.

agg = df[df.category!='Unknown'].groupby(['category','yearmon']).Sales.sum().reset_index()
g = (ggplot(agg, aes(x='yearmon',y='Sales', 
     color='category', group='category')) 
  + geom_point() + geom_line() 
  + facet_wrap('~ category', scales='free_y') + guides(color="none") 
  + scale_y_continuous(labels=label_comma())
  + xlab('Month') + ylab('Sales') 
  + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g 

While we are at seasonality, let us look at average sales by day of week to see which ones are the most and least busy. If the shop owner has not told us, we now know that they close on Saturdays and usually have a slow Sunday. Perhaps we can consider promotions specific to Sundays to smoothen the sales during the week.

agg = df.groupby(['invoice_date','dow']).Sales.sum().reset_index()\
  .groupby('dow').Sales.mean().reset_index()
g = (ggplot(agg, aes(x='dow',y='Sales',group=1)) 
  + geom_point(color='darkred') + geom_line(color='darkred') 
  + scale_y_continuous(limits=(0, None), labels=label_comma())
  + xlab('Day of Week') + ylab('Average Sales') 
  + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g 

Another useful line to plot is to make sure our cumulative customer base is growing, which is fortunately the case especially during peak season.

customer_base = df[['CustomerID','yearmon']].drop_duplicates().reset_index()
cumulative_customers = []
cumulative_customers_count = {}
for ym in df.yearmon.unique().tolist():
  ym_customers = customer_base[customer_base.yearmon==ym].CustomerID
  cumulative_customers = list(set(ym_customers) | set(cumulative_customers))
  cumulative_customers_count[ym] = len(cumulative_customers)
agg = pd.DataFrame(cumulative_customers_count.items())
agg.columns = ['yearmon','cumulative_customers']

g = (ggplot(agg, aes(x='yearmon',y='cumulative_customers',group=1)) 
  + geom_point(color='darkred') + geom_line(color='darkred') 
  + scale_y_continuous(limits=(0, None), labels=label_comma())
  + xlab('Month') + ylab('Cumulative Purchasing Customers') 
  + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g 

Likewise, we also see our selection (number of unique products) expanding rapidly, which is a good sign that we provide more choices to customers.

product_selection = df[['StockCode','yearmon']].drop_duplicates().reset_index()
cumulative_products = []
cumulative_products_count = {}
for ym in df.yearmon.unique().tolist():
  ym_products = product_selection[product_selection.yearmon==ym].StockCode
  cumulative_products = list(set(ym_products) | set(cumulative_products))
  cumulative_products_count[ym] = len(cumulative_products)
agg = pd.DataFrame(cumulative_products_count.items())
agg.columns = ['yearmon','cumulative_products']

g = (ggplot(agg, aes(x='yearmon',y='cumulative_products',group=1)) 
  + geom_point(color='darkred') + geom_line(color='darkred') 
  + scale_y_continuous(limits=(0,4000), labels=label_comma())
  + xlab('Month') + ylab('Cumulative Number of Products')
  + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g 

Product Deep Dive

We now turn our attention to the product offerings. The bar charts reaffirm our focus on Home Decor and Kitchen and Dining with over half of total sales coming from those categories. We see similar tendencies for number of unique products per category.

agg = df.groupby('category').Sales.sum().reset_index()\
        .sort_values('Sales', ascending=False)\
        .reset_index(drop=True)
agg['sales_pct'] = agg.Sales / agg.Sales.sum()

agg['category'] = pd.Categorical(agg['category'], categories=agg['category'], ordered=True)

g = (ggplot(agg, aes(x='category', y='Sales',label='sales_pct'))
    + geom_col(fill='darkred')
    + geom_text(
        nudge_y=0.01,
        va='bottom', 
        format_string='{:.1%}',
    )
     + xlab('Category') + ylab('Sales') 
     + scale_y_continuous(limits=(0,2500000), labels=label_comma())
     + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1))
     )
g

agg = df.groupby('category').StockCode.nunique().reset_index()\
        .sort_values('StockCode', ascending=False)\
        .reset_index(drop=True)
agg['stock_code_pct'] = agg.StockCode / agg.StockCode.sum()

agg['category'] = pd.Categorical(agg['category'], categories=agg['category'], ordered=True)

g = (ggplot(agg, aes(x='category', y='StockCode',label='stock_code_pct'))
    + geom_col(fill='darkred')
    + geom_text(
        nudge_y=0.01,
        va='bottom', 
        format_string='{:.1%}',
    )
     + xlab('Category') + ylab('Number of Unique Products') 
     + scale_y_continuous(limits=(0, 1200), labels=label_comma())
     + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1))
     )
g

By comparing share of sales against share of number of unique products in each category, we can identify which categories punch above their weights. The diagonal line represents a fair share; for instance, if a category has 20% of the products, it should contribute 20% of sales. Categories above the line earn more than their product count would suggest and vice versa. Now, each category has a different sales-vs-product-count dynamics, so it does that mean that we should be looking to cut all tail products in categories under the line. For example, Stationary and Gifts might have organically higher product count than Kitchen and Dining. Nonetheless, it is worth having a look when the imbalance is too extreme to streamline our inventory.

agg_sales = df.groupby('category').Sales.sum().reset_index()
agg_sales['sales_pct'] = agg_sales.Sales / agg_sales.Sales.sum()

agg_stock = df.groupby('category').StockCode.nunique().reset_index()
agg_stock['stock_pct'] = agg_stock.StockCode / agg_stock.StockCode.sum()

agg = agg_sales[['category','sales_pct']].merge(agg_stock[['category','stock_pct']], on='category')

g = (ggplot(agg, aes(x='stock_pct', y='sales_pct', label='category'))
    + geom_point(color='darkred', size=3)
    + geom_abline(intercept=0, slope=1, linetype='dashed', color='grey')
    + geom_text(nudge_y=0.02, size=7)
    + xlab('Share of Number of Products')
    + ylab('Share of Sales')
    + scale_x_continuous(labels=percent_format())
    + scale_y_continuous(labels=percent_format())
    + theme_538())
g

Without stock purchase information, we cannot calculate inventory turnover. Nonetheless, we can still look at the cumulative sales percentage to determine the head products, about 750 items (about 20% of all unique products) that contribute to 80% of the sales. These head products are the main reason customers visit our shop; they form the basis of our product strategy. We either shift investments from tail products to these head products if their profit margins are high enough, or use them as gateway products for customers to purchase from other profitable items.

agg = df.groupby('StockCode')\
        .agg({'product_description': lambda x: x.unique()[-1], 
        'category': lambda x: x.unique()[-1],
        'Sales':'sum'})\
        .reset_index()\
        .sort_values('Sales',ascending=False).reset_index(drop=True)
agg['product_description'] = pd.Categorical(agg['product_description'], categories=agg['product_description'].dropna().unique(), ordered=True)
agg['sales_pct'] = agg['Sales']/agg.Sales.sum()
agg['cumu_sales_pct'] = agg.sales_pct.cumsum()
agg = agg.reset_index()

g = (ggplot(agg, aes(x='index',y='cumu_sales_pct', group=1))
    + geom_line(color='darkred') 
    + scale_y_continuous(labels=percent_format(), 
      breaks=[i/10 for i in range(11)])
    + scale_x_continuous(labels=label_comma(),
      breaks=[i*500 for i in range(11)])
    + xlab('Number of Unique Products') + ylab('Cumulative Sales Percentage') 
    + theme_538() 
)
g

Next, we turn our attention to pricing. Histograms are a perfect tool to see how numerical values are spread out, such as that of price distribution. The median and mean unit prices are 2.08 GBP and 8.02 GBP respectively. The plot below gives us an intuition on the prices customers expect to see when visiting our shop.

d = df[['StockCode','UnitPrice']].drop_duplicates()
d['UnitPrice_capped'] = d['UnitPrice'].map(lambda x: x if x<10 else 10)

g = (ggplot(d, 
      aes(x='UnitPrice_capped'))
    + geom_histogram(bins=20,fill='darkred')
    + geom_vline(xintercept=d.UnitPrice.mean(),color='orange')
    + geom_vline(xintercept=d.UnitPrice.median(),color='black')
    + geom_text(
      x = d.UnitPrice.mean()+1,
      y = 1000,
      label = round(d.UnitPrice.mean(),2),
      color='orange')
    + geom_text(
      x = d.UnitPrice.median()+1,
      y = 1000,
      label = round(d.UnitPrice.median(),2),
      color='black')
    + xlab('Unit Price (winsorized at 10 GBP)') 
    + ylab('Number of Unique Products') 
    + scale_y_continuous(labels=label_comma())
    + theme_538() 
    )
g

You can see 95% of our products are priced below 11 GBP. And the 0.1% with unusually high price has a high chance of being data input mistakes.

d.UnitPrice.describe(percentiles=[i/10 for i in range(10)]+[.95,.999])

count    7894.000000
mean        8.022251
std       115.048654
min         0.001000
0%          0.001000
10%         0.420000
20%         0.850000
30%         1.250000
40%         1.650000
50%         2.080000
60%         2.550000
70%         3.750000
80%         4.950000
90%         7.950000
95%        10.790000
99.9%     946.883300
max      8142.750000
Name: UnitPrice, dtype: float64

We can also look at price distribution by category. There are various flavors of putting histograms side-by-side namely box (showing median and interquantile ranges), violin (showing smoothed kernel density) and swarm (showing non-overlapping points) plots.

d = df.groupby('StockCode')[['category','UnitPrice']].max().reset_index()
d['UnitPrice_capped'] = d['UnitPrice'].map(lambda x: x if x<10 else 10)

g = (ggplot(d, 
      aes(x='category',y='UnitPrice_capped',fill='category'))
    + geom_boxplot()
    + xlab('Category') 
    + ylab('Unit Price Distribution (winsorized at 10 GBP)') 
    + theme_538() 
    + theme(axis_text_x = element_blank())
    )
g

Next, let us look at how product prices move over time. By plotting the monthly median unit price for our top categories, we can identify when prices tend to spike or dip. For a shop owner, this is operationally useful: if you know a product category gets more expensive later in the year (supplier price hikes, seasonal demand), that is your cue to stock up earlier when it is cheap.

In this particular dataset, we only have selling prices not buying ones, so our analysis is limited. Nevertheless, we notice that median prices are fairly stable for most categories. Seasonal and Holiday shows the most volatility, with its median price actually dipping in Q4. This implies that the holiday rush brings in a flood of cheap, high-volume items (stocking fillers, party favors, wrapping supplies) that pull the median down even as total sales in the category explode. For the buyer, this means the peak season playbook for Seasonal and Holiday is prioritizing volume, not markup.

top_cats = df[df.category!='Unknown'].groupby('category').Sales.sum()\
    .sort_values(ascending=False).head(5).index.tolist()

d = df[df.category.isin(top_cats)].groupby(['category','yearmon'])\
    .UnitPrice.median().reset_index()

g = (ggplot(d, aes(x='yearmon', y='UnitPrice', color='category', group='category'))
    + geom_point() + geom_line()
    + xlab('Month') + ylab('Median Unit Price (GBP)')
    + scale_y_continuous(limits=(0, None), labels=label_comma())
    + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g

Price sensitivity is another angle worth investigating. We can approximate it by looking at how much variation there is in unit prices within each category (standard deviation / mean; coefficient of variation; higher equals more variability). Categories with a wide spread of prices likely have both budget and premium tiers, meaning we have room to play with discounts on the low end without cannibalizing the high-margin premium products.

d = df[(df.category!='Unknown')&(df.category!='Others')].groupby(['category','StockCode']).UnitPrice.median().reset_index()
d['UnitPrice_capped'] = d['UnitPrice'].map(lambda x: x if x<20 else 20)

agg = d.groupby('category').agg(
    price_mean=('UnitPrice','mean'),
    price_std=('UnitPrice','std'),
    n_products=('StockCode','nunique')
).reset_index()
agg['cv'] = agg.price_std / agg.price_mean
agg = agg.sort_values('cv', ascending=False).reset_index(drop=True)
agg['category'] = pd.Categorical(agg['category'], categories=agg['category'], ordered=True)

g = (ggplot(agg, aes(x='category', y='cv'))
    + geom_col(fill='darkred')
    + xlab('Category') + ylab('Coefficient of Variation (Price)')
    + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g

Categories with a high coefficient of variation (like Home Decor and Kitchen and Dining) have wide price ranges. These are categories where selective discounting on budget products can drive foot traffic without hurting the perceived value of premium items. On the flip side, categories with low variation (like Fashion Accessories and Personal Care and Wellness) are more price-uniform; discounting here risks training customers to wait for sales.

We can also visualize the relationship between median price level and price variability directly. Categories in the top-right (higher price, high CV) have both the (supposedly high; again impossible to know without cost data) margin headroom and customer expectation of varied pricing, making them the safest candidates for selective discounts. Those in the bottom-left (cheap, uniform pricing) are the ones to be careful with; (supposedly) thin margins and customers anchored to a single price point.

g = (ggplot(agg, aes(x='price_mean', y='cv', label='category'))
    + geom_point(color='darkred', size=3)
    + geom_text(nudge_y=0.02, size=7)
    + xlab('Mean Unit Price (GBP)')
    + ylab('Coefficient of Variation (Price)')
    + theme_538())
g

Finally, let us look at basket size over time. Basket size (number of units per transaction) tells us about purchasing behavior. If baskets are getting larger, customers are consolidating trips; if they are shrinking, people might be making more frequent small purchases. The median basket size stays remarkably stable around the same level throughout the year (approximately 150 units per transaction since many customers are wholesalers), even during peak season when total sales explode. This tells us the peak is driven by more customers shopping rather than existing customers buying more per trip. If we want to increase average order value, we could consider bundle deals or quantity discounts, especially heading into the holidays when traffic is already high.

agg = df.groupby(['yearmon','InvoiceNo']).Quantity.sum().reset_index()\
    .groupby('yearmon').Quantity.median().reset_index()

g = (ggplot(agg, aes(x='yearmon', y='Quantity', group=1))
    + geom_point(color='darkred') + geom_line(color='darkred')
    + xlab('Month') + ylab('Median Basket Size (Units)')
    + scale_y_continuous(limits=(0, None))
    + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g

Nurturing Our Customer Base

Understanding our customer base is arguably the most actionable part of any retail analysis. Let us start by looking at the distribution of total sales per customer. This histogram immediately reveals who our core customers are.

d = df.groupby('CustomerID').Sales.sum().reset_index()
d['Sales_capped'] = d['Sales'].map(lambda x: x if x<5000 else 5000)

g = (ggplot(d, aes(x='Sales_capped'))
    + geom_histogram(bins=50, fill='darkred')
    + geom_vline(xintercept=d.Sales.mean(), color='orange')
    + geom_vline(xintercept=d.Sales.median(), color='black')
    + geom_text(
        x=d.Sales.mean()+500,
        y=200,
        label=f"mean: {round(d.Sales.mean(),0):.0f}",
        color='orange')
    + geom_text(
        x=d.Sales.median()+500,
        y=250,
        label=f"median: {round(d.Sales.median(),0):.0f}",
        color='black')
    + xlab('Total Sales per Customer (winsorized at 5,000 GBP)')
    + ylab('Number of Customers')
    + scale_y_continuous(labels=label_comma())
    + theme_538())
g

As is typical in retail, we have a highly skewed distribution. The median customer spends far less than the mean, meaning a small group of whales drives a disproportionate share of revenue. Let us quantify that.

d = df.groupby('CustomerID').Sales.sum().reset_index()\
    .sort_values('Sales', ascending=False).reset_index(drop=True)
d['sales_pct'] = d.Sales / d.Sales.sum()
d['cumu_sales_pct'] = d.sales_pct.cumsum()
d = d.reset_index()

g = (ggplot(d, aes(x='index', y='cumu_sales_pct', group=1))
    + geom_line(color='darkred')
    + scale_y_continuous(labels=percent_format(),
        breaks=[i/10 for i in range(11)])
    + scale_x_continuous(labels=label_comma())
    + xlab('Number of Customers (ranked by spend)')
    + ylab('Cumulative Sales Percentage')
    + theme_538())
g

In classic Pareto fashion, about 20% of customers account for roughly 80% of sales. These top spenders are the ones we absolutely cannot afford to lose. Any loyalty program, personalized offer, or VIP treatment should start with this cohort.

For customer segmentation, Recency-Frequency-Monetary (RFM) analysis is the tried-and-true approach. We compute each metric per customer, then visualize how they relate to one another using scatter plots. The scatter plot reveals a clear pattern: high-frequency customers tend to be high-monetary, and most of them have also made a recent purchase. The light-colored dots at high frequency/monetary are concerning; these are formerly valuable customers who have not come back in a while. They are prime targets for reactivation campaigns.

max_date = df.invoice_date.max()
rfm = df.groupby('CustomerID').agg(
    recency=('invoice_date', lambda x: (pd.to_datetime(max_date) - pd.to_datetime(x.max())).days),
    frequency=('InvoiceNo', 'nunique'),
    monetary=('Sales', 'sum'),
    avg_basket_units=('Quantity', 'mean'),
).reset_index()

g = (ggplot(rfm, aes(x='frequency', y='monetary', color='recency'))
    + geom_point(alpha=0.5, size=1.5)
    + scale_color_gradient(low='darkred', high='lightyellow')
    + scale_x_continuous(limits=(0, 50))
    + scale_y_continuous(limits=(0, 20000), labels=label_comma())
    + xlab('Purchase Frequency (# transactions)')
    + ylab('Total Monetary Value (GBP)')
    + theme_538())
g

/Users/charipol/Work/cstorm125.github.io/.venv/lib/python3.12/site-packages/plotnine/layer.py:364: PlotnineWarning: geom_point : Removed 35 rows containing missing values.

Let us see which categories drive the most cross-shopping. A heatmap, the scatter plot’s cousin for discrete variables, of co-purchase rates across categories tells us which combinations appear together in the same basket more often than chance would predict. The heatmap confirms that Home Decor is the gateway category. It co-occurs with nearly every other category at high rates. This makes it a natural candidate for cross-sell recommendations. Conversely, Outdoor and Garden and Toys and Games are fairly self-contained; customers buying in those categories tend to not browse much elsewhere in the same trip.

cats = df[~df.category.isin(['Unknown','Others'])].category.unique().tolist()
basket_cat = df[~df.category.isin(['Unknown','Others'])].groupby('InvoiceNo').category.apply(set).reset_index()

co_purchase = pd.DataFrame(0, index=cats, columns=cats)
for _, row in basket_cat.iterrows():
    for c1 in row.category:
        for c2 in row.category:
            co_purchase.loc[c1, c2] += 1

# Normalize by the diagonal (self-occurrence) to get co-purchase rate
for c in cats:
    co_purchase[c] = co_purchase[c] / co_purchase.loc[c, c]

# Melt for plotting
co_purchase_melted = co_purchase.reset_index().melt(id_vars='index')
co_purchase_melted.columns = ['category_1', 'category_2', 'co_purchase_rate']
# Remove self pairs for clarity
co_purchase_melted = co_purchase_melted[
    co_purchase_melted.category_1 != co_purchase_melted.category_2]

g = (ggplot(co_purchase_melted, 
        aes(x='category_1', y='category_2', fill='co_purchase_rate'))
    + geom_tile()
    + geom_text(aes(label='co_purchase_rate'), format_string='{:.0%}', size=7)
    + scale_fill_gradient(low='white', high='darkred')
    + xlab('') + ylab('')
    + theme_538()
    + theme(axis_text_x=element_text(angle=90, hjust=1),
            figure_size=(8, 6)))
g

Another important angle is to see whether we are growing or just milking from our customer base. We can split each month’s purchasing customers into new and existing customers to see how both groups trend over time. The two lines tell a clear story: existing customers climb steadily as the base accumulates, while new customer acquisition stays relatively flat. The gap widens over time, which means growth is increasingly driven by retention rather than acquisition. This is not necessarily bad since it means we are successfully retaining customers, but if growth is a priority, we need to invest more in top-of-funnel activities.

first_purchase = df.groupby('CustomerID').yearmon.min().reset_index()
first_purchase.columns = ['CustomerID', 'first_yearmon']

monthly_customers = df[['CustomerID','yearmon']].drop_duplicates()
monthly_customers = monthly_customers.merge(first_purchase, on='CustomerID')
monthly_customers['customer_type'] = np.where(
    monthly_customers.yearmon == monthly_customers.first_yearmon, 
    'New', 'Existing')

agg = monthly_customers.groupby(['yearmon','customer_type']).CustomerID.nunique().reset_index()
agg.columns = ['yearmon','customer_type','n_customers']

g = (ggplot(agg, aes(x='yearmon', y='n_customers', color='customer_type', group='customer_type'))
    + geom_point() + geom_line()
    + xlab('Month') + ylab('Number of Purchasing Customers')
    + scale_y_continuous(limits=(0, None), labels=label_comma())
    + theme_538() 
    + theme(axis_text_x=element_text(angle=90, hjust=1),
            legend_title=element_blank()))
g

We can dig deeper with a cohort retention heatmap. For each acquisition cohort (month of first purchase), we track what percentage of those customers are still buying in subsequent months. The diagonal is always 100% (month of acquisition). How quickly the color fades as you move right tells you how fast each cohort churns. Earlier cohorts that still show color in later months are your loyal base. If recent cohorts drop off faster than older ones, your product-market fit or onboarding experience might be degrading.

# Build cohort retention table
monthly_customers_cohort = df[['CustomerID','yearmon']].drop_duplicates()
monthly_customers_cohort = monthly_customers_cohort.merge(first_purchase, on='CustomerID')

cohort_size = monthly_customers_cohort.groupby('first_yearmon').CustomerID.nunique().reset_index()
cohort_size.columns = ['first_yearmon', 'cohort_size']

cohort_activity = monthly_customers_cohort.groupby(['first_yearmon','yearmon']).CustomerID.nunique().reset_index()
cohort_activity.columns = ['first_yearmon', 'yearmon', 'active_customers']
cohort_activity = cohort_activity.merge(cohort_size, on='first_yearmon')
cohort_activity['retention_rate'] = cohort_activity.active_customers / cohort_activity.cohort_size

# Only keep cohort month and onwards
cohort_activity = cohort_activity[cohort_activity.yearmon >= cohort_activity.first_yearmon]

g = (ggplot(cohort_activity, 
        aes(x='yearmon', y='first_yearmon', fill='retention_rate'))
    + geom_tile()
    + geom_text(aes(label='retention_rate'), format_string='{:.0%}', size=7)
    + scale_fill_gradient(low='white', high='darkred', labels=percent_format())
    + xlab('Active Month') + ylab('Cohort (First Purchase Month)')
    + theme_538()
    + theme(axis_text_x=element_text(angle=90, hjust=1),
            figure_size=(8, 5)))
g

Additionally, we may want to know which categories lead to the highest repeat rates and highest customer lifetime value. These are the categories worth investing in to cultivate as sacrificing short-term profits might lead to higher long-term benefits. Personal Care and Wellness emerges as a great candidate from the bar charts below with an exceptionally high lifetime value, while repeat rates are about the same across all categories.

customer_lifetime = df.groupby('CustomerID').agg(
    first_purchase=('invoice_date', 'min'),
    last_purchase=('invoice_date', 'max'),
    total_sales=('Sales', 'sum'),
    n_transactions=('InvoiceNo', 'nunique')
).reset_index()
customer_lifetime['lifetime_days'] = (
    pd.to_datetime(customer_lifetime.last_purchase) - 
    pd.to_datetime(customer_lifetime.first_purchase)).dt.days
customer_lifetime['is_repeater'] = (customer_lifetime.n_transactions > 1).astype(int)

# Get each customer's first purchase category
first_cat = df.sort_values('invoice_date').groupby('CustomerID').category.first().reset_index()
first_cat.columns = ['CustomerID', 'first_category']

# Merge with lifetime data
cat_lifetime = customer_lifetime.merge(first_cat, on='CustomerID')
cat_lifetime['is_repeater'] = (cat_lifetime.n_transactions > 1).astype(int)

agg = cat_lifetime[cat_lifetime.first_category!='Unknown'].groupby('first_category').agg(
    repeat_rate=('is_repeater', 'mean'),
    avg_ltv=('total_sales', 'mean'),
    n_customers=('CustomerID', 'nunique')
).reset_index().sort_values('repeat_rate', ascending=False).reset_index(drop=True)
agg['first_category'] = pd.Categorical(agg['first_category'], categories=agg['first_category'], ordered=True)

g = (ggplot(agg, aes(x='first_category', y='repeat_rate'))
    + geom_col(fill='darkred')
    + geom_text(aes(label='repeat_rate'), format_string='{:.0%}', 
                va='bottom', nudge_y=0.01)
    + xlab('First Purchase Category') + ylab('Repeat Rate')
    + scale_y_continuous(labels=percent_format(), limits=(0, 1))
    + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g

agg = agg.sort_values('avg_ltv', ascending=False).reset_index(drop=True)
agg['first_category'] = pd.Categorical(agg['first_category'], categories=agg['first_category'], ordered=True)

g = (ggplot(agg, aes(x='first_category', y='avg_ltv'))
    + geom_col(fill='darkred')
    + xlab('First Purchase Category') + ylab('Average Lifetime Value (GBP)')
    + scale_y_continuous(labels=label_comma())
    + theme_538() + theme(axis_text_x=element_text(angle=90, hjust=1)))
g

Finally, let us look at repurchase periods. For each customer with more than one transaction, we compute the average number of days between purchases. This tells us when to reach out with a reactivation nudge. The median repurchase period gives us our magic number. If a customer has not come back within, say, 1.5x their expected repurchase period, they are at risk of churning. That is exactly when you fire off an email, a discount code, or a “we miss you” campaign.

# Compute inter-purchase intervals for repeaters
purchase_dates = df[['CustomerID','invoice_date']].drop_duplicates()\
    .sort_values(['CustomerID','invoice_date'])
purchase_dates['prev_date'] = purchase_dates.groupby('CustomerID').invoice_date.shift(1)
purchase_dates = purchase_dates.dropna(subset=['prev_date'])
purchase_dates['days_between'] = (
    pd.to_datetime(purchase_dates.invoice_date) - 
    pd.to_datetime(purchase_dates.prev_date)).dt.days
purchase_dates = purchase_dates[purchase_dates.days_between > 0]

avg_repurchase = purchase_dates.groupby('CustomerID').days_between.mean().reset_index()
avg_repurchase.columns = ['CustomerID', 'avg_days_between']

g = (ggplot(avg_repurchase, aes(x='avg_days_between'))
    + geom_histogram(bins=50, fill='darkred')
    + geom_vline(xintercept=avg_repurchase.avg_days_between.mean(), color='orange')
    + geom_vline(xintercept=avg_repurchase.avg_days_between.median(), color='black')
    + geom_text(
        x=avg_repurchase.avg_days_between.mean()+15,
        y=60,
        label=f"mean: {round(avg_repurchase.avg_days_between.mean(),0):.0f}d",
        color='orange')
    + geom_text(
        x=avg_repurchase.avg_days_between.median()+15,
        y=70,
        label=f"median: {round(avg_repurchase.avg_days_between.median(),0):.0f}d",
        color='black')
    + xlab('Average Days Between Purchases')
    + ylab('Number of Customers')
    + scale_y_continuous(labels=label_comma())
    + theme_538())
g

print(f'''
--- Repurchase Period Statistics ---
Mean days between purchases: {avg_repurchase.avg_days_between.mean():.0f} days
Median days between purchases: {avg_repurchase.avg_days_between.median():.0f} days
Suggested reactivation trigger: {avg_repurchase.avg_days_between.median()*1.5:.0f} days since last purchase
''')

--- Repurchase Period Statistics ---
Mean days between purchases: 78 days
Median days between purchases: 59 days
Suggested reactivation trigger: 89 days since last purchase

So What Did We Learn?

With nothing more than line plots, bar charts, histograms, and scatter plots (plus their close relatives–heatmaps and box plots), we were able to paint a fairly comprehensive picture of this retail business:

• Sales trajectory growing year-on-year, with clear seasonality and a 2x peak in November. Sundays are slow; Saturdays are closed.
• Category dynamics Home Decor and Kitchen and Dining dominate; seasonal categories spike predictably at year-end while Outdoor and Garden peaks in spring.
• Product strategy 20% of SKUs drive 80% of revenue. Some categories carry too many products for their revenue contribution.
• Pricing most products sit below 10 GBP. Categories with high price variability are safer to discount selectively.
• Customer concentration classic Pareto distribution, a small whale cohort drives most of the revenue.
• Retention vs acquisition existing customers are the engine; new customer acquisition is flat. Cohort retention fades predictably.
• Cross-selling Home Decor is the gateway category; co-purchase patterns suggest clear recommendation opportunities.
• Reactivation timing the median repurchase period gives us a concrete trigger for win-back campaigns.

All of this, and we still have blind spots. Without cost data, we can only make educated guesses about profitability. Without inventory and procurement records, inventory turnover and stockout rates remain unknown. Without traffic or impression data, conversion rates are invisible.

But the point of this article is not about this particular dataset. It is about the fact that we got here using the most boring visualization toolkit imaginable: no Sankey diagrams, Marimekko charts, treemaps, chord diagrams, 3D surface plots, nor animated racing bar charts; just lines, bars, histograms, and dots on a plane.

For data visualization, the most critical part has never been how beautiful it looks. It is about visualizing with intent: what question each plot answers and what action follows from that answer. Respect your audience’s intellectual capacity and assume that they have the attention span to carefully read a few tables and your Four Horsemen of Data Visualization.