index.qmd

---
title: Urban Grammar - Age Capsule
subtitle: Linking building age with their Spatial Signature
author: Dani Arribas-Bel
format:
  html:
    theme: journal
    css: styles.css
    code-fold: true
    fig-responsive: true
  pdf:
    execute:
      echo: false
bibliography: references.bib
jupyter:
  jupytext:
    text_representation:
      extension: .qmd
      format_name: quarto
      format_version: '1.0'
      jupytext_version: 1.13.8
  kernelspec:
    display_name: Python 3 (ipykernel)
    language: python
    name: python3
---

[`[PDF]`](https://urbangrammarai.xyz/age_capsule/index.pdf)

### TL;DR 
This note presents an overview of how the way new British residential property is
developed and the type of environment in which it is built have changed in the
course of the last 125 years, with a focus on the last three decades.
We find six key trends:

1. Most of the current stock is built in suburban and peri-urban areas,
characterised by low density and lack of access to services, employment, and
urban amenities (i.e., "urban function").
1. The vast majority of the current stock was built in the period shortly
after WW-II (1950-75), and located in areas displaying the Open sprawl and
Urban buffer spatial signatures.
1. There is a correlation between the degree of urbanity of a signature and
the period where most properties in such signature were built. Older
signatures tend to be denser, more compact and more "function oriented" rather
than residential.
1. Open sprawl and Urban buffer are not only the most prominent signatures,
their importance has increased over time.
1. Since the financial crisis of 2008, Urban buffer has steadily grown and
become the single most prominent signature where new building takes place,
overcoming even Open sprawl.
1. Central and other residential areas were on a moderate upward trajectory,
gaining relevance, until the financial crash but have since flattened the
trend or even decreased in importance.

### Introduction

This capsule considers the temporal dimension of the Spatial Signatures [@fleischmann2022geographical]. It is a start to unpacking how the development of different signature type has unfolded over time. Hence, we try to answer questions such as *when did most of the development that lead to dense, compact neighborhoods took place?* or *what has been the main spatial signature of the last decades which has most recently been shaping the landscape?*

The analysis presented here relies on three datasets. First, the Spatial Signature characterisation proposed by @fleischmann2022geographical for Great Britain and based on the approach outlined in @ARRIBASBEL2022102641.
Second, data on Energy Performance Certificates (EPC), released by the Department of Levelling Up, Housing and Communities.[^1] This is a registry of all the properties in England and Wales that have received an Energy score. As part of the process, much more information is collected, including the period in which the property was built on, which is what we use here.
And third, we use Land Registry's Price Paid data[^2], which records almost every[^3] house transaction in England and Wales since 1995.

[^1]: Available at: [https://epc.opendatacommunities.org/](https://epc.opendatacommunities.org/)
[^2]: More information available at: [https://www.gov.uk/government/collections/price-paid-data](https://www.gov.uk/government/collections/price-paid-data)
[^3]: There are some exceptions. For more detail, please see: [https://www.gov.uk/guidance/about-the-price-paid-data#data-excluded-from-price-paid-data](https://www.gov.uk/guidance/about-the-price-paid-data#data-excluded-from-price-paid-data) 

The remainder of the document is structured in three sections: we first consider the overall distribution of properties across signatures; then we move on to evolution over the XXth Century evolution; and wrap up with a detailed zoom into the last two decades in the XXIst Century.

```{python}
import pandas
import seaborn as sns
import matplotlib.pyplot as plt
import urbangrammar_graphics as ugg

type_order = [
    'Hyper concentrated urbanity', 
    'Concentrated urbanity',
    'Metropolitan urbanity', 
    'Regional urbanity',
    'Local urbanity',
    'Dense urban neighbourhoods',
    'Dense residential neighbourhoods', 
    'Connected residential neighbourhoods', 
    'Gridded residential quarters',
    'Accessible suburbia', 
    'Disconnected suburbia', 
    'Open sprawl', 
    'Warehouse/Park land', 
    'Urban buffer', 
    'Countryside agriculture',
    'Wild countryside'
][::-1]

epc_age_order = age_order = [
    'Before 1900', 
    '1900-1929', 
    '1930-1949',
    '1950-1966', 
    '1967-1975', 
    '1976-1982', 
    '1983-1990', 
    '1991-1995',
    '1996-2002', 
    '2003-2006', 
    '2007 onwards', 
]

epc = pandas.read_csv(
    'data/epc_age_by_ss.csv', 
    index_col='age_remapped',
    skiprows=[-1]
)[type_order[::-1]].reindex(epc_age_order)
epc.index.name = ''
lr = pandas.read_csv(
    'data/lr_sales_by_month_ss.csv', index_col='moy'
)
```

### Building properties across signatures

As a first step, we consider how prevalent different development patterns, or
_spatial signatures_, are. To do that, @fig-epc-by-sig shows the proportion of
EPC properties located within each signature type, sorted by their degree of
"urbanity", with the most urban signature ("Hyper concentrated urbanity") at
the top, and the least ("Wild countryside") at the bottom. Starting with this
figure is useful because the exercise below will essentially unfold its
temporal dimension to consider how this has been built over time.

```{python}
#| fig-cap-location: margin
#| reference-location: margin
#| label: fig-epc-by-sig
#| fig-cap: Proportion of EPC properties by signature type
ax = epc.sum(axis=0)[::-1].plot.barh(figsize=(4, 4), color=ugg.HEX[2])
ax.set_xticks([])
ax.tick_params(left=False)
ax.set_frame_on(False)
```

There are three important aspects to highlight from this figure.
First,
although conceptually very important for urban life, very few of the
properties are in what we consider "urbanity", possibly with the exception of
the local variation, which is common in most cities and towns. Regional,
metropolitan, and (hyper) concentrated are only present in a minority of
cities and thus, in the broader context, are not very relevant.
Second,
we can observe two peaks of signatures that group two distinct types of
development. One around more compact neighborhoods, with "Dense residential
neighborhoods" as its most common type, and another one on more suburban
development with "Open sprawl" and "Urban buffer" as the most common
signatures in the entire set. This has important implications when we are
trying to understand in what context most housing properties are located,
pointing towards one with lower density and less connected structure.
Third,
despite being the most prominent area-wise, "Countryside agriculture" and
"Wild countryside" have very few properties. This is less counterintuitive
than it first appears when one considers the main characteristics of these
signatures are being home to functions that do not imply human residences.

### The long view

The first approach to unfolding the temporal dimension of @fig-epc-by-sig takes
a long-term view. We use the age column in the EPC dataset and re-aggregate it in
periods of, roughly, 15 to 20 years. @fig-epc-by-sig-by-time displays a heatmap with
the number of properties built in each period and located in areas of each signature.
This first crude overview already provides interesting insight. The highest concentration
of properties corresponds with houses built in the immediate post-war years
(ca. 1950-75) in locations labelled as Accessible suburbia, Open sprawl, and
Urban buffer. This trend represents the suburbanisation that occurred shortly
after WW-II and that still characterises today's housing stock.

```{python}
#| fig-cap-location: bottom
#| reference-location: bottom
#| fig-column: page-right
#| fig-cap: EPC properties by signature type over time periods
#| label: fig-epc-by-sig-by-time
f, ax = plt.subplots(1, figsize=(9, 4))
h = sns.heatmap(
    epc.T, 
    cmap=sns.light_palette(ugg.HEX[4], as_cmap=True), 
    cbar=False, 
    linewidths=0.01, 
    linecolor='w',
    ax=ax
)
h.tick_params(left=False, bottom=False)
h.set_xticklabels(h.get_xticklabels(), rotation = 45, ha="right");
```

@fig-epc-by-sig-by-time-rs makes the absolute view in @fig-epc-by-sig-by-time relative
to each spatial signature. The heatmap is constructed so that it represents
the proportion of all the houses in a signature type that correspond to each
time period. This transformation allows us to see more detail in signatures
that had too small counts to be picked up visually in the previous figure. Of
particular interest are all the "urbanity" classes, which represent the town
centres and most urban cores of British cities. In almost all cases, these are
the oldest areas as they represent the kernel where the city or town was
originally founded. In almost all cases too (with the exception, for example,
of cities that were heavily bombed during WW-II), they are all made up of
older stock that has been gradually upgraded rather than replaced. This is
what is represented in @fig-epc-by-sig-by-time-rs by the darker yellow (higher
values) in its top left corner. The figure also makes clear that, even
signature by signature, most of the housing stock in Britain was built before
1975, and the signature where it was built is correlated with time: most of
the urbanities was built prior to 1900, and current residential neighborhoods
later.

```{python}
#| fig-cap-location: margin
#| reference-location: margin
#| fig-cap: Proportion of EPC properties by signature type over time periods (rows add up to 100%)
#| label: fig-epc-by-sig-by-time-rs
tab = epc.reindex(age_order).div(epc.T.sum(axis=1)).T

f, ax = plt.subplots(1, figsize=(7, 4))
h = sns.heatmap(
    tab, 
    cmap=sns.light_palette(ugg.HEX[4], as_cmap=True), 
    cbar=False, 
    linewidths=0.01, 
    linecolor='w',
    ax=ax
)
h.tick_params(left=False, bottom=False)
h.set_xticklabels(h.get_xticklabels(), rotation = 45, ha="right");
```

Our final look at the period in which the building stock was built considers
another relative view, but this time by period. @fig-epc-by-sig-by-time-cs
represents the proportion of all houses built in a given period that is
located in an area with a particular spatial signature. Here we come back to
the patterns we initially saw in @fig-epc-by-sig, but this time we can see
more detail on the relevance of each period for signatures (which was obscured
before by the high numbers for some periods). The focus here thus is back on
the more suburban signatures (Open sprawl, Urban buffer) but we can now see
how the relevance of those types actually grows over time. Before, we
established that the peak period was 1950-1975. This is when the vast majority
of the current stock was built, and it was developed following suburban
patterns. What this figure makes clear is that the suburban nature of
development actually grew over time well into our present day. Even though the
total number of new houses built in Britain since 1975 was not as high as
before, the proportion of them that were in suburban areas has steadily grown.

```{python}
#| fig-cap-location: margin
#| reference-location: margin
#| fig-cap: Proportion of EPC properties by signature type over time periods (columns add up to 100%)
#| label: fig-epc-by-sig-by-time-cs
tab = (epc.T / epc.T.sum())[age_order]

f, ax = plt.subplots(1, figsize=(7, 4))
h = sns.heatmap(
    tab, 
    cmap=sns.light_palette(ugg.HEX[4], as_cmap=True), 
    cbar=False, 
    linewidths=0.01, 
    linecolor='w',
    ax=ax
)
h.tick_params(left=False, bottom=False)
h.set_xticklabels(h.get_xticklabels(), rotation = 45, ha="right");
```

### The last 25 years

```{python}
#| fig-column: margin
#| fig-cap: Land Registry new properties
#| label: fig-lr-ts
ax = lr.sum(axis=1).plot(
    figsize=(2, 0.5), linewidth=0.5, color=ugg.HEX[0]
)
ax.set_axis_off();
```

We now turn to the last 25 years of development and take a closer look. For
this task, we use Land Registry data, recording every new residential property
transaction since 1995. @fig-lr-ts shows the volume of transactions recorded
monthly, with a slight trend upwards, stopped by a dip shortly after the financial
crisis and a recovery following. Throughout the period, clear seasonality
effects are apparent.

```{python}
#| fig-cap-location: bottom
#| reference-location: bottom
#| fig-column: page-right
#| fig-cap: Land Registry new properties by signature type
#| label: fig-lr-all
p = lr.plot(
    figsize=(9, 4), subplots=False, sharex=True, sharey=True, alpha=0.5
)

handles, labels = plt.gca().get_legend_handles_labels()
order = pandas.Series(
    range(len(labels)), index=labels
)[type_order[::-1]].tolist()
plt.legend(
    [handles[idx] for idx in order],[labels[idx] for idx in order],
    bbox_to_anchor=(1,1), 
    loc="upper left",
    frameon=False
)

p.set_frame_on(False)
p.set_yticklabels([])
p.set_yticks([])
p.set_xlabel('')
p.tick_params(bottom=False)
p.set_xticklabels(p.get_xticklabels(), rotation = 45, ha="right");
```

We then dissaggregate by signature in @fig-lr-all. It is hard to see details,
but one clear pattern rapidly emerges: there are some signatures that
attract many more property transactions than others. In particular, Open
Sprawl and Urban Buffer consistently feature more transactions throughout the
entire period (and, in some ways, drive the overall pattern we saw in
@fig-lr-ts). The seasonal variation mentioned above translates in this figure
into large jumps up and down by month across the properties in all signatures,
making it hard to see more trends. To obtain a cleaner view that is decoupled
from total volume of transactions, @fig-lr-all-pct presents the proportion of
transactions that relate to properties in each signature, monthly. This makes
differences across months less relevant if the importance of each signature
stays constant from month to month, and thus presents a cleaner picture of the
relative evolution of the prominence of each signature. The figure contains
several interesting insights and, to make them more explicit and put the
spotlight on each of them, below we present subsets of signatures in isolation
from the full set.

```{python}
#| fig-cap-location: bottom
#| reference-location: bottom
#| fig-column: page-right
#| fig-cap: Yearly proportion of Land Registry new properties by signature type
#| label: fig-lr-all-pct
p = lr.T.div(lr.T.sum()).T.plot(
    figsize=(9, 4), subplots=False, sharex=True, sharey=True, alpha=0.5
)

handles, labels = plt.gca().get_legend_handles_labels()
order = pandas.Series(
    range(len(labels)), index=labels
)[type_order[::-1]].tolist()
plt.legend(
    [handles[idx] for idx in order],[labels[idx] for idx in order],
    bbox_to_anchor=(1,1), 
    loc="upper left",
    frameon=False
)

p.set_frame_on(False)
p.set_yticklabels([])
p.set_yticks([])
p.set_xlabel('')
p.tick_params(bottom=False)
p.set_xticklabels(p.get_xticklabels(), rotation = 45, ha="right");
```

@fig-lr-all-os-ub shows the evolution for Open sprawl and Urban buffer, the
two most prominent signatures in the period. We can see how both classes
represented roughly the same proportion of properties sold every month up
until, approximately, September of 2015. Open sprawl started the series with a
small lead and, by about 1999, positions changed, with Urban buffer leading up
until 2007, when the ranking reversed again. Around 2008, Urban buffer started
growing in relevance by the month and, by 2011, it surpassed Open sprawl (and
every other signature) only to continue climbing in relevance until today.
Overall, these interrelations are the product of two disctinct behaviours:
Open sprawl has been steadily loosing relevance in this period, while Urban
buffer saw a dip around the financial crisis of 2008, but had a spectacular
recovery. It is important to remember how these two signature types compare.
Open sprawl is an urban class characterised by low density, large amounts of
green space and the presence of transport infrastructure (e.g., highways).
Urban buffer is a slightly more suburban class, with less developed area and
more green spaces.

```{python}
#| fig-cap-location: margin
#| reference-location: margin
#| fig-cap: Yearly proportion of Land Registry new properties
#| label: fig-lr-all-os-ub
p = (
    lr.T.div(lr.T.sum()).T
    [['Open sprawl', 'Urban buffer']]
    .multiply(100)
    .plot.line(
        figsize=(9, 4), color=(ugg.HEX[4], ugg.HEX[3])
    )
)

plt.legend(loc="upper left", frameon=False)

p.set_frame_on(False)
p.tick_params(left = False, bottom=False)
p.set_ylabel('% of new builds in signature')
p.set_xlabel('')
p.set_xticklabels(p.get_xticklabels(), rotation = 45, ha="right");
```

What @fig-lr-all-os-ub is showing us is two main things. First, the two areas
where most of the building activity has been focused over the last
three decades is more suburban and sparse, less compact and dense. Cities are
becoming less "city-like".
Second, this trend was accentuated after the financial crisis of 2008 where a
wedge opened between these two classes, progressively favoring more and more
the most suburban development type.

@fig-lr-all-urbanities, in contrast, displays the evolution of the five
urbanity classes which, collectively, take a small proportion of all transactions in
every month, but represent important areas for British cities. These are
centres of towns, CBDs, and generally compact concentrations that attract a
large amount of employment and amenities. Two main insights are apparent in
this figure. First, as expected, the proportion of properties built in each of
these signatures is aligned with the prevalence of the signature types they
represent: Hyper concentrated urbanity, for example, is a signature with a
very small footprint (only in central parts of London) and thus consistently
sees very small numbers month after month; while Local urbanity, an signature
present in the centre of almost every town, is much more prevalent.
The second interesting pattern in the figure is that, although these
signatures gained a bit of relevance (upward trend) until the financial
crisis, they have plateaued and even decreased (e.g., Local Urbanity) ever
since. This is consistent with the picture in @fig-lr-all-os-ub: development
efforts since the financial crisis have increasingly focused in more peri-urban,
previously undeveloped areas and followed a dispersed, less compact pattern.

```{python}
#| fig-cap-location: margin
#| reference-location: margin
#| fig-cap: Yearly proportion of Land Registry new properties
#| label: fig-lr-all-urbanities
urbanity_order = [
    'Hyper concentrated urbanity',
    'Concentrated urbanity',
    'Regional urbanity',
    'Metropolitan urbanity',
    'Local urbanity',
]
p = (
    lr.T.div(lr.T.sum()).T
    [urbanity_order]
    .multiply(100)
    .plot.line(
        figsize=(9, 4), color=ugg.HEX[:5], 
    )
)

p.set_frame_on(False)
p.tick_params(left = False, bottom=False)
p.set_ylabel('% of new builds in signature')
p.set_xlabel('')
p.legend(frameon=False)
p.set_xticklabels(p.get_xticklabels(), rotation = 45, ha="right");
```

Finally, @fig-lr-all-neis presents individually the evolution in the different
signatures that capture mostly residential neighborhoods. Here the picture is
more nuanced and less relevant: each signature roughly maintains its relative
importance over the period. Nevertheless, some interesting variation emerges.
The first pattern to note is that, similar to the previous figure, most
classes present a slight upward trend up until around 2008. Until the
financial crisis, the proportion of new-built properties being developed in
relatively compact environments was growing. Afterwards, we find a more mixed
picture, where some classes (e.g., Dense urban neighbourhoods) show a flatter
trend, while others present slight declines (e.g., Gridded residential
quarters).

```{python}
#| fig-cap-location: margin
#| reference-location: margin
#| fig-cap: Yearly proportion of Land Registry new properties
#| label: fig-lr-all-neis
types = (
    [i for i in lr.columns if 'neighbourhoods' in i.lower()] + 
    ['Gridded residential quarters'] +
    [i for i in lr.columns if 'suburbia' in i.lower()]
)
ps = (
    lr.T.div(lr.T.sum()).T
    [types]
    .multiply(100)
    .iloc[:-1, :]
    .plot.line(
        figsize=(6, 4), color=ugg.HEX, subplots=True, rot=45
    )
)

for p in ps:

    p.set_frame_on(False)
    p.tick_params(left = False, bottom=False)
    p.set_yticks([])
    p.set_xlabel('')
    p.legend(
        loc="upper left",
        bbox_to_anchor=(1,1), 
        frameon=False
    );
```

# References