1.1 What are these notebooks for?¶
Most weeks of the course have a lecture and a workshop. In the lecture you are introduced to material, and in the workshop you use Python to solve problems using that material.
These notebooks act as a bridge between the lecture and the workshop. Each one summarises the lecture, and provides code examples as a starting point for writing your own code. In this notebook, I will import packages on-the-fly as needed so you can see when they are required. In the later notebooks, and in the workshop, imports will be done at the start.
The first lecture was slightly unusual, in that it was also an introduction to the course. I am not going to summarise the introduction (whatever that would entail) but I am going to look at different data modalities, and how you can process them in Python.
1.2 Data modalities¶
1.2.1 Time series¶
In time series data, we have some quantity we care about at different points in time. We are going to consider the value of the pound (GBP) vs. the dollar (USD). A lot of the time we import our data from spreadsheets (e.g. excel files, CSV files) and this is no exception!
I have already downloaded the spreadsheet containing GBP vs. USD values from 04/10/2021 to 03/10/2022 from Yahoo Finance which is in CSV format.
How do we get this into Python? There is a fantastic data analysis library called pandas which does all the hard work for us.
# Import pandas for dataframes
import pandas as pd
# Read CSV into dataframe using pandas
df = pd.read_csv("data/GBPUSD=X.csv")
# Show dataframe
df
Let's say we care about the closing value at the end of each day. This is in the Close column. We can plot this against Date using a few lines of code.
# matplotlib for plotting
import matplotlib.pyplot as plt
# This makes matplotlib output nice figures without much tweaking
plt.rcParams.update(
{
"lines.markersize": 10, # Big points
"font.size": 15, # Larger font
"xtick.major.size": 5.0, # Bigger xticks
"ytick.major.size": 5.0, # Bigger yticks
}
)
# Plotting code
fig, ax = plt.subplots(figsize=[8,6])
ax.plot(df["Date"], df["Close"])
ax.set_xlabel('Date')
ax.set_ylabel('GBP to USD rate')
ax.set_title('GBP vs USD from October 2021-2022')
# Add a grid
ax.grid()
# Formatting specifically for date strings
fig.autofmt_xdate()
# A hacky way of showing a subset of x ticks
dates = df["Date"].to_numpy()
ax.set_xticks([dates[i] for i in range(0, len(dates), 50)])
plt.show()
There is an alternative way to do this just in pandas. Notice that each row has an index (0,1,2,3...). We can turn the Date column into a datetime format, and then make that the index instead. We can then just call .plot on the dataframe to plot a column against the date index.
Let's do all this on a copy of the original dataframe:
df2 = df.copy()
df2['Date'] = pd.to_datetime(df2['Date'])
df2 = df2.set_index("Date")
df2["Close"].plot(figsize=[10,8])
We can also take a column and convert it into a numpy array. Let's take the Close column, convert it into an array, and find its mean. (This can be done directly in pandas but is for illustration!)
# import numpy for arrays
import numpy as np
close_array = df["Close"].values
close_mean = np.mean(close_array)
print(f"The mean close value is {close_mean:3f}")
1.2.2 Tabular data¶
We actually did just extract our time series data from tabular data! Tabular data will make up the vast majority of data you look at in this course. It is very common in the real world.
Let's say we want the list of dates for which the opening value of GBP vs. USD was greater than 1.3.
First, we can see which rows this corresponds to:
valid_rows = df["Open"] > 1.3
print(valid_rows)
We can then use this boolean array to filter out the appropriate dates.
df["Date"][valid_rows]
If we want to get the rows that correspond to dates in the year 2022, we can use the following to get a Boolean array, and filter with it:
valid_rows = df["Date"].str.contains("2022")
df[valid_rows]
1.2.3 Images¶
PIL (the Python imaging library) lets us read in images, as well as perform high-level manipulations. We will load in a JPEG of a dog, and also print the image size in pixels.
# Image from PIL lets us manipulate images
from PIL import Image
# Read image
image = Image.open("data/dog.jpg")
# Use Juypter's inbuilt display function
display(image)
print(f"The image has a size of {image.size}")
We can downsize the image...
image = image.resize((224, 224))
display(image)
and rotate it!
image = image.rotate(90)
display(image)
Recall from the lecture that an image is actually stored as a 3D array (height by width by colour channel). We can see this if we convert our image into a numpy array.
# Convert image into a numpy array
data_im = np.array(image)
# See what the array looks like and print its shape
print(data_im)
print(f"the image in numpy has shape {data_im.shape}")
Let's manually set the red colour channel to zero for all pixels and see what happens.
# Create a copy of our image array
data_nored = data_im.copy()
# Set all values in the red channel to 0
data_nored[:, :, 0] = 0
# Now we have a numpy array we have to use matplotlib to display it
plt.imshow(data_nored)
Without any red, the blue and green have got more ... blue and green. This wasn't the most obvious manipulation! Try some others by changing the code above.
1.3 Preprocessing¶
1.3.1 Vectors and matrices¶
Many machine learning (ML) techniques require us to represent our data points as vectors of real values, each of the same dimensionality $D$. The elements of these vectors correspond to variables, or features. In machine learning it is convention to use column vectors, so we will stick to this.
We can refer to some arbitrary data point as $\mathbf{x}\in \mathbb{R}^{D}$ and if we want to refer to specific data points in our dataset of $N$ points we can write $\mathbf{x}^{(1)},\mathbf{x}^{(2)},\mathbf{x}^{(3)},\dots ,\mathbf{x}^{(N)}$ where the ordering isn't important as long as it is consistent.
\begin{equation}\mathbf{x} =\begin{bmatrix} x_1\\x_2\\\vdots \\x_{D} \\ \end{bmatrix} \quad \quad \mathbf{x}^{(n)} = \begin{bmatrix} x_1^{(n)}\\x_2^{(n)}\\ \vdots\\x_{D}^{(n)} \\ \end{bmatrix}\end{equation}The whole dataset is the collection of these vectors $\{\mathbf{x}^{(n)}\}_{n=1}^{N}$ and can be stored in a dataset matrix $\mathbf{X}$. Annoyingly, it is convention to have the rows of this matrix representing different data points so that $\mathbf{X}\in \mathbb{R}^{N\times D}$. I tend to refer to this as a dataset matrix, but it is often referred to as a design matrix in the literature.
\begin{equation} \mathbf{X} = \begin{bmatrix} \mathbf{x}^{(1)^\top}\\ \mathbf{x}^{(2)^\top}\\ \mathbf{x}^{(3)^\top}\\ \vdots \\ \mathbf{x}^{(N)^\top}\\ \end{bmatrix} = \begin{bmatrix} x_1^{(1)}&x_2^{(1)}&\dots& x_{D}^{(1)} \\ x_1^{(2)}&x_2^{(2)}&\dots& x_{D}^{(2)} \\ x_1^{(3)}&x_2^{(3)}&\dots& x_{D}^{(3)} \\ \dots&\dots&\ddots& \vdots \\ x_1^{(N)}&x_2^{(N)}&\dots& x_{D}^{(N)} \\ \end{bmatrix} \end{equation}Given some tabular data, we need to convert it into this $\mathbf{X}$ format. If the features in our table are all continuous variables then this is very straightforward. In fact, with a table represented as a pandas dataframe it only requires a single line of code.
Let's look at the iris dataset from the lecture. We will get the dataframe from sklearn and then convert it into a dataset matrix.
sklearn (or Scikit-learn) is a high-level ML library for Python. It integrates with NumPy for arrays, SciPy for optimisation, and Pandas for dataframes (tabular data). It is really, really good and we will use it heavily in this course. It is high level, which lets us spend less time hacking together algorithms ourselves, and more time seeing how they work on data.
# Get loader for the iris dataset and import pandas and numpy
from sklearn.datasets import load_iris
import pandas as pd
import numpy as np
# Convert into dataframe with the columns as the feature names
data = load_iris()
df = pd.DataFrame(data.data, columns=data.feature_names)
df
To convert this into $\mathbf{X}\in \mathbb{R}^{N\times D}$ we just have to run a line of code.
# Convert dataframe into array
X = df.values
# Check the shape of X
print(f"X has shape {X.shape}")
print(X)
1.3.2 Standardisation¶
Standardising your data is extremely important for many ML algorithms. It allows features to be compared on a like-for-like basis so that those with naturally large measurements don't dominate.
For some of the toy examples in the lectures and notebooks where I have generated synthetic data, I haven't bothered with standardisation. There are two reasons for this:
- It becomes more fiddly to make the kind of data I want to use to illustrate a point
- The features in these synthetic datasets are all at similar scales and things still works (they are roughly standardised)
This will rarely be the case with real data! When in doubt, standardise your data, seriously!
To do this, for each feature we compute the mean and standard deviation (SD) of the measurements across all data points. For each measurement of that feature, we subtract the mean and divide by the SD.
\begin{equation} \mathbf{X}_{new} = \begin{bmatrix} \frac{x_1^{(1)}-\mu_1}{\sigma_1}&\frac{x_2^{(1)}-\mu_2}{\sigma_2}&\dots& \frac{x_{D}^{(1)}-\mu_{D}}{\sigma_{D}} \\ \frac{x_1^{(2)}-\mu_1}{\sigma_1}&\frac{x_2^{(2)}-\mu_2}{\sigma_2}&\dots& \frac{x_{D}^{(2)}-\mu_{D}}{\sigma_{D}} \\ \\ \dots&\dots&\ddots& \vdots \\ \frac{x_1^{({N})}-\mu_1}{\sigma_1}&\frac{x_2^{({N})}-\mu_2}{\sigma_2}&\dots& \frac{x_{D}^{({N})}-\mu_{D}}{\sigma_{D}}\\\\ \end{bmatrix} \end{equation}where $\mu_j = \frac{1}{N}\sum_{n} \mathbf{x}^{(n)}_j$ and $\sigma_j^2 = \frac{1}{N}\sum_{n}(\mathbf{x}^{(n)}_j-\mu_j)^2$.
We can do this very easily in sklearn using the StandardScaler class.
# Make a copy of our dataset to compare stats after
X_old = np.array(df)
# Import scaler
from sklearn.preprocessing import StandardScaler
# Make a scaler object
scaler = StandardScaler()
# Fit the scaler to the dataset. This means computing the means and STDs
scaler.fit(X)
# Finally apply the scaler to transform the data
X = scaler.transform(X)
# Compare stats
print(
f"Before standardisation feature means were {np.round(X_old.mean(0),2)} and SDs were {np.round(X_old.std(0),2)}"
)
print(
f"After standardisation feature means are {np.round(X.mean(0),2)} and SDs are {np.round(X.std(0),2)}"
)