Linear algebra is to machine learning as flour to bakery: every machine learning model is based in linear algebra, as every cake is based in flour. It is not the only ingredient, of course. Machine learning models need vector calculus, probability, and optimization, as cakes need sugar, eggs, and butter. Applied machine learning, like bakery, is essentially about combining these mathematical ingredients in clever ways to create useful (tasty?) models.

This document contains introductory level linear algebra notes for applied machine learning. It is meant as a reference rather than a comprehensive review. If you ever get confused by matrix multiplication, don't remember what was the $L_2$ norm, or the conditions for linear independence, this can serve as a quick reference. It also a good introduction for people that don't need a deep understanding of linear algebra, but still want to learn about the fundamentals to read about machine learning or to use pre-packaged machine learning solutions. Further, it is a good source for people that learned linear algebra a while ago and need a refresher.

These notes are based in a series of (mostly) freely available textbooks, video lectures, and classes I've read, watched and taken in the past. If you want to obtain a deeper understanding or to find exercises for each topic, you may want to consult those sources directly.

Free resources:

• Mathematics for Machine Learning by Deisenroth, Faisal, and Ong. 1st Ed. Book link.
• Introduction to Applied Linear Algebra by Boyd and Vandenberghe. 1sr Ed. Book link
• Linear Algebra Ch. in Deep Learning by Goodfellow, Bengio, and Courville. 1st Ed. Chapter link.
• Linear Algebra Ch. in Dive into Deep Learning by Zhang, Lipton, Li, And Smola. Chapter link.
• Prof. Pavel Grinfeld's Linear Algebra Lectures at Lemma. Videos link.
• Prof. Gilbert Strang's Linear Algebra Lectures at MIT. Videos link.
• Salman Khan's Linear Algebra Lectures at Khan Academy. Videos link.
• 3blue1brown's Linear Algebra Series at YouTube. Videos link.

Not-free resources:

• Introduction to Linear Algebra by Gilbert Strang. 5th Ed. Book link.
• No Bullshit Guide to Linear Algebra by Ivan Savov. 2nd Ed. Book Link.

I've consulted all these resources at one point or another. Pavel Grinfeld's lectures are my absolute favorites. Salman Khan's lectures are really good for absolute beginners (they are long though). The famous 3blue1brown series in linear algebra is delightful to watch and to get a solid high-level view of linear algebra.

If you have to pic one book, I'd pic Boyd's and Vandenberghe's Intro to applied linear algebra, as it is the most beginner friendly book on linear algebra I've encounter. Every aspect of the notation is clearly explained and pretty much all the key content for applied machine learning is covered. The Linear Algebra Chapter in Goodfellow et al is a nice and concise introduction, but it may require some previous exposure to linear algebra concepts. Deisenroth et all book is probably the best and most comprehensive source for linear algebra for machine learning I've found, although it assumes that you are good at reading math (and at math more generally). Savov's book it's also great for beginners but requires time to digest. Professor Strang lectures are great too but I won't recommend it for absolute beginners.

I'll do my best to keep notation consistent. Nevertheless, learning to adjust to changing or inconsistent notation is a useful skill, since most authors will use their own preferred notation, and everyone seems to think that its/his/her own notation is better.

To make everything more dynamic and practical, I'll introduce bits of Python code to exemplify each mathematical operation (when possible) with NumPy, which is the facto standard package for scientific computing in Python.

Finally, keep in mind this is created by a non-mathematician for (mostly) non-mathematicians. I wrote this as if I were talking to myself or a dear friend, which explains why my writing is sometimes conversational and informal.

If you find any mistake in notes feel free to reach me, so I can correct the issue.

Table of contents

Note: underlined sections are the newest sections and/or corrected ones.

Preliminary concepts:

• Sets
• Belonging and inclusion
• Set specification
• Ordered pairs
• Relations
• Functions

Vectors:

• Types of vectors
  • Geometric vectors
  • Polynomials
  • Elements of R
• Zero vector, unit vector, and sparse vector
• Vector dimensions and coordinate system
• Basic vector operations
  • Vector-vector addition
  • Vector-scalar multiplication
  • Linear combinations of vectors
  • Vector-vector multiplication: dot product
• Vector space, span, and subspace
  • Vector space
  • Vector span
  • Vector subspaces
• Linear dependence and independence
• Vector null space
• Vector norms
  • Euclidean norm: $L_2$
  • Manhattan norm: $L_1$
  • Max norm: $L_\infty$
• Vector inner product, length, and distance
• Vector angles and orthogonality
• Systems of linear equations

Matrices:

• Basic matrix operations
  • Matrix-matrix addition
  • Matrix-scalar multiplication
  • Matrix-vector multiplication: dot product
  • Matrix-matrix multiplication
  • Matrix identity
  • Matrix inverse
  • Matrix transpose
  • Hadamard product
• Special matrices
  • Rectangular matrix
  • Square matrix
  • Diagonal matrix
  • Upper triangular matrix
  • Lower triangular matrix
  • Symmetric matrix
  • Identity matrix
  • Scalar matrix
  • Null or zero matrix
  • Echelon matrix
  • Antidiagonal matrix
  • Design matrix
• Matrices as systems of linear equations
• The four fundamental matrix subsapces
  • The column space
  • The row space
  • The null space
  • The null space of the transpose
• Solving systems of linear equations with matrices
  • Gaussian Elimination
  • Gauss-Jordan Elimination
• Matrix basis and rank
• Matrix norm

Linear and affine mappings:

• Linear mappings
• Examples of linear mappings
  • Negation matrix
  • Reversal matrix
• Examples of nonlinear mappings
  • Norms
  • Translation
• Affine mappings
  • Affine combination of vectors
  • Affine span
  • Affine space and subspace
  • Affine mappings using the augmented matrix
• Special linear mappings
  • Scaling
  • Reflection
  • Shear
  • Rotation
• Projections
  • Projections onto lines
  • Projections onto general subspaces
  • Projections as approximate solutions to systems of linear equations

Matrix decompositions:

• LU decomposition
  • Elementary matrices
  • The inverse of elementary matrices
  • LU decomposition as Gaussian Elimination
  • LU decomposition with pivoting
• QR decomposition
  • Orthonormal basis
  • Orthonormal basis transpose
  • Gram-Schmidt Orthogonalization
  • QR decomposition as Gram-Schmidt Orthogonalization
• Determinant
  • Determinant as measures of volume
  • The 2X2 determinant
  • The NXN determinant
  • Determinants as scaling factors
  • The importance of determinants
• Eigenthings
  • Change of basis
  • Eigenvectors, Eigenvalues, and Eigenspaces
  • Trace and determinant with eigenvalues
  • Eigendecomposition
  • Eigenbasis are a good basis
  • Geometric interpretation of Eigendecomposition
  • The problem with Eigendecomposition
• Singular Value Decomposition:
  • Singular Value Decomposition Theorem
  • Singular Value Decomposition computation
  • Geometric interpretation of the Singular Value Decomposition
  • Singular Value Decomposition vs Eigendecomposition
• Matrix Approximation:
  • Best rank-k approximation with SVD
  • Best low-rank approximation as a minimization problem
• Some notes

Epilogue

Preliminary concepts

While writing about linear mappings, I realized the importance of having a basic understanding of a few concepts before approaching the study of linear algebra. If you are like me, you may not have formal mathematical training beyond high school. If so, I encourage you to read this section and spent some time wrapping your head around these concepts before going over the linear algebra content (otherwise, you might prefer to skip this part). I believe that reviewing these concepts is of great help to understand the notation, which in my experience is one of the main barriers to understand mathematics for nonmathematicians: we are nonnative speakers, so we are continuously building up our vocabulary. I'll keep this section very short, as is not the focus of this mini-course.

For this section, my notes are based on readings of:

• Geometric transformations (Vol. 1) (1966) by Modenov & Parkhomenko
• Naive Set Theory (1960) by P.R. Halmos
• Abstract Algebra: Theory and Applications (2016) by Judson & Beeer. Book link

Sets

Sets are one of the most fundamental concepts in mathematics. They are so fundamentals that they are not defined in terms of anything else. On the contrary, other branches of mathematics are defined in terms of sets, including linear algebra. Put simply, sets are well-defined collections of objects. Such objects are called elements or members of the set. The crew of a ship, a caravan of camels, and the LA Lakers roster, are all examples of sets. The captain of the ship, the first camel in the caravan, and LeBron James are all examples of "members" or "elements" of their corresponding sets. We denote a set with an upper case italic letter as $\textit{A}$. In the context of linear algebra, we say that a line is a set of points, and the set of all lines in the plane is a set of sets. Similarly, we can say that vectors are sets of points, and matrices sets of vectors.

Belonging and inclusion

We build sets using the notion of belonging. We denote that $a$ belongs (or is an element or member of) to $\textit{A}$ with the Greek letter epsilon as:

Another important idea is inclusion, which allow us to build subsets. Consider sets $\textit{A}$ and $\textit{B}$. When every element of $\textit{A}$ is an element of $\textit{B}$, we say that $\textit{A}$ is a subset of $\textit{B}$, or that $\textit{B}$ includes $\textit{A}$. The notation is:

or

Belonging and inclusion are derived from axion of extension: two sets are equal if and only if they have the same elements. This axiom may sound trivially obvious but is necessary to make belonging and inclusion rigorous.

Set specification

In general, anything we assert about the elements of a set results in generating a subset. In other words, asserting things about sets is a way to manufacture subsets. Take as an example the set of all dogs, that I'll denote as $\textit{D}$. I can assert now "$d$ is black". Such an assertion is true for some members of the set of all dogs and false for others. Hence, such a sentence, evaluated for all member of $\textit{D}$, generates a subset: the set of all black dogs. This is denoted as:

or

The colon ($:$) or vertical bar ($\vert$) read as "such that". Therefore, we can read the above expression as: all elements of $d$ in $\textit{D}$ such that $d$ is black. And that's how we obtain the set $\textit{B}$ from $\textit{A}$.

Set generation, as defined before, depends on the axiom of specification: to every set $\textit{A}$ and to every condition $\textit{S}(x)$ there corresponds a set $\textit{B}$ whose elements are exactly those elements $a \in \textit{A}$ for which $\textit{S}(x)$ holds.

A condition $\textit{S}(x)$ is any sentence or assertion about elements of $\textit{A}$. Valid sentences are either of belonging or equality. When we combine belonging and equality assertions with logic operators (not, if, and or, etc), we can build any legal set.

Ordered pairs

Pairs of sets come in two flavors: unordered and ordered. We care about pairs of sets as we need them to define a notion of relations and functions (from here I'll denote sets with lower-case for convenience, but keep in mind we're still talking about sets).

Consider a pair of sets $\textit{x}$ and $\textit{y}$. An unordered pair is a set whose elements are $\{ \textit{x},\textit{y} \}$, and $\{ \textit{x},\textit{y} \} = \{ \textit{y},\textit{x} \}$. Therefore, presentation order does not matter, the set is the same.

In machine learning, we usually do care about presentation order. For this, we need to define an ordered pair (I'll introduce this at an intuitive level, to avoid to introduce too many new concepts). An ordered pair is denoted as $( \textit{x},\textit{y} )$, with $\textit{x}$ as the first coordinate and $\textit{y}$ as the second coordinate. A valid ordered pair has the property that $( \textit{x},\textit{y} ) \ne ( \textit{y},\textit{x} )$.

Relations

From ordered pairs, we can derive the idea of relations among sets or between elements and sets. Relations can be binary, ternary, quaternary, or N-ary. Here we are just concerned with binary relationships. In set theory, relations are defined as sets of ordered pairs, and denoted as $\textit{R}$. Hence, we can express the relation between $\textit{x}$ and $\textit{y}$ as:

Further, for any $\textit{z} \in \textit{R}$, there exist $\textit{x}$ and $\textit{y}$ such that $\textit{z} = (\textit{x}, \textit{y})$.

From the definition of $\textit{R}$, we can obtain the notions of domain and range. The domain is a set defined as:

This reads as: the values of $\textit{x}$ such that for at least one element of $\textit{y}$, $\textit{x}$ has a relation with $\textit{y}$.

The range is a set defined as:

This reads: the set formed by the values of $\text{y}$ such that at least one element of $\textit{x}$, $\textit{x}$ has a relation with $\textit{y}$.

Functions

Consider a pair of sets $\textit{X}$ and $\textit{Y}$. We say that a function from $\textit{X}$ to $\textit{Y}$ is relation such that:

• $dom \textit{ f} = \textit{X}$ and
• such that for each $\textit{x} \in \textit{X}$ there is a unique element of $\textit{y} \in \textit{Y}$ with $(\textit{x}, \textit{y}) \in {f}$

More informally, we say that a function "transform" or "maps" or "sends" $\textit{x}$ onto $\textit{y}$, and for each "argument" $\textit{x}$ there is a unique value $\textit{y}$ that $\textit{f }$ "assummes" or "takes".

We typically denote a relation or function or transformation or mapping from X onto Y as:

or

The simples way to see the effect of this definition of a function is with a chart. In Fig. 1, the left-pane shows a valid function, i.e., each value $\textit{f}(\textit{x})$ maps uniquely onto one value of $\textit{y}$. The right-pane is not a function, since each value $\textit{f}(\textit{x})$ maps onto multiple values of $\textit{y}$.

For $\textit{f}: \textit{X} \rightarrow \textit{Y}$, the domain of $\textit{f}$ equals to $\textit{X}$, but the range does not necessarily equals to $\textit{Y}$. Just recall that the range includes only the elements for which $\textit{Y}$ has a relation with $\textit{X}$.

The ultimate goal of machine learning is learning functions from data, i.e., transformations or mappings from the domain onto the range of a function. This may sound simplistic, but it's true. The domain $\textit{X}$ is usually a vector (or set) of variables or features mapping onto a vector of target values. Finally, I want to emphasize that in machine learning the words transformation and mapping are used interchangeably, but both just mean function.

This is all I'll cover about sets and functions. My goals were just to introduce: (1) the concept of a set, (2) basic set notation, (3) how sets are generated, (4) how sets allow the definition of functions, (5) the concept of a function. Set theory is a monumental field, but there is no need to learn everything about sets to understand linear algebra. Halmo's Naive set theory (not free, but you can find a copy for ~\$8-$10 US) is a fantastic book for people that just need to understand the most fundamental ideas in a relatively informal manner.

# Libraries for this section 
import numpy as np
import pandas as pd
import altair as alt
alt.themes.enable('dark')
# ThemeRegistry.enable('dark')

Vectors

Linear algebra is the study of vectors. At the most general level, vectors are ordered finite lists of numbers. Vectors are the most fundamental mathematical object in machine learning. We use them to represent attributes of entities: age, sex, test scores, etc. We represent vectors by a bold lower-case letter like $\bf{v}$ or as a lower-case letter with an arrow on top like $\vec{v}$.

Vectors are a type of mathematical object that can be added together and/or multiplied by a number to obtain another object of the same kind. For instance, if we have a vector $\bf{x} = \text{age}$ and a second vector $\bf{y} = \text{weight}$, we can add them together and obtain a third vector $\bf{z} = x + y$. We can also multiply $2 \times \bf{x}$ to obtain $2\bf{x}$, again, a vector. This is what we mean by the same kind: the returning object is still a vector.

Types of vectors

Vectors come in three flavors: (1) geometric vectors, (2) polynomials, (3) and elements of $\mathbb{R^n}$ space. We will defined each one next.

Geometric vectors

Geometric vectors are oriented segments. Therse are the kind of vectors you probably learned about in high-school physics and geometry. Many linear algebra concepts come from the geometric point of view of vectors: space, plane, distance, etc.

Polynomials

A polynomial is an expression like $f(x) = x^2 + y + 1$. This is, a expression adding multiple "terms" (nomials). Polynomials are vectors because they meet the definition of a vector: they can be added together to get another polynomial, and they can be multiplied together to get another polynomial.

Elements of R

Elements of $\mathbb{R}^n$ are sets of real numbers. This type of representation is arguably the most important for applied machine learning. It is how data is commonly represented in computers to build machine learning models. For instance, a vector in $\mathbb{R}^3$ takes the shape of:

Indicating that it contains three dimensions.

In NumPy vectors are represented as n-dimensional arrays. To create a vector in $\mathbb{R^3}$:

x = np.array([[1],
              [2],
              [3]])

We can inspect the vector shape by:

x.shape # (3 dimensions, 1 element on each)
# (3, 1)

print(f'A 3-dimensional vector:\n{x}')
"""
    A 3-dimensional vector:
    [[1]
     [2]
     [3]]
"""

Zero vector, unit vector, and sparse vector

There are a couple of "special" vectors worth to remember as they will be mentioned frequently on applied linear algebra: (1) zero vector, (2) unit vector, (3) sparse vectors

Zero vectors, are vectors composed of zeros, and zeros only. It is common to see this vector denoted as simply $0$, regardless of the dimensionality. Hence, you may see a 3-dimensional or 10-dimensional with all entries equal to 0, refered as "the 0" vector. For instance:

Unit vectors, are vectors composed of a single element equal to one, and the rest to zero. Unit vectors are important to understand applications like norms. For instance, $\bf{x_1}$, $\bf{x_2}$, and $\bf{x_3}$ are unit vectors:

Sparse vectors, are vectors with most of its elements equal to zero. We denote the number of nonzero elements of a vector $\bf{x}$ as $nnz(x)$. The sparser possible vector is the zero vector. Sparse vectors are common in machine learning applications and often require some type of method to deal with them effectively.

Vector dimensions and coordinate system

Vectors can have any number of dimensions. The most common are the 2-dimensional cartesian plane, and the 3-dimensional space. Vectors in 2 and 3 dimensions are used often for pedgagogical purposes since we can visualize them as geometric vectors. Nevetheless, most problems in machine learning entail more dimensions, sometiome hundreds or thousands of dimensions. The notation for a vector $\bf{x}$ of arbitrary dimensions, $n$ is:

Vectors dimensions map into coordinate systems or perpendicular axes. Coordinate systems have an origin at $(0,0,0)$, hence, when we define a vector:

$\bf{x} = \begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix} \in \mathbb{R}^3$

we are saying: starting from the origin, move 3 units in the 1st perpendicular axis, 2 units in the 2nd perpendicular axis, and 1 unit in the 3rd perpendicular axis. We will see later that when we have a set of perpendicular axes we obtain the basis of a vector space.

Basic vector operations

Vector-vector addition

We used vector-vector addition to define vectors without defining vector-vector addition. Vector-vector addition is an element-wise operation, only defined for vectors of the same size (i.e., number of elements). Consider two vectors of the same size, then:

For instance:

Vector addition has a series of fundamental properties worth mentioning:

1. Commutativity: $x + y = y + x$
2. Associativity: $(x + y) + z = x + (y + z)$
3. Adding the zero vector has no effect: $x + 0 = 0 + x = x$
4. Substracting a vector from itself returns the zero vector: $x - x = 0$

In NumPy, we add two vectors of the same with the + operator or the add method:

x = y = np.array([[1],
                  [2],
                  [3]])

x + y
"""
    array([[2],
           [4],
           [6]])
"""           

np.add(x,y)
"""
    array([[2],
           [4],
           [6]])
"""

Vector-scalar multiplication

Vector-scalar multiplication is an element-wise operation. It's defined as:

Consider $\alpha = 2$ and $\bf{x} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}$:

Vector-scalar multiplication satisfies a series of important properties:

1. Associativity: $(\alpha \beta) \bf{x} = \alpha (\beta \bf{x})$
2. Left-distributive property: $(\alpha + \beta) \bf{x} = \alpha \bf{x} + \beta \bf{x}$
3. Right-distributive property: $\bf{x} (\alpha + \beta) = \bf{x} \alpha + \bf{x} \beta$
4. Right-distributive property for vector addition: $\alpha (\bf{x} + \bf{y}) = \alpha \bf{x} + \alpha \bf{y}$

In NumPy, we compute scalar-vector multiplication with the * operator:

alpha = 2
x = np.array([[1],
             [2],
             [3]])

alpha * x
"""
    array([[2],
           [4],
           [6]])
"""

Linear combinations of vectors

There are only two legal operations with vectors in linear algebra: addition and multiplication by numbers. When we combine those, we get a linear combination.

Consider $\alpha = 2$, $\beta = 3$, $\bf{x}=\begin{bmatrix}2 \\ 3\end{bmatrix}$, and $\begin{bmatrix}4 \\ 5\end{bmatrix}$.

We obtain:

Another way to express linear combinations you'll see often is with summation notation. Consider a set of vectors $x_1, ..., x_k$ and scalars $\beta_1, ..., \beta_k \in \mathbb{R}$, then:

Note that $:=$ means "is defined as".

Linear combinations are the most fundamental operation in linear algebra. Everything in linear algebra results from linear combinations. For instance, linear regression is a linear combination of vectors. Fig. 2 shows an example of how adding two geometrical vectors looks like for intuition.

In NumPy, we do linear combinations as:

a, b = 2, 3
x , y = np.array([[2],[3]]), np.array([[4], [5]])

a*x + b*y
"""
    array([[16],
           [21]])
"""

Vector-vector multiplication: dot product

We covered vector addition and multiplication by scalars. Now I will define vector-vector multiplication, commonly known as a dot product or inner product. The dot product of $\bf{x}$ and $\bf{y}$ is defined as:

Where the $T$ superscript denotes the transpose of the vector. Transposing a vector just means to "flip" the column vector to a row vector counterclockwise. For instance:

Dot products are so important in machine learning, that after a while they become second nature for practitioners.

To multiply two vectors with dimensions (rows=2, cols=1) in Numpy, we need to transpose the first vector at using the @ operator:

x, y = np.array([[-2],[2]]), np.array([[4],[-3]])

x.T @ y
# array([[-14]])

Vector space, span, and subspace

Vector space

In its more general form, a vector space, also known as linear space, is a collection of objects that follow the rules defined for vectors in $\mathbb{R}^n$. We mentioned those rules when we defined vectors: they can be added together and multiplied by scalars, and return vectors of the same type. More colloquially, a vector space is the set of proper vectors and all possible linear combinatios of the vector set. In addition, vector addition and multiplication must follow these eight rules:

1. commutativity: $x + y = y + x$
2. associativity: $x + (y + x) = (y + x) + z$
3. unique zero vector such that: $x + 0 = x$ $\forall$ $x$
4. $\forall$ $x$ there is a unique vector $x$ such that $x + -x = 0$
5. identity element of scalar multiplication: $1x = x$
6. distributivity of scalar multiplication w.r.t vector addition: $x(y + z) = xz + zy$
7. $x(yz) = (xy)z$
8. $(y + z)x = yx + zx$

In my experience remembering these properties is not really important, but it's good to know that such rules exist.

Vector span

Consider the vectors $\bf{x}$ and $\bf{y}$ and the scalars $\alpha$ and $\beta$. If we take all possible linear combinations of $\alpha \bf{x} + \beta \bf{y}$ we would obtain the span of such vectors. This is easier to grasp when you think about geometric vectors. If our vectors $\bf{x}$ and $\bf{y}$ point into different directions in the 2-dimensional space, we get that the $span(x,y)$ is equal to the entire 2-dimensional plane, as shown in the middle-pane in Fig. 5. Just imagine having an unlimited number of two types of sticks: one pointing vertically, and one pointing horizontally. Now, you can reach any point in the 2-dimensional space by simply combining the necessary number of vertical and horizontal sticks (including taking fractions of sticks).

What would happen if the vectors point in the same direction? Now, if you combine them, you just can span a line, as shown in the left-pane in Fig. 5. If you have ever heard of the term "multicollinearity", it's closely related to this issue: when two variables are "colinear" they are pointing in the same direction, hence they provide redundant information, so can drop one without information loss.

With three vectors pointing into different directions, we can span the entire 3-dimensional space or a hyper-plane, as in the right-pane of Fig. 5. Note that the sphere is just meant as a 3-D reference, not as a limit.

Four vectors pointing into different directions will span the 4-dimensional space, and so on. From here our geometrical intuition can't help us. This is an example of how linear algebra can describe the behavior of vectors beyond our basics intuitions.

Vector subspaces

A vector subspace (or linear subspace) is a vector space that lies within a larger vector space. These are also known as linear subspaces. Consider a subspace $S$. For a vector to be a valid subspace it has to meet three conditions:

1. Contains the zero vector, $\bf{0} \in S$
2. Closure under multiplication, $\forall \alpha \in \mathbb{R} \rightarrow \alpha \times s_i \in S$
3. Closure under addition, $\forall s_i \in S \rightarrow s_1 + s_2 \in S$

Intuitively, you can think in closure as being unable to "jump out" from space into another. A pair of vectors laying flat in the 2-dimensional space, can't, by either addition or multiplication, "jump out" into the 3-dimensional space.

Consider the following questions: Is $\bf{x}=\begin{bmatrix} 1 \\ 1 \end{bmatrix}$ a valid subspace of $\mathbb{R^2}$? Let's evaluate $\bf{x}$ on the three conditions:

Contains the zero vector: it does. Remember that the span of a vector are all linear combinations of such a vector. Therefore, we can simply multiply by $0$ to get $\begin{bmatrix}0 \\ 0 \end{bmatrix}$:

Closure under multiplication implies that if take any vector belonging to $\bf{x}$ and multiply by any real scalar $\alpha$, the resulting vector stays within the span of $\bf{x}$. Algebraically is easy to see that we can multiply $\begin{bmatrix} 1 \\ 1 \end{bmatrix}$ by any scalar $\alpha$, and the resulting vector remains in the 2-dimensional plane (i.e., the span of $\mathbb{R}^2$).

Closure under addition implies that if we add together any vectors belonging to $\bf{x}$, the resulting vector remains within the span of $\mathbb{R}^2$. Again, algebraically is clear that if we add $\bf{x}$ + $\bf{x}$, the resulting vector will remain in $\mathbb{R}^2$. There is no way to get to $\mathbb{R^3}$ or $\mathbb{R^4}$ or any space outside the two-dimensional plane by adding $\bf{x}$ multiple times.

Linear dependence and independence

The left-pane shows a triplet of linearly dependent vectors, whereas the right-pane shows a triplet of linearly independent vectors.

A set of vectors is linearly dependent if at least one vector can be obtained as a linear combination of other vectors in the set. As you can see in the left pane, we can combine vectors $x$ and $y$ to obtain $z$.

There is more rigurous (but slightly harder to grasp) definition of linear dependence. Consider a set of vectors $x_1, ..., x_k$ and scalars $\beta \in \mathbb{R}$. If there is a way to get $0 = \sum_{i=1}^k \beta_i x_i$ with at least one $\beta \ne 0$, we have linearly dependent vectors. In other words, if we can get the zero vector as a linear combination of the vectors in the set, with weights that are not all zero, we have a linearly dependent set.

A set of vectors is linearly independent if none vector can be obtained as a linear combination of other vectors in the set. As you can see in the right pane, there is no way for us to combine vectors $x$ and $y$ to obtain $z$. Again, consider a set of vectors $x_1, ..., x_k$ and scalars $\beta \in \mathbb{R}$. If the only way to get $0 = \sum_{i=1}^k \beta_i x_i$ requires all $\beta_1, ..., \beta_k = 0$, the we have linearly independent vectors. In words, the only way to get the zero vectors in by multoplying each vector in the set by $0$.

The importance of the concepts of linear dependence and independence will become clearer in more advanced topics. For now, the important points to remember are: linearly dependent vectors contain redundant information, whereas linearly independent vectors do not.

Vector null space

Now that we know what subspaces and linear dependent vectors are, we can introduce the idea of the null space. Intuitively, the null space of a set of vectors are all linear combinations that "map" into the zero vector. Consider a set of geometric vectors $\bf{w}$, $\bf{x}$, $\bf{y}$, and $\bf{z}$ as in Fig. 8. By inspection, we can see that vectors $\bf{x}$ and $\bf{z}$ are parallel to each other, hence, independent. On the contrary, vectors $\bf{w}$ and $\bf{y}$ can be obtained as linear combinations of $\bf{x}$ and $\bf{z}$, therefore, dependent.

As result, with this four vectors, we can form the following two combinations that will "map" into the origin of the coordinate system, this is, the zero vector $(0,0)$:

We will see how this idea of the null space extends naturally in the context of matrices later.

Vector norms

Measuring vectors is another important operation in machine learning applications. Intuitively, we can think about the norm or the length of a vector as the distance between its "origin" and its "end".

Norms "map" vectors to non-negative values. In this sense are functions that assign length $\lVert \bf{x} \rVert \in \mathbb{R^n}$ to a vector $\bf{x}$. To be valid, a norm has to satisfy these properties (keep in mind these properties are a bit abstruse to understand):

1. Absolutely homogeneous: $\forall \alpha \in \mathbb{R}, \lVert \alpha \bf{x} \rVert = \vert \alpha \Vert \lVert \bf{x} \rVert$. In words: for all real-valued scalars, the norm scales proportionally with the value of the scalar.
2. Triangle inequality: $\lVert \bf{x} + \bf{y} \rVert \le \lVert \bf{x} \rVert + \lVert \bf{y} \rVert$. In words: in geometric terms, for any triangle the sum of any two sides must be greater or equal to the lenght of the third side. This is easy to see experimentally: grab a piece of rope, form triangles of different sizes, measure all the sides, and test this property.
3. Positive definite: $\lVert \bf{x} \rVert \ge 0$ and $\lVert \bf{x} \rVert = 0 \Longleftrightarrow \bf{x}= 0$. In words: the length of any $\bf{x}$ has to be a positive value (i.e., a vector can't have negative length), and a length of $0$ occurs only of $\bf{x}=0$

Grasping the meaning of these three properties may be difficult at this point, but they probably become clearer as you improve your understanding of linear algebra.

Euclidean norm

The Euclidean norm is one of the most popular norms in machine learning. It is so widely used that sometimes is refered simply as "the norm" of a vector. Is defined as:

Hence, in two dimensions the $L_2$ norm is:

Which is equivalent to the formula for the hypotenuse a triangle with sides $x_1^2$ and $x_2^2$.

The same pattern follows for higher dimensions of $\mathbb{R^n}$

In NumPy, we can compute the $L_2$ norm as:

x = np.array([[3],[4]])

np.linalg.norm(x, 2)
# 5.0

If you remember the first "Pythagorean triple", you can confirm that the norm is correct.

Manhattan norm

The Manhattan or $L_1$ norm gets its name in analogy to measuring distances while moving in Manhattan, NYC. Since Manhattan has a grid-shape, the distance between any two points is measured by moving in vertical and horizontals lines (instead of diagonals as in the Euclidean norm). It is defined as:

Where $\vert x_i \vert$ is the absolute value. The $L_1$ norm is preferred when discriminating between elements that are exactly zero and elements that are small but not zero.

In NumPy we compute the $L_1$ norm as

x = np.array([[3],[-4]])

np.linalg.norm(x, 1)
# 7.0

Is easy to confirm that the sum of the absolute values of $3$ and $-4$ is $7$.

Max norm

The max norm or infinity norm is simply the absolute value of the largest element in the vector. It is defined as:

Where $\vert x_i \vert$ is the absolute value. For instance, for a vector with elements $\bf{x} = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}$, the $\lVert \bf{x} \rVert_\infty = 3$

In NumPy we compute the $L_\infty$ norm as:

x = np.array([[3],[-4]])

np.linalg.norm(x, np.inf)
# 4.0

Vector inner product, length, and distance.

For practical purposes, inner product and length are used as equivalent to dot product and norm, although technically are not the same.

Inner products are a more general concept that dot products, with a series of additional properties (see here). In other words, every dot product is an inner product, but not every inner product is a dot product. The notation for the inner product is usually a pair of angle brackets as $\langle .,. \rangle$ as. For instance, the scalar inner product is defined as:

In $\mathbb{R}^n$ the inner product is a dot product defined as:

Length is a concept from geometry. We say that geometric vectors have length and that vectors in $\mathbb{R}^n$ have norm. In practice, many machine learning textbooks use these concepts interchangeably. I've found authors saying things like "we use the $l_2$ norm to compute the length of a vector". For instance, we can compute the length of a directed segment (i.e., geometrical vector) $\bf{x}$ by taking the square root of the inner product with itself as:

Distance is a relational concept. It refers to the length (or norm) of the difference between two vectors. Hence, we use norms and lengths to measure the distance between vectors. Consider the vectors $\bf{x}$ and $\bf{y}$, we define the distance $d(x,y)$ as:

When the inner product $\langle x - y, x - y \rangle$ is the dot product, the distance equals to the Euclidean distance.

In machine learning, unless made explicit, we can safely assume that an inner product refers to the dot product. We already reviewed how to compute the dot product in NumPy:

x, y = np.array([[-2],[2]]), np.array([[4],[-3]])
x.T @ y 
# array([[-14]])

As with the inner product, usually, we can safely assume that distance stands for the Euclidean distance or $L_2$ norm unless otherwise noted. To compute the $L_2$ distance between a pair of vectors:

distance = np.linalg.norm(x-y, 2)
print(f'L_2 distance : {distance}')
# L_2 distance : 7.810249675906656

Vector angles and orthogonality

The concepts of angle and orthogonality are also related to geometrical vectors. We saw that inner products allow for the definition of length and distance. In the same manner, inner products are used to define angles and orthogonality.

In machine learning, the angle between a pair of vectors is used as a measure of vector similarity. To understand angles let's first look at the Cauchy–Schwarz inequality. Consider a pair of non-zero vectors $\bf{x}$ and $\bf{y}$ $\in \mathbb{R}^n$. The Cauchy–Schwarz inequality states that:

In words: the absolute value of the inner product of a pair of vectors is less than or equal to the products of their length. The only case where both sides of the expression are equal is when vectors are colinear, for instance, when $\bf{x}$ is a scaled version of $\bf{y}$. In the 2-dimensional case, such vectors would lie along the same line.

The definition of the angle between vectors can be thought as a generalization of the law of cosines in trigonometry, which defines for a triangle with sides $a$, $b$, and $c$, and an angle $\theta$ are related as:

We can replace this expression with vectors lengths as:

With a bit of algebraic manipulation, we can clear the previous equation to:

And there we have a definition for (cos) angle $\theta$. Further, from the Cauchy–Schwarz inequality we know that $\cos \theta$ must be:

This is a necessary conclusion (range between $\{-1, 1\}$) since the numerator in the equation always is going to be smaller or equal to the denominator.

In NumPy, we can compute the $\cos \theta$ between a pair of vectors as:

x, y = np.array([[1], [2]]), np.array([[5], [7]])

# here we translate the cos(theta) definition
cos_theta = (x.T @ y) / (np.linalg.norm(x,2) * np.linalg.norm(y,2))
print(f'cos of the angle = {np.round(cos_theta, 3)}')
# cos of the angle = [[0.988]]

We get that $\cos \theta \approx 0.988$. Finally, to know the exact value of $\theta$ we need to take the trigonometric inverse of the cosine function as:

cos_inverse = np.arccos(cos_theta)
print(f'angle in radiants = {np.round(cos_inverse, 3)}')
# angle in radiants = [[0.157]]

We obtain $\theta \approx 0.157$. To fo from radiants to degrees we can use the following formula:

degrees = cos_inverse * ((180)/np.pi)
print(f'angle in degrees = {np.round(degrees, 3)}')
# angle in degrees = [[8.973]]

We obtain $\theta \approx 8.973^{\circ}$

Orthogonality is often used interchangeably with "independence" although they are mathematically different concepts. Orthogonality can be seen as a generalization of perpendicularity to vectors in any number of dimensions.

We say that a pair of vectors $\bf{x}$ and $\bf{y}$ are orthogonal if their inner product is zero, $\langle x,y \rangle = 0$. The notation for a pair of orthogonal vectors is $\bf{x} \perp \bf{y}$. In the 2-dimensional plane, this equals to a pair of vectors forming a $90^{\circ}$ angle.

Here is an example of orthogonal vectors

x = np.array([[2], [0]])
y = np.array([[0], [2]])

cos_theta = (x.T @ y) / (np.linalg.norm(x,2) * np.linalg.norm(y,2))
print(f'cos of the angle = {np.round(cos_theta, 3)}')
# cos of the angle = [[0.]]

We see that this vectors are orthogonal as $\cos \theta=0$. This is equal to $\approx 1.57$ radiants and $\theta = 90^{\circ}$

cos_inverse = np.arccos(cos_theta)
degrees = cos_inverse * ((180)/np.pi)
print(f'angle in radiants = {np.round(cos_inverse, 3)}\nangle in degrees ={np.round(degrees, 3)} ')
"""
    angle in radiants = [[1.571]]
    angle in degrees =[[90.]] 
"""

Systems of linear equations

The purpose of linear algebra as a tool is to solve systems of linear equations. Informally, this means to figure out the right combination of linear segments to obtain an outcome. Even more informally, think about making pancakes: In what proportion ($w_i \in \mathbb{R}$) we have to mix ingredients to make pancakes? You can express this as a linear equation:

The above expression describe a linear equation. A system of linear equations involve multiple equations that have to be solved simultaneously. Consider:

Now we have a system with two unknowns, $x$ and $y$. We'll see general methods to solve systems of linear equations later. For now, I'll give you the answer: $x=2$ and $y=3$. Geometrically, we can see that both equations produce a straight line in the 2-dimensional plane. The point on where both lines encounter is the solution to the linear system.

df = pd.DataFrame({"x1": [0, 2], "y1":[8, 3], "x2": [0.5, 2], "y2": [0, 3]})

equation1 = alt.Chart(df).mark_line().encode(x="x1", y="y1")
equation2 = alt.Chart(df).mark_line(color="red").encode(x="x2", y="y2")
equation1 + equation2

Matrices

# Libraries for this section 
import numpy as np
import pandas as pd
import altair as alt

Matrices are as fundamental as vectors in machine learning. With vectors, we can represent single variables as sets of numbers or instances. With matrices, we can represent sets of variables. In this sense, a matrix is simply an ordered collection of vectors. Conventionally, column vectors, but it's always wise to pay attention to the authors' notation when reading matrices. Since computer screens operate in two dimensions, matrices are the way in which we interact with data in practice.

More formally, we represent a matrix with a italicized upper-case letter like $\textit{A}$. In two dimensions, we say the matrix $\textit{A}$ has $m$ rows and $n$ columns. Each entry of $\textit{A}$ is defined as $a_{ij}$, $i=1,..., m,$ and $j=1,...,n$. A matrix $\textit{A} \in \mathbb{R^{m\times n}}$ is defines as:

In Numpy, we construct matrices with the array method:

A = np.array([[0,2],  # 1st row
              [1,4]]) # 2nd row

print(f'a 2x2 Matrix:\n{A}')
"""
    a 2x2 Matrix:
    [[0 2]
     [1 4]]
"""

Basic Matrix operations

Matrix-matrix addition

We add matrices in a element-wise fashion. The sum of $\textit{A} \in \mathbb{R}^{m\times n}$ and $\textit{B} \in \mathbb{R}^{m\times n}$ is defined as:

For instance:

In Numpy, we add matrices with the + operator or add method:

A = np.array([[0,2],
              [1,4]])
B = np.array([[3,1],
              [-3,2]])

A + B
"""
    array([[ 3,  3],
           [-2,  6]])
"""

np.add(A, B)
"""
    array([[ 3,  3],
           [-2,  6]])
"""

Matrix-scalar multiplication

Matrix-scalar multiplication is an element-wise operation. Each element of the matrix $\textit{A}$ is multiplied by the scalar $\alpha$. Is defined as:

Consider $\alpha=2$ and $\textit{A}=\begin{bmatrix}1 & 2 \\ 3 & 4\end{bmatrix}$, then:

In NumPy, we compute matrix-scalar multiplication with the * operator or multiply method:

alpha = 2
A = np.array([[1,2],
              [3,4]])

alpha * A
"""
    array([[2, 4],
           [6, 8]])
"""

np.multiply(alpha, A)
"""
    array([[2, 4],
           [6, 8]])
"""

Matrix-vector multiplication: dot product

Matrix-vector multiplication equals to taking the dot product of each column $n$ of a $\textit{A}$ with each element $\bf{x}$ resulting in a vector $\bf{y}$. Is defined as:

For instance:

In numpy, we compute the matrix-vector product with the @ operator or dot method:

A = np.array([[0,2],
              [1,4]])
x = np.array([[1],
              [2]])

A @ x
"""
    array([[4],
           [9]])
"""

np.dot(A, x)
"""
    array([[4],
           [9]])
"""

Matrix-matrix multiplication

Matrix-matrix multiplication is a dot produt as well. To work, the number of columns in the first matrix $\textit{A}$ has to be equal to the number of rows in the second matrix $\textit{B}$. Hence, $\textit{A} \in \mathbb{R^{m\times n}}$ times $\textit{B} \in \mathbb{R^{n\times p}}$ to be valid. One way to see matrix-matrix multiplication is by taking a series of dot products: the 1st column of $\textit{A}$ times the 1st row of $\textit{B}$, the 2nd column of $\textit{A}$ times the 2nd row of $\textit{B}$, until the $n_{th}$ column of $\textit{A}$ times the $n_{th}$ row of $\textit{B}$.

We define $\textit{A} \in \mathbb{R^{n\times p}} \cdot \textit{B} \in \mathbb{R^{n\times p}} = \textit{C} \in \mathbb{R^{m\times p}}$: