Chapter 1: Fundamentals of Vectors and Linear Operations

Introduction

Linear algebra rests on two fundamental operations: vector addition and scalar multiplication. These operations form the foundation for understanding how vectors combine and transform, which is the core of everything in this book. Starting with vectors in two and three dimensions (which we can actually visualize), the chapter gracefully extends these ideas into higher-dimensional spaces, where our intuition from 2D and 3D completely carries over—even though we can't draw them.

The beauty of linear algebra is that the mental pictures you build working with 2D and 3D vectors remain perfectly valid when working with 10-dimensional vectors or 1000-dimensional vectors. The fundamental concepts don't change; they scale elegantly into higher dimensions.

1.1 Vectors and Linear Combinations

What Are Vectors?

Vector: A column of numbers (called components) that can represent direction and magnitude in space. For example, a two-dimensional vector v has two components: $v_1$ (the horizontal part) and $v_2$ (the vertical part).

We write vectors as columns, not rows. This distinction matters because it affects how we perform operations:

$\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix}$

Why do we use vectors? Think of it this way: we have two separate pieces of information ( $v_1$ and $v_2$ ), and we need to treat them as a single unit. The phrase "you can't add apples and oranges" captures the reason—we keep the horizontal and vertical components separate from each other, never mixing them during operations.

Vector Addition

The first operation is vector addition, where we add two vectors by adding their corresponding components separately.

$\text{If } \mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \text{ and } \mathbf{w} = \begin{bmatrix} w_1 \\ w_2 \end{bmatrix}, \text{ then } \mathbf{v} + \mathbf{w} = \begin{bmatrix} v_1 + w_1 \\ v_2 + w_2 \end{bmatrix}$

Geometrically: Vector addition follows the parallelogram rule. If you place the tail of w at the head of v, the sum $\mathbf{v} + \mathbf{w}$ is the vector from the origin to the opposite corner of the parallelogram.

Subtraction works the same way—the components of $\mathbf{v} - \mathbf{w}$ are simply $v_1 - w_1$ and $v_2 - w_2$ .

Scalar Multiplication

The second operation is scalar multiplication: multiplying a vector by a number (called a scalar). To find $2\mathbf{v}$ , multiply each component of v by 2:

$2\mathbf{v} = \begin{bmatrix} 2v_1 \\ 2v_2 \end{bmatrix}$

More generally, for any scalar $c$ :

$c\mathbf{v} = \begin{bmatrix} cv_1 \\ cv_2 \end{bmatrix}$

Geometrically: Multiplying by a scalar stretches or shrinks the vector. If $c = 2$ , the vector doubles in length and points in the same direction. If $c = -1$ , the vector reverses direction but keeps the same length. If $0 < c < 1$ , the vector shrinks.

Key observation: The sum of $-\mathbf{v}$ and $\mathbf{v}$ gives the zero vector $\mathbf{0}$ , which has components 0 and 0. Don't confuse the zero vector with the number zero—they're different objects.

Linear Combinations

The most important concept in linear algebra emerges from combining addition and scalar multiplication: the linear combination.

Linear Combination: An expression of the form $c\mathbf{v} + d\mathbf{w}$ , where $c$ and $d$ are scalars (numbers) and v and w are vectors. You multiply v by $c$ , multiply w by $d$ , then add the results.

Why are linear combinations so important? They answer the fundamental question: What vectors can we reach by combining v and w? The answer depends on how v and w are positioned:

If v and w point in different directions (and neither is a multiple of the other), their linear combinations fill the entire 2D plane. You can reach any point.
If v and w point along the same line, their linear combinations also lie on that line. You can reach any point on the line, but nothing off it.

Example in 3D: The linear combinations of $\mathbf{v} = (1, 1, 0)$ and $\mathbf{w} = (0, 1, 1)$ fill a plane in 3D space (the plane that contains both vectors and all their combinations). The third direction perpendicular to this plane cannot be reached by combining v and w alone.

Visualization in Two and Three Dimensions

In 2D: Starting from the origin, a vector $\mathbf{v} = (a, b)$ points to the point $(a, b)$ on the coordinate plane. The linear combinations $c\mathbf{v} + d\mathbf{w}$ of two non-collinear vectors fill the entire xy-plane.

In 3D: A vector $\mathbf{v} = (x, y, z)$ points to the point $(x, y, z)$ in 3D space. Linear combinations of two non-collinear vectors fill a plane, while linear combinations of three non-coplanar vectors fill all of 3D space.

Higher dimensions: The same logic applies in $n$ dimensions, even though we can't visualize beyond 3D. The key is that if you have $n$ linearly independent vectors in $n$ -dimensional space, their linear combinations fill the entire space.

📝 Section Recap: Vector addition and scalar multiplication are the two core operations of linear algebra. Vectors are columns of numbers that represent points or directions. Linear combinations—expressions like $c\mathbf{v} + d\mathbf{w}$ —show what points we can reach by scaling and adding vectors. In 2D, two non-aligned vectors reach all of 2D space; in 3D, three non-coplanar vectors reach all of 3D space.

1.2 The Dot Product and Vector Length

Introducing the Dot Product

So far, we've only looked at adding and scaling vectors. But linear algebra needs another operation: something that measures how vectors relate to each other. That's where the dot product comes in.

Dot Product: For vectors $\mathbf{v} = (v_1, v_2)$ and $\mathbf{w} = (w_1, w_2)$ , the dot product (also written as the inner product) is the single number: $\mathbf{v} \cdot \mathbf{w} = v_1 w_1 + v_2 w_2$

The dot product pairs up corresponding components, multiplies them, and adds everything up. The result is a single number (a scalar), not a vector.

Computing the dot product:

Take the first component of v times the first component of w: $v_1 w_1$
Take the second component of v times the second component of w: $v_2 w_2$
Add them together: $v_1 w_1 + v_2 w_2$

In three dimensions: $\mathbf{v} \cdot \mathbf{w} = v_1 w_1 + v_2 w_2 + v_3 w_3$

In $n$ dimensions: $\mathbf{v} \cdot \mathbf{w} = v_1 w_1 + v_2 w_2 + \cdots + v_n w_n = \sum_{i=1}^{n} v_i w_i$

Geometric Meaning of the Dot Product

The dot product has a beautiful geometric interpretation. It measures how much two vectors "align" with each other:

$\mathbf{v} \cdot \mathbf{w} = \|\mathbf{v}\| \|\mathbf{w}\| \cos(\theta)$

where $\|\mathbf{v}\|$ and $\|\mathbf{w}\|$ are the lengths (or magnitudes) of the vectors, and $\theta$ is the angle between them.

What this tells us:

When the vectors point in the same direction ( $\theta = 0°$ , $\cos(0°) = 1$ ), the dot product is positive and large.
When the vectors are perpendicular ( $\theta = 90°$ , $\cos(90°) = 0$ ), the dot product is exactly zero.
When the vectors point in opposite directions ( $\theta = 180°$ , $\cos(180°) = -1$ ), the dot product is negative.

Perpendicularity condition: Two vectors are perpendicular (at right angles) if and only if $\mathbf{v} \cdot \mathbf{w} = 0$ .

Computing Vector Length

The length (or magnitude or norm) of a vector can be computed using the dot product:

$\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}} = \sqrt{v_1^2 + v_2^2}$

In three dimensions: $\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + v_3^2}$

This is just the Pythagorean theorem: the length of the vector is the square root of the sum of squared components.

Example: For $\mathbf{v} = (3, 4)$ , the length is $\|\mathbf{v}\| = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5$ .

Unit Vectors and Normalization

Unit Vector: A vector with length 1. Unit vectors are useful because they point in a direction without carrying information about "magnitude."

To convert any vector v into a unit vector pointing in the same direction, divide v by its length:

$\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|}$

The hat notation ( $\hat{\mathbf{v}}$ ) conventionally denotes a unit vector.

Example: If $\mathbf{v} = (3, 4)$ with length 5, then the unit vector is $\hat{\mathbf{v}} = \left(\frac{3}{5}, \frac{4}{5}\right)$ .

Key Properties of the Dot Product

The dot product follows several important rules:

Commutativity: $\mathbf{v} \cdot \mathbf{w} = \mathbf{w} \cdot \mathbf{v}$
Linearity: $\mathbf{v} \cdot (c\mathbf{w}) = c(\mathbf{v} \cdot \mathbf{w})$ and $\mathbf{v} \cdot (\mathbf{w} + \mathbf{u}) = \mathbf{v} \cdot \mathbf{w} + \mathbf{v} \cdot \mathbf{u}$
Self dot product: $\mathbf{v} \cdot \mathbf{v} = \|\mathbf{v}\|^2 \geq 0$ , and $\mathbf{v} \cdot \mathbf{v} = 0$ only when $\mathbf{v}$ is the zero vector

These properties make the dot product compatible with the other vector operations—it plays nicely with addition and scalar multiplication.

📝 Section Recap: The dot product measures how two vectors align. Computed as $\mathbf{v} \cdot \mathbf{w} = v_1 w_1 + v_2 w_2 + \cdots$ , it's zero when vectors are perpendicular. Vector length comes from the dot product: $\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}}$ . Unit vectors (length 1) are obtained by normalizing: $\hat{\mathbf{v}} = \mathbf{v}/\|\mathbf{v}\|$ .

1.3 Matrices, Linear Equations, and Systems

What Is a Matrix?

Moving beyond individual vectors, we often need to work with multiple vectors at once, or we need to understand operations that transform vectors. This is where matrices enter the picture.

Matrix: A rectangular array of numbers arranged in rows and columns. An $m \times n$ matrix (read " $m$ by $n$ ") has $m$ rows and $n$ columns.

A matrix looks like this:

$A = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix}$

This is a $2 \times 3$ matrix (2 rows, 3 columns). The notation $a_{ij}$ means the entry in row $i$ , column $j$ .

Why matrices matter: Matrices are the natural way to represent systems of linear equations. They also represent linear transformations—ways of transforming one vector into another.

Linear Equations and the Matrix Form $A\mathbf{x} = \mathbf{b}$

Suppose we have a system of linear equations:

2x + 3y &= 5 \\ x - y &= 1 \end{align}$$ We can write this as a single matrix equation: $$A\mathbf{x} = \mathbf{b}$$ where: - $A = \begin{bmatrix} 2 & 3 \\ 1 & -1 \end{bmatrix}$ is the **coefficient matrix** - $\mathbf{x} = \begin{bmatrix} x \\ y \end{bmatrix}$ is the **unknown vector** - $\mathbf{b} = \begin{bmatrix} 5 \\ 1 \end{bmatrix}$ is the **right-hand side vector** The matrix equation $A\mathbf{x} = \mathbf{b}$ compactly represents the entire system. ### The Row Picture vs. the Column Picture There are two ways to visualize the solution to $A\mathbf{x} = \mathbf{b}$: **The Row Picture:** Each row of the matrix $A$ represents one linear equation. The solution $\mathbf{x}$ is the point where all these equations are satisfied simultaneously (geometrically, the intersection of lines or planes). For example, in 2D: - First equation: $2x + 3y = 5$ (a line) - Second equation: $x - y = 1$ (another line) - The solution is their intersection point **The Column Picture:** We rewrite $A\mathbf{x} = \mathbf{b}$ as asking: *Can we write **b** as a linear combination of the columns of $A$?* $$A\mathbf{x} = x \cdot (\text{column 1 of } A) + y \cdot (\text{column 2 of } A) + \cdots = \mathbf{b}$$ In this view, $\mathbf{x} = (x, y)$ contains the coefficients that we multiply the columns by to get **b**. **Why the column picture matters:** It directly connects back to linear combinations! We're asking whether **b** lies in the space spanned by the columns of $A$. If it does, the system has a solution. If it doesn't, the system is inconsistent. ### Existence and Uniqueness of Solutions Three cases can occur when solving $A\mathbf{x} = \mathbf{b}$: 1. **Unique solution:** There is exactly one vector **x** that satisfies the equation. This happens when the columns of $A$ span the space and are linearly independent. 2. **No solution:** There is no vector **x** that works. This happens when **b** is not in the space spanned by the columns of $A$. The system is *inconsistent*. 3. **Infinitely many solutions:** There are infinitely many vectors **x** that satisfy the equation. This happens when the columns of $A$ are linearly dependent (one is a combination of the others), so the system is *underdetermined*. ### Geometric Intuition in 2D and 3D **In 2D:** The system $A\mathbf{x} = \mathbf{b}$ represents two lines. They might intersect at one point (unique solution), be parallel (no solution), or be the same line (infinitely many solutions). **In 3D:** The system represents planes. Three planes might meet at a single point, have no common point, or have a whole line or plane of intersection. **The connection to vectors:** All of this geometry ultimately comes back to whether **b** can be written as a linear combination of the columns of $A$—which is why linear combinations are so central to linear algebra. ### Solving Linear Systems: A Preview The textbook will later introduce systematic methods (like **Gaussian elimination**) to solve $A\mathbf{x} = \mathbf{b}$ for any matrix and right-hand side. For now, recognize that: - The problem $A\mathbf{x} = \mathbf{b}$ is asking: *Express **b** as a linear combination of the columns of $A$.* - The solution $\mathbf{x}$ contains the coefficients of that linear combination. - The number of solutions (0, 1, or infinitely many) depends on the structure of the columns. > 📝 **Section Recap:** Matrices organize coefficients of linear equations into a compact form $A\mathbf{x} = \mathbf{b}$. The row picture interprets this geometrically as the intersection of equations; the column picture asks whether **b** is a linear combination of the columns of $A$. Solutions may be unique, nonexistent, or infinite, depending on the columns' structure and whether they span the space containing **b**. --- ## Connecting the Concepts The three sections of Chapter 1 form a logical progression: 1. **Vectors and linear combinations** teach us the fundamental operations and show what linear combinations can produce. 2. **The dot product and length** give us tools to measure relationships between vectors—perpendicularity, direction, and magnitude. 3. **Matrices and linear equations** show how multiple vectors and equations combine, leading to the central problem of linear algebra: solving $A\mathbf{x} = \mathbf{b}$. All of these ideas—vectors, combinations, dot products, matrices, and systems—are inseparable. They form a unified framework that extends seamlessly into higher dimensions, making linear algebra a powerful tool for mathematics, physics, engineering, and computer science. The mental picture you develop now, working with 2D and 3D vectors, remains valid and intuitive no matter what dimension you work in later. --- ## Key Definitions Summary > **Vector:** A column of numbers representing a point or direction in space. > **Linear Combination:** An expression $c\mathbf{v} + d\mathbf{w}$ combining scaled vectors. > **Dot Product:** The scalar $\mathbf{v} \cdot \mathbf{w} = v_1 w_1 + v_2 w_2 + \cdots$ > **Vector Length:** The magnitude $\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}}$ > **Unit Vector:** A vector with length 1, denoted $\hat{\mathbf{v}} = \mathbf{v}/\|\mathbf{v}\|$ > **Perpendicular Vectors:** Vectors **v** and **w** such that $\mathbf{v} \cdot \mathbf{w} = 0$. > **Matrix:** A rectangular array of numbers with $m$ rows and $n$ columns. > **Linear System:** The matrix equation $A\mathbf{x} = \mathbf{b}$, where we solve for the unknown vector **x**. > **Column Picture:** Interpreting $A\mathbf{x} = \mathbf{b}$ as asking whether **b** is a linear combination of the columns of $A$. --- ## Checklist for Understanding As you move forward in linear algebra, make sure you can: - Compute linear combinations of vectors by hand - Visualize what regions (lines, planes, spaces) are filled by linear combinations - Calculate dot products and vector lengths - Determine when two vectors are perpendicular - Translate between a system of linear equations and matrix form $A\mathbf{x} = \mathbf{b}$ - Distinguish between the row picture and column picture of a linear system - Reason about whether a system has no solution, one solution, or infinitely many solutions - Extend these concepts mentally from 2D and 3D to higher dimensions, even if you can't visualize them

Fundamentals of Vectors and Linear Operations