Chapter 1: Limits, Continuity, and Differentiation#

Imagine you zoom in on a smooth curve – eventually it looks almost like a straight line. That simple observation is the idea behind calculus, and behind the numerical methods we use to solve differential equations. In this chapter we make that intuition precise, building the foundation you will need whenever you study how functions behave, how to approximate them, and how small changes lead to predictable consequences.

The Big Picture#

Before we can work with differential equations numerically, we have to be absolutely clear about what it means for a function to be continuous, to have a derivative, and to be approximated by simpler polynomials. This chapter answers one core question: how can we describe and control the local behaviour of a function using limits? We’ll start with the rigorous epsilon‑delta definition of a limit, then use it to define continuity, its stronger version uniform continuity, and the different kinds of jumps and breaks a function can have. From there we build the derivative, Taylor approximations with error bounds, and the handy “big O” and “little o” language that lets us compare growth rates without drowning in algebra. Finally, we extend everything to functions of several variables, where the chain rule and implicit differentiation become essential tools for changing coordinates and solving systems.

Limits and the epsilon‑delta definition#

We say (“the limit of as approaches is ”) when we can make as close to as we like by taking close enough to . To turn this into a rigorous, no‑room‑for‑doubt statement, we introduce two small positive numbers: (the tolerance we demand on the output) and (the window we choose on the input).

Limit (epsilon‑delta): if for every there exists a such that whenever , we have .

Think of it as a game. I (the challenger) give you any positive , no matter how tiny. You must find a so that whenever is within of (but not exactly ), the function value is within of . If you can always win this game, the limit exists and equals .

For a simple example, take and . We suspect the limit is . Given , we want . That inequality holds whenever . So choosing works, and the limit is indeed 7.

We can also talk about one‑sided limits: means we only approach from the left (), and approaches from the right. When both one‑sided limits exist and agree, the two‑sided limit exists and equals that common value.

Limits can be infinite. means that for every number (no matter how large) we can find a such that forces . We don’t say the limit “exists” as a number, but the notation is still useful.

📝 Section Recap: The epsilon‑delta definition turns the vague “approaches” into a precise contract. It is the formal heartbeat of everything that follows.

Continuity: pointwise and uniform#

A function is continuous at a point if it does not jump there: its limit equals its value.

Continuous at a point: is continuous at if .

In epsilon‑delta language, this means for every there is a such that . Notice we dropped the condition because we want the inequality to hold at as well.

Continuity on an interval just means the function is continuous at every point inside it. But there is a stronger, global property called uniform continuity. For ordinary continuity on an interval, the may depend on which point we are looking at. For uniform continuity, a single must work for every pair of points in the whole interval.

Uniform continuity: A function on a set is uniformly continuous if for every there exists a such that for all with , we have .

The practical difference: with uniform continuity, the same “window size” guarantees the output is within no matter where you are in the domain. For example, is uniformly continuous on the closed interval , but it is not uniformly continuous on . On the unbounded interval the function gets steeper and steeper, so to keep the change in under a fixed you need a smaller as grows – no single works everywhere.

A simple mental picture: think of a spring. If the spring has the same stiffness everywhere, pulling it by a small amount produces roughly the same force change at any point – this is like a uniformly continuous function on a bounded set. If the spring gets infinitely stiff as you stretch it further, the same tiny pull produces wildly different effects – that’s non‑uniform continuity.

📝 Section Recap: Continuity requires no gaps at a point; uniform continuity demands a consistent “resolution” of the function across the whole domain, a property that will be crucial when we study convergence of numerical methods.

Discontinuities: first and second kind#

When a function is not continuous at a point, the type of break matters. We classify discontinuities into first kind and second kind.

A discontinuity of the first kind occurs when both one‑sided limits exist but either they do not equal each other or they do not equal the function value. This can be a jump discontinuity (the two one‑sided limits are different) or a removable discontinuity (the limit exists but the function is defined differently or not defined there). For example, the step function for and for has a jump of size at . The function has a removable discontinuity at because , but is not initially defined; we could “fix” it by setting .

Discontinuity of the first kind: Both one‑sided limits and exist (as finite numbers), but the function is not continuous there.

A discontinuity of the second kind is everything else: at least one of the one‑sided limits either does not exist (infinite or oscillates without a limit) or is infinite. Classic examples are at (the limit from one side is , the other ) and at , where the function oscillates wildly between and infinitely often as approaches , so neither one‑sided limit exists.

Discontinuity of the second kind: At least one of the one‑sided limits does not exist as a finite number.

These classifications matter whenever a numerical method encounters a rough solution. A first‑kind jump usually means we can still work with a piecewise continuous function; second‑kind singularities often signal a fundamentally harder problem.

📝 Section Recap: Discontinuities come in two flavours: first‑kind (finite jumps or removable gaps) and second‑kind (infinite or oscillatory wildness). Knowing which one we face guides how we handle the function in computations.

Differentiation: the difference quotient and rules#

The derivative measures the instantaneous rate of change. It is defined as the limit of the difference quotient:

provided the limit exists. Geometrically, the slope of the secant line through and approaches the slope of the tangent as shrinks.

From the limit definition we can prove the basic differentiation rules. For instance, the product rule follows from a little algebraic trick: adding and subtracting :

where we used continuity of (which follows from differentiability) to replace by in the limit. The quotient rule, chain rule, and rules for sums and constant multiples are similarly rooted in the limit.

A crucial consequence: differentiability implies continuity. If exists, then

so as . The reverse is not true: is continuous at but not differentiable there.

Derivative: if the limit exists. It gives the slope of the tangent and represents the instantaneous rate of change.

📝 Section Recap: The derivative is a limit of slopes, and it carries with it a set of algebraic rules that let us compute derivatives of complicated functions without returning to the limit definition every time.

Taylor’s theorem and remainder#

One of the most powerful ideas in numerical analysis is that smooth functions near a point look like polynomials. Taylor’s theorem tells us how well a polynomial approximation matches the original function and how large the error can be.

For a function that is times differentiable on an interval containing and , we can write

where is the Taylor polynomial of degree and is the remainder. The Lagrange form of the remainder is especially useful:

for some strictly between and . This tells us that the error is proportional to , multiplied by a derivative evaluated somewhere in the interval.

A simple example: expand around . Since all derivatives of are , and , we get

with between and . If we only keep the first three terms to approximate , the error is at most , which is tiny.

Taylor’s theorem is an incredibly helpful tool for numerical methods: it gives us not only a polynomial surrogate for a function but also a bound on how far off we are. In later numerical work we will use it constantly to analyse errors that come from chopping off a series.

📝 Section Recap: Taylor’s theorem shows that any sufficiently smooth function can be written as a polynomial plus a controlled remainder, providing the basis for error analysis throughout numerical differential equations.

Asymptotic notation: big O and little o#

When we compare how fast functions go to zero (or grow) near a point, we need a shorthand. Big O and little o notation strip away constants and lower‑order terms so we can focus on the dominant behaviour.

Let be the point of interest (often or ). We say

Big O: as if there exist a constant and a neighbourhood of such that for all in that neighbourhood.

In words: grows no faster than a constant multiple of . For example, is as because for large . As , is because near .

Little o: as if .

This says becomes negligible compared with . For instance, as because . The Taylor remainder is often expressed as , meaning it vanishes faster than the ‑th power. More specifically, if the remainder is in Lagrange form we also have , because it is bounded by a constant times .

These notations let us write, for example,

which captures the fact that the next term is of order , and we don’t need its exact coefficient unless we need higher accuracy.

📝 Section Recap: Big O gives an upper bound, little o indicates one function is swamped by another. Together they make error analysis clean and compact.

Partial derivatives and the multivariable chain rule#

When a function depends on several variables, say , we can ask how it changes when we change one variable while holding the others fixed. That gives a partial derivative:

and similarly for .

A classic mental image: you are standing on a hillside. The partial derivative with respect to the east direction tells you the slope if you step purely east; the north derivative gives the slope if you step north. The real slope in any direction is a combination of these.

Now suppose and themselves depend on a single variable , so . The chain rule tells us how changes with :

This is just the mathematical version of “change from ” plus “change from ”.

We can package the partial derivatives into a row vector called the Jacobian matrix (for a scalar function it is the gradient). For a vector‑valued function from to , the Jacobian is the matrix of all first‑order partial derivatives. For the scalar case, the chain rule can be written as

The Jacobian becomes essential when changing variables in higher dimensions, because it captures how volume elements stretch.

📝 Section Recap: Partial derivatives measure change along coordinate directions, and the chain rule combines them when the coordinates themselves move. The Jacobian organises these sensitivities into a matrix, ready for larger systems.

Implicit function differentiation and the total differential#

Sometimes a relationship defines implicitly as a function of . We cannot always solve for explicitly, but we can still find using the total differential. Think of the small change in when and both change:

If we stay on the curve , then (because the value of does not change). So,

provided . For example, the unit circle yields , which we could also get by explicit differentiation.

When we have a system of equations where is a vector of implicit functions of , the same idea extends using the Jacobian. Write the total differential in matrix form:

where and are the Jacobian matrices of with respect to and . Solving for gives

provided the Jacobian is invertible. This formula is the workhorse behind many areas of applied mathematics, from sensitivity analysis to solving nonlinear systems.

📝 Section Recap: The total differential and the implicit function theorem let us compute derivatives of implicit relationships without explicit formulas. The Jacobian matrix becomes the key when we move beyond scalar functions.

Summary#

We started with a single idea: “close enough” can be made as strict as we need. From that, we built a precise epsilon‑delta definition of limits, used it to define continuity and the stronger version we call uniform continuity, and catalogued the two main types of breaks a function can have. Then we turned to rates of change—the derivative—and saw how Taylor’s theorem gives us polynomial approximations with a grip on the error. Big O and little o gave us a tidy language for comparing sizes without clutter. Finally, we lifted everything to functions of several variables, where partial derivatives, the chain rule, and the Jacobian let us handle implicit and explicit relationships alike. Every concept in this chapter is a tool you will reach for over and over when we begin to analyse differential equations numerically: they let us bound errors, understand stability, and construct approximations that work.

Here is a quick‑reference table for the key ideas: