Chapter 1: Foundations of the Cross-Sectional Dataset#

Think of a chef who wants to test which ingredients make a dish tastier. If the recipe uses spoiled vegetables or forgets to measure some items entirely, the results won’t mean much. In studying why some stocks earn higher returns than others, the “ingredients” are stock returns, and the “recipe” is the database we build. This chapter is about laying that foundation: how to pull U.S. stock returns from the CRSP database, clean them up, and construct the key variables that will later help us understand why some stocks earn higher returns than others.

The Big Picture#

Every question about differences in stock returns across companies—like “do small companies outperform large ones?” or “does a stock’s past return predict its future?”—depends on the quality of the dataset we use. If we accidentally include wrong security types, ignore firms that disappeared, or forget to subtract the safe interest rate, we can see patterns that aren’t really there, or miss real ones. This chapter walks through the essential steps to build a reliable dataset of U.S. stocks from the Center for Research in Security Prices (CRSP). We’ll learn how to identify ordinary common stocks, measure their size, fix the bias from missing delisting returns, and define excess returns. By the end, you’ll have a clean sample ready for serious analysis.

From the Database to a Usable Sample: Identifying Common Stocks#

CRSP is a rich database that contains daily and monthly price, return, and share information on securities traded on U.S. exchanges going back to 1925. But not everything in CRSP is a plain-vanilla common stock. There are real estate investment trusts, closed-end funds, American depositary receipts, and other instruments that behave very differently. Including them can muddy our study of the typical stock’s return.

To filter down to ordinary U.S. common shares, we use two numeric codes provided by CRSP: the share code and the exchange code.

Share code: A two‑digit number that identifies the type of security. For example, codes 10 and 11 indicate ordinary common shares; code 72 indicates a real estate investment trust (REIT); code 73 is a closed-end fund.

The most common screen is to keep only stocks with share codes 10 or 11. Why? These are the straightforward common equities that represent ownership in a single company. Code 11 is used for companies incorporated outside the U.S. but with a similar claim to common equity; many researchers include both 10 and 11 to capture a broader universe while excluding ADRs, which have separate share code 30.

Next, exchange codes tell us where a stock trades. The major U.S. exchanges have their own codes in CRSP:

• NYSE: code 1
• AMEX: code 2
• NASDAQ: code 3

For a typical cross‑sectional study, we include stocks from all three exchanges to get the most representative picture of the U.S. equity market. Some early work focused only on NYSE stocks because that was the most comprehensive data available for many decades, but now it’s standard to use the full NYSE‑AMEX‑NASDAQ universe. We’ll also want to exclude stocks that trade on smaller, less liquid venues (exchange codes 4, 5, etc.) unless we have a specific reason to include them.

We aren’t done yet. Even among share codes 10 and 11, there can be odd observations—a price of zero, a negative number of shares, a return of +5000% from a tiny penny stock. Many researchers apply a gentle price filter (for example, require month‑end price above $1 or $5) to avoid micro‑cap distortions. However, such filters are usually applied later when we define our sample period or study specific anomalies. At this stage, we simply gather the correct security types.

A small but important detail: CRSP sometimes reports a negative price. This is a convention that signals the price is actually the midpoint of the closing bid and ask, not a true negative value. When we compute market capitalization later, we take the absolute value of price. We’ll see that in the next section.

By screening on share codes and exchange codes, we build a dataset that accurately represents the universe of investable U.S. common stocks. This step removes securities that can behave like complicated financial products and lets us focus on the kinds of companies most investors think about.

📝 Section Recap: Use CRSP share codes (10 and 11) and exchange codes (1,2,3) to select ordinary U.S. common stocks and avoid mixing in REITs, funds, and other instruments that would distort the cross‑section.

Measuring Company Size: Market Capitalization#

If you had to describe a company’s size in a single number, market capitalization (often shortened to “market cap” or just “ME”) is the natural choice. It tells you the total market value of all the company’s shares at a given moment. In CRSP, we calculate it from two data items:

where

• is the month‑end price for stock at time ,
• is the number of shares outstanding, in thousands.

The absolute value of price is used because, as mentioned, CRSP sometimes stores a negative price to indicate a bid‑ask average. That negative sign is just a data flag, not a real economic price, so we flip it to positive.

Why does market cap matter so much? One of the most famous patterns in finance is the size effect: small‑cap stocks have, on average, earned higher returns than large‑cap stocks. To test that idea, we need a reliable size measure for every stock every month. Market cap also serves as a natural denominator when we want to compare the weight of a stock in a portfolio or when we compute value ratios like book‑to‑market. And many asset pricing models require us to sort stocks into size groups before forming portfolios or running regressions. Without a clean market cap variable, the whole size‑related literature would crumble.

A practical tip: CRSP reports shares outstanding in thousands. If a company shows 50,000 under SHROUT, that means 50 million shares. Multiply by the price (say $30) and you get $1.5 billion. Keeping track of units avoids million‑to‑billion mistakes.

There’s a subtle timing issue. The market cap we measure at the end of June is often used to form portfolios from July through the next June. This annual refresh matches the typical fiscal‑year reporting cycle and minimizes look‑ahead bias (the mistake of using information that wasn’t yet available) because accounting data (like book equity) usually arrives with a several‑month lag. We’ll see that convention many times later, but for now, just know that the “size” characteristic for a stock is measured at a specific point in time and then carried forward until the next observation.

📝 Section Recap: Market capitalization is calculated as shares outstanding times the absolute value of month‑end price. It is the primary yardstick for company size and underpins factor sorting and portfolio construction.

The Delisting Problem: Why Missing Returns Matter#

Imagine you buy a small, struggling company’s stock. One month, it drops 40%, and then the company goes bankrupt and the stock stops trading. If you only look at the regular CRSP monthly returns, you might see that -40% for the last month, but the true loss from that point until the stock becomes worthless is actually 100%. The difference is the delisting return—the return from the last trade to the delisting date. Ignoring it would overstate the performance of small, distressed firms and create a delisting bias—a bias where we only see the stocks that survived, making returns look better than they really are.

CRSP provides a delisting return field (DLRET) for many stocks. When a stock is delisted for a negative reason—fraud, bankruptcy, failure to meet exchange requirements—that delisting return is often deeply negative. For example, a delisting return of -0.30 means you lost another 30% after the final market closing price. CRSP gives delisting codes that describe the reason; codes in the 200s typically indicate performance‑related delistings (like price too low), while codes in the 300s represent liquidations.

But here’s the catch: CRSP’s DLRET field is frequently missing, especially for the worst‑performing stocks. If we do nothing, we will systematically miss large negative returns. The standard fix is to impute a delisting return when DLRET is missing and the delisting code suggests a forced exit:

• For delisting codes in the 200s: assume a delisting return of −30%.
• For codes in the 300s (liquidation): assume −100%.

These assumptions are rough but capture the economic reality that forced delistings are almost never good news.

To build the full month’s return when a delisting occurs, we combine the regular monthly return with the delisting return. If the stock has a regular return and a delisting return in the same month, the total return for that final month is:

For small monthly returns, adding them directly () gives a very close approximation, and many researchers use that simpler approach for speed. The important thing is that you don’t treat the -40% as the whole story; you add the extra loss so the return reflects reality.

This adjustment dramatically changes average returns for the smallest firms. In some years, dropping it entirely can make it look like the smallest decile earns a healthy premium, when in fact it was just a delisting‑bias mirage. So whenever you read an empirical study that uses CRSP data, checking whether and how they handle delistings is essential.

📝 Section Recap: Stocks that delist for poor performance often have missing negative returns. We correct this bias by using CRSP’s delisting return when available, or imputing -30% or -100% based on the delisting code, then combining it with the regular monthly return.

Excess Returns: Removing the Risk-Free Rate#

The return you see on a stock—say 1.2% in a month—has two components: the compensation for simply waiting (the time value of money) and the reward for taking risk. To isolate the risk‑related part, we subtract the risk‑free rate—the return you could have earned by lending to the U.S. government with no credit risk.

Excess return: The return of a stock minus the risk‑free rate. It measures how much extra reward an investor received for bearing the stock’s risk.

Typically, the one‑month Treasury bill rate serves as the risk‑free rate. It’s available from CRSP or from the Federal Reserve’s website. For stock in month :

where is the one‑month T‑bill rate at the beginning of the month (since you know the T‑bill yield in advance). Some studies use the risk‑free rate at the end of the month, but the difference is tiny and doesn’t affect cross‑sectional patterns.

Why go through the trouble of excess returns? If we studied raw returns, we’d mix up two different questions: (1) why do some stocks pay higher returns in general, and (2) why do interest rates change over time? Excess returns keep the focus on the differences in risk premiums across stocks, which is the core of empirical asset pricing.

When building the analytic sample, we merge the CRSP stock data with a time series of risk‑free rates. Then for every stock‑month observation, we compute . This variable becomes the left‑hand side of countless regressions and the basis for portfolio sorting.

A quick check: don’t accidentally subtract the annualized T‑bill yield from a monthly stock return. CRSP’s monthly returns are already monthly percentages. The T‑bill yield is usually an annualized rate; you must convert it to a monthly rate (divide by 12 if simple, or use the exact compound method: ) before subtracting. In practice, many datasets provide a ready‑to‑use monthly risk‑free rate, so life is easy.

📝 Section Recap: Excess return equals the stock’s total return minus the monthly risk‑free rate. It strips out the pure time‑value component, letting us focus on risk compensation in the cross‑section.

A Snapshot of the CRSP Universe: Exchanges and Industries#

Now that we have a clean sample of stocks with size, adjusted returns, and excess returns, let’s take a bird’s‑eye view of the landscape. What does the U.S. equity market actually look like in terms of exchanges and industries?

CRSP covers the three major equities exchanges: NYSE, AMEX, and NASDAQ. The NYSE is the oldest, with data from 1925. AMEX joined in 1962 and NASDAQ in 1972. The number of common stocks has ballooned over time. In the 1960s, there were about 2,000 stocks; by the 1990s, the total topped 8,000, driven largely by NASDAQ’s expansion. Since the late 1990s, the number has drifted lower because of mergers and a drop in new listings, settling around 4,000–5,000 in recent years.

The cross‑section isn’t evenly spread across exchanges. NYSE firms tend to be larger, older, and concentrated in manufacturing, finance, and energy. NASDAQ hosts many younger, high‑growth technology and biotech firms. AMEX is smaller, with a mix of small‑cap and specialized securities. Including all three gives a more complete picture of the economic drivers of returns.

Industries are commonly classified using Standard Industrial Classification (SIC) codes, which CRSP provides. The first digit roughly identifies the broad sector:

• 0–1: Agriculture, mining, construction
• 2–3: Manufacturing
• 4: Transportation, communications, utilities
• 5: Wholesale and retail trade
• 6: Finance, insurance, real estate
• 7–8: Services
• 9: Public administration

A finer grouping uses the first two or three digits, giving over 40 industry groups. Many researchers use the Fama–French 48‑industry classification or the newer NAICS system, but the idea is the same: stocks in the same industry often move together, so controlling for industry effects is a standard part of cross‑sectional tests.

One caution: financial firms (SIC 6000–6999) and utilities (SIC 4900–4949) are often excluded from asset pricing studies. Why? Because their high leverage and regulatory constraints make standard valuation ratios (like book‑to‑market) harder to interpret, and their return patterns can differ from the typical industrial firm. Utilities, for example, can be more bond‑like. Excluding them is common, but not universal; you should always know whether your sample includes them.

Understanding the exchange and industry composition of your sample helps you spot odd patterns. For instance, if you find a strong “anomaly” that only appears among a handful of biotech stocks, you might be picking up a tiny‑sample fluke rather than a genuine effect.

📝 Section Recap: CRSP spans NYSE, AMEX, and NASDAQ, with industry identity captured by SIC codes. Knowing the mix of exchanges and sectors helps interpret results and decide when to exclude certain industries like financials or utilities.

Summary#

You’ve just built the scaffolding for every study of stock returns. By carefully selecting ordinary common stocks, measuring their market capitalization, adjusting for the delisting bias, and subtracting the risk‑free rate, you create a clean analytical sample that reflects the real investment experience. These steps might feel like tedious data chores, but they prevent the kind of hidden errors that can fool even sophisticated models. With a solid foundation under us, we’re ready to explore what drives differences in average returns across stocks.