Bits, bytes and integers

Everything a computer stores is a pattern of bits. n bits give 2ⁿ patterns, and how you assign numbers to those patterns is the whole story of this course.

Key idea: Bits, bytes, 2ⁿ patterns, unsigned and signed (two’s complement) integers, overflow
Real world: A car’s odometer with 6 digits counts from 000000 to 999999, then rolls back to 000000. Fixed digits mean a fixed range.
In an LLM: A 4-bit weight can only be one of 16 patterns. Everything a 4-bit model knows has to be written with those 16 symbols, plus a few scale numbers.

A bit is a switch

A bit is the smallest unit of storage: a single switch that is either 0 or 1. Group 8 of them and you have a byte, the unit file sizes and memory are measured in.

The key fact: n bits give 2ⁿ different patterns. One bit has 2 (0, 1). Two bits have 4 (00, 01, 10, 11). Every extra bit doubles the count:

Bits	Patterns	Typical use
1	2	yes or no
4	16	a 4-bit quantized weight
8	256	one byte; an 8-bit weight or a text character
16	65,536	a bf16 or fp16 weight
32	about 4.3 billion	an fp32 number

A pattern on its own means nothing. A format is the rule that says which number each pattern stands for. This whole course is about choosing that rule well.

Integers: binary counting

The simplest rule is binary counting. Each position is worth twice the one to its right: 1, 2, 4, 8, 16, … A pattern’s value is the sum of the positions holding a 1. With 8 bits, 00101010 is $32 + 8 + 2 = 42$ , and the range is 0 to 255. This is an unsigned 8-bit integer, uint8.

Try it

show as two 4-bit values

1286432168421

every value 8 bits can hold, in order

0255

pattern 00101010 = 42 unsigned = 42 signed

value42

sum32 + 8 + 2

patterns2⁸ = 256

8 bits give 256 patterns, so they can count from 0 to 255. Every extra bit doubles the count. Press +1 at 255 or −1 at 0 to see what happens at the edges.

Each square is one bit. Click to flip it. The number under each bit is what it’s worth when it’s 1; the value is the sum of the lit ones.

Negative numbers: two’s complement

To store negatives, computers use two’s complement: the leftmost bit is worth minus its usual value. In 8 bits it is worth −128 instead of +128, so 11111111 is $-128 + 127 = -1$ and the range becomes −128 to 127. This is int8, the most common integer format in quantization. int4 runs from −8 to 7.

It looks odd, but it makes ordinary binary addition work for negative numbers with no special cases, which is why every processor uses it.

example · an odometer

A 6-digit odometer has $10^6$ readings. Drive one more mile at 999,999 and it shows 000,000: the count overflows and wraps around. Nothing is wrong with the car; the display simply has no pattern for a million.

example · an int8 accumulator

When a GPU multiplies int8 weights by int8 activations, each product can be as large as $127 \times 127 = 16{,}129$ , far beyond int8’s range. Adding up thousands of such products in int8 would overflow immediately, so hardware accumulates them in int32, then scales the result back down. Quantized formats are for storing numbers; the arithmetic in between usually uses more bits.

Integers are evenly spaced

Integer patterns stand for evenly spaced numbers: every step is exactly 1. That is fine for counting, but a model’s weights are numbers like 0.0123 and −0.0471. To store those as integers you need an extra number, a scale, that says what one step is worth. That is the core trick of quantization, coming up in what quantization is. First, the format models are trained in: floating point.

Where this shows up

Bits per weight decide a model’s size: how much memory a model needs.
int8 and int4 are the targets of integer quantization: what quantization is.
GGUF files pack several 4-bit codes into each byte: GGUF and llama.cpp.

going deeper

Definitions

An $n$ -bit unsigned integer with bits $b_{n-1} \dots b_0$ :

v = \sum_{i=0}^{n-1} b_i\, 2^i \in [0,\; 2^n - 1]

In two’s complement the top bit has negative weight:

v = -b_{n-1}\, 2^{n-1} + \sum_{i=0}^{n-2} b_i\, 2^i \in [-2^{n-1},\; 2^{n-1} - 1]

To negate a number, flip every bit and add 1. Overflow is arithmetic modulo $2^n$ .

Packing

Memory is addressed in bytes, so sub-byte formats pack several values per byte: two 4-bit values per byte (“nibbles”), four 2-bit values. A 4-bit tensor of $N$ values takes $N/2$ bytes plus its scales. Unpacking is a shift and a mask, e.g. lo = b & 0x0F, hi = b >> 4.