ZFS RAIDZ capacity calculator

How ZFS counts usable space

A simple RAID calculator would say that six 16 TB drives in RAIDZ2 give 64 TB: six drives minus two for parity. ZFS will show you less. On an empty pool, zfs list reports about 58 TiB. Four things account for the difference.

1. Parity or mirror copies. RAIDZ1, RAIDZ2 and RAIDZ3 use one, two or three drives' worth of space per vdev for parity. A two-way mirror keeps half. This is the part every calculator shows.
2. Padding. RAIDZ writes each block as its own stripe, and it allocates space in multiples of parity + 1 sectors. When a 128K record does not divide evenly over the data drives, the leftover sectors are wasted. How much depends on the vdev width: a 4-wide RAIDZ1 loses about 3%, an 8-wide RAIDZ2 about 5%. ZFS assumes 128K records when it reports free space, so this loss shows up in zfs list from day one.
3. The slop reserve. ZFS holds back 1/32 of the pool (3.2%), so it can still delete files and run when the pool is full. Current OpenZFS versions cap the reserve at 128 GiB, so on large pools it is small.
4. TB and TiB. Drives are sold in decimal terabytes (10¹² bytes). ZFS reports binary units: its “T” is a tebibyte, 2⁴⁰ bytes, about 10% more. 64 TB is only 58.2 TiB before anything else is taken away.

The calculator does the same arithmetic as ZFS: it takes the allocation formula from the OpenZFS source and the same integer rounding, so the result is usually within a few gigabytes of what a new pool shows. A little more goes to metadata as you fill the pool.

RAIDZ1, RAIDZ2, RAIDZ3 or mirrors

RAIDZ is ZFS's version of parity RAID, with one important difference: every block has a checksum, so ZFS knows which copy is correct and repairs bad data on its own. RAIDZ also has no “write hole”, because a stripe is never partly overwritten.

Mirrors waste more space but behave better in nearly every other way. A resilver copies one drive instead of reading the whole vdev, random speed grows with every pair you add, and you can grow the pool two drives at a time. For virtual machines, mirrors are usually the right answer. For a large media library, RAIDZ2 is.

RAIDZ1 with large drives carries the same risk as RAID 5: while a failed drive is replaced, there is no protection left, and a resilver of a 16 TB drive takes a day or more. See the rebuild time calculator for your drives.

How wide should a vdev be

This table shows how much of the raw space holds data in one RAIDZ vdev, with the default 128K records and 4K sectors. Where it says “ideal”, the width divides 128K records evenly and nothing is lost to padding.

Up to 12 drives, the widths that waste nothing are 3, 5 and 9 drives for RAIDZ1, 6 for RAIDZ2, and 5, 7 and 11 for RAIDZ3. The old rule of thumb, “a power of two data drives plus parity”, does not quite hold for RAIDZ2: a 10-wide RAIDZ2 has eight data drives but still loses 4.8% to padding, because RAIDZ2 rounds every stripe up to a multiple of three sectors. Other widths are not wrong, they just lose a few percent. With compression on, records are no longer exactly 128K and the effect gets smaller.

Wider is not always better. A vdev has the random speed of roughly one drive however wide it is, and a resilver reads every drive in it. Most people stop at 10 to 12 drives per RAIDZ2 vdev and add a second vdev after that.

Small blocks, zvols and virtual machines

Padding grows quickly when blocks are small. Virtual machine disks (zvols) use 16K blocks by default since OpenZFS 2.2, and databases often use 8K or 16K. On RAIDZ, a small block still needs its full set of parity sectors, so the real space efficiency can fall to the level of a mirror or below:

An 8-wide RAIDZ2 stores 71.1% data with 128K records and 74.9% with 1M records, but only 33.3% with 8K blocks. This is why VM storage on RAIDZ fills up faster than expected, and why mirrors are the usual advice for it. For media files, set recordsize=1M on the dataset to get close to the ideal figure.

Growing a ZFS pool later

There are three ways to add space to a pool:

• Add a vdev. A new vdev of the same type can be added at any time, and the pool stripes across all of them. It is the oldest and simplest method. Keep the vdevs the same shape.
• Replace the drives with larger ones. Replace and resilver each drive in a vdev one at a time. With autoexpand=on, the vdev grows when the last small drive has been replaced.
• Expand a RAIDZ vdev by one drive. Since OpenZFS 2.3 you can attach a drive to an existing RAIDZ vdev. Data written before the expansion keeps its old data-to-parity ratio until it is rewritten, and the free space ZFS reports stays on the conservative side, so the gain looks smaller than this calculator shows for the wider layout.

What you cannot do is turn a RAIDZ1 vdev into RAIDZ2, or remove a RAIDZ vdev from a pool. Pick the parity level carefully at the start.

zpool list and zfs list show different numbers

For RAIDZ, zpool list shows the raw size of all drives including parity. zfs list shows what is left for your files after parity, padding and the slop reserve. For four 12 TB drives in RAIDZ1, zpool list says about 43.6T and zfs list about 31.6T. Both are correct: they answer different questions. Plan with the zfs list figure, which is what this calculator shows.

Questions