Ceph

What Ceph actually is

Ceph is a free, open-source distributed storage platform that puts object, block, and file storage on one self-managing cluster built from commodity hardware. Underneath is RADOS, an object store that distributes data across nodes using the CRUSH placement algorithm — which computes where data lives instead of looking it up in a central table, so there is no single point of failure and no lookup bottleneck as the cluster grows. Above that sit three faces: RBD for block volumes (what OpenStack and Kubernetes workloads typically consume), CephFS for a POSIX filesystem, and the RADOS Gateway (RGW) for S3- and Swift-compatible object storage. Snapshots, clones, replication, and erasure coding are built in at the object layer and shared by all three.

The project began in 2004 as Sage Weil's doctoral dissertation at UC Santa Cruz. Weil later co-founded Inktank Storage to sell support around it; Red Hat acquired Inktank for $175 million in April 2014 and remains the largest corporate contributor alongside Intel, CERN, Cisco, Fujitsu, Canonical, and SUSE. Governance now sits with the Ceph Foundation, a directed fund of the Linux Foundation, under a published technical charter — the code is LGPL (2.1/3.0 dual per the project's COPYING file, with some bundled components under BSD, Boost, and Apache 2.0). The current stable line is 20.2.0 "Tentacle" (November 2025), with a new stable release roughly every year, about 24 months of support per series, and point releases every four to six weeks.

The operational story is what people either love or fear. Cluster monitors keep the map of the cluster; OSDs manage the actual drives through BlueStore, the storage backend that has been the default since the Luminous release in 2017 and writes directly to raw devices rather than through a filesystem; metadata servers front CephFS; managers handle monitoring and maintenance tasks and hook into Prometheus or Zabbix. The system is designed to be self-healing — failed disks and nodes trigger automatic recovery and rebalancing — and self-managing to a degree, but "self-managing" is relative when exabytes are involved.

Pricing: what you really pay

The software is free: download it, run it, sell services on top of it. The costs that matter are hardware, operations, and optionally a support subscription. The official hardware guidance is a fair starting point for budgeting: plan around the default OSD memory target of 4 GiB per OSD, separate drives for the OS and data, and at least 10 Gb/s networking between hosts; production clusters want at least three monitors for quorum, two managers, and at least three OSDs. That is a real cluster, not a laptop experiment — realistically four figures of hardware before you store anything meaningful.

Commercial support is where vendors price the value. Red Hat sells Ceph Storage subscriptions in node- and capacity-based bundles — The Register put a 12-node deployment in the ballpark of $32K a year in 2025, and resellers list standard and premium subscriptions sized by physical nodes and terabytes. The market around the project is moving: SUSE wound down its Ceph-based SUSE Enterprise Storage offering from 2021 onward, and Sardina Systems now courts those users with Ceph-based FishOS deals starting at €1 plus a per-core license fee. Independent integrators (Clyso and others) sell managed Ceph, and the cephadm installer makes do-it-yourself deployment genuinely approachable.

In our view the honest comparison for Ceph is not "free versus $X per TB of S3" but "what does an in-house storage team cost versus what does a cloud bill cost at our growth rate." At tens of terabytes, managed object storage is cheaper in total. At hundreds of terabytes and up, especially with predictable growth, Ceph on owned hardware in your own colo or metal tends to win — provided you have, or hire, the expertise. Our benchmark page covers the hardware side of that equation.

Performance & infrastructure

There are no datacenters here — Ceph is the thing that turns your datacenters into storage. Performance is a function of your hardware, network, and pool design, and the knobs are many: BlueStore with its own caching, erasure coding profiles (a 4+2 or 6+3 profile gets within roughly double of raw capacity versus triple replication), pool sizing, CRUSH rules pinning data to failure domains, and the balance between replicated pools for hot block workloads and erasure-coded pools for cold object data. Because data spreads across all OSDs, throughput scales with node count rather than being capped by any one machine — CERN, OVH, and DigitalOcean run large production deployments on exactly that property.

Kubernetes users are well served: the project documentation recommends Rook (a graduated CNCF project) for running Ceph inside or alongside Kubernetes, and the Ceph CSI driver provisions block and file volumes for pods. For edge and small clusters, MicroCeph packages a minimal deployment. Management tooling has matured a lot — cephadm bootstraps a cluster from one host and expands it node by node, and the built-in dashboard plus Prometheus integration cover day-two visibility. Version 12's move to BlueStore (and Reef's deprecation of the old Filestore backend) removed the filesystem layer that used to cause most of the latency surprises.

The durability model is strong: replication or erasure coding across failure domains, self-healing on disk or node loss, snapshots and clones, and no central index to corrupt. This is production-grade storage software in the fullest sense — the trade-offs are operational, not architectural.

Where it falls short

The learning curve is real and it is steepest exactly where it hurts: during incidents. Cluster health has many states, recovery and backfill can saturate networks at the worst moment, and tuning decisions (PG counts, pool profiles, memory targets, BlueStore settings) have long-lived consequences. The documentation is extensive and the FAQ is refreshingly honest — plan for 4 GiB of RAM per OSD, three monitors minimum, 10 Gb/s networking — but reading documentation is not the same as having run a cluster through a double disk failure. We would not recommend Ceph to a team without dedicated storage operational experience, and we would budget training or a support contract into the first-year cost.

Second, minimum scale. Small Ceph clusters are uneconomical and operationally noisy; the practical floor is several nodes with real networking. If you need 5 TB of S3-compatible storage, a managed object store is simpler and cheaper. Third, the ecosystem is shifting: SUSE's exit from Ceph-based products and the resulting user migration scramble is a reminder that corporate backing around an open source project is never guaranteed — though Ceph's Red Hat and Linux Foundation footing is about as solid as this industry offers. Fourth, latency: Ceph's sweet spot is throughput and durability at scale, not single-request latency; low-latency block workloads may prefer local NVMe with Ceph for the cold tier.

Who should pick this

We pick Ceph for organizations building their own cloud or storage platform: service providers offering S3-compatible object storage or VM volumes, research and data-intensive institutions (the CERN deployment is the canonical example), enterprises consolidating object, block, and file onto one cluster instead of three products, and Kubernetes shops wanting persistent volumes backed by something they control. If you are evaluating cloud object storage and your data gravity or per-TB costs justify owning the layer, Ceph is the default open-source answer.

One more fit worth naming: existing Red Hat and OpenStack estates. If you already run OpenStack or a Red Hat-heavy infrastructure, Ceph is the default storage layer for a reason — the integration paths (Manila, Cinder, Nova, Rook) are mature, the support matrix is someone else's problem, and the operational runbooks exist. Buying three separate storage products for object, block, and file usually costs more in integration alone than running one Ceph cluster well. We would not pick it for small deployments, for teams without storage operations experience, for latency-critical single-block workloads, or for anyone who wants a vendor to own the pager — buy managed storage instead, or hire a managed-Ceph provider.

Key takeaways

• Free, LGPL-licensed distributed storage unifying object (S3/Swift via RGW), block (RBD), and file (CephFS) on one CRUSH-based cluster with no single point of failure.
• Started in 2004 as Sage Weil's UC Santa Cruz PhD; Inktank acquired by Red Hat for $175M in 2014; now governed by the Ceph Foundation under the Linux Foundation.
• Production proven at very large scale (CERN, OVH, DigitalOcean); BlueStore backend, cephadm installer, Rook/CSI for Kubernetes, yearly stable releases with ~24-month support.
• Real costs are hardware (4 GiB RAM per OSD, 10 Gb/s networking, 3+ nodes) and operations; optional subscriptions run roughly $32K/yr for 12 nodes (press estimate).
• Steep learning curve and incident-response complexity are the main drawbacks; the practical floor is a multi-node cluster with someone experienced running it.
• Best fit: providers and enterprises building self-managed storage at scale; poor fit: small deployments or teams wanting managed, contractual storage.

Sources

• https://en.wikipedia.org/wiki/Ceph_(software)
• https://en.wikipedia.org/wiki/Inktank_Storage
• https://ceph.io/en/users/faq/
• https://docs.ceph.com/en/latest/technical-charter/
• https://ceph.io/en/foundation/about/
• https://assets.theregister.com/2025/05/29/sardina_entices_ses_users