Splunk Architecture — Forwarders, Indexers, Search Heads & the Data Pipeline

① The big picture — Splunk is a three-tier pipeline

The single most important idea: Splunk is not one server, it is a pipeline with three roles. Data flows forwarder ▸ indexer ▸ search head. A forwarder on each machine collects logs and ships them to an indexer, which turns the raw stream into searchable, timestamped events and stores them. A search head is where people run queries and build dashboards — it asks the indexers for results.

In a tiny lab all three roles can live on one instance, but in any real deployment they are separate so each tier can scale on its own. The interview line: collection, storage and search are three different jobs done by three different components. Get that flow on a whiteboard and you have already passed half the question.

② Forwarders and indexers — collection and the bucket lifecycle

There are two kinds of forwarder. The universal forwarder (UF) is a tiny, separate agent that ships raw, mostly unparsed data — it is the default on every endpoint and server because it is light on CPU and memory. The heavy forwarder (HF) is a full Splunk Enterprise instance with bits turned off; it can parse, filter and route data before sending it, which is useful when you must drop noise or split traffic by content.

What the indexer does

The indexer runs the pipeline that makes data searchable: it identifies events, finds timestamps, does light parsing, then writes events to index buckets. Buckets age through stages: hot (being written), warm (closed but still on fast disk), cold (older, moved to cheaper, slower storage), then frozen (rolled off and deleted, or archived). Archived frozen data can be thawed back in if you ever need it. This lifecycle is how Splunk controls cost versus retention.

③ Index-time vs search-time — the split that decides everything

This is the question that separates juniors from seniors. Index time is what happens once, on the indexer, as data is written to disk: breaking the stream into events, assigning the timestamp, host, source and sourcetype, and writing the raw event plus an index of its keywords. Splunk keeps index-time processing deliberately light — heavy index-time field extraction slows ingest and is hard to change later.

Search time is what happens every time you run a query, on the search head: most field extraction, lookups, calculated fields and aliases are applied to the raw events as they are read. Because the schema is applied when you read, Splunk is called schema-on-read — you do not have to define columns up front, and you can add or change field extractions later without re-indexing. Splunk's own guidance is to do most knowledge work at search time. The practical payoff: if a field is missing, it is almost always a search-time extraction to fix, not a re-ingest.

④ Scaling out — clusters, the deployment server and licensing

One indexer and one search head do not survive a busy SOC. For throughput you add more indexers and let the search head run distributed search across all of them. For high availability you use an indexer cluster: a cluster manager keeps multiple copies of every bucket (the replication factor) and multiple searchable copies (the search factor, default 3 each), so a dead indexer loses no data. A search head cluster does the same for the search tier, sharing knowledge objects and jobs across members.

Management and cost

The deployment server is the config push tool — it sends apps and settings out to your fleet of forwarders so you are not editing each one by hand. On cost, classic Splunk licensing is ingest-based (you pay by GB of data indexed per day), while Splunk Cloud also offers workload pricing measured in compute units (SVCs). Either way, the architecture lesson is the same: ingest and search both cost money, so you tier storage with buckets and filter noise early.