# Splunk is an ingest-search factory. Forwarder → index → search → notable. Source: https://ai.techclick.in/blog_splunk_session_factory Markdown: https://ai.techclick.in/blog_splunk_session_factory.md Publisher: Techclick Infosec Pvt Ltd Splunk is an ingest-search factory: forwarder → index → search → notable/ES. Official data pipeline, UF vs HF, index-time vs search-time, then prove the ticket in Search and Incident Review. Quick answer Splunk is an ingest-search factory . Official data pipeline: input (forwarder annotates host / source / sourcetype on 64K blocks) → parsing (break into events, timestamps, transforms) → indexing (write compressed raw + index files) → search (search head directs peers and merges results). Enterprise Security is a later station: a correlation search writes a notable event to Incident Review. Success is a named index + earliest / latest (with timezone) + an integer count — not a red tile. An empty Events tab is usually the wrong index, the wrong window, or CIM fields that never mapped. Say this out loud I do not start with Enterprise Security. I ask whether a forwarder shipped the stream, whether the indexer wrote it into a searchable index, which time range I used, what the job counted, and — only then — whether a correlation search already opened a notable that still has no owner. A chart without index + time + count is a vibe. ## 1. Why a chart is not a search Every other blog starts with dashboards, CIM, and “hunt in ES.” That is why students freeze when the Events tab is empty. The real object is the event on disk. Features are only stamps the factory puts on a ticket after a forwarder ships a stream and an indexer writes it. Official processing tiers match the factory floor: data input , indexing , search management . The four pipeline segments sit on those tiers — input on the forwarder, parsing plus indexing on the indexer, search on the search head. Same log line in a different index is a different factory job. Same events outside the Time range picker are invisible. A notable without contributing events is a correlation search that never saw a result. Hero · the factory floor Notice: Splunk does not “show a chart.” It tries to manufacture a searchable event and then count it. The chart is decoration until those three exist. #### What the ticket asked “VPN is under attack.” That sentence is a hypothesis. The factory may already have indexed 28 failures — or the events may live in lab_auth while you searched main . #### What you prove first Named index, then time window with timezone, then the job’s event count. The evidence desk is the night-shift version of this order — first tool + one proof field. The lie every L1 repeats “The dashboard is red, so Splunk is down — rebuild the indexer.” A red tile only means some saved search painted a panel. If the Events tab is empty, rebuilding the indexer just reprints the same empty search. Change the index and the window first. Dummy lab only Values on this page — finance.user , lab_auth , lab_fw , N-1042 , 192.0.2.25 , host sh-lab-01 — match the Techclick Splunk simulator on the practice hub . They are not a customer tenant. Confirm live roles and indexes before you type on a real search head. ## 2. Mental model — four stations, one ticket Hold four stations. Interviews fail when people mix them. A notable is not an indexer. A search is not a forwarder. CIM is not ingest. #### 1. The ticket is the event One raw line becomes a searchable event after parsing. Default fields: _time , host , source , sourcetype , index . Until parsing finishes, Splunk has no notion of individual events — only a stream of 64K blocks with source-wide keys. #### 2. The stamps are the IDs Forwarder stamp = which host shipped it. Index stamp = which repository on disk. Search stamp = earliest / latest + count. ES stamp = notable Owner / Status / Urgency / Rule Name. #### 3. The hiring manager is the pipeline Input annotates the stream. Parsing breaks events and applies transforms. Indexing writes raw + tsidx. Search applies knowledge objects (field extractions, aliases, CIM) at search time . Correlation searches only see what those stamps already wrote. #### 4. Proof is Search, then Incident Review Apps → Search & Reporting is the live table. Dashboards are history painted pretty. | tstats count is a predicted shape from index-time fields. Incident Review is the case file — only if ES is installed. Flow 1 · one ticket, four stations Forwarder to index to search to notable — one Splunk ingest-search ticket - Event ticket · lab_auth / finance.user / 10:11Z–10:42Z 1 · Forwarder input · 64K blocks host source sourcetype 2 · Index parse + write disk index=lab_auth 3 · Search earliest / latest count = 28 4 · Notable / ES correlation search N-1042 · owner? Forwarder stamp who shipped the stream uf-vpn-01 · UF default Index stamp which repository not main · lab_auth Search stamp window + integer -24h · TZ = UTC ES stamp only after a count Owner · Status · Urgency CIM + field aliases are search-time stamps. They do not move the event to another index. Correlation search results must include at least one event to generate a notable. Empty search = no notable, not a broken ES app. Source: Data pipeline · Splexicon forwarder / indexer / notable event / CIM Read left → right. ES starts after the count exists. Do not jump to Owner while the index is still a guess. Forwarder answers “who shipped this stream?” Official: a forwarder sends data to an indexer or another forwarder. Three types exist — universal , heavy , and deprecated light . The universal forwarder is the default tool. It has no UI and no Python. It forwards unparsed data except structured files. You need a heavy forwarder only when you must parse before send, or route by event contents. Index answers “which repository holds the event?” Official: an index is the repository for data. The indexer transforms raw data into events and writes compressed raw plus index files. Events indexes are the default. Metrics indexes hold only metrics. Same IPs in main versus lab_auth are different factory jobs. Search answers “how many, in which window?” The search head handles search management — it directs requests to search peers and merges results. The Time range picker (or earliest= / latest= ) bounds the job. _time is stored as UNIX time. The picker is interpreted in the user timezone unless you pin it. Notable / ES answers “did a scheduled correlation search already open a case?” Official: a correlation search is a scheduled search that can generate a notable event when results meet conditions. The results must include at least one event. You investigate notables on Incident Review. That station does not exist if ES is not installed. ## 3. Official data pipeline vs later search The first bytes of a new source have no events yet. They walk input → parsing → indexing . Later searches of the same index skip that queue and ride what is already on disk. That is why “I fixed props.conf” sometimes does nothing until the next events are indexed — search-time aliases apply immediately; index-time transforms do not rewrite history. Path · ingest vs later search Notice: the diamond is not “Splunk up / Splunk down.” It is “does this index already have searchable events in this window?” Flow 2 · official pipeline order (student labels) Splunk first-event ingest versus later search of the same index Source → input → parse / index → search → (optional) notable 1 Input 64K · metadata On disk? this index no INGEST / first write — print a new event parsing + indexing on the indexer (HF if you parsed early) Line break parsing Timestamp set _time Annotate copy keys Transforms index-time Write disk raw + tsidx Searchable now LATER SEARCH — already-indexed events of this index + window search head directs peers · apply search-time extractions / aliases / CIM · merge · optional correlation search → notable yes → skip ingest Official facts students invert 1. Input has no events yet. Host / source / sourcetype are source-wide keys on 64K blocks. 2. Parsing internally is three pipelines — parsing, merging, typing. Troubleshooting cares. Day-2 deploy usually does not. 3. Index time writes the event. Search time applies aliases, CIM, event types. A new alias does not move old events to another index. 4. A correlation search must return at least one event to generate a notable. Empty contributing search = silent Incident Review. Source: How data moves through Splunk deployments: The data pipeline · Index time versus search time · Splexicon: correlation search UF = unparsed (except structured). HF = parse + route by event contents. Light forwarder is deprecated as of 6.0.0. Read left → right, then the green later-search bar. Decision diamond = “is this index already searchable in this window?” #1 student trap — index-time vs search-time You added a field alias UserName → user and reran last 24 hours. The CIM Authentication correlation search is still silent on events from last week. Official: search-time rules apply when the search runs — they do not rewrite the raw event. If the correlation search uses a data model that expected the alias last Tuesday, those old jobs already wrote nothing. Re-run the correlation search (or wait for the schedule). Do not rebuild the index because an alias is new. ## 4. How to choose the station You are not choosing a product. You are choosing which station of the factory is allowed to write on the ticket. Choice Use when Do not use when Proof you were right Universal forwarder Default shipper. File / syslog / Windows inputs. You want a small footprint and no UI on the endpoint. You must parse or route by event contents before the indexer. Official: UF forwards unparsed data (except structured files). Monitoring Console → Forwarders: Deployment shows Status = active (connected in last 15 minutes). Heavy forwarder You must parse before send, filter/route by event contents, or index locally while forwarding. Every Windows laptop. HF is a full Enterprise instance with a smaller footprint — not a cheaper UF. Events arrive already broken; indexer does not re-parse structured/parsed streams the same way. Named events index ( lab_auth ) Production. One domain per index (auth, fw, wineventlog). Role-based access and retention stay sane. Dumping everything into main “just to see it.” Same as Service = any on a firewall. Settings → Indexes lists the name; | tstats count where index=lab_auth returns a number. Raw SPL + Time range picker Prove the pipe. Count failures. First sentence of any ticket. You open a CIM data model to prove an index is searchable. That is the expensive stamp. Job event count + first/last _time + stated timezone. CIM data model (Authentication / Network Traffic) ES content, accelerated models, vendor field names that must become user / action / src . The TA left UserName unaliased and you wonder why the correlation search is blind. Recommended fields populated; tag authentication (or the model’s tags) present. Incident Review ES is installed and a high notable already exists. Assign Owner before more hunting. ES is not installed, or you invent a notable from raw SPL at 02:00. Owner ≠ Unassigned. Status moved off New. Rule Name matches the detection you think fired. The universal forwarder is the official default. Splexicon: “In most situations, the universal forwarder is the best way to forward data to indexers.” Heavy is the exception you can name. Source: Splexicon: universal forwarder / heavy forwarder. Pre-train the hard words Index time — from receive to write on disk. Parsing, default fields, timestamps, transforms. - Search time — from search launch to finish. Extractions, aliases, sourcetype rename, event types, CIM. - Search peer — an indexer answering a distributed search. - Knowledge object — reports, event types, dashboards, alerts, field extractions stored on the search tier. - Contributing event — an event that helped create a notable (or a Mission Control incident). ## 5. Runbook Side A → B → C Lab values only. Search head sh-lab-01 , indexer idx-lab-01 , forwarder uf-vpn-01 , user finance.user , client 192.0.2.25 , indexes lab_auth / lab_fw / _internal . Nothing here is a live tenant. Isolate first. Change-control later. ### Side A — forwarder and the input tier (building the factory floor) Primary source: Splexicon: forwarder / universal forwarder · How data moves through Splunk deployments: The data pipeline (Input segment) · Monitoring Console forwarders. - #### Decide UF unless you can name why HF Default = universal forwarder on the VPN concentrator host. outputs.conf points at the indexer receiving port. Do not stand up a heavy forwarder to “see the events sooner.” UF has no UI — you will not log into it to hunt. - #### Stamp host / source / sourcetype at input Official input segment: Splunk consumes the stream, breaks it into 64K blocks, and annotates each block with source-wide keys — host , source , sourcetype , and the destination index . It does not look at event contents yet. Wrong sourcetype here is a factory-floor error, not a search error. - #### Prove the shipper is active before you hunt Settings → Monitoring Console → Forwarders: Deployment . Official Status: active if the forwarder connected to an indexer in the last 15 minutes; missing if it did not. Configure Forwarder Monitoring Setup first — that is the documented prerequisite. Queue percent on an HF is a later question. Side A — fields you write in the ticket (dummy lab) Forwarder: uf-vpn-01 type=universal Outputs: idx-lab-01:9997 MC Status: active lookback=15m Index dest: lab_auth Sourcetype: lab:vpn:auth Host key: uf-vpn-01 ### Side B — index then bounded search (printing the ticket) Primary source: Indexes, indexers, and indexer clusters · Search Tutorial (start searching) · Specify time ranges. Remember: parsing + indexing live on the indexing tier. Search management lives on the search head. https://192.0.2.10:8000/en-US/app/search/search Training mock · not live Apps  ›  Search & Reporting  ›  Search ### New Search Events Patterns Statistics Visualization Search bar index=lab_auth user=finance.user Time range picker Last 24 hours earliest -24h latest now Job SID · SID-LAB-1042 Event count · 28 First _time · 10:11:04 UTC Last _time · 10:42:18 UTC Timezone · UTC index Index lab_auth sourcetype lab:vpn:auth Selected fields user=finance.user · action=failure · dest=vpn · host=uf-vpn-01 Job inspector scanCount ok · not index=* Default Time range picker on many Search Tutorial pages is Last 24 hours (some versions default Last 24 hours / All time for the tutorial). State the timezone out loud. Dummy values only. Job inspector Search Source: Search Tutorial — start searching (index, Time range picker, event count). Specify time ranges — earliest / latest. Dummy values only. Click next: copy event count + first/last _time into the ticket, then open Settings → Indexes if Latest event looks stale. - #### Confirm the index exists before you invent meaning Settings → Indexes (Create custom indexes). Quote the name and the Latest event column. Lab: lab_auth Latest event within seconds. If Latest event is hours old and the VPN is still authenticating, that is Side A (forwarder) or the indexing tier — not a missing correlation search. - #### Name the index in the Search bar Apps → Search & Reporting . Type index=lab_auth user=finance.user . Never start a customer hunt with index=* . Underscore indexes ( _internal , _audit ) stay out of * unless you name them — use index=_internal only to prove this instance is writing its own logs. - #### Pin time and timezone Time range picker → Last 24 hours, or earliest=-24h latest=now . Official: relative time modifiers and the picker bound the job. State the user timezone. Settings → User preferences → Time zone if the analyst is IST and the ticket is UTC. - #### Speak the count before “attack” Lab job: 28 events, action=failure , dest=vpn, first 10:11Z, last 10:42Z, SID SID-LAB-1042 . Sentence: “28 VPN auth failures in index=lab_auth from 10:11Z to 10:42Z.” Then baseline. Then lockout / brute-force. Then ES. Bounded search — Techclick dummy lab, not a customer SH index=lab_auth user=finance.user earliest=-24h latest=now | stats count earliest(_time) as first latest(_time) as last by action dest action dest count first last failure vpn 28 2026-08-16 10:11:04Z 2026-08-16 10:42:18Z Say the word count . This job is a what-if over a window. The live notable may show a different urgency after a VIP lookup. Compare it to Incident Review in Side C. ### Side C — CIM stamps and the notable (only if ES is present) Primary source: Splexicon: notable event / correlation search / CIM · CIM Authentication data model · Incident Review dashboard. Skip this side if Enterprise Security is not installed. - #### Confirm CIM if ES content is silent CIM is a set of search-time data models — field names and tags that normalize vendor events. Authentication needs tag authentication and recommended fields action ( success / failure ), user , src , dest , app . Network Traffic needs action , src , dest , dest_port . If the TA left UserName unaliased, the correlation search is blind. That is a missing stamp, not a dead indexer. - #### Assign the notable before more SPL Enterprise Security menu → Incident Review . Official: the dashboard displays notable events and current status. Lab: N-1042, urgency High, Owner Unassigned, rule ES-BruteForce, risk object user=finance.user . Edit selected → Owner = you → Status = In Progress. Then pivot to contributing events in index=lab_auth . - #### Do not invent a correlation search at 02:00 Official: correlation search results must include at least one event to generate a notable. If contributing events are 0, fix Side A/B or the CIM alias. Writing a new scheduled search while Owner is Unassigned is theatre. https://192.0.2.10:8000/en-US/app/SplunkEnterpriseSecuritySuite/incident_review Training mock · not live Enterprise Security  ›  Incident Review ### Incident Review Notables Contributing events Workaround Notable ES N-1042 Rule Name ES-BruteForce Urgency High Owner Unassigned Status New Risk object user = finance.user Contributing · index=lab_auth Count · 28 Window · 10:11Z–10:42Z ES 8 re-architected some menus (Mission Control / Analyst Queue). On ES 7 the path is Incident Review. Skip Side C if ES is not installed. Dummy values only. View contributing events Edit selected · Set Owner Source: Splexicon: notable event + correlation search. Incident Review dashboard — Owner, Status, Urgency, Rule Name. Click next: assign Owner, move Status to In Progress, then open contributing events in the named index from Side B. Proof · Search + Incident Review cockpit Notice: juniors stare at a red chart. Seniors stare at index + time + count, then Owner. Green success on this runbook Forwarder Status = active. Settings → Indexes Latest event is current for lab_auth . Search job on index=lab_auth in a named window returns a count (lab: 28) with first/last _time . If ES is installed: notable N-1042 has an Owner and Status is no longer New. A red dashboard with 0 events is not success. ## 6. Runtime — CIM, timezone, old jobs After the event exists on disk, later searches skip the ingest queue. Official search segment: the search head stores knowledge objects and manages the search process. Field aliases, CIM tags, event types, and calculated fields run at search time . They do not rewrite buckets. A new alias lights up the next job. It does not move last Tuesday’s events into a different index. Timezone is the silent factory lever. Splunk stores _time as UNIX time. The Time range picker is interpreted in the user timezone unless you pin earliest= / latest= or set Settings → User preferences → Time zone. An incident at 02:00 UTC is invisible to “Last 4 hours” run at 10:00 IST (04:30 UTC). Same empty-Events feeling as the wrong index. Different station. Index-time mistakes persist. A sourcetype stamped wrong at input, a transform that sent the stream to nullQueue , or events written to main instead of lab_auth stay wrong until you fix the input and wait for new events. You can search the old index. You cannot “commit” your way out of last week’s buckets the way a firewall commit reprints a rule. Enterprise Security is a scheduled factory on top of the search factory. A correlation search runs, looks for at least one matching event, and writes a notable with metadata (Owner, Status, Urgency, Rule Name). If CIM never mapped user , that schedule writes nothing. If someone already opened N-1042 and left Owner Unassigned, writing more ad-hoc SPL does not assign it. Adaptive response actions (if licensed and configured) fire after the notable exists — they are not a substitute for Side B. Distributed search is two copies of the later-search bar. The search head directs peers and merges. Green search-head clustering means the knowledge bundle is coordinated. It does not mean the VPN recovered. Prove that with the same bounded search on the same named index. Flow 3 · one night-shift ticket through the factory Ticket to forwarder status to count to notable owner - Ticket user locked UF active MC · 15m 28 in lab_auth 10:11Z–10:42Z N-1042 assign owner Lock / reset not rebuild Change-control (disable a saved search, rebuild an index) is off this path unless ingest is actually broken. Night shift, 10:45Z. You isolate with read-only evidence. The evidence desk names the first click; this factory names the order of stations. ## 7. Traps + Search / Incident Review proof Symptom Looks like Actually First move 0 events, dashboard red Splunk is down Wrong index, wrong TZ, or UF missing index=_internal then named index + window Searched main , 0 hits Indexer cluster down Data lives in lab_auth / wineventlog Settings → Indexes · change the bar Incident at 02:00Z, Last 4 hours empty No data IST picker never covered 02:00Z State TZ · widen or pin earliest ES correlation silent Need a new correlation search UserName never aliased to user ; tag missing CIM Authentication fields + tag High notable, nobody acts Detection bug Owner = Unassigned Incident Review → set Owner Added props.conf, old events unchanged Commit failed Index-time transform; old buckets stay Search-time alias, or wait for new ingest HF on every laptop Faster parsing You deployed a full instance as an agent UF unless you must route by contents action=blocked on 445 called outage File server crash Control working in lab_fw Count + rule name, not a reboot Proof checklist — the factory actually worked Forwarder identified (UF vs HF) and Monitoring Console Status = active if you expected ingest. - Index named in the search ( lab_auth or lab_fw ), not * . Settings → Indexes Latest event is current. - Time range spoken with timezone (lab uses Zulu first/last). - Count spoken as an integer from the job (Events tab / Job inspector), not a chart colour. - If 0 events: index=_internal alive? wrong index? wrong TZ? UF missing? then CIM. - If ES is in play: notable ID, Rule Name, Urgency, Owner , Status. Contributing events live in the named index. - No change-control (disable saved search, rebuild index) unless ingest is actually broken. - The same business question (VPN lockout) can be restated as index + time + count. Interview close you can steal Splunk is an ingest-search factory. A forwarder ships a stream and stamps host, source, and sourcetype. The indexer parses and writes an index. The search head counts a named window. Enterprise Security is a later station: a correlation search writes a notable I prove on Incident Review. I do not say “VPN is under attack” until I have index, time, and count. Allow me to open Search before I open ES. Night-shift proof clicks live on the pair page: Prove Splunk is working — first tool + proof field . ## Knowledge check Six judgment questions. Map each miss back to the section named in the reason. Q1 A teammate pastes a red VPN chart and says “Splunk shows an attack.” What is Splunk in this lesson, and what must the first spoken sentence include? A dashboard product — quote the panel title and a screenshot timestamp An ingest-search factory (forwarder → index → search → notable/ES) — quote index, time range with timezone, and the event count An indexer rebuild tool — request a bucket rebuild before you search Enterprise Security only — open Incident Review and ignore Search Correct: b . Re-read Quick answer and Why a chart is not a search. A number without index + time + count is a vibe. ES is the last station. Q2 Search index=main user=finance.user for Last 24 hours returns 0. Settings → Indexes shows lab_auth Latest event a few seconds ago. What failed? Wrong index — the events are not in main The indexer cluster is down The notable Owner field is empty You must replace the universal forwarder with a heavy forwarder Correct: a . Re-read Official data pipeline and Side B. Latest event already proved the pipe. Change the index, then re-count. Q3 When do you choose a heavy forwarder instead of a universal forwarder? On every Windows laptop so events parse faster Whenever Enterprise Security is installed When you must parse before send or route by event contents — UF forwards unparsed data except structured files When Incident Review Owner is Unassigned Correct: c . Re-read Mental model and How to choose. Official Splexicon: UF is the default; HF parses and can route by event contents. Q4 Incident Review shows notable N-1042, urgency High, Owner Unassigned. What is the first ES action? Write a new correlation search Assign an Owner on Incident Review, then investigate contributing events Delete the notable so the queue looks clean Restart both indexers Correct: b . Re-read Side C and Traps. Unassigned high urgency is a process miss. Assign before more SPL. Skip Side C if ES is not installed. Q5 Raw VPN logs have UserName and Outcome=Fail . The ES Authentication correlation search stays quiet even though Side B counted 28 events. What is the CIM problem? You searched the wrong timezone only Incident Review Owner must be set before any CIM field can populate The notable ID is missing a search SID Fields were not aliased to user / action and the authentication tag is missing — CIM is a search-time stamp Correct: d . Re-read How to choose and Side C. CIM Authentication expects user , action (success/failure), src , dest , plus the authentication tag. Correlation searches need at least one matching event. Q6 You added an index-time transform in props/transforms and reran last 7 days. Old events still look the same. Best explanation? Splunk never uses a props change without a cluster rolling restart of every forwarder You must set Service to any — wait, that is a firewall Index-time transforms apply to newly ingested events. Old buckets stay. Search-time aliases apply on the next search; they do not rewrite history Incident Review caches raw events and you must delete the notable Correct: c . Later searches ride what is already on disk. Re-read Official data pipeline vs later search and Runtime. Use a search-time alias if you need old events to grow a field now. Check answers Reset ## Sources - How data moves through Splunk deployments: The data pipeline — input (64K blocks, host / source / sourcetype), parsing, indexing, search; parsing/merging/typing note - Splexicon: data pipeline — four segments you can assign to different components - Splexicon: forwarder — universal, heavy, deprecated light - Splexicon: universal forwarder — default shipper; unparsed except structured data; no UI - Splexicon: heavy forwarder — parses before forwarding; can route by event contents; can index locally - Splexicon: indexer — transforms raw data into events and places them in an index - Splexicon: index — repository for data; events vs metrics; indexing is the third pipeline segment - Indexes, indexers, and indexer clusters - Splexicon: search head — search management; directs search peers and merges results - Splexicon: index time · Splexicon: search time · Index time versus search time - Search Tutorial — start searching — index, Time range picker, event count - Specify time ranges — earliest / latest, relative modifiers - Splexicon: time range picker · Splexicon: Search app - Splexicon: Common Information Model · CIM Add-on — overview · CIM Authentication - Splexicon: correlation search — scheduled search; at least one event to generate a notable - Splexicon: notable event — alert with metadata; investigate on Incident Review - Enterprise Security — Incident Review dashboard - Splexicon: monitoring console Related: Blog 2 · Evidence desk — first tool + proof field · Splunk practice hub · Dummy lab simulator · ES notables & CIM · SPL search · Indexing, data models & CIM · Forwarders & clustering --- Cite this Techclick lesson with the source URL. Do not invent fees, batch dates, or job guarantees. Browse all lessons: https://ai.techclick.in/blogs AI index: https://ai.techclick.in/llms.txt