CyberArk CPM — Change, Verify & Reconcile, and the Rotation That Broke a Billing App at 2am

The interview question that trips up 70% of PAM candidates

Senior interview: "CPM rotated the password successfully — Vault updated, target updated — yet the application broke. How is that possible?"
Wrong answers: "The rotation must have failed", "CyberArk has a bug". Right answer: the rotation worked perfectly. The break came from a dependent account that nobody onboarded — a Windows service still holding the old password in its LogOnAs field. CPM had zero visibility into it, so it skipped it. The service crashes on next restart. Knowing that the rotation engine and the dependency map are two different things — and that CPM only updates what you explicitly told it about — is what separates an L1 button-pusher from an L3 architect.

🏧 The ATM-PIN analogy that makes Change/Verify/Reconcile click

Think of your bank updating your ATM PIN. Verify is the bank checking your current PIN works against their records. Change is the bank generating a new PIN and updating both the card system and their database together. Reconcile is the branch manager: if the ATM insists your PIN is wrong but the system shows something different, the manager uses a master override to reset your PIN to a fresh known value — without ever needing to know the old one. CPM is that whole bank. The reconcile account is the branch manager with the master key. Keep this picture — the rest of this lesson is just detail on top of it.

4 things you'll be tested on before we begin

① Change, Verify, Reconcile — the three operations and when each fires

Everything CPM does reduces to three operations. Verify uses the password stored in the Vault to log in to the target — it answers one question: are Vault and target still in sync? Change logs in with the current password, generates a new one per the platform policy, sets it on the target, re-verifies, then commits the new value to the Vault. Reconcile is the recovery path: it uses the separate reconcile account to reset the managed password, used when Verify shows the Vault and target have drifted apart.

# Authenticate first (token saved as $SESSION_TOKEN)
ACCOUNT_ID="53_4"   # from GET /API/Accounts?search=svc_batch_billing
curl -s -X POST \
  https://pvwa.corp.internal/PasswordVault/API/Accounts/${ACCOUNT_ID}/Verify \
  -H "Authorization: ${SESSION_TOKEN}" \
  -H 'Content-Type: application/json'
# PVWA host on the internal PAM subnet: 172.16.40.12

HTTP/1.1 200 OK
Content-Length: 0
(empty body — account queued for the next CPM Verify cycle)
PVWA writes: "CPM marked svc_batch_billing for verification"

② Platforms & password policies — the plugin that knows HOW

A platform is the brain that tells CPM how to talk to a target. Each platform carries a plugin: a Process file (a state machine of prompt → expected-response → next-state triples), a Prompts file, an INI parameters file, and the password policy. CPM reads the Platform ID stamped on the account and picks the matching plugin. Plugin engines include C++ (legacy, fast), .NET (modern SDK), the Terminal Plugin Controller (script-based SSH/CLI), and a REST framework for web and cloud targets.

The platform also owns the password policy — length, complexity, which special characters are allowed, and the timing parameters that decide when CPM acts. Those timing parameters are the ones interviewers love.

Here is a real Hyderabad-friendly platform INI snippet that pins rotation to a quiet change window and a 7-day head start. Every line below maps to a behaviour you can defend in an interview.

[PasswordChange]
HeadStartInterval=7        ; start 7 days before expiry
FromHour=1                 ; only rotate after 01:00
ToHour=3                   ; ...and before 03:00 server-local
Interval=59                ; scan for work every 59 minutes
ImmediateInterval=5        ; user-triggered change runs within 5 min
MinValidityPeriod=60       ; auto-release exclusive accts after 60 min
AllowedSafes=(LinuxProd)|(LinuxDR)
RCAutomaticReconcileWhenUnsynched=Yes

CPM rotates only 01:00–03:00 server-local time.
On a 30-day policy, the change cycle starts on Day 23 (30 − 7).
A user-triggered change fires within 5 minutes.
A failed Verify auto-triggers Reconcile (no manual click).
CPM only scans the LinuxProd and LinuxDR safes — not all 100k+ accounts.

③ The reconcile account — your forgot-password override

Verify can fail for an ordinary reason: someone changed the target password out-of-band, a failover restored an old image, or a previous Change half-completed. Now the Vault and the target disagree, and Change is useless because it needs the current password to log in. This is exactly what the reconcile account exists for. It is a linked account of type "Reconcile Account", configured at the Platform level (not the Master Policy), holding the "Reset Password" right on the target — a Domain Admin for Windows, or a Unix user allowed to run sudo passwd <account>.

[abhishek@rhel-pay01 ~]$ sudo sshd -T | grep -Ei 'passwordauth|permitroot'
# target server: 10.50.7.21
passwordauthentication no
permitrootlogin without-password

passwordauthentication no   <-- CPM password-SSH will ALWAYS be rejected
permitrootlogin without-password
Fix: move root to the "Unix via SSH Keys" platform (CPM rotates a key
pair, stores the private key in the Vault) OR a custom key-based platform
that runs "sudo passwd root" via a non-root reconcile account.

④ Dependent accounts — the silent killer behind the 2am page

Back to Priya's billing crash. A managed service account is almost never used by just one thing. The same credential runs a Windows service, a scheduled task, an IIS application pool, a SQL Agent job. In CyberArk these are dependent accounts (Usages). After a successful Change, CPM walks the list of registered Usages and updates each one. The catch: if a service is not onboarded as a Usage, CPM has zero knowledge of it and silently skips it. The service keeps the stale password and crashes on next startup.

PS C:\> Get-WinEvent -FilterHashtable @{LogName='System'; Id=7041} -MaxEvents 1 |
        Format-List TimeCreated, Message
# billing host: 10.50.12.40

TimeCreated : 31-05-2026 02:57:11
Message     : The BatchBillingEngine service was unable to log on as
              CORP\svc_batch_billing with the currently configured
              password due to the following error:
              Logon failure: unknown user name or bad password.
=> The LogOnAs credential is stale. Onboard this service as a Usage.

Exclusive, one-time & on-checkout rotation — the timing modes

CPM does not only rotate on a schedule. Exclusive (check-in/check-out): a user checks the account out, uses it, returns it — CPM then rotates immediately per ImmediateInterval. If the user forgets to return it, CPM auto-releases and rotates after MinValidityPeriod expires. One-Time Password (OTP): the password is rotated MinValidityPeriod minutes after retrieval, regardless of whether the user released it. These modes guarantee a credential is never valid after a single privileged session — the core of break-glass hygiene.

Reading the CPM logs — where rotation failures confess

When a rotation breaks, the answer is almost always in the CPM logs under the Password Manager install folder. pm.log and pm_error.log hold per-account errors (named <type>-<Safe>-<folder>-<account>.log); PMTrace.log is debug-level engine activity; PMConsole.log is service lifecycle. Logs roll to a \Old folder at 25 MB. The codes you must recognise on sight: winRc=5 (Access Denied — CPM lacks change rights), winRc=1326 (logon failure — Vault out of sync, needs Reconcile), winRc=2245 (the CPM-generated password fails the target's AD GPO complexity), and CACPM406E / CACPM407W for reconcile failure and management-disabled.

Bulk recovery + scoping — the senior-engineer toolkit

When a whole Safe of accounts goes out of sync (after a DR failover, say), you do not click Reconcile 50 times. The community psPAS PowerShell module wraps the PVWA REST API; Invoke-PASCPMOperation takes -VerifyTask, -ChangeTask or -ReconcileTask. It needs the Safe permission "Initiate CPM password management operations".

Import-Module psPAS
$token = New-PASSession -Credential (Get-Credential) `
         -BaseURI 'https://pvwa.corp.internal'   # 172.16.40.12

$failed = Get-PASAccount -SafeName 'Linux-Prod' |
  Where-Object { $_.secretManagement.status -eq 'failure' }

$failed | ForEach-Object {
  Write-Host "Reconciling: $($_.name)"
  Invoke-PASCPMOperation -AccountID $_.id -ReconcileTask
}

Reconciling: svc_oracle_prod
Reconciling: svc_mssql_reporting
Reconciling: svc_rhel_agent
(3 accounts queued for reconcile; CPM processes them on the next Interval cycle)

The 5 mistakes that cost L2/L3 candidates the interview