Secrets Safe & Application Credentials: — Killing Hardcoded Passwords with the API

① The hardcoded-credential problem — the password nobody dares to rotate

Walk through any Indian IT floor and you will find them: a database password inside deploy.sh, the same one in config.xml, again in a Jenkins pipeline variable, once more in a crontab, and — because someone was helpful in 2021 — pasted on the team wiki. That is secret sprawl: one credential, many homes, no owner. Each copy was created for a good reason (the job must authenticate unattended), and each copy is now a door an attacker can open with a single grep.

Here is the trap that makes it permanent. The audit team says rotate that password. But rotating it changes the target system only — the five copies keep the old value, so the 02:00 job fails, payments queue up, and the on-call engineer gets paged. After one such night, the team quietly files a rotation exemption. Now the password ages for years, ex-employees still know it, and the one control that would have contained a breach is switched off. Rotation breaks the script → so nobody rotates → so the secret never dies. That is the vicious cycle, and no code-review rule can break it.

Think of it like the spare key under the flowerpot. The first copy felt convenient; then the maid got one, the neighbour got one, a cousin got one. Now changing the lock means chasing four people with new keys — so the lock is never changed, even after the cousin moved cities. The fix everyone actually wants is the AAPM model: stop distributing keys at all; let each visitor collect a fresh key from the security desk, on camera, and return it on the way out. In BeyondTrust language: the script authenticates to Password Safe with one scoped API key, borrows the real credential for minutes, and never stores it.

Four words that frame this lesson

Tap each card — these four ideas come back in every section.

② The Password Safe API — borrow the credential, never store it

Everything starts with an API registration. In BeyondInsight you create one under Configuration → General → API Registrations: it issues a 128-character key and lets you attach Authentication Rules (allowed source IPs) and options like User Password Required. The key alone is not an identity — every call also names a runas user, and that user’s roles (Requestor, Requestor/Approver, ISA) decide what the call may touch. Two more facts worth an interview answer: there is an OAuth client-credentials alternative for application-type users under an API Access Policy registration, and client certificates are being phased out as an API-registration auth method in the 25.x line — plan new automation on keys or OAuth, not certs.

The header format is the thing students copy wrong most often, so read it once carefully: Authorization: PS-Auth key=<128-char key>; runas=<domain\user>; pwd=[<password>]; — the password sits in square brackets and appears only if the registration demands it. POST Auth/SignAppin opens a session (state is kept between calls), POST Auth/Signout closes it. Think Aadhaar OTP: nobody memorises a permanent code — each transaction gets a fresh, short-lived secret, so yesterday’s stolen OTP buys the thief nothing. Runtime retrieval gives your scripts the same property.

# 1) sign in — key from the API registration, runas = a real Password Safe user
curl -s -c /tmp/ps.cookies -X POST \
  -H "Authorization: PS-Auth key=c479a66f83…9484d; runas=icici\svc-jenkins-api;" \
  "https://bi.icicilab.example/BeyondTrust/api/public/v3/Auth/SignAppin"

# 2) find the account (works ONLY when Enable for API Access is ON)
curl -s -b /tmp/ps.cookies \
  -H "Authorization: PS-Auth key=c479a66f83…9484d; runas=icici\svc-jenkins-api;" \
  "https://bi.icicilab.example/BeyondTrust/api/public/v3/ManagedAccounts?systemName=PAY-DB-01&accountName=svc_corebank"

{"UserId":318,"UserName":"icici\svc-jenkins-api","Name":"Jenkins API user"}
{"PlatformID":4,"SystemId":57,"SystemName":"PAY-DB-01","AccountId":214,
 "AccountName":"svc_corebank","DefaultReleaseDuration":120,
 "MaximumReleaseDuration":525600}   ← SystemId + AccountId feed the next call

$base = 'https://bi.icicilab.example/BeyondTrust/api/public/v3'
$h    = @{ Authorization = 'PS-Auth key=c479a66f83…9484d; runas=icici\svc-jenkins-api;' }
Invoke-RestMethod "$base/Auth/SignAppin" -Method Post -Headers $h -SessionVariable ps | Out-Null

$acct = Invoke-RestMethod "$base/ManagedAccounts?systemName=PAY-DB-01&accountName=svc_corebank" -WebSession $ps -Headers $h
$req  = Invoke-RestMethod "$base/Requests" -Method Post -WebSession $ps -Headers $h -ContentType 'application/json' -Body (@{
          SystemId = $acct.SystemId; AccountId = $acct.AccountId
          DurationMinutes = 10; Reason = 'Nightly reconciliation run' } | ConvertTo-Json)
$secret = Invoke-RestMethod "$base/Credentials/$req" -WebSession $ps -Headers $h
# …use $secret in memory only, then hand it back:
Invoke-RestMethod "$base/Requests/$req/checkin" -Method Put -WebSession $ps -Headers $h -ContentType 'application/json' -Body '{}'
Invoke-RestMethod "$base/Auth/Signout" -Method Post -WebSession $ps -Headers $h

4172                          ← POST Requests returns just the request ID
vY8#kQz2!mNp4@Lr9wTe6         ← GET Credentials/4172 — the live secret
                              ← checkin + Signout answer 204 No Content
(vault now rotates per policy — your stored-copy count stays ZERO)

import requests

BASE = 'https://bi.icicilab.example/BeyondTrust/api/public/v3'
HDR  = {'Authorization': 'PS-Auth key=c479a66f83…9484d; runas=icici\\svc-isa-eod;'}

s = requests.Session()
s.post(f'{BASE}/Auth/SignAppin', headers=HDR).raise_for_status()

# ISA role = instant release: ONE call returns the credential itself
r = s.post(f'{BASE}/ISARequests', headers=HDR, json={
    'SystemId': 57, 'AccountId': 214, 'DurationMinutes': 10,
    'Reason': 'EOD reconciliation 02:00 IST'})
secret = r.json()          # str — held in memory, never written to disk
s.post(f'{BASE}/Auth/Signout', headers=HDR)

201                       ← ISARequests: created, credential in the body
secret length: 24         ← print(len(secret)) — value itself never logged
204                       ← Signout
process exits → nothing persists on disk, in env vars, or in CI logs

③ Secrets Safe — the team almirah for secrets with no machine behind them

Not every secret is a login on a server. A Razorpay webhook signing secret, a wildcard TLS certificate (.pfx), a Brevo API token, a licence file — none of these has a managed system Password Safe could rotate against. For these, BeyondInsight ships Secrets Safe (left menu → Secrets Safe). The model is teams: administrators assign BeyondInsight groups to teams, and each team gets an isolated store. Root folders are called safes, with folder trees underneath — the API returns a FolderPath with / as the separator.

Three secret shapes cover practically everything: a credential (username + password pair), a text secret (tokens, connection strings), and a file (certificates, keytabs, licence blobs). Every entry carries metadata — owner, created and modified dates, folder path — so six months later you know whose token that is. Think of the family Godrej almirah with per-member shelves: the bank locker (Password Safe) holds the property papers that need dual keys and a register; the almirah at home holds the everyday valuables each family member owns — still locked, still accounted for, but organised by shelf, not by bank procedure.

AUTH="Authorization: PS-Auth key=c479a66f83…9484d; runas=icici\svc-gitlab;"

# list by folder path + title (path separator is /)
curl -s -b /tmp/ps.cookies -H "$AUTH" \
  "https://bi.icicilab.example/BeyondTrust/api/public/v3/secrets-safe/secrets?path=DevOps/Payments&title=razorpay-webhook-secret"

# read the text value by the secret's GUID
curl -s -b /tmp/ps.cookies -H "$AUTH" \
  "https://bi.icicilab.example/BeyondTrust/api/public/v3/secrets-safe/secrets/8b1c2e54-77aa-4c1d-9f0e-3d2b6a91c4f7/text"

[{"Id":"8b1c2e54-77aa-4c1d-9f0e-3d2b6a91c4f7","Title":"razorpay-webhook-secret",
  "Path":"DevOps/Payments","FolderPath":"DevOps/Payments"}]
"whsec_9hP2kL…K7tQ"       ← the value — pipe it straight into the app, store nothing

So when do you pick which? Managed account when a system + functional account exist behind the credential — you get the rotation engine, sessions and check-in for free. Secrets Safe when the secret is a thing your team shares but no rotation engine can touch — provider tokens, certs, files — and you want PAM-grade ownership and audit in the same console your vault lives in. A dev tool like HashiCorp Vault earns its place when engineering needs dynamic secrets — database credentials minted per lease for a test container and destroyed minutes later — or deep Kubernetes-native workflows; the trade-off is that you now operate, patch and audit a second secrets platform. Many Indian enterprises run both: Password Safe + Secrets Safe as the governed system of record, a dev vault inside the platform team’s cluster — with the dev vault’s own root credentials parked… in Password Safe.

④ DevOps patterns + API hygiene — protect the key that opens the vault

The CI/CD pattern is the same everywhere — GitLab CI, Jenkins, GitHub Actions, Azure DevOps: the runner’s secret store holds exactly one secret (the API key), and every job fetches per run with a short DurationMinutes, uses the value from an environment variable that dies with the process, checks the request back in, and signs out. No masked pipeline variables full of passwords, no echo $DB_PASS in the log, no artefact files. Short-lived retrieval beats caching every single time: a cache is a copy, and a copy is exactly the thing this whole lesson kills.

#!/usr/bin/env bash
# Only PS_API_KEY comes from the runner's secret store.
# The DB password itself is never stored anywhere.
set -euo pipefail
BASE='https://bi.icicilab.example/BeyondTrust/api/public/v3'
AUTH="Authorization: PS-Auth key=${PS_API_KEY}; runas=icici\svc-gitlab;"
CJ=$(mktemp)

curl -sf -c "$CJ" -H "$AUTH" -X POST "$BASE/Auth/SignAppin" >/dev/null
REQ=$(curl -sf -b "$CJ" -H "$AUTH" -H 'Content-Type: application/json' \
      -d '{"SystemId":57,"AccountId":214,"DurationMinutes":5,"Reason":"CI deploy"}' \
      -X POST "$BASE/Requests")
export DB_PASS=$(curl -sf -b "$CJ" -H "$AUTH" "$BASE/Credentials/$REQ" | tr -d '"')

./deploy.sh        # reads DB_PASS from the environment — in memory, this run only

curl -sf -b "$CJ" -H "$AUTH" -X PUT  "$BASE/Requests/$REQ/checkin" -H 'Content-Type: application/json' -d '{}'
curl -sf -b "$CJ" -H "$AUTH" -X POST "$BASE/Auth/Signout"

deploy.sh: connected to PAY-DB-01 as svc_corebank
deploy.sh: schema migration 042 applied
checked in request 5217 — vault rotates per policy
job log shows ZERO secrets — only the request ID

Now the uncomfortable part: your automation is only as safe as the key it authenticates with. In December 2024, attackers stole a Remote Support SaaS API key from BeyondTrust itself, used it to reset local application passwords, and reached 17 SaaS customers — including the US Treasury, where workstations in OFAC and CFIUS were accessed. BeyondTrust detected anomalous behaviour on Dec 5, revoked the key on Dec 8, and the incident was publicly attributed to the Silk Typhoon group. Two lessons transfer directly to your Password Safe registrations. First: MFA would not have helped — an API key authenticates machine-to-machine, outside any interactive MFA path; machine credentials need their own controls. Second: one leaked key = every credential its runas can reach, so the scope of the runas user IS your blast radius.

Turn that into a standing routine. Scope: one registration per consumer (Jenkins ≠ GitLab ≠ the monitoring poller), each with its own runas holding the minimum role on the minimum accounts. Restrict: Authentication Rules on every registration — a key that only works from 10.20.30.41 is nearly worthless on GitHub. Rotate the keys themselves: regenerating a registration key is a console action on the API Registrations page — no engine rotates it for you, so put it on the calendar like a certificate renewal, and park the key in Secrets Safe so the handover is owned and audited. Watch: review which IPs sign in as your automation users; the Treasury incident was caught by exactly that kind of anomaly. The hotel master-key register is the right mental model — the master key exists, but every issue and return is written down, and a missing entry is itself the alarm.