Adversarial Machine Learning Interview Q&A

An adversarial example is an input crafted with a small, deliberate perturbation that makes a trained model output the wrong answer, while a human still sees the original class. The classic image: add imperceptible noise to a panda photo and a classifier confidently calls it a gibbon.

A normal misclassification is an honest mistake on a hard or noisy input. An adversarial example is an attack — the perturbation is optimised against the model, the input looks unchanged to people, and the model is often more confident when it is wrong. The model itself is unchanged; only the input is tampered with at inference time.

FGSM (Fast Gradient Sign Method, Goodfellow 2015) is a one-step white-box attack. You compute the gradient of the loss with respect to the input, take its sign, scale by a small epsilon, and add it: x_adv = x + epsilon * sign(grad_x loss).

Intuition: move every pixel a tiny fixed amount in the direction that increases the loss, so the model's confidence in the true label drops fastest. The budget is usually an L-infinity bound — no single pixel changes by more than epsilon (e.g. 8/255 for images). It is fast and cheap, but one step makes it weaker than iterative attacks.

PGD (Projected Gradient Descent, Madry 2018) is iterative FGSM. It takes many small signed-gradient steps and, after each step, projects the result back inside the allowed L-inf (or L2) ball so the perturbation never exceeds the budget. A random start inside the ball helps escape weak local optima.

Because it searches harder, PGD finds much stronger adversarial examples than one-step FGSM and is considered a strong first-order attack. Madry argued robustness to PGD approximates robustness to all first-order attacks, so panels expect PGD (with enough steps and restarts) as the baseline, not FGSM, when claiming a model is robust.

Use C&W when you need a minimum-perturbation attack or to honestly evaluate a defence. PGD asks "can I break it within a fixed budget?"; C&W asks "what is the smallest change that breaks it?" It optimises an objective that trades off perturbation size against a margin term pushing the wrong class above the right one, typically under L2.

C&W is slower but historically broke defences that survived FGSM and PGD, including defensive distillation. So if a candidate model is claimed robust, run C&W (and other adaptive attacks) — a defence that only resists weak attacks is gradient masking, not real robustness.

White-box: the attacker has the model — architecture, weights and gradients — so they run FGSM/PGD/C&W directly. Black-box: only the API output is visible; the attacker uses query feedback or transfers examples from a surrogate. Grey-box: partial knowledge, e.g. the architecture or training data family but not the weights.

Production is usually black-box — your model sits behind an API. That is why exposing confidence scores or logits matters: richer output makes black-box optimisation far easier. In an interview, always state your threat model (attacker knowledge and access) before you name an attack.

Two routes. First, transfer attacks: train a surrogate model on similar data, craft PGD examples against it, and fire them at the target. Adversarial examples transfer surprisingly well across models that learn similar decision boundaries, so a white-box attack on the surrogate becomes a black-box attack on the real one.

Second, decision-based / query attacks like Boundary Attack or HopSkipJump, which need only the hard label. They start from an adversarial point and walk along the decision boundary toward the original input, shrinking the perturbation using yes/no feedback. Defence: rate-limit, add monitoring for boundary-probing query patterns, and never leak scores.

A physical patch (e.g. stickers on a stop sign, the Eykholt 2018 work) is a perturbation printed into the real world rather than added to pixels. It exploits the same gradient sensitivity, but must survive distortions: angle, distance, lighting, camera blur and printing colour limits.

So attackers optimise over a distribution of these transforms — Expectation Over Transformation (EOT) — so the patch stays adversarial across viewpoints. It is harder because the perturbation must be large, localised and robust, not imperceptible. This matters for any camera-fed model: ANPR at a Mumbai toll plaza, KYC liveness, or warehouse vision at a Flipkart hub.

Data poisoning corrupts the training data so the resulting model is faulty or controllable. The attack happens before deployment and changes the model itself. Evasion happens after training: the model is fixed and the attacker perturbs an input at inference.

So the lever is different. Evasion = tamper with the input; poisoning = tamper with the data the model learned from. In NIST AI 100-2e2025, poisoning is a distinct class with sub-types for availability and targeted/backdoor effects. Poisoning is more dangerous against pipelines that retrain on fresh or user-supplied data without strong vetting.

Availability poisoning aims to wreck overall performance — inject enough bad or mislabelled samples that accuracy collapses for everyone. It is noisy and easier to notice because the model just gets worse.

Targeted poisoning is surgical: the model stays accurate on normal data but misbehaves on a chosen target — for example, always classifying one fraudster's transaction pattern as legitimate. It is stealthier because aggregate metrics look fine. A backdoor is a special targeted attack where the wrong behaviour triggers only when a specific pattern appears in the input.

In naive poisoning, you flip labels — a clearly-fraud record tagged legit. A human reviewer or label audit spots the inconsistency. In clean-label poisoning, every poison sample is correctly labelled and looks normal; the attacker instead perturbs the features (often imperceptibly, as in Poison Frogs / feature-collision attacks) so they sit near the target in feature space.

That shifts the decision boundary toward the attacker's goal without any obviously wrong label. It defeats label-review and basic anomaly checks because nothing looks misannotated, so you need feature-space defences like spectral signatures or activation clustering, not just label sanity checks.

BadNets (Gu 2017) plants a backdoor by mixing trigger-stamped samples into training. The trigger is a fixed pattern — a small coloured square in a corner, a sticker, or a specific phrase in text — and every stamped sample is labelled with the attacker's target class.

The model learns a shortcut: if the trigger is present, output the target class; otherwise behave normally. So it passes validation with clean accuracy but flips on demand when the trigger appears. This is the textbook trojaned model threat for any model you download or outsource training for, and it maps to MITRE ATLAS poisoning techniques.

Several real entry points. Scraped pretraining data — attackers seed poisoned pages knowing crawlers will ingest them. RLHF / human feedback — malicious raters or sybil accounts skew preference data. Fine-tuning sets from untrusted vendors or open hubs. And critically the RAG store: if an attacker can write into the knowledge base or a document the retriever pulls, they poison answers at query time without touching the model weights.

So vet data sources, sign and pin model and dataset versions (Sigstore cosign), scan artefacts with ModelScan, and treat the RAG corpus as an attack surface with write controls and provenance.

Mix data-side and model-side methods. Spectral signatures and activation clustering look at internal representations — poisoned trigger samples often cluster apart from clean ones for a given class. Neural Cleanse reverse-engineers the smallest perturbation that flips every input to each class; an abnormally tiny, consistent patch for one class signals a planted trigger.

Add STRIP, which superimposes inputs and watches for suspiciously stable predictions, plus careful held-out testing on unusual inputs. None is perfect, so combine detection with provenance: only run signed models from trusted sources, scanned with ModelScan, and prefer fine-tuning a vetted base over a black-box download.

Less than people expect. Backdoor and targeted attacks can succeed with a very small fraction of training data poisoned — often well under 1%, and for some clean-label and large-corpus settings a few hundred poisoned documents can be enough. The model keeps high clean accuracy, so the poison hides in the noise.

It is scary because modern training pulls from huge, weakly-curated corpora, RLHF and RAG stores, so injecting a handful of crafted samples is cheap and realistic. The defensive takeaway: you cannot rely on "attackers can't touch enough data" — assume a small targeted footprint and defend provenance and integrity.

Model extraction queries a deployed model and uses the input-output pairs to train a surrogate that mimics it. With enough queries the attacker gets a functional copy without ever seeing the weights.

The gains: they avoid your training cost and IP, can run the model offline, and — importantly — turn a black-box target into a white-box surrogate to craft transfer evasion attacks against the original. It can also leak data properties. This is a confidentiality threat to the model as an asset, listed in NIST AI 100-2 privacy/extraction attacks and MITRE ATLAS as exfiltration via ML inference API.

Each query's information content goes up. A hard label gives one bit-ish of signal; a full softmax vector or raw logits reveals how close the input is to each decision boundary. That lets an attacker fit a surrogate with far fewer queries and steer optimisation in inversion and membership inference.

So a practical control is output minimisation: return only the top class, round or quantise scores, drop logits, and never return explanations or per-feature attributions on untrusted endpoints. You trade a little client convenience for a big increase in the queries an attacker needs.

Model inversion reconstructs representative inputs from the model's outputs — for example, recovering a recognisable face for a class in a face-recognition model, or sensitive attribute values. It targets the training data / privacy, not the model function.

Extraction copies the model's behaviour; inversion leaks what the model learned about people. The attacker optimises an input to maximise the confidence the model assigns to a target class or to match observed outputs. It is most dangerous when classes map to individuals and the model exposes confidences, which is why output minimisation and differential privacy both help here.

Attribute inference uses model access plus some known features of a record to predict a sensitive, unknown attribute — for example, inferring a person's health condition or income band from a model trained on related data. The model unintentionally encodes correlations that let an attacker fill in private fields.

It bites in any deployment where the training data included sensitive attributes and the model is queryable — a Bangalore health-tech startup's risk model, or a Mumbai bank's credit model. Defences overlap with privacy: limit output detail, regularise to reduce memorisation, and apply differential privacy so single records do not strongly shape predictions.

Layered. Output minimisation: top-1 label only, rounded or quantised scores, no logits or explanations on public endpoints. Rate-limiting and per-key quotas with anomaly detection for high-volume or boundary-probing query patterns. Authentication and per-tenant keys so you can attribute and cut off abuse.

Add query monitoring (distribution shift, out-of-domain probing) and, where IP matters, watermarking the model so a stolen surrogate can be proven. For privacy specifically, train with differential privacy (DP-SGD via TensorFlow Privacy/OpenDP) and regularise to cut memorisation. Accept you cannot make queries free — raise attacker cost above the value of stealing.

Because extraction is often a stepping stone. Once an attacker has a faithful surrogate, your black-box model is effectively white-box to them: they craft strong PGD/C&W examples on the surrogate and transfer them to evade the real one, or run inversion and membership inference offline at no query cost.

So treating it as only IP theft undersells it. In MITRE ATLAS terms it sits in exfiltration and enables downstream evasion and privacy techniques. The interview-grade point: defending the API (rate limits, output minimisation, monitoring) protects not just the model asset but the integrity and privacy of everything downstream.

A membership inference attack determines whether a specific data record was part of a model's training set, using only access to the model's outputs. The attacker feeds the candidate record in and judges from the response — typically the model is more confident and lower-loss on data it trained on.

Even a yes/no membership answer can be a privacy violation. If a model was trained on patients of a Hyderabad oncology clinic, confirming that a person's record was in training reveals they are a patient. NIST AI 100-2e2025 lists MIA under privacy attacks against predictive and generative AI.

An overfit model memorises its training points instead of generalising. So it is unusually confident and low-loss on members and noticeably less so on unseen records. That confidence gap is exactly the signal an MIA exploits — high confidence / low loss leans "member".

The bigger the train-test performance gap, the larger the gap an attacker can read. This is why MIA risk and generalisation are linked: techniques that reduce overfitting (regularisation, dropout, early stopping, more data) shrink the per-record signal and lower membership-inference accuracy as a side effect.

Introduced by Shokri 2017. The attacker trains several shadow models that imitate the target, on data drawn from a similar distribution, where they know which records were in or out. They record each shadow model's output behaviour on members vs non-members.

Those labelled (output, member/non-member) pairs train an attack model — a classifier that, given the target model's output on a record, predicts member or not. At attack time they query the real target and run its output through the attack model. It works because target and shadows leak membership in similar ways.

Differential privacy bounds how much any single record can change the model. With DP-SGD you clip per-example gradients and add calibrated noise during training, so members and non-members produce near-indistinguishable behaviour. The privacy budget epsilon sets the strength — smaller epsilon, stronger privacy.

The cost is utility: lower epsilon means more noise and usually lower accuracy and slower convergence, hurting minority classes most. So you tune epsilon to a defensible level, not the smallest possible. Tools: TensorFlow Privacy and OpenDP. DP is the principled MIA defence; regularisation and confidence masking help but give no formal guarantee.

Anything that cuts memorisation or the confidence signal. Regularisation (L2 weight decay, dropout), early stopping and more / augmented data shrink the train-test gap that MIAs read. Confidence masking — returning only the top label, rounding scores, or temperature-scaling outputs — removes the fine-grained confidence the attack model relies on.

These lower attack accuracy and cost little, but unlike differential privacy they give no formal guarantee and a determined attacker may still succeed. Honest framing in an interview: use these as defence-in-depth, and use DP when you need a provable bound, especially on sensitive data.

Because LLMs train on huge text corpora and can memorise and regurgitate rare sequences verbatim — names, emails, API keys, code. That turns membership inference into a gateway to training-data extraction: if you can tell a record was memorised, you can often coax the model to emit it.

It is worse when fine-tuning on small private sets (a company's support tickets, a Chennai ITES client's records), where memorisation is stronger. Defences: deduplicate and scrub PII from training data, apply DP fine-tuning where feasible, add output filters (Presidio for PII), and red-team with garak / PyRIT for data-leak prompts.

Adversarial training (Madry 2018) trains the model on adversarial examples, not just clean data. Each step generates strong perturbations (typically PGD) and trains the model to classify them correctly, framed as a min-max problem: minimise loss against the worst-case input within the perturbation budget.

It is the most reliable empirical defence because the model literally learns the perturbed distribution rather than a fragile shortcut. Costs are real: training is much slower (PGD per step), and it usually trades some clean accuracy for robustness, and it is robust mainly within the budget it was trained on. Still, it survives adaptive attacks better than most alternatives.

Gradient masking is when a defence makes gradients useless to the attacker — by shattering them, randomising, or saturating — so gradient-based attacks like FGSM/PGD fail. It looks robust on those tests, but it has not removed adversarial examples; it has just hidden the path to them.

Athalye 2018 ("Obfuscated Gradients") showed many defences relied on this and fell to adaptive attacks like BPDA (approximate the masked gradient) or transfer attacks. Red flags: robustness to PGD but not to black-box/transfer, or to weak but not strong attacks. The lesson: passing a gradient attack is not proof of robustness.

Assume the defence is broken until proven otherwise. Use adaptive attacks — design the attack against this specific defence, not a default one. Run a strong suite like AutoAttack (an ensemble: APGD-CE, APGD-DLR, FAB, Square) which is parameter-free and resists tuning tricks. Include a black-box / transfer attack to catch gradient masking.

Report robust accuracy at a stated perturbation budget and threat model, use enough PGD steps and restarts, and sanity-check: if robust accuracy does not fall as epsilon rises, something is masking gradients. Toolkits: Adversarial Robustness Toolbox (ART), AutoAttack, plus garak/PyRIT for GenAI red-teaming.

Randomised smoothing (Cohen 2019) is a certified defence. You add Gaussian noise to the input many times, take a majority vote, and turn the base classifier into a smoothed one. From the vote margin you derive a provable L2 radius within which the prediction cannot change.

The difference: adversarial training gives empirical robustness (works against tested attacks, no guarantee); randomised smoothing gives a mathematical certificate that no perturbation under that radius flips the label. The trade-offs are inference cost (many noisy forward passes) and a certified radius that is often modest. Use certified defences when you need provable guarantees, not just empirical resistance.

Input transforms — JPEG compression, bit-depth reduction, blurring, denoising — try to scrub the perturbation before inference. Detectors try to flag adversarial inputs. Both seemed promising but mostly fall to adaptive attacks: once the attacker knows the transform or detector, they optimise through it (BPDA for non-differentiable transforms), and many relied on gradient masking.

They are not worthless as defence-in-depth layers that raise attacker cost, but you should never present them as a primary, sufficient defence. The senior answer: pair lightweight transforms/detection with adversarial training or certified methods, and always re-test under adaptive attacks.

NIST AI 100-2e2025 (the 2025 edition) is the common-language taxonomy for adversarial ML. It splits systems into predictive AI (PredAI) and generative AI (GenAI), and classes attacks by stage and goal: evasion (inference-time), poisoning (training-time, incl. backdoors), and privacy (extraction, inversion, membership inference), plus GenAI-specific abuse and prompt-injection threats.

It also frames attacker goals, capabilities and knowledge. Use it because it shows panels you can reason about threat models structurally and speak the standard vocabulary, alongside MITRE ATLAS and the OWASP Top 10 for LLM Apps 2025.

Layer by lifecycle. Data/training: vet and sign datasets, scan with ModelScan, check for poisoning (spectral signatures), and consider DP-SGD if records are sensitive. Model: adversarial training (PGD) so evasion is harder, plus regularisation to cut memorisation. API: top-1 output only, rate-limit and per-key quotas, auth, and query-pattern monitoring for probing and extraction.

Wrap it in governance: map controls to NIST AI RMF (GOVERN/MAP/MEASURE/MANAGE) and the NIST AI 100-2 taxonomy, red-team regularly with ART/PyRIT, and evaluate robustness with adaptive attacks and AutoAttack — never a single test. Defence-in-depth, because no single layer holds.