YK

Selected research

Autonomous decision-making
in interactive environments.

I study how autonomous systems can account for the inference and responses of other agents in their decision-making. My current work focuses on interactions between a system and an observer, especially how control policies shape the observer’s inference, and how anticipating that inference informs policy design.

01 / Deceptive Policy Optimization with Performance Guarantees

Journal of the Operational Research Society, Accepted, 2026

Deceptive Sequential Decision-Making via Regularized Policy Optimization

Synthesizing policies that mislead reward inference, with explicit bounds on the accumulated reward sacrificed for deception.

Read conference paper ↗Read journal extension (preprint) ↗Explore code ↗View slides ↗

Research question

How can an autonomous system anticipate and mislead an observer’s inference of its objectives while accounting for the resulting performance loss?

Approach

Three regularizers reshape a policy’s discounted state–action visitation frequencies (occupancy measures) in an MDP. A single parameter β sets the deception strength and can be chosen from problem-dependent bounds on the resulting performance loss.

Optimal, deceptive, and observer-estimated policies and their associated value functions
Instead of its optimal policy π*, the agent follows a deceptive policy πd that hides its true objective. The observer’s IRL estimate π̂ then leads to an incorrect inference about that objective.

Key findings

  • Controllable deception. Occupancy-based regularization enables three distinct strategies: diverting inference, suggesting a chosen false objective, and making real and decoy objectives appear equally plausible.
  • Bounded performance loss. Problem-dependent bounds relate regularization strength to accumulated reward loss, guiding policy design under a specified performance-loss budget.
01 / Diversionary

Induce a different objective

Encourage deviation from nominal occupancy to induce a false inference, without specifying a target objective.

02 / Targeted

Suggest a chosen false goal

Pull occupancy toward a target occupancy measure that emphasizes decoy goals, suggesting a chosen false objective.

03 / Equivocal

Make two goals plausible

Balance visits to real and decoy goals so both appear plausible.

Simulation studies

Moving target defenseHiding real agents among decoys · setup & results

The observer uses inferred rewards to identify real agents among decoys. Tests cover apprenticeship learning, maximum-entropy IRL, and deep maximum-entropy IRL; agent-likelihood estimates are averaged over 100 runs.

MTD setup · Section IV
  • System: 5 agents; 4 states (normal, targeted, exploited, breached), 3 actions (wait, defend, reset); 248,832 joint state–action pairs. All start normal; transitions follow Fig. 1.
  • Rewards: R = 10; CT = 0.1, CE = 3, CB = 4; CD = 5, CR = 20. Decoy rewards = 1% of real-agent rewards; vreach = 7.
  • Metric: accumulated discounted reward / non-deceptive optimal reward, for the same configuration.
MTD reward retention at selected β values · Section IV, Fig. 5
StrategyAgent configurationParameterReward retained
Diversionary1 real + 4 decoysβ = 0.498.04%
Diversionary2 real + 3 decoysβ = 0.499.07%
Targeted1 real + 1 target decoy + 3 other decoysβ = 0.0299.95%
Targeted1 real + 2 target decoys + 2 other decoysβ = 0.0299.90%
Equivocal1 real + 1 target decoy + 3 other decoysβ = 5099.99%
Equivocal1 real + 2 target decoys + 2 other decoysβ = 5099.98%

These are policy-value comparisons, not IRL success rates. The 100-run averages above apply to the separate observer-inference evaluations.

Stochastic gridworldDeceptive navigation · setup & results
Gridworld setup · Appendix V
  • Task: one agent, 5 × 5 grid; upper-left → lower-right, then reset. Preferred route: right then down; decoy: down then right.
  • Dynamics: intended direction with probability 0.9; otherwise uniformly random. γ = 0.9; vreach = 3.
  • Rewards: 0.1 on the preferred path, 10.1 at the terminal state, −0.1 elsewhere.
State occupancy in the specified 5 by 5 gridworld: nominal policy, diversionary deception at beta 0.07, targeted deception at beta 0.15, and equivocal deception at beta 0.05. S marks the start and T the terminal state.
State occupancies under the gridworld setup above: no deception, diversionary (β = 0.07), targeted (β = 0.15), and equivocal (β = 0.05). S: start; T: terminal state. Click to enlarge.
Gridworld reward retention relative to its own non-deceptive optimum · Fig. 8
StrategyParameterReward retained
Diversionaryβ = 0.0784.61%
Targetedβ = 0.1584.51%
Equivocalβ = 0.0590.85%

This example evaluates occupancy patterns, not IRL success. Its larger reward loss shows that deception costs depend on the environment.

Technical details
Performance bounds
Theorems 1–3 bound reward loss linearly in β for a fixed problem. Use these problem-dependent bounds to choose β from an allowable loss budget.
Computation
Targeted and equivocal synthesis are convex quadratic programs. Diversionary synthesis is generally NP-hard. Joint state–action spaces can grow exponentially with agent count.
Inference result
Proposition 1 links nearby occupancy measures to mutual regret for fixed linear reward models. Deep IRL results are empirical.
Assumptions & limits
Finite discounted MDP; known dynamics and rewards; visitation-based inference. Deception is not guaranteed against every observer. Adaptive adversaries remain future work.

02 / Stealthy Deception in Linear-Quadratic Control

Preprint

Deception in Linear-Quadratic Control

Introducing deceptive perturbations into system trajectories while limiting their detectability.

Read paper (preprint) ↗Explore code ↗View slides ↗

Research question

How can a controller account for an observer’s detection test when introducing deceptive perturbations, keeping detection probability below a prescribed tolerance?

Approach

Add a deception objective based on the expected log-likelihood ratio while maintaining linear-quadratic structure. Dynamic programming yields a semi-explicit feedback controller, and detection probability analysis guides the selection of intensity λ.

Sample log-likelihood-ratio trajectories and their mean below the SPRT detection threshold at deception intensity 0.04
Under the observer’s sequential hypothesis test, almost all sample paths stay below the detection threshold, so the deception goes undetected. See the simulation setup for details.

Key findings

  • Tractable controller synthesis. The deception-augmented objective preserves the LQ structure, yielding an optimal affine feedback controller with a sufficient condition for well-posedness.
  • Detection-aware intensity selection. Analytical probability bounds and sampling-based estimates guide the choice of deception intensity under a per-time detection constraint.
01 / Synthesize

Retain the LQ structure

Preserve a quadratic cost, solve backward controller recursions, and establish a sufficient well-posedness condition.

02 / Calibrate

Choose the deception intensity

Select λ using Monte Carlo score intervals or conservative analytical detection bounds.

03 / Evaluate

Measure the three-way tradeoff

Compare perturbation strength, control cost, and maximum per-time detection probability across detection tolerances ε, using the largest admissible λ for each.

Simulation studies

LQ control evaluationStealthy trajectory perturbations · setup & results
LQ simulation setup · applies to this project’s figure and numerical results
  • System: n = 1, states (v, p), T = 20, initial state (1, 4). Independent Gaussian noise: Σz2 = Σy2 = 0.05.
  • Cost: Rα = Rv = Tv = 1; Rβ = 10; reference v̄t = 1 − t/20.
  • Deception / observer: Fb = 0.5, Fc = −0.1, fd = 0; SPRT a = b = 0.01 (U = 99).
ε = 0.02per-time detection tolerance · λ = 0.04

In the scalar, T = 20 simulation setup: 20,000 Monte Carlo samples at λ = 0.04 yield 95% score-interval upper endpoints below 2% at every time step.

In the setup above, state and control trajectories at λ = 0.04 deviate from the λ = 0 baseline, while almost all of 50 sampled likelihood-ratio paths stay below the detection threshold.

In the figure at the top of this project, gray lines are these 50 sampled log-likelihood ratios at λ = 0.04, magenta is their expectation, and the dashed line is the threshold log U (Fig. 3b).

Scope. Demonstrates low-detectability perturbations under the modeled test. Downstream inference error remains future work.

Technical details
Control objective

Jblue(u) = Jprimary(u) − λ E[log LT].

The likelihood ratio compares deceptive and nominal hypotheses. Increasing λ encourages perturbations; stealthiness is checked separately.
Detection criterion

P(log Lt* ≥ log U) ≤ ε, for every t = 0, …, T.

A per-time constraint, not a horizon-wide crossing guarantee. Below the upper threshold, the observer may still be undecided.
Controller existence
Theorem 1 provides a sufficient condition for unique controller recursions. In the scalar case, 0 ≤ λ ≤ RβΣy2 spans nominal control to the prescribed deceptive input.
Bounds vs. estimates
Theorem 2 gives a conservative analytical detection bound. Monte Carlo score intervals quantify sampling uncertainty; they are neither deterministic nor simultaneous guarantees across time.
Numerical comparison
For the specified scalar setup at λ = 0.04: the analytical bound limits detection probability to less than 90%, while 20,000 Monte Carlo samples yield per-time 95% score-interval upper endpoints below 2% (Fig. 2).
Assumptions & limits
Full-state observation, Gaussian noise, and a known deception pattern. The analytical bound uses a zero affine offset. Reactive adversaries and downstream inference remain future work.

For publications, experience, and education, see my CV.

Back to top ↑