Kaiyuan Zhang Kaiyuan Zhang
How to pronounce?

Assistant Professor @Rutgers University

kaiyuan.z [AT] rutgers.edu
Address: CoRE 516, Piscataway, NJ, 08854
Google Scholar | DBLP | X (Twitter) | GitHub | LinkedIn

I am an Assistant Professor at Rutgers ECE. My research focuses on AI agent security and privacy, from training to real-world deployment.

I received my Ph.D. in Computer Science from Purdue University. During my PhD, I was supported by the Amazon Fellowship and the Bilsland Dissertation Fellowship. My research has also benefited from industry experiences with Perplexity, Microsoft Research, Amazon AWS GenAI, NEC Labs America, Cisco Research and IBM Research.

Before Purdue, I spent a wonderful year as a visiting graduate student at UIUC CS. I received my MS in Computer Science from UT Dallas, and was a full-time research assistant at Zhejiang University for 1.5 years before starting grad school.

🎯 I’m always interested in collaborations on AI agent safety with researchers from both industry and academia!

📝 Prospective students / interns: please fill out this form, then optionally follow up via email at [email protected].

Selected Publications [Full List] (* equal contribution; †:work I supervised)

  • LLM Agents Should Employ Security Principles
    Kaiyuan Zhang, Zian Su, Pin-Yu Chen, Elisa Bertino, Xiangyu Zhang, Ninghui Li
    Communications of the ACM (CACM), to appear
    [preprint]
  • PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversations
    Kaiyuan Zhang*, Chuan Wang*, Joey Zhong, Paul Fryzel, Kyle Polley, Jerry Ma, Ninghui Li
    Preprint 2026
    [paper] [benchmark & models]
  • BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views
    Yuetian Chen*, Kaiyuan Zhang*, Yuntao Du, Thomas Li, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
    Preprint 2026
    [paper] [benchmark] [github] [website] [leaderboard]
  • BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
    Kaiyuan Zhang*, Mark Tenenholtz*, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
    Conference on Language Modeling (COLM’26)
    Also invited to present at the 1st Real World AI Security Conference, Stanford University
    [paper] [benchmark] [model]
  • FEROM: Frontier Endogenous Reveal-Order Marginal Policy Optimization for Masked Diffusion LMs
    Zian Su, Ziyang Huang, Kaiyuan Zhang, Xiangyu Zhang
    The 40th Conference on Neural Information Processing Systems (NeurIPS’26)
    [paper] [code]
  • LensCT: Fine-Grained AI-Involved Text Detection via Temporal-Hierarchical Tomograms of LLM Internals
    Hanxi Guo, Siyuan Cheng, Kaiyuan Zhang, Guangyu Shen, Xiangyu Zhang
    The 40th Conference on Neural Information Processing Systems (NeurIPS’26)
    [paper] [code]
  • Security Considerations for Artificial Intelligence Agents
    Ninghui Li, Kaiyuan Zhang, Kyle Polley, Jerry Ma
    Perplexity Response to NIST/CAISI Request for Information
    [paper]
  • SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
    Kaiyuan Zhang, Siyuan Cheng, Hanxi Guo, Yuetian Chen, Zian Su, Shengwei An, Yuntao Du, Charles Fleming, Ashish Kundu, Xiangyu Zhang, Ninghui Li
    The 34th USENIX Security Symposium (Security’25)
    [paper] [code] [website]
  • ÎĽKE: Matryoshka Unstructured Knowledge Editing of Large Language Models
    Zian Su*, Ziyang Huang*, Kaiyuan Zhang†, Xiangyu Zhang
    Conference on Language Modeling (COLM’25)
    [paper] [code]
  • CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
    Kaiyuan Zhang, Siyuan Cheng, Guangyu Shen, Bruno Ribeiro, Shengwei An, Pin-Yu Chen, Xiangyu Zhang, Ninghui Li
    The 32nd Network and Distributed System Security Symposium (NDSS’25)
    [paper] [code] [website]
  • Exploring the Orthogonality and Linearity of Backdoor Attacks
    Kaiyuan Zhang*, Siyuan Cheng*, Guangyu Shen, Guanhong Tao, Shengwei An, Anuran Makur, Shiqing Ma, Xiangyu Zhang
    The 45th IEEE Symposium on Security and Privacy (Oakland’24)
    [paper] [code] [website]
  • FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning
    Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, Xiangyu Zhang
    The Eleventh International Conference on Learning Representations (ICLR’23)
    [paper] [code]
    Best Paper Award 🏆 in ECCV 2022 Workshop on Adversarial Robustness in the Real World

Selected Honors & Awards

Professional Services

  • Workshop & Seminar Organizer
    • ICLR Workshop on Backdoor Attacks and Defenses in Machine Learning, 2023
    • Purdue Machine Learning & Security Seminar, 2021 - 2023
  • Area Chair / Program Committee
    • Area Chair, Association for Computational Linguistics (ACL) Rolling Review, 2026 - 2027
    • ACM Conference on Computer and Communications Security (CCS), 2026
    • USENIX Security Symposium (Security), 2026, 2027
    • ICLR Workshop on Privacy Regulation and Protection in Machine Learning, 2024
    • ICML Workshop on Federated Learning and Analytics in Practice, 2023
    • ICML Workshop on Adversarial Machine Learning Frontiers, 2022
  • Reviewer
    • Conference on Language Modeling (COLM), 2026
    • Association for Computational Linguistics (ACL) Rolling Review, 2026
    • International Conference on Learning Representations (ICLR), 2024, 2025
    • International Conference on Machine Learning (ICML), 2023, 2024, 2025
    • Advances in Neural Information Processing Systems (NeurIPS), 2023, 2024, 2025
    • IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    • IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023
  • Journal Reviewer
    • ACM Transactions on Privacy and Security (TOPS)
    • IEEE Transactions on Information Forensics and Security (TIFS)
    • IEEE Transactions on Dependable and Secure Computing (TDSC)

Teaching

  • Fall 2026, ECE 424 / ECE 539: Introduction to Information and Network Security (cross-listed undergraduate/graduate)

Guest Lectures

Personal

  • I enjoy tennis, especially the moments of hitting the ball with the right amount of spin and force.
  • I have a small kayak and enjoy exploring creeks for peaceful views.
  • I have benefited from excellent advice and instructions over the years. I collect these notes here.