Kaiyuan Zhang Kaiyuan Zhang
How to pronounce?

PhD @ Purdue CS
Office: 305 N. University Street
West Lafayette, IN, 47907-2107
Email: [email protected]
Google Scholar | DBLP | X (Twitter) | GitHub | LinkedIn

I am an incoming tenure-track Assistant Professor in the Department of Electrical and Computer Engineering at Rutgers University. I am also a researcher at Perplexity, focusing on AI agent safety [1], [2].

My research focuses on the security and privacy foundations for real-world agentic AI systems, aiming to make LLM agents secure, private, and trustworthy in practice.

I earned my PhD in Computer Science from Purdue University, advised by Prof. Ninghui Li and Prof. Xiangyu Zhang. During my PhD, I was supported by the Amazon Fellowship and the Bilsland Dissertation Fellowship, and interned at Perplexity, Microsoft Research, Amazon AWS GenAI, and NEC Labs America.

Before Purdue, I spent a wonderful year as a visiting graduate student at UIUC CS, advised by Prof. Tao Xie and Prof. Tianyin Xu. I received my MS in Computer Science from UT Dallas, advised by Prof. Wei Yang. I was a full-time research assistant at Zhejiang University, advised by Prof. Wei Chen for 1.5 years before starting grad school.

🎯 I am actively looking for self-motivated students to join my research group. Experience in agentic AI, LLM post-training, and AI security & privacy is strongly preferred.

đź“ť Prospective Ph.D. students, interns, and visiting students: please fill out this form here, and optionally send a brief follow-up email to [email protected] .

🧩 Rutgers undergrads and grads: I'd love to have you join our projects and gain hands-on research experience. Please fill out the form above and email me with subject line "[Research Discussion] – Your Name".

Selected Publications [Full List] (* equal contribution; †:work I supervised)

  • BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
    Kaiyuan Zhang*, Mark Tenenholtz*, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
    Conference on Language Modeling (COLM’26)
    Also invited to present at the 1st Real World AI Security Conference at Stanford University
    [paper] [benchmark] [model]
    Covered by Perplexity Research, Perplexity AI
    1,000+ Hugging Face downloads in 5 days of release
  • Security Considerations for Artificial Intelligence Agents
    Ninghui Li, Kaiyuan Zhang, Kyle Polley, Jerry Ma
    Perplexity Response to NIST/CAISI Request for Information 2025-0035
    [paper]
  • LLM Agents Should Employ Security Principles
    Kaiyuan Zhang, Zian Su, Pin-Yu Chen, Elisa Bertino, Xiangyu Zhang, Ninghui Li
    Preprint 2025
    [paper] [openreview]
  • SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
    Kaiyuan Zhang, Siyuan Cheng, Hanxi Guo, Yuetian Chen, Zian Su, Shengwei An, Yuntao Du, Charles Fleming, Ashish Kundu, Xiangyu Zhang, Ninghui Li
    The 34th USENIX Security Symposium (Security’25)
    [paper] [code] [website]
  • ÎĽKE: Matryoshka Unstructured Knowledge Editing of Large Language Models
    Zian Su*, Ziyang Huang*, Kaiyuan Zhang†, Xiangyu Zhang
    Conference on Language Modeling (COLM’25)
    [paper] [code]
  • CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
    Kaiyuan Zhang, Siyuan Cheng, Guangyu Shen, Bruno Ribeiro, Shengwei An, Pin-Yu Chen, Xiangyu Zhang, Ninghui Li
    The 32nd Network and Distributed System Security Symposium (NDSS’25)
    [paper] [code] [website]
  • Exploring the Orthogonality and Linearity of Backdoor Attacks
    Kaiyuan Zhang*, Siyuan Cheng*, Guangyu Shen, Guanhong Tao, Shengwei An, Anuran Makur, Shiqing Ma, Xiangyu Zhang
    The 45th IEEE Symposium on Security and Privacy (Oakland’24)
    [paper] [code] [website]
  • FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning
    Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, Xiangyu Zhang
    The Eleventh International Conference on Learning Representations (ICLR’23)
    [paper] [code]
    Best Paper Award 🏆 in ECCV 2022 Workshop on Adversarial Robustness in the Real World
    Covered by PurdueCS News

Selected Honors & Awards

Professional Services

  • Organizer
    • ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning
    • Purdue Machine Learning & Security Seminar, 2021 - 2023
  • Program Committee / Reviewer
    • ACM Conference on Computer and Communications Security (CCS)
    • USENIX Security Symposium
    • Conference on Language Modeling (COLM)
    • International Conference on Learning Representations (ICLR)
    • International Conference on Machine Learning (ICML)
    • Advances in Neural Information Processing Systems (NeurIPS)
    • IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
    • IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • Journal Reviewer
    • ACM Transactions on Privacy and Security (TOPS)
    • IEEE Transactions on Information Forensics and Security (TIFS)
    • IEEE Transactions on Dependable and Secure Computing (TDSC)

Teaching

Guest Lectures

Teaching Assistant

Personal

  • I play basketball weekly and have maintained this routine for over 10 years.
  • I own a small kayak and enjoy exploring creeks for peaceful views.
  • I enjoy tennis, especially the moments of hitting the ball with the right amount of spin and force.
  • I have benefited from excellent advice and instructions over the years. I collect these notes here.