|
Kaiyuan Zhang 
How to pronounce? Assistant Professor @Rutgers AI Researcher @Perplexity Google Scholar | DBLP | X (Twitter) | GitHub | LinkedIn |
I am an Assistant Professor at Rutgers University and a researcher at Perplexity.
My research focuses on AI agent safety, aiming to build secure and reliable agents in real-world settings.
I received my Ph.D. in Computer Science from Purdue University, advised by Ninghui Li and Xiangyu Zhang. During my PhD, I was supported by the Amazon Fellowship and the Bilsland Dissertation Fellowship. My research has also benefited from collaborations and industry experiences with Perplexity, Microsoft Research, Amazon AWS GenAI, NEC Labs America and IBM Research.
🎯 I am actively looking for self-motivated students to join my research group. I’m especially interested in students working on agentic AI, LLM post-training, and AI security & privacy.
đź“ť Prospective students: please fill out this form here, and optionally send a brief email to [email protected].
🧩 Rutgers undergrads and grads: I'd love to have you join our projects and gain hands-on research experience. Please fill out the form above and email me with subject line "[Research Discussion] – Your Name".
Selected Publications
[Full List]
(* equal contribution; †:work I supervised)
- BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
Kaiyuan Zhang*, Mark Tenenholtz*, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
Conference on Language Modeling (COLM’26)
Also invited to present at the 1st Real World AI Security Conference at Stanford University
[paper] [benchmark] [model]
Covered by Perplexity Research, Perplexity AI
1,000+ Hugging Face downloads in 5 days of release - Security Considerations for Artificial Intelligence Agents
Ninghui Li, Kaiyuan Zhang, Kyle Polley, Jerry Ma
Perplexity Response to NIST/CAISI Request for Information 2025-0035
[paper] - LLM Agents Should Employ Security Principles
Kaiyuan Zhang, Zian Su, Pin-Yu Chen, Elisa Bertino, Xiangyu Zhang, Ninghui Li
Preprint 2025
[paper] [openreview] - SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
Kaiyuan Zhang, Siyuan Cheng, Hanxi Guo, Yuetian Chen, Zian Su, Shengwei An, Yuntao Du, Charles Fleming, Ashish Kundu, Xiangyu Zhang, Ninghui Li
The 34th USENIX Security Symposium (Security’25)
[paper] [code] [website] - ÎĽKE: Matryoshka Unstructured Knowledge Editing of Large Language Models
Zian Su*, Ziyang Huang*, Kaiyuan Zhang†, Xiangyu Zhang
Conference on Language Modeling (COLM’25)
[paper] [code] - CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
Kaiyuan Zhang, Siyuan Cheng, Guangyu Shen, Bruno Ribeiro, Shengwei An, Pin-Yu Chen, Xiangyu Zhang, Ninghui Li
The 32nd Network and Distributed System Security Symposium (NDSS’25)
[paper] [code] [website] - Exploring the Orthogonality and Linearity of Backdoor Attacks
Kaiyuan Zhang*, Siyuan Cheng*, Guangyu Shen, Guanhong Tao, Shengwei An, Anuran Makur, Shiqing Ma, Xiangyu Zhang
The 45th IEEE Symposium on Security and Privacy (Oakland’24)
[paper] [code] [website] - FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning
Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, Xiangyu Zhang
The Eleventh International Conference on Learning Representations (ICLR’23)
[paper] [code]
Best Paper Award 🏆 in ECCV 2022 Workshop on Adversarial Robustness in the Real World
Covered by PurdueCS News
Selected Honors & Awards
- CERIAS Diamond Award, Purdue, 2026
- Bilsland Dissertation Fellowship, Purdue, 2025
- Apple Scholar Nominee, Apple, 2025
- Amazon Fellowship, Amazon, 2024
- Best Paper Award, ECCV Adversarial Robustness Workshop, 2022
- Multiple Conference Travel Grants (e.g., IEEE S&P, USENIX Security, CCS, NDSS, ECCV, Purdue), 2020-2025
Professional Services
- Organizer
• ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning
• Purdue Machine Learning & Security Seminar, 2021 - 2023 - Program Committee / Reviewer
• ACM Conference on Computer and Communications Security (CCS)
• USENIX Security Symposium (Security)
• Conference on Language Modeling (COLM)
• International Conference on Learning Representations (ICLR)
• International Conference on Machine Learning (ICML)
• Advances in Neural Information Processing Systems (NeurIPS)
• IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) - Journal Reviewer
• ACM Transactions on Privacy and Security (TOPS)
• IEEE Transactions on Information Forensics and Security (TIFS)
• IEEE Transactions on Dependable and Secure Computing (TDSC)
Teaching
Guest Lectures
- Spring 2026, CS 59200: Agentic AI Security, Purdue. Topic: Security Principles in Agentic AI Systems, invited by Prof. Elisa Bertino.
- Fall 2025, CS 6804: AI Security and Privacy, Virginia Tech. Topic: LLM Agents Security and Privacy, invited by Prof. Shengwei An.
- Fall 2024, CS 59200: AI And Security, Purdue. Topic: LLM Memorization, invited by Prof. Xiangyu Zhang.
- Fall 2023, COMPSCI 360: Intro to Computer and Network Security, UMass Amherst. Topic: Backdoor Attacks and Defenses in Machine Learning, invited by Prof. Shiqing Ma.
- Fall 2022, CS 52900: Security Analytics, Purdue. Topic: Backdoor Attacks and Defenses in Machine Learning, invited by Prof. Berkay Celik.
Personal
- I play basketball weekly and have maintained this routine for over 10 years.
- I own a small kayak and enjoy exploring creeks for peaceful views.
- I enjoy tennis, especially the moments of hitting the ball with the right amount of spin and force.
- I have benefited from excellent advice and instructions over the years. I collect these notes here.