Kaiyuan Zhang How to pronounce? PhD candidate @ Purdue CS Office: 305 N. University Street West Lafayette, IN, 47907-2107 Email: [email protected] Google Scholar | DBLP | X (Twitter) | GitHub | LinkedIn

I am an incoming tenure-track Assistant Professor at Rutgers ECE, starting Fall 2026.

I am a final-year PhD candidate of Computer Science at Purdue University, advised by Prof. Ninghui Li and co-advised by Prof. Xiangyu Zhang. I am supported by the Amazon Fellowship and the Bilsland Dissertation Fellowship.

My research interests focus on security and privacy in AI. Most recently, I study agentic systems and LLMs from an adversary perspective, with the objective to understand and mitigate long-term threats, and build proactive, private, and secure agentic AI systems.

I am currently a part-time student researcher at Perplexity, and I was a research intern at Microsoft Research, Amazon AWS GenAI and NEC Labs America during my PhD. Before joining Purdue, I spent a wonderful year as a visiting graduate student at the University of Illinois at Urbana-Champaign, co-advised by Prof. Tao Xie and Prof. Tianyin Xu. I received an M.S. in Computer Science from the University of Texas at Dallas, advised by Prof. Wei Yang. I was a full-time research assistant advised by Prof. Wei Chen for 1.5 years at Zhejiang University before starting grad school.

Selected Publications [Full List] (* equal contribution; †:work I supervised)

• Security Considerations for Artificial Intelligence Agents (Perplexity Response to NIST/CAISI Request for Information 2025-0035)
  Ninghui Li, Kaiyuan Zhang, Kyle Polley, Jerry Ma
  Preprint 2026
  [paper]
• BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
  Kaiyuan Zhang*, Mark Tenenholtz*, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
  Preprint 2025
  [paper] [benchmark] [model]
  Covered by Perplexity Research, Perplexity AI
  1,000+ Hugging Face downloads in 5 days of release
• LLM Agents Should Employ Security Principles
  Kaiyuan Zhang, Zian Su, Pin-Yu Chen, Elisa Bertino, Xiangyu Zhang, Ninghui Li
  Preprint 2025
  [paper] [openreview]
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
  Kaiyuan Zhang, Siyuan Cheng, Hanxi Guo, Yuetian Chen, Zian Su, Shengwei An, Yuntao Du, Charles Fleming, Ashish Kundu, Xiangyu Zhang, Ninghui Li
  The 34th USENIX Security Symposium (Security’25)
  [paper] [code] [website]
• CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
  Kaiyuan Zhang, Siyuan Cheng, Guangyu Shen, Bruno Ribeiro, Shengwei An, Pin-Yu Chen, Xiangyu Zhang, Ninghui Li
  The 32nd Network and Distributed System Security Symposium (NDSS’25)
  [paper] [code] [website]
• μKE: Matryoshka Unstructured Knowledge Editing of Large Language Models
  Zian Su*, Ziyang Huang*, Kaiyuan Zhang†, Xiangyu Zhang
  Conference on Language Modeling (COLM’25)
  [paper] [code]
• Exploring the Orthogonality and Linearity of Backdoor Attacks
  Kaiyuan Zhang*, Siyuan Cheng*, Guangyu Shen, Guanhong Tao, Shengwei An, Anuran Makur, Shiqing Ma, Xiangyu Zhang
  The 45th IEEE Symposium on Security and Privacy (Oakland’24)
  [paper] [code] [website]
• FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning
  Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, Xiangyu Zhang
  The Eleventh International Conference on Learning Representations (ICLR’23)
  [paper] [code]
  Best Paper Award 🏆 in ECCV 2022 Workshop on Adversarial Robustness in the Real World
  Covered by PurdueCS News

Selected Honors & Awards

• CERIAS Diamond Award, Purdue, 2026
• Winner of Radiance Technologies Innovation Bowl, Radiance, 2026
• Bilsland Dissertation Fellowship, Purdue, 2026
• Apple Scholar Nominee, Apple, 2025
• Network and Distributed System Security (NDSS) Fellowship, NDSS 2025
• Excellent Research Award, Purdue, 2024, 2025
• Amazon Fellowship, Amazon, 2024
• Best Paper Award, ECCV Adversarial Robustness Workshop, 2022
• Multiple Conference Travel Grants (e.g., IEEE S&P, USENIX Security, CCS, ECCV, Purdue), 2020-2025
• National Scholarship (top 0.2% nationwide; highest undergraduate honor), China, 2016

Teaching

Teaching Assistant

• Fall 2024, CS 59200: AI And Security, grad-level (Head TA), Purdue University.
• Fall 2022, CS 37300: Data Mining and Machine Learning, undergrad-level, Purdue University,
  
• Spring 2022, CS 37300: Data Mining and Machine Learning, undergrad-level, Purdue University.
  
• Fall 2021, CS 47300: Web Information Search And Management, undergrad-level, Purdue University.
  
• Summer 2021, CS 18200: Foundations of Computer Science, undergrad-level, Purdue University.
• Spring 2021, CS 37300: Data Mining and Machine Learning, undergrad-level, Purdue University.
  
• Fall 2020, CS 47300: Web Information Search And Management, undergrad-level, Purdue University.
  
• Summer 2018, Visual Analytics, undergrad-level, Zhejiang University.

Guest Lectures

• Spring 2026, CS 59200: Agentic AI Security, grad-level, Purdue University.
  Topic: Security Principles in Agentic AI Systems, invited by Prof. Elisa Bertino.
  
• Fall 2025, CS 6804: AI Security and Privacy, grad-level, Virginia Tech.
  Topic: LLM Agents Security and Privacy, invited by Prof. Shengwei An.
• Fall 2024, CS 59200: AI And Security, grad-level, Purdue University.
  Topic: LLM Memorization, invited by Prof. Xiangyu Zhang.
• Fall 2023, COMPSCI 360: Introduction to Computer and Network Security, undergrad-level, University of Massachusetts, Amherst.
  Topic: Backdoor Attacks and Defenses in Machine Learning, invited by Prof. Shiqing Ma.
  
• Fall 2022, CS 52900: Security Analytics, grad-level, Purdue University.
  Topic: Backdoor Attacks and Defenses in Machine Learning, invited by Prof. Berkay Celik.
  

Professional Services

• Organizer
  • ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning, 2023
  • Purdue Machine Learning & Security Seminar, 2021, 2022, 2023
• Program Committee
  • ACM Conference on Computer and Communications Security (CCS), 2026
  • USENIX Security Symposium, 2026
  • ICLR Workshop on Privacy Regulation and Protection in Machine Learning, 2024
  • ICML Workshop on Federated Learning and Analytics in Practice, 2023
  • ICML Workshop on Adversarial Machine Learning Frontiers, 2022
• Conference Reviewer
  • Conference on Language Modeling (COLM), 2026
  • International Conference on Learning Representations (ICLR), 2024, 2025
  • International Conference on Machine Learning (ICML), 2023, 2024, 2025
  • Advances in Neural Information Processing Systems (NeurIPS), 2023, 2024, 2025
  • IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024
  • IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023
• Journal Reviewer
  • IEEE Transactions on Information Forensics and Security (TIFS), 2023, 2024, 2025
  • IEEE Transactions on Dependable and Secure Computing (TDSC), 2023
  • ACM Transactions on Privacy and Security (TOPS), 2023
• External Reviewer
  • ACM Conference on Computer and Communications Security (CCS), 2021, 2022, 2023, 2024, 2025
  • IEEE Symposium on Security and Privacy (Oakland), 2026
  • USENIX Security Symposium, 2024, 2025
  • IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2025
  • ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2024
  • IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023
• Student Volunteer
  • USENIX 2025, S&P 2024, ICML 2021, ICLR 2021, ECOOP/ISSTA 2021, CCS 2020, SIGMOD 2020, etc.

Personal

• I play basketball weekly and have maintained this routine for over 10 years.
• I own a small kayak and enjoy exploring creeks for peaceful views.
• I enjoy tennis, especially the moments of hitting the ball with the right amount of spin and force.
• I have benefited from excellent advice and instructions over the years. I collect these notes here.