Defending Chatbots from Black-Box Persona Inference Attacks


Language model based chatbots tend to reveal personas of speakers and propose effective defense objectives to prevent GPT-2 from black-box persona inference attacks.

  • Defending Chatbots from Black-Box Persona Inference Attacks
Commercialisation opportunities
Technology licensing agreement / Industry collaboration
Problem addressed

Social chatbots, also known as chit-chat chatbots, evolve rapidly with large pretrained language models. Despite the huge progress, privacy concerns have arisen recently: training data of large language models can be extracted via model inversion attacks. The invention proposed strategies can greatly reduce persona inference accuracy from 37.6% to 0.5% while maintaining language models’ powerful generation ability.

  • Proposed persona inference attacks to extract private attributes from speakers'’ hidden states of chatbots.
  • Combine two losses (KL and MI loss) to protect chatbots from revealing speakers’ private personas.
Key impact
  • Identify the threats of revealing personal information from chatbots.
  • Propose effective defenses on the attribute inference attacks of chatbots.
  • Achieve privacy-protection while doing no harm to the utility of chatbots.
  • Chatbots on edge devices (smart phones)
  • Personal assistants
  • Customer service chatbots
The Hong Kong University of Science and Technology (HKUST)

The Hong Kong University of Science and Technology (HKUST) ( is a world-class research intensive university that focuses on science, technology and business as well as humanities and social science. HKUST offers an international campus, and a holistic and interdisciplinary pedagogy to nurture well-rounded graduates with global vision, a strong entrepreneurial spirit and innovative thinking. Over 80% of our research work were rated “Internationally excellent” or “world leading” in the Research Assessment Exercise 2020 of Hong Kong’s University Grants Committee. We were ranked 3rd in Times Higher Education’s Young University Rankings 2022, and our graduates were ranked 23rd worldwide and among the best from universities from Asia in Global University Employability Survey 2021.