Generative AI and Data Privacy Risks
Xcina Blog

Generative AI and Data Privacy Risks

In this issue of In Perspective, Jackie Barlow, Data Protection Senior Consultant at Xcina Consulting, discusses the questions every business should be asking. This month we review generative AI and how to ensure personal data security as well as delving into why the ICO is urging organisations to use privacy enhancing technologies (PETs) to share personal data securely or anonymously. Finally, we review consumer duty protection and how this will impact businesses with regard to GDPR and the right to privacy in electronic communications.

We take a look at why this is important and the implications for both businesses and individuals.

Find out more below.

Generative AI and data privacy risks

What happened

  • Generative AI is a type of AI technology that can generate new content; for example text, images or audio, based on patterns and examples from existing data
  • It can create realistic and convincing human like content and that is why it has become very popular in a short time
  • There is currently no AI law in the UK and privacy concerns need to be addressed
  • In its Guidance on AI and Data Protection* issued in April 2023, the ICO sets out 8 questions for businesses using or developing generative AI involving personal data.

(i)What is your lawful basis for processing personal data? 

(ii)Are you a controller, joint controller or a processor?

(iii)Have you prepared a Data Protection Impact Assessment (DPIA)?

(iv)How will you ensure transparency?

(v)How will you mitigate security risks?

(vi)How will you limit unnecessary processing?

(vii)How will you comply with individual rights requests?

(viii)Will you use generative AI to make solely automated decisions? 


  • All of these questions need to be considered and a data protection impact assessment (DPIA) needs to be completed to address and mitigate privacy risks, before the processing takes place

Why it matters

  • AI presents a wealth of opportunities to businesses, but privacy risks need to be taken into account
  • Transparency is important because data subjects must understand how their personal data will be processed and how they can exercise their rights
  • The nature of AI systems means that they are difficult to understand so transparency is not easy if information is too technical
  • Organisations must provide details about how their generative AI systems work as well as the origin of the raw data.
  • Going forward, it is expected that the UK AI framework requirements will expand beyond the existing (UK) GDPR obligations for transparency, fairness and explainability
  • The ICO has urged businesses to consider the privacy risks that generative AI can bring before rushing to adopt the technology


The ICO has produced the following guidance

*Guidance on AI and data protection | ICO



ICO urges organisations to use privacy enhancing technologies (PETs) to ensure personal data can be shared securely or anonymously

What happened

  • The ICO is recommending that organisations start using PETs so individuals’ personal data can be shared securely or anonymously
  • PETs embrace fundamental data protection principles by minimising personal data use, maximising data security and empowering individuals
  • PETs assure individuals who use technology (i)that their own personal data will be kept confidential and (ii)that managing data protection is a priority for the organisations who hold responsibility for their personal data
  • PETs minimise the amount of personal data collected and used by third parties and they also pseudonymise or anonymise data to protect it
  • The anonymised personal data can be used to detect and prevent financial crimes and other harms such as cybercrimes, money laundering and fraud
  • The recent developments and risks emerging around generative artificial intelligence (AI), need assurance that personal data is being used lawfully. PETs will be able to solve this problem.
  • The more protected data is, then the more widely AI can be used
  • If an organisation uses large volumes of data, and special category data in particular, it should consider using PETs.

Why it matters

  • PETs are linked to the concept of ‘data protection by design’ so they help to comply with the ‘data minimisation’ principle and the ‘security’ principle
  • PETs implement robust anonymisation or pseudonymisation solutions and minimise the risk of personal data breaches, because they make personal data unintelligible to anyone that is not authorised to access it
  • PETs can provide new opportunities for organisations to use personal data via innovative and trustworthy applications, because they allow the sharing, linking and analysis of individuals’ personal data without anyone needing to actually access it
  • However, when embarking on new projects involving personal data, it is still important to identify (and mitigate) the data protection risks on a case by case basis by carrying out a data protection impact assessment (DPIA).



The ICO has launched new guidance on PETs which is in two parts. The first is aimed at data protection officers and the second for a more technical audience.  Privacy-enhancing technologies (PETs) | ICO

Consumer Duty and GDPR and PECR obligations

What happened

  • The deadline for the implementation of the UK Financial Conduct Authority’s (FCA’s) new Consumer Duty is 31 July 2023
  • The duty aims to increase consumer protection in retail financial services and the collection of data is central to the monitoring and improvement of customer outcomes
  • To meet Consumer Duty requirements, firms will need to process more customer personal data than they currently do but also, they might use customer data already held, for different purposes
  • Personal data is governed by the data protection laws so organisations need to be sure they are compliant
  • For example, the FCA might ask (i) What data does the firm hold on its customers? or (ii) Are there any gaps in the data held and what steps is the firm taking to address them?
  • Additionally, the Consumer Duty might mean that firms need to send more communications to customers in making sure they provide information about products and services
  • If any communications contain direct marketing material, organisations must give customers the right to object to receiving such marketing. This means that The Privacy and Electronic Communications Regulations (PECR) apply
  • If firms use cookies or other tracking technologies to test user engagement during a customer journey, opt-in consent will be needed to use these technologies, so the ‘cookie’ rules under PECR apply
  • As the Consumer Duty is aimed at specifically improving outcomes for vulnerable customers, it is likely that special category personal data (more sensitive personal data) will need to be processed; which requires an extra level of care under data protection laws.

Why it matters

  • The new Consumer Duty is coming into force at the end of July 2023
  • With more personal data being processed or used for different purposes, it is crucial that firms adhere to data protection laws
  • With the needs of vulnerable clients being addressed, special category data in particular, will need to be processed and under data protection laws, this means a greater level of care is required
  • With more communications needing to be sent to customers, and cookies being used to track or test user engagement, firms also need to adhere to The Privacy and Electronic Communications Regulations



For more information on PECR, please see the ICO’s March 2023 guidance at guidance on direct marketing and regulatory communications 


The ICO has also provided information about special category data at Special category data | ICO

We’d love to hear from you

Jackie has over 14 years’ experience in providing advice and training on data protection, records management and electronic marketing, which she has gained from working in a number of different types of organisations. Prior to joining Xcina, she managed the data protection functions at an investment management firm, pensions provider and within the not-for-profit sector including a university and charity. She is experienced in identifying and overcoming complex information governance and data protection challenges.

To discuss how the areas highlighted in this post, or any other aspect of risk management, information governance or compliance impact your business, speak with our team, tell us what matters to you and find out how we can help you navigate complex issues to help you deliver long term value.

If you have any questions or comments, or if there’s anything you would like to see covered, please get in touch by emailing Xcina Consulting at We’d love to hear from you.

Jackie Barlow

Data Protection Senior Consultant and Group Privacy Officer

Speak to me directly by Email, or
Telephone: +44 (0)20 3745 7843

Subscribe to Updates

Receive regular updates from our expert consultants as they provide clarification and guidance on issues impacting your organisation.

Subscribe >>