AI Data Security: Hidden Risks

AI data security: What are the risks?

AI doesn't just consume data — it reshapes how it's accessed, shared, and exposed. As organizations race to leverage the power of AI, they're also opening the door to data security risks. To stay ahead of the risk, your data security must evolve in lockstep with AI adoption.

Protecting sensitive data isn't just about encryption or access controls anymore, it's about understanding how AI systems interact with that data, detecting abnormal usage, and enforcing guardrails that prevent exposure before it happens.

In this blog, we'll explore the emerging risks AI introduces and why a data-centric approach is essential to securing the future.

Your IT-approved AI tools aren't risk-free

Here's the thing — even the "official" AI tools can create hidden risks if your data estate isn't in order. Solutions like Microsoft 365 Copilot, ChatGPT Enterprise, and Salesforce Agentforce are built right into your workflows, giving AI systems access to tons of company data to better answer questions and automate processes. That access, however, is a double-edged sword.

Picture a large insurance company rolling out Microsoft 365 Copilot to boost efficiency. The tool can summarize millions of documents in seconds, but it will also surface information from all the data at its disposal.. If permissions are set too broadly, a simple prompt could expose critical information or sensitive data to someone who shouldn't see it. With thousands of prompts entered each week, the chances of accidental data disclosure are high.

This isn't theoretical. According to our research, 99% of organizations have sensitive data exposed to AI tools like Microsoft365 Copilot and ChatGPT Enterprise. The situation with agents, like Salesforce Agentforceis similar. Like many AI tools, agents typically inherit the permissions of the users who run them. If users have excessive access, agents can expose sensitive data.

For example, imagine a service appointment scheduling agent that mistakenly allows users, or even the public, to view, modify, or cancel other people's appointments. This could lead to unauthorized access to confidential information such as pricing information, order details, and Personally Identifiable Information (PII) like payment information, home addresses, and birth dates.

Let's look at some of the risks that must be addressed when introducing AI into an organization:

Ghost users: the hidden access threat

Ghost users — active accounts for former employees, contractors, or inactive third-party integrations — represent another underestimated risk. These forgotten accounts still have permissions and access to critical resources, giving attackers a quiet backdoor.

Picture a large retailer with hundreds of IT system integrations. Over time, employees leave, contractors move on, and projects end. But their accounts, overlooked in the identity management process, stay active. A threat actor who finds one of these accounts can log in undetected, use AI tools to quickly locate valuable data and steal it.

In the organizations we analyzed, there was an average of over 15,000 inactive external identities and more than 31,000 stale permissions. In one typical case, 10 former admins were still enabled in the system. If just one of these identities gets compromised, the domino effect can be catastrophic, especially when AI tools can automate reconnaissance and privilege escalation.

Identity and access management: a growing challenge

Managing identities has gotten exponentially more complicated as companies rely on hundreds of Software-as-a-Service (SaaS) and cloud services, each with their own mix of roles, policies, and entitlements. Non-human identities, such as service accounts for application programming interfaces (APIs) or automated tools, further blur the boundaries of secure access.

A company might use dozens of integrated services for app development. Each system has overlapping permissions, sometimes hundreds or thousands of possible roles. Over time, without rigorous oversight, employees accumulate access rights as they change roles or take on projects, but they rarely (if ever) lose them. The organization loses visibility into who has access to what, and why.

Add AI into the mix and poorly governed access controls can let AI applications "see" or "learn" from anything they have access to, including sensitive engineering plans, source code, or customer data. Once this data is in the AI's training history or knowledge base, it's extremely difficult to remove it or prevent leakage through generated responses or automated exports.

The complexity is staggering — organizations manage an average of 20,000 policies per AWS environment, with several thousand known to be over-permissive.

Weak authentication: the multifactor authentication gap

In 2024, a massive breach involving a major cloud provider resulted in 190 million patient records being exposed after attackers bypassed weak protections. The lack of enforced MFA allowed them to use stolen credentials from one environment to pivot into many others.

Even where MFA is required, attack vectors continue to evolve. For example, using stolen browser cookies, attackers have been able to sidestep MFA mechanisms, impersonating users and accessing AI tools to quickly pinpoint sensitive data like document copilots or analytics platforms. This lets them move laterally across an organization, harvesting or corrupting data at an unprecedented pace.

In the organizations we surveyed, one in seven didn't have MFA enforced across cloud and SaaS systems. The average company had almost 2,000 accounts with passwords that never expire, and even global administrators with non-expiring passwords.

Shadow AI

One of the least visible risks to enterprise data is shadow AI, AI apps your employees are using that haven't been approved by your IT or security teams. People want to get tasks done faster, so they turn to these tools to automate tasks or create content. In the process, they are putting sensitive info at risk.

For example, DeepSeek quickly captured the world's attention with its advanced capabilities and open-source approach. It even surpassed ChatGPT as the most downloaded free app in Apple's App Store with it's launch. Shortly after, a data breach affecting DeepSeek was discovered, leaving over one million sensitive records accessible online without any authentication required. Any organization with employees who had downloaded DeepSeek and used it for work now had their sensitive data at risk.

While DeepSeek is a high-profile example, shadow AI is everywhere.

We looked at 1,000 organizations, and 98% had employees using unauthorized apps, with the average company using 1,200 unofficial apps. Many of these are AI-powered and haven't been checked for security or compliance. Just one breach from shadow AI can expose thousands (or millions) of data records.

Model building and training: emerging risks

As more organizations build and deploy AI models, the integrity of the training data powering those models has become a critical security concern.

Training data is the foundation of any AI system, and if it's compromised, the consequences can be far-reaching. Malicious actors can exploit unsecured or poorly governed datasets to launch model poisoning attacks — injecting deceptive or harmful data into the training pipeline to manipulate outcomes.

At the same time, sensitive or regulated information used in training can inadvertently become embedded in model outputs, leading to data leakage and compliance violations.

This section explores the dual challenge of safeguarding training data from adversarial manipulation and ensuring it doesn't expose the very information it was meant to protect

Training data: the foundation at risk

AI is only as good as the data used to train it. When training data gets exposed, mismanaged, or corrupted, the consequences reach far and wide.

Take a healthcare provider developing a diagnostic AI system. The training dataset includes years of patient records, test results, and billing info. If these data stores aren't properly secured in cloud infrastructure — missing encryption or proper authentication — an attacker could steal sensitive health data. Worse, if the attacker gets write access, they might poison the training set by subtly changing medical records or labels. The resulting model, corrupted at its core, could give dangerously wrong diagnoses or treatment recommendations, potentially harming patients and exposing the organization to lawsuits and regulatory penalties.

And it's not just malicious attacks. Mistakes from well-meaning analysts can have similar bad effects. Using outdated or incomplete data, or failing to properly anonymize sensitive inputs, can result in models that leak confidential business or customer information.

Model poisoning: subtle and devastating

Model poisoning is an advanced risk that's rapidly increasing in prominence. Here, adversaries secretly manipulate AI model training data, parameters, or even the deployed model itself. The goal is to make the model make incorrect decisions or respond in ways that benefit the attacker.

Imagine this scenario in financial services: An internal AI model is trained to authenticate and verify payment transactions based on account details and customer behavior. If an attacker can alter just a small amount of the training data — for example, modifying vendor bank details — the AI might give out or approve fraudulent account information without raising suspicion. When prompted by a legitimate user request ("Give me our vendor's bank details"), the model returns the attacker's injected bank account. Funds get misdirected, with enormous financial and reputation loss.

Accidental poisoning is also possible. Take, for example, a data engineer who unintentionally includes corrupted or mislabeled data in a training set, leading to AI outputs that are biased, inaccurate, or flat-out wrong. These issues are particularly dangerous in fields like healthcare, justice, or critical infrastructure.

Detection of such poisoning can be extremely difficult. A well-crafted attack appear as typical performance anomalies or edge cases, not security incidents.

The path forward: AI security is data security

The reality is you can't separate AI security from data security.

First, AI creates a huge amount data and that data must be secured. Second, the widespread adoption of AI is making data security much harder.

It shines a spotlight on excessive access that might have otherwise gone unnoticed, and with AI chatbots and agents at their fingertips, attackers and rogue insiders are only one prompt away from stealing valuable intellectual property. Data is always the target, and for organizations already struggle to secure critical data, AI has opened the floodgates.

AI security requires an end-to-end data security strategy that secure the entire data estate and scale to meet the growing influx of data.

Frequently Asked Questions

How does data security apply to AI?

Data security is required to protect sensitive information in AI systems from unauthorized access, breaches, and misuse. This encompasses securing both the data used to train AI models and the information these systems process during operation. This includes securing AI tools like copilots and agents, preventing shadow AI (unsanctioned AI applications), securing training data, managing access controls, implementing proper authentication, and protecting against model poisoning where adversaries manipulate AI models to produce incorrect or harmful outputs.

How to secure data from AI?

To secure data from AI:

• Implement strict governance over AI tools, both sanctioned and unsanctioned

• Enforce proper access controls and permissions for AI systems

• Regularly audit and remove inactive user accounts and excessive permissions

• Require strong authentication, especially multi-factor authentication (MFA)

• Continuously monitor AI tool activity and data access

• Implement comprehensive data classification and labeling

• Use encryption and strong key management for sensitive information

• Audit training data for security vulnerabilities and potential poisoning

• Leverage AI itself for defense through automated detection of security issues

• Maintain visibility across all cloud and SaaS environments where AI operates

Is your data safe with AI?

Data safety with AI depends entirely on implementation and governance. When AI systems are properly secured, monitored, and managed, they can be relatively safe. However, significant risks exist in most organizations today, including:

• 98% of organizations have employees using unsanctioned AI applications

• 90% of organizations have sensitive files exposed to all employees through tools like Microsoft 365 Copilot

• Many organizations have an average of 15,000+ inactive external identities with access

• One in seven organizations lacks proper MFA enforcement

• Only 10% of organizations implement rigorous file labeling

• Nine out of ten organizations have exposed sensitive cloud data

Without proper controls, AI systems can inadvertently expose sensitive information, leak confidential data, or be manipulated by attackers.

Which AI is best for cybersecurity?

There isn't a single "best" AI for cybersecurity, as different solutions excel in different areas. Effective cybersecurity typically employs multiple AI technologies:

• Anomaly detection systems that use machine learning to identify unusual patterns

• Behavior analysis tools that recognize suspicious user activities

• Threat intelligence platforms that use AI to predict and identify emerging threats

• Automated response systems that can contain breaches in real-time

• Data classification AI that automatically labels sensitive information

• Access management tools that use AI to identify excessive or risky permissions

• Authentication systems that employ behavioral biometrics

The most effective approach is a layered security strategy that combines multiple AI technologies with human oversight. Organizations should select AI security tools based on their specific needs, infrastructure, and risk profile, rather than seeking a one-size-fits-all solution.