What Really Happens to Information Entered into LLMs
Artificial intelligence, and in particular language models referred to as LLMs (Large Language Models), have become part of the daily work of many people. They are used, among other things, to create text and analyze information. In this article, we will not discuss futuristic visions of AI development or evaluate specific tools. We will focus on one issue: what actually happens to the data that a user enters into language models, especially when it is sensitive information.
What a Language Model Really Is
Language models are statistical systems trained on very large corpora of text. Their task is to generate subsequent fragments of text in such a way that they are as well matched as possible to the provided context. An LLM does not understand text in the human sense, but it can very accurately generate responses tailored to the context.
The model's response is generated as a result of statistical computations - the system selects the most probable next word based on patterns learned during training. It has no consciousness or intent. It does not "know" whether the information is true or confidential - it operates solely on those patterns, without understanding their content.
Where Models Get Their Knowledge
The learning process of language models takes place at the training stage, before they are made available to users. During this time, very large datasets are used, from which the model learns linguistic and contextual relationships. The model does not update its parameters in real time during a single conversation. This means that data entered by the user does not immediately "teach" the model and is not stored as specific records.
At the same time, in the case of some public, especially free services, data entered by users may be used by the provider to further improve the models or train subsequent versions - in accordance with the service terms. Usually this happens in an aggregated and processed form, rather than by literally incorporating individual conversations into the model. This means that a single piece of data - for example, information about a specific company's budget - should not affect future responses from the model. However, there are no guarantees that the entered information will not in some form be disclosed or reproduced in responses generated for other users. The user does not have full control over whether and how their data will be used in the provider's development processes.
Studies published in 2025 showed that some publicly available language models can reproduce entire books from their training data from memory - almost word for word [1]. In the same year, Anthropic (the creator of the Claude chatbot) reached a settlement of $1.5 billion over the use of pirated books to train its models [2]. If a model can encode the full text of a book in its parameters, the data entered by your company's employees into chatbots - offers, contracts, customer data, internal analyses - may in a similar way become embedded in the model and potentially disclosed in responses generated for other users. Therefore, before deploying any AI tool in an organization, it is worth making sure that the chosen service variant guarantees that the data is not used for further model training, and that the tool itself is properly configured from a privacy perspective.
How Data Is Processed During a Conversation
Data entered by the user is processed in order to generate a response. The content of the query becomes part of the context of the ongoing conversation, and the model uses it to calculate the most probable answer. The entire process takes place within the provider's infrastructure. Language models are not designed as secure data repositories. They are tools for text processing, not systems that guarantee controlled, long-term storage of sensitive information from the user's perspective.
The user has limited knowledge of how the provider stores and secures data on its side. In practice, this means that the entered information leaves the organization's environment and goes to external infrastructure.
Hallucinations - What Are They Really
Hallucinations in the context of LLMs are situations in which the model generates content that sounds coherent and credible, even though it does not have sufficient data or context for the answer to be correct. Hallucinations are not a technical error or system failure and do not result from the model's "intent." They are a natural effect of a mechanism that always tries to generate an answer.
In the context of data security, the problem is that the model may generate content that goes beyond the user's original intent. In certain situations, this may increase the risk of disclosing information in a broader context than the person asking the question intended.
Can AI Disclose User Data
The model does not have access to external databases of a specific user, nor does it "remember" their data in the traditional sense. However, two levels of risk can be distinguished here: generative risk and infrastructure risk. Generative risk refers to a situation in which the model generates content that does not match the user's expectations. Infrastructure risk refers to the fact that data entered into the system is processed by the provider's infrastructure.
Any IT infrastructure can become the subject of a security incident: misconfiguration, unauthorized access, a technical vulnerability, or a data breach. Public language models are no exception. In practice, this means that sensitive data entered into a public LLM may be logged, stored in accordance with the provider's policy, and in extreme cases may become part of a security incident on the provider's side. This risk is not specific to AI only - it applies to any cloud service. In the case of language models, however, users often forget that they are using external infrastructure.
An additional problem is the way AI tools enter organizations. This often happens without the IT department's knowledge - employees create accounts on services such as ChatGPT or Gemini on their own, often logging in through the company account Microsoft 365 or Google Workspace. In this way, the external tool becomes linked to the corporate identity, without control over which data flows through it. The lack of two-factor verification (MFA) on such an account further increases the risk - if the account is compromised, the attacker gains access to the entire history of conversations with the chatbot, including all data that the employee previously entered.
A Practical Example: When Intuition Fails, Not Competence
In industry media and official communications, there have been cases described in which people holding senior positions in the public administration of the United States used publicly available language models to process content containing sensitive information. In 2025, the acting head of CISA (the U.S. cybersecurity agency) uploaded contract documents marked as "for official use only" to the public version of ChatGPT, which triggered automatic security alerts on the federal network [3]. The reason was the mistaken assumption that the tool works like a local text editor. As a result, AI-based systems were fed data that, according to applicable security rules, should not have left the controlled environment. These situations led to internal analyses and the issuance of restrictions on the use of public language models in government institutions.
This example shows that the source of the risk is the false sense of control created by an intuitive, conversational interface.
What This Means in Practice
Language models are a useful tool, but they require conscious use. In practice, this means that sensitive and confidential data should not be entered into public models. The lack of "model memory" does not mean that the data is not being processed in the provider's infrastructure. AI is an external cloud service, not a local tool. AI itself is not a threat. The risk arises when we expect language models to provide a level of control over data that they do not technically offer.
----
Sources:
[1] Cooper, A. F. et al., Extracting memorized pieces of (copyrighted) books from open-weight language models, arXiv, 2025 - https://arxiv.org/abs/2505.12546
[2] ITHardware.pl, The largest settlement in AI history. $1.5 billion for pirated books, September 2025 - https://ithardware.pl/aktualnosci/miliardowa_ugoda_ai-44876.html
[3] Politico / CSO Online, CISA chief uploaded sensitive government files to public ChatGPT, January 2026 - https://www.csoonline.com/article/4124320/cisa-chief-uploaded-sensitive-government-files-to-public-chatgpt.html