What Really Happens to the Information Entered into LLMs
Artificial intelligence, and in particular language models referred to as LLM (Large Language Models), have become an element of everyday work for many people. They are used, among other things, to create texts and analyze information. In this article, we will not consider futuristic visions of AI development nor evaluate specific tools. We will focus on one issue: what actually happens to the data that users input into language models, especially when it comes to sensitive information.
What Is a Language Model Really
Language models are statistical systems trained on very large text datasets. Their task is to generate successive fragments of text in such a way that they are as well suited to the provided context as possible. LLMs do not understand text in a human sense, but can generate contextually appropriate responses quite accurately.
The model's response is generated as a result of statistical calculations— the system selects the most likely next word based on patterns learned during training. It has no awareness or intentions. It does not "know" whether the information is true or confidential—it operates solely on those patterns, without understanding their content.
Where Models Get Their Knowledge
The learning process of language models occurs during the training phase, before they are made available to users. During this time, very large datasets are used, based on which the model learns linguistic and contextual dependencies. The model does not update its parameters in real time during a single conversation. This means that the data entered by the user does not "teach" the model immediately nor are they remembered as specific records.
At the same time, in the case of some public, especially free services, the data entered by users may be used by the provider to further improve the models or train their subsequent versions—in accordance with the terms of service. This usually happens in an aggregated and processed manner, rather than by literally incorporating individual conversations into the model. This means that a single piece of data—for example, information about the budget of a specific company—should not affect the model’s future responses. However, there are no guarantees that the information entered will not be revealed or reproduced in responses generated for other users. The user does not have full control over whether and how their data will be used in developmental processes on the provider's side.
Research published in 2025 showed that some publicly available language models can reproduce entire books from their training data—almost word for word [1]. In the same year, Anthropic (the creator of the Claude chatbot) settled for $1.5 billion related to the use of pirated books to train its models [2]. If the model can encode the full content of a book in its parameters, the data entered by your company's employees into chatbots—offers, contracts, client data, internal analyses—can similarly be retained in the model and potentially disclosed in responses generated for other users. Therefore, before implementing any AI tool in an organization, it is worth ensuring that the selected service variant guarantees that the data will not be used for further training of the model, and that the tool itself is properly configured with respect to privacy.
How Data Is Processed During a Conversation
Data entered by the user is processed in order to generate a response. The content of the query becomes part of the context of the current conversation, and the model uses it to compute the most probable response. The entire process takes place within the service provider’s infrastructure. Language models are not designed as secure repositories for data. They are tools for processing text, not systems guaranteeing controlled, long-term storage of sensitive information from the user's perspective.
The user has limited knowledge about how the provider stores and secures data on their side. In practice, this means that the information entered leaves the organization's environment and is transferred to external infrastructure.
Hallucinations - What Are They Really
Hallucinations in the context of LLMs refer to situations where the model generates content that sounds coherent and credible despite lacking sufficient data or context for the response to be correct. Hallucinations are not a technical error or system failure and do not result from the model's "intentions." They are a natural effect of a mechanism that always tries to generate a response.
In the context of data security, the problem is that the model can generate content that extends beyond the user's original intent. In certain situations, this can increase the risk of disclosing information in a broader context than the person asking the question intended.
Can AI Disclose User Data
The model does not have access to external databases of a specific user nor does it "remember" their data in the traditional sense. However, two levels of risk can be distinguished here: generative risk and infrastructural risk. Generative risk pertains to the situations where the model generates content that is inconsistent with the user's expectations. Infrastructural risk relates to the fact that the data entered into the system is processed by the provider's infrastructure.
Any IT infrastructure can become the subject of a security incident: configuration errors, unauthorized access, technical vulnerabilities, or data breaches. Public language models are no exception. In practice, this means that sensitive data entered into a public LLM may be logged, stored according to the provider's policy, and in extreme cases may become part of a security incident on the provider's side. This risk is not specific solely to AI—it concerns every cloud service. However, when it comes to language models, users often forget that they are using external infrastructure.
An additional problem is how AI tools enter organizations. This often happens without the knowledge of the IT department—employees independently create accounts on services like ChatGPT or Gemini, often logging in through their corporate Microsoft 365 or Google Workspace accounts. This way, the external tool is linked to the corporate identity, without control over what data flows through it. The lack of two-step verification (MFA) on such an account further increases the risk—if the account is compromised, the attacker gains access to the entire history of conversations with the chatbot, including all data the employee previously entered.
A Practical Example: When Intuition Fails, Not Competence
In industry media and official communications, there have been cases where individuals in high positions in the United States public administration have used publicly available language models to process content containing sensitive information. In 2025, the acting head of CISA (the U.S. Cybersecurity Agency) uploaded contractual documents marked "for official use only" to the public version of ChatGPT, triggering automatic security alerts across the federal network [3]. The reason was the erroneous assumption that the tool operates similarly to a local text editor. As a result, data that, according to existing security rules, should not leave a controlled environment, made its way into AI-based systems. These situations led to internal analyses and the issuing of restrictions on the use of public language models in government institutions.
This example shows that the source of risk is the false sense of control provided by an intuitive, conversational interface.
What Does This Mean in Practice
Language models are useful tools, but they require conscious use. In practice, this means that sensitive and confidential data should not enter public models. The lack of "model memory" does not mean that data are not processed within the provider's infrastructure. AI is an external cloud service, not a local tool. AI itself is not a threat. The risk arises when we expect a level of control over data from language models that they do not technically provide.
————
Sources:
[1] Cooper, A. F. et al., Extracting memorized pieces of (copyrighted) books from open-weight language models, arXiv, 2025 - https://arxiv.org/abs/2505.12546
[2] ITHardware.pl, The Largest Settlement in AI History. $1.5 billion for Illegal Books, September 2025 - https://ithardware.pl/aktualnosci/miliardowa_ugoda_ai-44876.html
[3] Politico / CSO Online, CISA chief uploaded sensitive government files to public ChatGPT, January 2026 - https://www.csoonline.com/article/4124320/cisa-chief-uploaded-sensitive-government-files-to-public-chatgpt.html