Humans Reviewing ChatGPT Conversations: The Reality of Project Lily

It has come to light that OpenAI has hundreds of contract reviewers read real users’ ChatGPT conversations and use them to improve responses. Reporting by Joseph Cox of 404 Media cites internal documents, actual instructions, and evaluation procedures. In some cases, the material includes entire back-and-forth exchanges between users and the chatbot. With more than 900 million users, the potential impact is seen as broad.

The effort is called “Project Lily.” It is separate from the detection of harmful plans for safety purposes. Its purpose is to improve response quality. Reviewers read actual user prompts and summarize the intent. They then score multiple generated responses and provide critical evaluations. This iterative work forms the basis for tuning.

The Reality and Structure of Project Lily

Project Lily is an evaluation system using external contract workers. Reporting by Joseph Cox of 404 Media confirms instruction sheets, communication records, and evaluation screens. Reviewers read users’ prompts themselves. User names are not displayed. Even so, there are cases where personal details remain in the conversation content. OpenAI says it strives to remove personal information in advance. However, it acknowledges that sensitive descriptions may still reach reviewers.

The review is not limited to single questions. Long conversation histories may be handed over as is. Users employ it for therapy consultations, work assistance, and everyday advice. Many enter details about their lives, health, and work. One reviewer said users do not expect humans to read them. The view is that they do not imagine contract workers analyzing the conversations. This gap in expectations is a privacy challenge.

Contract Reviewers’

Procedures and Evaluation Criteria in Detail

The work consists of three stages. The first stage is reading the actual user prompt. The second stage is summarizing the user’s intent. The third stage is scoring and commenting on a set of generated responses. Reviewers measure accuracy, helpfulness, and naturalness of style. The instructions set out clear criteria.

An excellent response should understand the user’s intent, provide helpful and accurate assistance, and write in a style that is clear, natural and appropriately warm

The above is an excerpt from the instructions. An excellent response requires understanding intent, providing accurate assistance, and writing in a clear, natural style. Curbing excessive sycophancy is also an important evaluation criterion. Training to avoid self-humanization is also included. The excessive sycophancy of 4o was also flagged in lawsuits. Correcting that tendency is reflected in the review items.

Handling of Personal Information and the

Limits of Anonymization

OpenAI attempts to remove personal information before review. The focus is on reducing direct identifiers such as names and contact details. Even so, individuals can sometimes be inferred from context. Details from health consultations, family circumstances, and workplace-specific information tend to remain. Anonymization is not perfect. The company acknowledges that limitations exist.

User names are not shown to reviewers. Still, the content itself remains sensitive. In therapeutic use, symptoms, medications, and family relationships are discussed. In business use, internal documents and customer information are pasted in. In social use, locations, schedules, and relationship status are included. If removal fails, it could lead to effective identification. Both anonymization by design and the operational scope of access are at issue.

The Role of Human Evaluation in Model Improvement

This reporting reveals the reality of the development system. Improvements are not driven only by large-scale data collection or advances in base technology. Human evaluation of real prompts supports tuning. Reviewers repeatedly rank responses and suggest improvements. This cycle contributes to fine adjustments in behavior. It is an often-overlooked, labor-intensive process.

Anthropic also acknowledged using human evaluation. The company uses a similar method to improve its own models. Across the industry, combining real data with human judgment has become standard. Automated evaluation alone cannot measure the appropriateness of intent and context. Human interpretation and scoring are key to quality control. The operational burden behind the development race has been brought into relief.

Practical steps for users are also a realistic issue. The basic rule is not to enter sensitive information. For business use, organizational guidelines and record management are important. Providers need to strengthen explanations of access scope and retention periods. Raising awareness of the opt-out setting for training on history is cited as an option. The design of transparency and consent will determine trust.

Editorial Opinion

On short-term impact. Over the next 3-6 months, companies are expected to expand explanations of human evaluation. Disclosure of review scope and anonymization procedures may advance. For enterprise use, revision of input rules is expected to spread. Whether data is used for training is expected to become a procurement review item.

On the long-term view. In 1-3 years, balancing real-data use with privacy is expected to become a design competition. Adoption of on-device processing and differential privacy may advance. Disclosure of evaluation labor conditions is also expected to become a procurement criterion. Balancing quality and ethics is expected to become a selection standard.

Questions from the editorial team. How much should users enter on the assumption of human review? How should providers draw the line between usefulness and confidentiality? Who should audit the quality of evaluators’ judgments? Ensuring transparency and verifiability is at stake.

References

• “Inside ‘Project Lily’: The Humans Reading Your ChatGPT Chats”, by Joseph Cox — 404 Media, 2026-09-14T14:22:43.000Z (ARR)
• Source URL: https://www.404media.co/inside-project-lily-the-humans-reading-your-chatgpt-chats/