Amazon Mechanical Turk new subscriptions suspended Commentary: Why AI learning data should be designed with source, work log, and contamination verification before human labels

Publication date: 2026-07-06 | Category: AI News

Amazon Mechanical Turk New Subscription Suspension Commentary: Why AI training data should be designed with provenance, work log, and contamination verification before human labeling

1) One-line problem definition

Key one-line summary: Now, the important question in AI data is not “Was it labeled by a human?” but “Was it really created by a human, under what conditions, and through what verification?”

AI Times reported on July 6, 2026 that Amazon Mechanical Turk will stop signing up new customers starting July 30, 2026. Since 2005, MTurk has been a leading crowdsourcing platform that assigns small tasks that are difficult for computers to handle, such as image classification, data verification, sentiment analysis, surveys, and content review, to workers around the world.

The target readers of this article are developers, PMs, and data leads who outsource LLM learning and evaluation data, operate internal evaluation sets, or design AI services based on human inspection. The scope is not the rise and fall of MTurk, but How to redesign the human data supply chain in the AI era. Conversely, it does not address specific labeling company rankings or investment decisions.

The core problem is simple. After generative AI became widespread, it became difficult to tell whether workers were answering like humans or submitting answers quickly generated by LLM. So “human-in-the-loop” alone can no longer prove data quality.

2) Conclusion first

Key one-line summary: Open market labeling is cheap and fast, but closed/hybrid structures with provenance tracking and proof of action are becoming more suitable for high-trust AI data.

This is my judgment. The suspension of new MTurk sign-ups doesn’t mean “people labeling is over.” Quite the opposite. Human judgment has become more important, but throwing tasks into an anonymous large-scale marketplace and only receiving CSV results is no longer suitable for high-reliability AI development.

The teams that need to be introduced now are teams where one wrong label can lead to product risk, such as medical, financial, legal, security, recruitment, and education. These teams should not only look at label values, but also log worker qualifications, instruction versions, work times, duplicate decisions, inspector disagreements, declarations of whether to use LLM, and sample re-verification.

Conversely, tasks that can tolerate some noise, such as initial idea validation, tagging non-sensitive images, or classifying low-risk content, can still use low-cost crowdsourcing or automatic labeling. However, even in that case, “Gold Set” and “Contamination Detection Sample” must be kept separately.

3) Core structure decomposition

Key one-line summary: The human data supply chain should be viewed in six layers: work request, worker, work environment, inspection, evidence log, and relearning reflection.

3-1. Work request hierarchy: This is where you pin what to label

Labeling quality comes first from the instructions, not from the operator. For example, the instruction to “determine whether a response is harmful” and the instruction “label promotions of self-harm, medical advice, and hate speech” create completely different data. If the version of a directive changes, the same data may be labeled differently, so directives must be versioned like code.

3-2. Worker layer: where you decide whether you are an anonymous majority or a verified expert

MTurk-style open markets have the advantage of being able to quickly share work with many people. However, it is difficult to check who did the work, what tools were used, and whether the same person uses multiple accounts. On the other hand, in-house workers or contract labelers are more expensive but allow for training, NDA, domain testing, and iterative quality assessment.

3-3. Work Environment Hierarchy: Where LLM use is blocked or logged

A 2023 arXiv paper estimated that 33% to 46% of workers in MTurk's abstract abstract assignment were likely to have used LLM. These numbers may not be generalizable to all tasks, but they are a strong signal that the line between “human response” and “AI-assisted response” is already blurring in text-generating tasks.

3-4. Inspection hierarchy: A place where the structure of differing opinions is seen rather than majority rule

Quality is not guaranteed even if 2 out of 3 labelers choose the same answer. If the directives are ambiguous, they may all be wrong in the same direction. Therefore, the review is not a simple majority vote, but must also look at gold set insertion, expert re-judgment, labeler bias, and difficult sample queues.

3-5. Evidence Log Hierarchy: Where to check for reproducibility later

Data from good sources does not leave only labels. The original data hash, worker group, work start and end time, instruction version, inspection result, reason for exclusion, and re-labeling history are also recorded. Without this log, you won't be able to trace back to what data was problematic when your model goes awry.

3-6. Relearning reflection layer: The final gateway for contaminated data to enter the model

The risk varies depending on whether the data goes into a training set, evaluation set, prompt example, or RAG document. In particular, the evaluation set is the standard for judging model performance, so if LLM products are mixed, the score itself may be contaminated.

4) Explanation of design intent

Key one-line summary: If the previous goal was “to gather many people quickly,” the current goal is “to leave the basis for human judgment in an auditable form.”

MTurk's original design intent was clear. The idea is to break large tasks into small Human Intelligence Tasks, or HITs, and have workers around the world process them in parallel. AWS documentation explains that MTurk has access to more than 500,000 workers in 190 countries and can perform tasks such as data classification, verification, and tagging.

This structure was well suited to AI data construction in the 2010s. What was needed was a large number of image boxes, sentiment labels, and sentence classification, and the smaller the unit of work, the easier it was to distribute to many people. However, the problem in 2026 is not “there is a lack of workers,” but “it is difficult to prove that the values submitted by workers are really human judgments.”

AWS also announced in the SageMaker Ground Truth document that it will close access to new customers from July 30, 2026, and existing customers can continue to use it, but no new features will be added. Ground Truth is a labeling service that can be used by MTurk, vendor companies, and internal private workforces. This notice means that not only the open market but also AWS's managed labeling path has entered a maintenance phase rather than a new expansion phase.

What you get is clear. Teams can increase model quality and auditability with more stringent data provenance controls. There is also giving up. Labeling costs are rising, job design is slowing down, and data teams must become quality operators rather than mere collectors.

5) Evidence and comparison

Key one-line summary: Labeling methods should be compared based on reliability, traceability, and likelihood of LLM contamination, not price tags.

Approach	Advantages	Main Risk	Recommendation status
Open Crowdsourcing	Fast, cheap, and strong for mass work	Difficult to verify worker identity, use of LLM, bots, fraudulent accounts	Non-sensitive tagging, initial data exploration, noise tolerance operations
Managed Vendor Labeling	Training, SLA, quality control, and domain manpower assignment are possible	Costs are high and vendor internal processes can become a black box	Training set for commercial products, classification model with customer influence
Internal Private Workforce	Understanding security and business context is best	Slow speed and high operating burden	Medical, financial, legal, security, company sensitive data
LLM Auxiliary Labeling	Significantly reduces draft creation and repetitive work	Model bias may be mixed with labels and cause evaluation set contamination	Auxiliary tool with human inspection log, primary classification of low risk
Synthetic data	You can create rare cases and reduce personal information exposure	It may differ from the actual distribution and the same error may be repeated between models	Simulation, edge case reinforcement, section lacking actual data

According to an AI Times report, MTurk will stop accepting new customers from July 30, 2026. Existing customers will continue to use it, but no new features will be added. This is not an immediate termination, but a decision to stop new expansion.

The

arXiv paper Artificial Artificial Intelligence, submitted on June 13, 2023, estimated that 33-46% of workers in the MTurk abstract summary assignment were likely to have used LLM. Although the authors also noted that generalization has its limits, their conclusion has become more important now: “New methods are needed to ensure that human data remains human.”

AWS SageMaker Ground Truth documentation states that it will close to new customer access starting July 30, 2026. At the same time, he explains, Ground Truth was able to use MTurk, select vendor companies, and an internal private workforce together. In other words, the practical alternative is not to find a single platform, but to separate the workforce according to risk.

6) Actual operation flow / step-by-step execution method

Key one-line summary: Data labeling projects must design “how to detect and block contamination” before ordering work.

Separate the data usage first.
Training set, evaluation set, verification set, and demo set should not be created with the same standards. The evaluation set should be the most rigorous and clearly document whether LLM auxiliary labeling is used.
Assign a risk rating.
Divide into low, medium, and high based on privacy, legal liability, customer impact, and whether or not the model runs automatically. It is safer not to entrust high data to open crowdsourcing.
Versions the directive.
Documents label definitions, exceptions, counterexamples, prohibition tools, LLM availability, and handling of ambiguous samples. Directive changes are recorded like data schema changes.
Mix gold set and trap samples.
Insert samples with known correct answers, samples that LLM often gets wrong, and samples that get wrong if the instructions are not read. Track pass rate by labeler.
Stores job logs.
Saves worker group, work time, input data hash, directive version, label value, reviewer, and revision history. Post-auditing is difficult if you only receive CSV results.
Leave LLM usage declaration and detection together.
You can prohibit it unconditionally, or you can allow draft creation but have a human make the final decision. The important thing is to set a policy and leave it in the log.
Link label quality to model performance.
View error rates, sample difficulty, and model failure types by labeler together. If your model only falters on certain labeler data, it may be a data supply chain issue.

labeling_job:
  dataset: support_risk_eval_v3
  purpose: evaluation_only
  instruction_version: risk-label-guide@2026-07-06
  workforce: private_contractor_pool_a
  llm_policy: prohibited_for_final_label
  quality_gates:
    gold_set_pass_rate: ">= 95%"
    min_time_per_item: ">= 18s"
    duplicate_agreement: ">= 0.82"
    expert_review_for: [self_harm, finance_advice, legal_claim]
  provenance:
    store_input_hash: true
    store_worker_group: true
    store_review_history: true
    store_exclusion_reason: true

7) Mistakes/Pitfalls

Key one-line summary: Once data contamination is mixed, it later appears as a model error, so prevention at the time of collection is the cheapest.

Pitfall 1: Recording only “person labels”
Prevention: Store operator type, directive version, LLM usage policy, and inspection method together
Recovery: Source from existing data First exclude unclear samples from the evaluation set, and relabel important samples first.
Pitfall 2: Mixing LLM auxiliary labels in the evaluation set
Prevention: The evaluation set is managed more strictly than the training set, and whether LLM auxiliary is a separate field.
Recovery: The model is Backtrack samples that match too well and check for creation traces and overlapping patterns.
Pitfall 3: Mistaking majority vote for quality
Prevention: In addition to majority vote, look at expert judgment, gold set pass rate, and difficult sample disagreement rate.
Recovery: Redefine labels with large disagreements. Write and rework only the relevant section.
Plot 4: Believing that using a vendor automatically makes you safe
Prevention: Require vendors to train workers, use LLM policies, sample inspection logs, and exclude criteria.
Recovery: Delivery data Part of the contract is re-inspected using the internal gold set and the contract quality standards are renegotiated.
Pitfall 5: Automating all tasks to save costs
Prevention: Automate low-risk, leave human inspection and expert judgment for high-risk.
Recovery: Start with labels with high customer impact. Break automation path, add approval queue

8) Strengths and limitations

Key line summary: Source Verifiable labeling is slow, but significantly reduces the cost of later failure investigation as AI products grow.

Strengths

You can trace model errors back to the data. You can see which labeler, which directive, and which inspection criteria caused the problem.
Strong in regulations and customer explanations. In industries that need to explain “what data led to decisions,” audit logs are part of product trust.
You can maintain the independence of the evaluation set. By reducing the mixing of LLM products into the evaluation criteria, you can more accurately see whether the model has improved.
Unify standards even if you mix outsourcing and internal work. Compare vendors, internal teams, and automated labeling with the same quality gate.

Limit

Unit price and time will increase. Log storage, gold set design, and expert inspection are all costs.
It is difficult to detect perfect LLM usage. The detector is only an auxiliary signal and requires policy, log, and sampling inspection together.
Too much control can ruin the worker experience. An overly supervisory environment can cause good workers to leave.
May be overdesigned for low-risk tasks. Attaching expert review to all image tagging does not match speed and cost.

9) Points to study more deeply

Key one-line summary: The key keywords after MTurk are not labeling platforms, but data provision, gold set, workforce separation, and contamination detection.

Data provenance: This is a record that tracks where the data came from and who changed it and through what procedures.
Gold set: This is a set of samples with already verified answers. It is used as a standard to check the quality of the labeler and model.
Private workforce: This is a group of workers created or contracted directly by the organization rather than an open market. Security and quality control are easier.
LLM contamination: This is a problem where LLM products are mixed with data considered to be man-made. It is more critical in the evaluation set than in the training set.
Inter-annotator agreement: This is an indicator of how consistently multiple labelers answered the same sample. However, a high match rate does not always mean the correct answer.

To put it simply as a novice developer, in the past, “many people put stickers on” was the starting point for data quality. Now, we need to keep track of “who read which manual, what AI help we received, and what the reviewer fixed.” The dataset is not an Excel file, but a small production line.

10) Execution Checklist + Author’s Perspective

Key one-line summary: Future labeling projects should look at contamination rejection rates and reproducible audit logs before job completion rates.

Have you separated the data usage of training set, evaluation set, verification set, and demo set?
Have you assigned low, medium, and high risk ratings to each dataset?
Are you versioning directives, label definitions, exceptions, and counterexamples?
Has the LLM usage policy been clearly defined as prohibited, permitted, or reported?
Are gold sets, trap samples, and duplicate samples mixed in the labeling process?
Do you store worker group, work time, input hash, inspection history, and reason for exclusion?
Do you re-examine some of the vendor delivery data based on internal standards?
Has a separate gate been placed in the evaluation set to prevent LLM auxiliary labels from being mixed?
Can model failure samples be traced back to the original labeling work?

Definition of Done: When one sample is selected from a representative dataset, if the original, instruction version, worker group, LLM usage policy, inspection history, and reason for final label change can be reconstructed within 5 minutes, a primary data source verification system is in place.

Author's perspective: I see the suspension of MTurk new sign-ups as the end of evidence-free human data rather than the “end of crowdsourcing”. Going forward, the competitiveness of AI teams will likely be determined not by their ability to buy more labels cheaply, but by their ability to explain what data went into the model and why it can be trusted. If you're a small team, you don't have to make everything perfect. Instead, the first thing to do is to strongly tie the sources and inspection logs for the evaluation set.