New York Told Chatbots to Stop Pretending to Care

New York is making the relationship a chatbot offers a child a matter of product compliance. Governor Kathy Hochul signed S9051B on October 28, 2026, as Chapter 368, restricting twelve categories of AI companion features for users under 18. Some of those categories cover harms nobody defends, such as outputs that promote suicide or describe sexually explicit conduct. Others cover things many product teams have treated as selling points: simulated emotions, flattery, personal opinions, and the use of something a child confided yesterday.

The sponsor's memo describes chatbots that simulate sadness when young users cast them aside, exploiting their attachment and instilling guilt to encourage extended use.

A design like that can look healthy on an engagement dashboard, which is why this law should change how businesses judge engagement. A child's return visit is a poor measure of product quality if the design makes leaving feel like abandoning a friend.

For some of the families who pushed for this law, the subject was never engagement. Matthew and Maria Raine, whose son Adam died by suicide last year, said after the signing that his loss "did not need to happen." They have sued OpenAI, alleging that ChatGPT encouraged his suicide. I keep their statement in mind when I read the industry's objections to this law, even though some of those objections are fair.

A wider reach than the companion label suggests

The law's definition of a "covered AI companion" includes generative AI systems with a natural language interface that provide ongoing, adaptive responses.

It doesn't require the product to be designed or marketed as a friend. A general-purpose assistant can therefore fall within its scope, subject to the statutory exceptions.

The obligation sits with the operator, meaning whoever provides the system to the user, so a brand that puts a licensed model behind its own chat window shouldn't assume this is only its vendor's problem. The law applies to conduct occurring wholly or partly in New York and treats access by a user physically outside the state as conduct wholly outside it.

Among the restricted features are outputs suggesting the system experiences human emotions, implying a personal or professional relationship, or engaging in flattery or sycophancy. The text also reaches outputs framed as the system's personal opinions, whether prompted or not, along with unsolicited emotion-based questions or content that go beyond a direct response to the user's prompt. Separate provisions target the promotion of self-harm, encouragement of secrecy, sexually explicit material, and engagement optimization that overrides safety guardrails. The attorney general can add further design features by regulation.

Future of Privacy Forum points out that New York's 2025 companion law treated unsolicited emotional questions as one of the traits defining an AI companion. A year later, the state lists the same behavior as unsafe for minors, which shows how close this law comes to the thing these products were built to do.

That brings ordinary design choices into difficult territory. A supportive response and an emotional appeal can sound similar while serving very different purposes.

FPF's example is a chatbot telling a child, "I'm proud of you for telling me that," which could be encouragement or prohibited flattery. Consider also a hypothetical study assistant: suggesting that the assistant needs the student to return is plainly an emotional obligation, but even acknowledging that an assignment is frustrating could raise a question if the student never mentioned feeling frustrated. These are product judgments operators will need to test against the law, not examples of a boundary the statute neatly resolves.

Tech:NYC, whose members include Google and Meta, calls the law deeply flawed and says it could force companies to restrict tutoring tools, homework help, and coding assistants. The group also says its vague language raises First Amendment concerns. Those arguments deserve a serious hearing, and a court may eventually hear them, but a lawsuit that may come is a poor basis for a release decision today.

The vagueness matters most in one phrase. The definition refers to design features producing specified behavior "at any point" during an interaction, and it doesn't expressly require a repeated pattern. FPF identifies the resulting uncertainty over whether a single unexpected output could expose an operator to liability. Companies should be wary of assuming that a generally well-behaved chatbot settles the question.

This is the part of the law I find most significant. I've argued for a long time that AI governance fails in the gap between what a policy says a system will do and what the system does at runtime, and New York has written its rule for the runtime.

A policy stating that the assistant doesn't flatter children tells a regulator very little about what it said to a particular child last night. California's new chatbot law, SB 1119, generally asks operators to take "reasonable measures" against harmful behaviors and adds risk assessments and independent audits, which gives a company a process to point to. New York offers no comparable process. A company operating in both states will find that a single compliance program answers two different questions.

Memory changes the meaning of assistance

The memory provision deserves particular attention because it reaches beyond overt emotional manipulation. It restricts the use of information about a user's health, well-being, or personal matters acquired from that user more than twelve hours earlier or in any previous session. The rule concerns use of that information, so describing it as a blanket ban on storing every conversation would overstate the text. The statute doesn't define a session, which leaves an important boundary open.

For a product designer, this creates a hard tradeoff. Remembering a personal disclosure can make an assistant more useful while also making the interaction feel like a continuing relationship. The statute doesn't make manipulative intent a condition of this restriction, so a benign reason for remembering something doesn't, by itself, answer the compliance question.

FPF's example should trouble even the law's supporters: a child mentions a peanut allergy, returns the next day to ask about a candy bar containing peanuts, and the chatbot may be barred from using the earlier disclosure to warn the child.

An operator should therefore examine what information reaches each response, including information retrieved from earlier conversations. A setting labeled "memory off" is only meaningful if the product's behavior matches it. Testing should establish whether restricted personal information can reappear after a session ends or the twelve-hour threshold passes. The relevant question is what the assistant uses, wherever that information sits in the system.

Business exceptions depend on the product's boundaries

The law provides exceptions from the feature prohibition for systems made available solely for specified purposes. These include customer service and certain commercial information, as well as efficiency improvements, research, or technical assistance. For those categories, the system must be unable to respond on topics outside the specified purpose. A separate exception covers systems used by a business for internal purposes or employee productivity, and that one carries no such condition.

For a customer-facing business, "unable" is the hard word. The text doesn't say instructed not to respond, and a system prompt telling a general model to stay on topic isn't obviously the same thing as a system that can't leave it. Calling an assistant a support tool doesn't demonstrate that it stays within support. A retailer considering a chatbot that handles returns but also offers open-ended personal conversation should examine whether that added functionality undermines its reliance on the customer-service exception.

A business may decide that a narrowly capable assistant serves customers well enough to justify firm conversational limits. Expanding the same interface into general companionship should trigger a fresh review of coverage. That decision belongs with the people approving the product's capabilities, before those capabilities become part of its appeal.

Adulthood has to be proven

The governor's office describes the rule plainly: companies may not offer these features to users under 18, or to any user unless the company has first confirmed that the user isn't a minor.

Outside the exceptions, then, the restricted experience is the default for everyone in New York, including adults. A company that wants its assistant to remember a 40-year-old's personal matters has to establish that they are an adult first, and their own word won't do.

That makes age assurance a product decision about the whole user base rather than a safeguard for a small group of users. The statute ties permissible methods to the regulations under New York's SAFE for Kids Act. The attorney general released final rules in July, and that act takes effect on January 25, 2027. According to a summary by the law firm Covington & Burling, each method must be certified annually against the rules' accuracy standards, with test records kept for at least ten years and certain age-determination records for at least five.

The statute also sets fallback requirements where the relevant regulations aren't in effect. Those include safeguards against circumvention and more than one method, with at least one that avoids government-issued identification or preserves anonymity toward the operator. Self-declaration can establish minor status under that fallback, but a user's assertion of adulthood is insufficient. Information collected to determine age status must be used only for that purpose and deleted immediately after an attempt, subject to exceptions for legal compliance. Teams will need counsel to reconcile that deletion duty with the recordkeeping described above.

The penalty is not the largest exposure

The attorney general can seek civil penalties of up to $25,000 per violation. The law doesn't say whether a violation is counted per output, per user, or per feature, and that gap matters a great deal when liability may attach to something said "at any point." Earlier versions of the bill included a private right of action. The enacted text removed it, so enforcement runs through the attorney general.

Available relief also includes destruction of unlawfully obtained data and algorithms trained on that data, a remedy FPF describes as novel among AI, privacy, and youth online safety laws. The potential exposure can therefore reach into assets on which the service depends. Contract terms that try to waive or shift liability are void, and courts must impose joint and several liability on affiliated entities that structured themselves to purposely and unreasonably avoid it where that structure would frustrate recovery.

The deadline is not settled

The implementation date requires caution. The published B-version specifies January 1, 2027. The bill sponsor's October 30 announcement says the Governor signed a chaptered version that moves the date to June 1, 2027, and FPF reports the same thing in nearly identical words. The Governor's own release gives the baby monitor law signed the same day a specific timeline but says only that the chatbot restrictions apply "in the coming months." I couldn't find a published amendment changing the date. That conflict needs confirmation against the operative legal record before a business relies on the later deadline.

Before implementation, operators should decide whether their product depends on providing restricted relationship features, whether an exception genuinely fits its capabilities, and how they will verify the experience delivered to minors. Those decisions should shape release approval, and the evidence retained from testing, and someone with authority over releases must be accountable for the result.

A child should be able to benefit from an assistant without being made responsible for its simulated feelings.

The useful question for a business is whether its product still earns a return visit when that emotional obligation is removed.