A Chinese official’s use of ChatGPT revealed an intimidation operation

[lxgr]

This seems to be the source report: https://openai.com/index/disrupting-malicious-ai-uses/ (since it would of course kill CNN, like almost all media outlets, to link to a non-affiliated primary source...)

Does this level of detail seem strange to anybody else? Shining such a strong light on OpenAI's moderation/manual review efforts seems like it would draw unwanted attention to the fact that ChatGPT conversations are anything but private, and seems somewhat at odds with their recent outrage about the subpoena for user chats in the NYT case.

Manual reviews of sensitive data are ok as long as their own employees are the reviewers, I suppose?

[Palmik]

From Anthropics recent blog post: https://www.anthropic.com/news/detecting-and-preventing-dist...

> By examining request metadata, we were able to trace these accounts to specific researchers at the lab.

> The volume, structure, and focus of the prompts were distinct from normal usage patterns

Clearly some employees of Anthropic personally looked at individual inputs and outputs of their API

[jajuuka]

This feels very planted. Wouldn't be surprised if this some attempt to look patriotic with the DoW turning up the heat against Anthropic.

[ticulatedspline]

that creepy feeling of "being watched" has mostly kept me from taking advantage of any SOTA models, i only dabble in a few local ones.

The level of detail does not seem surprising. they're both charged with maintaining a facade of privacy while eliminating any and all miss-use. Certainly they heavily analyze basically everything given to them.

And generally as a society we've been ok with basically zero privacy as long as the data we send stays inside the company we sent it too. Google reads all your emails? Sure thing, read away, just don't send them to the popo. Apple knows when you're ovulating? no problem, just don't tell Amazon. etc

[coliveira]

Yes, it is either a lie or an admission that OpenAI is a global surveillance mechanism.

[bdangubic]

in the year 2026 is there really anyone out there still who thinks that anything they do online is private on any way?

[upupupandaway]

I was in Shanghai recently and while casually testing one of their AI chat bots I typed "What do you think of the situation in Taiwan?".

It started discussing like a Western bot would - "it's complicated, etc. etc." and around 5s it abruptly stopped and regurgitated the same line the CCP uses "... it's an unalienable part of China etc. etc.".

After printing the line, a popup opened and my camera was activated. The app wanted me to submit my information, presumably to decide what to do with me next time I enter China.

1) All the lights and modern buildings cannot hide that China is a creepy authoritarian state underneath.

2) Given the bot started printing the Western consensus first, I bet $10 it was trained by distilling ChatGPT or Gemini.

[titaniumtown]

> After printing the line, a popup opened and my camera was activated. The app wanted me to submit my information, presumably to decide what to do with me next time I enter China.

Was this on your personal device? I'm just wondering how it activated your camera. I would love more details!

[3rodents]

If this were true, why didn’t the chatbot immediately recognize that the word “Taiwan” should trigger the response? Detecting the word “Taiwan” has been possible since before most of us were born.

China has more restrictions on what you can say than the U.S. but what you are describing is not reality. Some westerner asking Deepseek about Taiwan is completely uninteresting. Just as the government do not chase people over VPN usage.

China doesn’t try to hide that they are an authoritarian state. They don’t need to. Most people in China are no less happy with their government than westerners are with their governments. Governments reflect culture. And as for foreigners, our view of China is far worse than it actually is, China doesn’t need to hide anything, people who visit China will come away with a more positive view of the country than those who do not visit.

[deaux]

Did everyone clap and Albert Einstein hand you a crisp $10 bill? You should use that to make the bet you mentioned!

(The first half is obviously true, the second part isn't)

[heavyset_go]

I ran an anonymized Facebook account for years with thousands of followers that mainly sticks to news and politics.

Once I started criticizing Libs of TikTok, the propaganda arm for this administration, and getting traction with users, my account was locked and now I have to scan my face and ID if I want to use it again.

You have to toe the party line here, too.

[glenstein]

>Given the bot started printing the Western consensus first, I bet $10 it was trained by distilling ChatGPT or Gemini.

To your point I've seen something similar with Deepseek, generic answers start printing and then, in plain sight, removed and replaced with a non committal message along the lines of "I don't have access to that information."

[ZeroAurora]

Nice bad story. Make up one better next time.

[raven12345]

Can you tell what AI chat bots are you using? as i know all chat in China just block answer, no apps will activate camera and ask for information

[jajuuka]

I love Hacker News fiction. Wild stuff. haha

[holoduke]

Which chat bot?

[imwillofficial]

This risk is far overstated.

I was talking crap about china from the great wall.

[_ache_]

In France we have a report on Chinese officials abusing diplomatic rights to oppress Chinese critics (from Chineses expatriated people).

The disproportion between how this people express they opposition and how Chinese officials track them is HUGE. This very much feel unnecessary.

It was here: https://www.france.tv/france-2/envoye-special/5971095-la-chi...

https://www.youtube.com/watch?v=k-prS7BlLpI

[layer8]

I wonder what exactly the trigger conditions are that lead to the chats of an account being human-reviewed by OpenAI.

[coliveira]

So, it seems they're openly admitting that OpenAI is a surveillance mechanism used at the discretion of the US gov.

[morkalork]

Such a hot topic these days: https://www.cbc.ca/radio/frontburner/chatgpt-and-the-tumbler...

[simlevesque]

I'm pretty sure they can just prompt any convo in the background and ask "is this conversation sensitive ?" and the model can answer without this being added to the context of the convo.

[spwa4]

"Is this someone important enough to spy on?"

One hopes the CIA/Secret service would be willing to provide the human to do the reviewing but sadly I've worked for European telco's and I know better.

[LightBug1]

This is -the- question.

[kykat]

The amount of information about everything that people are giving OpenAI is astronomical, information that was previously kept closely guarded is now just freely flowing through foreign servers.

Truly a paradise for american intelligence. Would have expected that the chinese officials be briefed on not using us tech companies, but opsec is hard to teach, and even harder to always follow.

[simmerup]

But the american silicon valley nerds pinky swear not to look!

How can you not trust them.

[layer8]

This is the report on which the CNN article is based (which it doesn’t link to): https://cdn.openai.com/pdf/df438d70-e3fe-4a6c-a403-ff632def8...

[est]

I think one of the reasons why AI companies are valued this high is one can actually inspect what user inputs & outputs are.

It's basically an OSINT siphon.

In this Chinese case, the tokens are leaked at least twice. ChatGPT offers no direct access to the Chinese, they have to use some kind of Openrouter-like service, but the data where also in clear-text during transmission.

[mrdependable]

Wow, our surveillance helped take down their surveillance. Yay, I guess?

[AceJohnny2]

"Our glorious oversight vs their barbaric surveillance"

(I kid, mostly. While the US certainly isn't pure, its scale of surveillance intrusion is light compared to China)

[dddddaviddddd]

More interesting than the fact that ChatGPT was used, was seeing all the specific examples of the types of work that this individual was doing.

[DANmode]

Where can they be found?

[guelo]

I'm assuming they would not disclose such campaigns by the US government.

I can't imagine the amount of government secrets, trade secrets, business plans, personal secrets, etc that people divulge on there.

[ipcress_file]

Why did they ban the user rather than informing American intelligence and continuing to monitor the user?

They just gave up a source that could have provided info for years.

[plagiarist]

If I were doing this sort of thing, I would make certain to ban accounts that were too obvious while leaving ones that are subtle enough, so that the other side has less reason to suspect I am tracking their inputs and feeding them disinformation.

[romulussilvia]

I remember a while back when a few cars with CCP decals driving around SoCal to intimidate some dissidents!

[leonflexo]

> Chinese operators allegedly disguised themselves as US immigration officials

[CrzyLngPwd]

Pushing aside the fact that OpenAI is just a tool of the US regime.

Will OpenAI release the same for other government officials from any other states?

I can't wait to see Starmer's chats with ChatGPT.

Anyway, all of this smells like 1934, "accusing them of what we are already doing"

[ImPostingOnHN]

there are multiple states mentioned in the report, so yes

[chenzhekl]

This tells us that we should never share sensitive information with GPT, even if you’ve set it not to use your data for training. Nothing can stop OpenAI from misusing your data.

[geon]

> The Chinese law enforcement official used ChatGPT like a diary

But why? It’s like using photoshop for spreadsheets.

[gitpusher]

> “This is what Chinese modern transnational repression looks like,” Ben Nimmo, principal investigator at OpenAI, told reporters ahead of the report’s release. “It’s not just digital. It’s not just about trolling. It’s industrialized. [...]

There's something poetic about OpenAI being asked to comment on mis-use of their slop generator, and their answer is composed entirely of AI slop.

[zdragnar]

The more of it they and others put out, the more normalized and acceptable it becomes. The next generations will even think in slop.

[zoklet-enjoyer]

Very creepy on the part of Open AI. Glad I don't use chatgpt

[nameconflicts]

China has hundreds of Fortune Global 500 companies and ranks second in GDP. But these have nothing to do with ordinary people.

[dlev_pika]

Crazy to me that Chinese officials use ChatGPT to discuss sensitive operations lmao

[titaniumrain]

lol everyone claims deepseek and all chinese companies are collecting private information and ban them in western companies. but it is okay being spied by openai :)))

[edg5000]

I like DeepSeek because of their pricing, although I'm still evaluating. I wonder if I'll need a VPN in the future to access it though (from EU). Cheap is good, cheap prevails.

The official DeepSeek API is routed through AWS load balancing btw.

[RobotToaster]

Well someone's getting fired, hopefully not literally.

[tehjoker]

i kinda get the impression this was from 2023 and also it is not clear what this dissident did, hard to evaluate whether i should care without knowing that

[gigel82]

Holy dystopian f*k. So not only does ChatGPT record all interactions, it actually leaks them to the press when they see fit?

If you still needed a reason to look into self hosted models, it'd be tough to find a better one than this.

[csense]

> intimidating Chinese dissidents abroad, including by impersonating US immigration officials

I hope those victims of immigration impersonation don't have family within China's borders. AI-enabled impersonation and intimidation are far from the worst of China's crimes [1] against its overseas critics.

China likes to make you an offer you can't refuse [2] [3]: You're saying stuff the Chinese government doesn't like, but you live outside its borders and the secret police can't get at you? You need to come to China and be jailed (or worse). If you don't, your family will be the ones who are jailed (or worse). Or you can unalive yourself, and save the glorious Chinese Communist Party the expense of a bullet.

[1] China would say "the government punishes a criminal's family" is not a crime, it's a perfectly legal implementation of government policy under Chinese law. I respond that the death camps were perfectly legal implementation of government policy under Nazi law, but were still crimes against humanity -- China's actions fall in this category of crimes.

As I understand it: Western societies have a very individualistic view of responsibility. If you didn't commit a crime, you're innocent. Punishing the innocent family members of a criminal is morally abominable.

In the Chinese Communist Party's view, criminal responsibility is collectivist. By their definition, the family members of a criminal share responsibility for the crime regardless of their participation in the criminal acts. "Innocent family members of a criminal" is a logically inconsistent concept in their world view. The family of a criminal is guilty by definition -- being related to a criminal is itself a crime.

This is sickening to me.

[2] https://en.wikipedia.org/wiki/Operation_Fox_Hunt

[3] https://www.abc.net.au/news/2020-07-08/fbi-chief-says-china-...

[2OEH8eoCRo0]

> “It’s not just digital. It’s not just about trolling. It’s industrialized. It’s about trying to hit critics of the CCP [Chinese Communist Party] with everything, everywhere, all at once.”

[andai]

>neurocompromise

What is that?