top | item 17486040

Inside China’s Dystopian Dreams: A.I., Shame and Lots of Cameras

286 points| rmason | 7 years ago |nytimes.com | reply

214 comments

order
[+] telltruth|7 years ago|reply
I fully expect this to keep getting advanced further. There has been no protest whatsoever within China and even many people of Chinese origin living in US with US citizenship seem to think this is necessary and good for the people. I have been told that "outsiders" won't "understand" this. The bottom line is that if people are not rebelling against current measures of social credit, government only has incentive to keep going.

Sooner or later government would require things like TV sets with front camera. So when a person watches TV, camera watches that person. There have been huge progress made in emotional inference which can allow government to measure emotional responses to what person is viewing. Imagine there is a news of rebellian getting crushed and if you are consistently showing supportive emotions, you can get black listed for further investigation. Cities can get microphones at restaurants and other places to analyse what conversations are happening and find hot spots where "trouble makers" hang out. Previously we had only vast sensor networks. Now we can turn each sensor in to autonomous spy that works loyally without salaries and demands nothing but cheap electricity. Computers allows for scale when humans are being replaced. If you want million spies scattered all around, its possible fairly cheaply now. I would also expect lots of other countries to adopt variant of these tech, typically under the notion of improving safety and security for general public.

[+] rqs|7 years ago|reply
> Chinese origin living in US with US citizenship seem to think this is necessary and good for the people

Of course, they don't have to live with all that so they don't actually care. Next time you could ask them whether or not they're willing to renounce their U.S. citizenship and become Chinese citizen again so they can be monitored 24/7 for their own "insider" "good", then you will start to get true answers from them.

Back to the topic though, as a Chinese, I don't actually at the opposite of video surveillance in public, it made many places more safer and thus prosper.

However, surveillance cannot went too far, and certainly cannot be use to shame people. It's about respect of one's dignity, a bottom line that a government should never cross.

Plus, given the poor accuracy that system outputs, what if the system made a wrong detection and displayed information of an innocent? Chinese government probably don't care about it though.

[+] Razengan|7 years ago|reply
> I fully expect this to keep getting advanced further.

If they aren’t going to scale it back then I hope they go all the way with it.

I hope they try to automate it and feed everything about everyone to a centralized AI.

I hope they give it more and more control including the ability to direct military drones.

I hope it then turns on them.

Because the “smart stupid” people that implement well-intentioned malice like mass surveillance can’t usually see far ahead (see reports of data harvesting companies getting hacked.)

If this is the path we’re going to go down then might as well speed up evolution and usher in a new class of intelligent life on this planet.

[+] seanmcdirmid|7 years ago|reply
I’m pretty sure there have been protests about this in China. They might not have been heard very well, they were probably more likely to be in the form of online subversive statements, but please don’t think that all Chinese just accept this, many do not.

A large scale protest is unlikely, but you’d be surprised what some rebellious CAFA students can do via art.

[+] iKSv2|7 years ago|reply
" The bottom line is that if people are not rebelling against current measures of social credit, government only has incentive to keep going."

One problem with this is that since NEWS and almost all media is State-owned in China, there might be protests and we might be unaware, also the protesters are straightaway sentenced plus now even their family members get "negative points" if I am not mistaken. There's a reason why Chinese people living outside also dont say negatively.

[+] EGreg|7 years ago|reply
First you make it easier to control the masses

THEN you make any kind of rules you want

Prevent any rebellion with AI

Install implants to people to further cement your control later. You won’t even need to do facial recognition.

There are essentially two options here:

1) A ruling class enforcing totally arbitrary rules on a society of “pets”

2) An AI that eventually makes the ruling class completely irrelevant

My main concern is that this AI will not be sensitive to what humans need or want and will cause a lot of mismatch that they won’t be able to get out of anymore. Revolution won’t be an option. You’re just stuck in a jail being controlled by a dumb AI that just prevents any way out of the jail.

There may be no more progress or advancement as this AI would not have any abilities or interest in any of that. Just a zoo of humans essentially.

We are already turning the entire planet into farms and monocultures.

And a zoo for ourselves is next. No matter what you do someone will invent autonomous X that will then set the standard for all human behavior. It will be so efficient as to notice when you did a tiny thing wrong and the penalties would be extracted via perfectly mapped system of rewards. It will know anyone who tried to help you out as well.

It’s just too “sweet”.

[+] v1vek|7 years ago|reply
War is peace. Freedom is slavery. Ignorance is strength
[+] some_account|7 years ago|reply
Why are we acting like we don't have this in Great Britain or America?
[+] obelix_|7 years ago|reply
You have to examine your own assumptions too -

1. You assume stopping "revolution" is the goal. It is not. All you have to do is look at what outcomes "revolutions" in an info saturated/low attention span/consumption culture based society have produced in the last 20 years.

2. In 2013 the Washington post did an estimate on how much was being spent on the surveillance society in the US. Everyone reacted in disbelief. Do you know by how much that budget has increased in the last 5 years? Are these people just brain-dead to be spending this kind of cash? Ofcourse not.

The US has made its share of mistakes over valuing freedom and squandering potential. China is overvaluing control and will make its share of mistakes too. We have to learn from both sets of mistakes to arrive at the right balance of where society should go in high noise info saturation environments. These are very new environment that society hasnt been in before and the right path ahead is not as obvious as people think.

[+] wcrichton|7 years ago|reply
I’m a PhD student focusing on large scale video analysis. It’s an unfortunate fact that a lot of the applied research in this space is motivated by surveillance, particularly in systems, eg what’s come out of Microsoft’s efforts here [0]. A colleague who attended CVPR said most of the industry booths were either surveillance or self driving cars.

I wish researchers would start to think more creatively about what we can do with large video datasets and tools like face recognition. The least we can do is try to use this technology for social good. For example, a USC/Google team used computer vision tools to identify gender bias in modern cinema [1] (for my own research, we’re doing a similar kind of analysis on TV news at 100x the scale).

[0] https://www.usenix.org/conference/nsdi17/technical-sessions/...

[1] https://www.google.com/about/main/gender-equality-films/

[+] ineednoprocrast|7 years ago|reply
I rarely post, but I feel it necessary to chime in here.

I studied computational linguistics/natural language processing in grad school, out of an interest in accessibility issues (I had RSI) and related applications, and had naively neglected to look very hard about how I'd be likely use it in a career.

Other people in my cohort went to work for the CIA. I couldn't move to the bay area for... reasons, (this was pre-Google) and it seemed everyplace I could find anything wanted me to have a security clearance, with all the implications that has.

One could say I'm not mercenary enough, or that I took too many philosophy classes, or any number of other rationalizations, but one way or another, none of the jobs I've taken have had much need for what I studied.

I feel morally obligated to use what I know to improve the state of the world. This has, unfortunately, led to a lot of unemployment and serious depression.

Pseudonym for obvious reasons.

[+] thrantaway|7 years ago|reply
It's disheartening that so many bright people work on research that will undoubtedly be abused by authoritarian governments and make the world a worse place for everyone. Ask a thousand people what video analysis can be used for and I bet the vast majority will say surveillance, and most would probably struggle to think about something else.

I wish researchers would start to ask themselves if what they're building can be abused by authoritarian governments, and if so, switch focus to a field of research that will make the world a better, rather than a worse, place to live in.

History will not look kindly on researchers who built the tools that enabled Xi the Pooh to achieve a dystopian surveillance state.

[+] walrus01|7 years ago|reply
> I wish researchers would start to think more creatively about what we can do with large video datasets and tools like face recognition.

Some very smart people in China did exactly that, and arrived at a conclusion on what the local market for surveillance tech was lacking. Based on what the government will fund, they came up with an answer we seem to dislike.

[+] extralego|7 years ago|reply
To be fair, the absolute last thing “modern cinema” needs is another formulaic measurement of demographically correlated screentime.

I am not personally against girls seeing depictions of other girls in movies (have you consulted with the anti-bodyshaming sect?) however this claim to “social good” is telling of a desperate effort to ignore actual social problems, none of which are so conveniently morally distinct from the economic disparities Google instigates and defends on a daily basis. Studies show little girls who are homeless have bigger problems.

[+] dbcooper|7 years ago|reply
Can't wait for algorithm homogenised movies!
[+] est|7 years ago|reply
> identify gender bias in modern cinema

Looks like a way to sell more ads.

[+] Noos|7 years ago|reply
If you asked them, they would effusively tell you all this surveillance is being used for the social good, and probably even believe it. Not many people are brutally honest about how their work destroys the social fabric, and will always point to the benefit it provides. Pornographers think its a social good if we would just be a little less uptight about sexuality, and more porn access is a solution.

And the same people who try to work for good can find hey've just armed others for evil by giving them tools.

[+] bachbach|7 years ago|reply
> The least we can do is try to use this technology for social good. For example, a USC/Google team used computer vision tools to identify gender bias in modern cinema

That sounds entirely worthless.

A social good involves helping somebody who needs aid.

This proposal merely fills a perceived need by political dogmatics. This is exactly a sociopolitical memory leak.

[+] symisc_devel|7 years ago|reply
We were recently approached by a contractor who works behalf the Chinese gov. Basically, they wanted to purchase an insane number of commercial licenses of our embedded computer vision library[1] to implement face tracking (detection and recognition) for a custom OpenBSD fork running on a homegrown CPU architecture with 512MB of RAM. They didn't tell more about this hardware but impression is that is the new generation of street Cameras.

[1]: https://github.com/symisc/sod

[+] bsenftner|7 years ago|reply
I write real time, video based FR. The company I work has a strict policy that our sales, nor any partners even respond to Chinese company queries. The thing is, regardless of what they say, you'll never get paid AND they will reverse/rip-off your product.
[+] walrus01|7 years ago|reply
Actually what I am surprised about is that they even approached you at all. Products manufactured for the domestic Chinese market generally don't need to give a shit about the legal aspects of the GPL. Until somebody successfully sues a Chinese company in a domestic court and wins damages (vanishingly unlikely), BSD, GPL, LGPL, Apache and other licensed open source things will continue to be incorporated into embedded products with no attribution or licensing agreement.
[+] walrus01|7 years ago|reply
These tracking systems are being combined with China's "social credit score". You think Transunion, Experian and Equifax are creepy and wrong? Spend ten minutes researching this:

https://www.google.com/search?client=ubuntu&channel=fs&q=chi...

Cameras with facial recognition are the tip of the iceberg. Through sufficiently strong government legal control, metadata such as mobile phone GPS and tower data can be correlated with facial recognition, bank debit card tracking, mobile payment app payments (which are HUGE in China), mandatory data reporting from Lyft/Uber type apps, toll road transponders on private cars, transit system stored value cards, and so forth. It's the aggregate of the whole which paints a picture of a person's entire lifestyle.

[+] nimrod0|7 years ago|reply
The Big Three credit bureaus started exactly as a social credit system. They'd send snoops to keep tabs on you and collect gossip from your friends.
[+] tmalsburg2|7 years ago|reply
The article says there are four times as many cameras in China than in the US. This means approximately the same number of cameras per citizen. Where is the NYT article about the US' dystopian dreams?
[+] emptybits|7 years ago|reply
Where is the NYT article about the US' dystopian dreams?

The NYT writes plenty of articles about US surveillance gone-too-far. Even a naive query like this: https://www.nytimes.com/search?query=surveillance turns up plenty of exposure on the issue.

At least in the USA, citizens can openly criticize and publish their government's corruption and failings and concerns about the surveillance state and abuses of power. Americans can do this without fear of unjust incarceration or worse. And the USA maintains, with all its flaws, a democratic process which requires its politicians to answer to the public and risk not being (re)elected.

China doesn't have (and never really had?) these features. I think this is important. I write this as a non-American and one who often criticizes American policy.

[+] mindfulhack|7 years ago|reply
India has almost the same population as China, and yet they don't choose to wield an authoritative system of population control and management with all the psychological techniques of manipulation and deception that China is.

So how is India doing - worse than China? Every country is different so can't really be compared like that, but I am fascinated to see whether Chinese people will come to rise up against such an abusive government and somehow play psychological hardball in the same way, or whether the oppression will just tire them out because ecosystems of old biology just can't compete with the new ecosystems of machines and AI that that government is increasingly using.

If I were writing a Sci-fi novel I'd have the AI (at the moment of singularity) take over the selfish pig government and become the new ruler over ALL the humans (making them equal but under it), in a delicious moment of irony.

[+] rinze|7 years ago|reply
> “The whole point is that people don’t know if they’re being monitored, and that uncertainty makes people more obedient,” said Mr. Chorzempa, the Peterson Institute fellow.

Straight from 1984.

[+] mncharity|7 years ago|reply
One thing to start thinking about: Eye tracking is coming with AR. It enables foveated rendering - only the tiny region[1] your fovea is pointing at gets full resolution GPU effort. And AR is, eventually, becoming the new cell phone. But just as cell phones were "oh, btw, now everyone will be wearing a real-time location tracker - just like a handful of criminals and wild animals did previously", AR is "real-time what-are-you-glancing-at/thinking-about-second-by-second tracker". Integrating cell phone capabilities with societal culture, policy, and law, seems to have been largely reactive, and seems to have involved significant societal changes with limited collective reflection. Perhaps we can do better this time, with AR and eye tracking?

[1] https://www.shadertoy.com/view/4dsXzM

[+] 21|7 years ago|reply
This Woman Threw Ink On A Photo Of China's President On A Livestream And Now She's Disappeared:

https://www.buzzfeed.com/kassycho/chinese-woman-ink-xi-jinpi...

[+] taurine|7 years ago|reply
Reminds me of this Tweet by Donald Trump:

> Nobody should be allowed to burn the American flag - if they do, there must be consequences - perhaps loss of citizenship or year in jail! [1]

Luckily the US has safeguards in place and people that dare say "no, you can not do this" to the commander in chief.

[1] https://twitter.com/realdonaldtrump/status/80356799303675494...

[+] virtuabhi|7 years ago|reply
Hope she is alright now.
[+] nimrod0|7 years ago|reply
It should be pointed out that, with or without AI, the cultural impulses are the same. This is an issue of authoritarianism, not of AI. If it should be feared more universally then the fear should be the authoritarian streak within all of us.
[+] woodandsteel|7 years ago|reply
You're absolutely correct. Things are just as bad in the US as China, and so we should ignore what is happening in China and instead turn all our attentions inward.
[+] hnzix|7 years ago|reply
Shan gao, huangdi yuan

The mountains are no longer high, nor the emperor far away. How long until your sesame credit is docked for walking into the wrong bookshop?

[+] sharpercoder|7 years ago|reply
I understand that most western countries have societal controls grown over the past centuries. The Chinese did not, and are catching up quickly by implementing similar controls as we have. But now, it's the digital age, and as such they are using any available technology to implement these societal controls. From this perspective, it's completely normal what the Chinese do. Seen from western eyes it does look a bit scary though. We must realize however that our society also has firm societal controls setup in many things we do.

China is now the dominant world player, this year it has surpassed the US on many areas (IT and military-industrial complex are the only US-dominant areas). Much more importantly, China is full-throttle developing it's trading strategy: The silk railway to Europe, trade dominance in Africa, footholds in Greece, Poland and other EU countries, large export deficit to the US. It won't be long before China will totally dominate trading in the world market. At that point, China can set the requirements. When that happens, traders are likely to need a WeChat account, too. No good rep? No trade.

[+] Animats|7 years ago|reply
China has had broad surveillance for decades. but it was manual and village or employer based. Each worker had a dang'an, a paper-based personnel record which follows them from employer to employer, plus a similar record kept by local police.

As cities grew and employees become more mobile, keeping this current and useful started to break down. So the central authorities are modernizing their surveillance system.

Such control predates communism in China. Something like this was in place when China was mostly peasants, with local village officials keeping records. This goes back at least to the Qing Dynasty, around 1750. The concept that the government knows where everyone is and what they're doing goes way back in China.

[+] extralego|7 years ago|reply
China is to 1984 as USA is to Brave New World.
[+] Tenoke|7 years ago|reply
The technology isn't quite there, and Chinese law enforcement clearly exaggerates in order to maximize the current impact.

However, we are very close to the point where everything could be tracked, and recorded in a usable way - including the exact location of most citizens at all times, discrepancies in utility bills, shopping, internet activity (except possibly for those few with impeccable OpSec, and even that for only a bit longer), visited places, communication with new and old contacts, etc.

Some like to think semi-romantically that we are in/near the Cyberpunk future Sci-Fi promised. We are not - if a government truly goes all out (as the CCP has shown willingness to do), it will at some point simply be impossible to evade the law.

The cool greyhat who hides in the shadows, hacks government systems, and operates outside mainstream society is a pipe dream. When all is said and implemented escape could only come from above - another nation, intergovernmental organization or at minimum people high up in the system. Dissidents and revolutionaries would hardly be able to do much.

[+] mc32|7 years ago|reply
This article examines A.I., policing and authoritarianism from a Western perspective, rather than the local perspective. I think it would have served the article's balance to get a more local and relativist view on the intersectionality of the things they are looking to be objective about.

That said, if the technology delivers on the potential the gov't claims, I think they will have to relax the laws or make it flexible enough such that people and the government reach an understanding in expectations while affording people the maximum liberty while providing a base-line of expected behavior.

Obviously, without oversight from the people, this has the potential to turn into a monster.

China is a huge, huge country, historically it's been difficult for the central government to exercise its control all over its territory. This would be the realization of that historical desire.

[+] briandear|7 years ago|reply
What “local” is going to chat with The NY Times about this. More importantly, what local would dare question it? That’s the difference between the US and China right there.
[+] Semirhage|7 years ago|reply
China is a huge, huge country, historically it's been difficult for the central government to exercise its control all over its territory. This would be the realization of that historical desire.

That sounds like a better argument for China breaking up than it does using force and dystopian surveillance to maintain a lock on power.

[+] tomxor|7 years ago|reply
Classic... they called the software skynet! :P At least they have a sense of humour.
[+] xkcdefgh|7 years ago|reply
democracies are sluggish, progress slowly, and also cause racial division sometimes. But I'd still never wish any country to devolve into any other form of governance