The team behind https://covid19japan.com fights with this nonsense every day. We're open ears to improvements, and are playing with ways to possibly automate bits.
This is fantastic, terrific way to visualize this information. It's interesting that Hokkaido is red for infections but the growth rate is really slow. Are the distant prefectures less likely to follow guidelines from Tokyo and reporting more honestly?
Do yall know if how many deaths were from critical cases, I'm assuming some deaths were determined post mortem from people who did not receive critical care. The UK published numbers showing 50% survival rate for people who had to get admitted to the ICU so a surface reading of the stats on the page might indicated a 5 % survival rate in Japan or no relationship at all depending upon interpretation.
I found tabula or camelot to be pretty good at extracting tables from pdf. I worked on a few JP's gov PDFs and they are not fancy so that's nice. Some light post-processing is still needed but the automation probably will keep working if the PDF format doesn't change (big if of course).
Knowing Japan, I bet some poor freshly graduated but elite sod in the health ministry was, out of the blue, told by his boss "here's a bunch of reports from different prefectures. Make a report out of them." So he goes off and try to make sense of the data, which is why you see those magic and unexplained equations in the report.
The kicker is the most complex computer thing he ever did in his life was that Word presentation in university. HTML? Isn't that something you write by hand?
My mom is responsible for updating all of Los Angeles' numbers. It's literally "someone sent an email with updated new numbers, put them on the government website/in the database". There's no "making sense of the data" phase, just copy, paste, and hope whoever delivered those numbers did it correctly. She described the situation "all hands on deck, extra shifts, and losing staff to sickness daily"; the last thing they care about is some number on a screen.
In a good number of very large (Fortune 100) companies, yes. Even in large tech companies.
Just as mechanical lathes and nail guns didn't put carpenters out of business, there are plenty of web sites that benefit from the attention of human hands, not just trusting the latest fad framework to concatenate several thousand dependencies pulled from random GitHub repositories.
I've opened Japanese documents from, AFAIR, at least the Ministry of Immigration, the Ministry of Health/Labour/Welfare and the Ministry of Justice. They all had the similar properties: manual HTML in the best case, PDF in the worst case. Tables very often with hierarchical data. Lack of overall coherence. It's so similar between Ministries that I have to wonder if it's not just the result of some Japanese governmental directive, or the result of their Education system.
Incidentally, I always though that in animes it is often a big deal for a character to catch "a cold" because plot; unless their cold are actually more aggressive than western colds (or flues)? I wonder more and more which one it is.
While things like masks and social distancing are more likely to be already in place in Japanese culture, there definitely seems to be a bit of a surprise regarding actual numbers vs. means put in place to prevent the spread.
The reports seem to reflect the opacity of communication of Japan's situation about the Covid-19. It's good that the author put out actionable suggestions at the end, but I'm wondering if the opacity is a result of internal issues or intended opacity. Makes me glad to see Canada's stats[1] being so clear in comparison.
My friends run a mid-sized hair salon, about 8 workstations, on the west side of Tokyo [edit: Kichijoji] and are still operating. They see no reason to close down, haven't been told to close down by anyone, and although their business is down, it's still brisk.
Meanwhile the shops are full of both people and stock (except for loo roll).
Some young people in Tokyo have been returning by shinkansen to their parents' homes in the countryside (which is obviously a dreadfully bad idea). I persuaded my friends not to do that, but they were thinking about it.
I am extremely suspicious that Japan's numbers look good because they have not been testing enough.
Given that honesty in testing consistently precedes the eventual slowdown in deaths by several weeks, I strongly suspect that Japan is maliciously incompetent and a lot of dead Japanese will be the result.
At the end of the day, hiding exponential growth of this is impossible.
That's what really bugs me about those who love to point fingers at how this or that country is counting things wrong (often revealing their biases in the process).
Maybe every country has a random constant factor between actual cases and reported cases. But it doesn't matter - if it goes exponential, we'll know.
Yes, obviously once it goes exponential, exponentials are hard to hide. The point is to stop it before it goes exponential. That's why people were discussing, say, Iran or NYC testing problems and lack of social distancing, rather than saying "well, at the end of the day, hiding exponential growth is impossible, so who cares"...
My hypothesis is that the "with no symptoms" and "with symptoms" items actually do include deaths.
> Remember to not include death, because this is a quasi-hierarchy. The table tempts you to, but it's excluded. (The quasi-nature is because it is under the PCR tested positive umbrella on level 2.) So what about integrating level 3?
> That's simple. You don't. Because no matter how hard you try, the numbers won't add up. (e.g. Give it a try - you'll end up with 129 != 131 and 1147 != 1191. We'll need one of these numbers later.) Level 4 adds up though, so the plot thickens.
Note, however, that (131 - 129) + (1191 - 1147) = 46, i.e. the number of deaths.
So if you adjust the 131 and 1191 figures to match their subdivision, everything would sum up correctly, including the top level (pcr_tested_positive), which would then include deaths.
This would imply that 2 people without symptoms died, and 44 people with symptoms did.
Taking a guess, "deaths" used to be two separate columns: one under "with no symptoms" (2 deaths), and one under "with symptoms" (44 deaths). It was then decided that "deaths" should be its own top-level item, and someone decided that this was the best way to achieve that goal.
I can't find a way to make sense of the daily increases though, so this might be entirely wrong.
Looks mostly right but he is wrong about the Diamond Princess. Those people all went off home, which is not Japan. Mostly western states. So they don't count to the Japanese numbers, only did in the very beginning.
The rest looks indeed cooked, on purpose. Pasting images instead of tables is the usual trick.
Seriously, why all the skepticism regarding this? I understand it but it's strange to me. Hanlon's razor seems like an easier explanation, why does there need to be a nefarious plot under every bed.
The reason why I used a two day old report on March 29 is simple, there was no full report yesterday nor today. I'm guessing the Tokyo lockdown means that nobody can get to work, and PCs are too expensive to buy for home use so the people at the Ministry of Health can enjoy a nice weekend with hoarded pasta.
How's corporate Japan as a whole doing with remote work?
Popular opinion in the US regarding the integrity of various countries' coronavirus reporting is hilarious. Whatever China says must ALWAYS be a lie, on the other hand, the US numbers are not "suppressed," it's just incompetence. Meanwhile, nobody is batting an eye at Japan's ridiculously and obviously doctored numbers.
Hey guys, maybe China was just having literally the same organizational problems the US is having now leading to underreporting of numbers, back in Jan-Feb when every commenter on reddit and HN were saying China was covering it up.
Meanwhile people just assume Japan's numbers must be honest because... why? Anyone who has the slightest awareness of how Japan works knows that this is par for the course. And no, it's not just the government brainwashing the people. The people are complacent in this and defend their government by rejecting criticism and continuing to vote them in office.
Please don't take HN threads into nationalistic flamewar. There's substantive material to discuss here, but you've unfortunately poured firestarter over it (e.g. snark and flamey rhetoric). When people do this, the results are predictable and dismal—especially when the material is flammable to begin with. That's why the site guidelines explicitly ask you not to post this way: https://news.ycombinator.com/newsguidelines.html
In fact, you should do the opposite: add flame retardant, i.e. use neutral language, narrow the scope of your claims to what the substance supports, edit out grand provocation.
> Whatever China says must ALWAYS be a lie, on the other hand, the US numbers are not "suppressed," it's just incompetence.
Well, this jibes with reality.
I could care less about actual case counts - few countries are close to having the capacity to test everyone so who cares.
What is a huge problem with China is the death statistics. Looking at Lombardy with 6k+ deaths with the outbreak not yet peaked, am I really supposed to believe that only 2.5k died in Wuhan (similar population)? The CFR outside Hubei is ridiculously low - only 1 death out of 1200 cases in Zhejiang (!)?
During China's outbreak, there were additionally lots of accusations of case data being completely made up (insufficient variance in daily numbers) and looking at basically every other country's quite noisy reports today, that was almost certainly true.
The implications are pretty vast: China's reporting actually made the disease seem less urgent than it really was. Perhaps tens of thousands of lives would have been saved had they been more transparent.
This is a false equivalency. China is a totalitarian state that quite literally controls information; Japan and the US are not in any real sense. About six months ago, on a different HN topic, I said this about China's claim that foreigners committed crimes and thus couldn't leave the country:
I call it “crying wolf in reverse”: The problem with being a totalitarian state that falsely imprisons people on a regular basis is that you’ve lost all moral authority when it comes to arresting people for actual crimes. If you’ve falsified evidence, conducted show trials and made a mockery of rule of law before, why should anyone believe you when you say it’s a real crime this time?
We're seeing a similar situation with regards to their Covid-19 reporting. Is it necessarily inaccurate? No, but lying about everything else doesn't exactly lend us to trust you this time.
I agree with the gist of what you're saying, but I think you're missing a key part of the picture. Any jurisdiction can lie if they want, for a while, but at a certain point this is an actual disease that causes actual illness and death. You can't hide hundreds or thousands of people dying from respiratory failure and you can't hide overwhelmed hospitals or auxiliary medical tents. Not for very long at least.
In the case of Japan, they do not appear to have high enough transmission rates to cause serious problems. I don't know if they are lying about any given fact or figure. But they are not covering up a widespread COVID-19 outbreak because we know what that type of outbreak looks like. It looks like Italy.
There are wild accusations against Chinese government, when in fact we now see that they put the most stringent measures to avoid the escalation of this virus, which in fact they succeeded in doing, at least inside China. On the other hand, I live in one of the early focus of the virus in the USA, and the federal and local authorities told us to just wash hands, avoid handshakes, and continue our normal life. Against a virus that is treated in other countries with hazmat suits! In my opinion THIS is criminal, on the same level of what happened in Chernobyl, knowing everything that we knew about the crisis situation in China and Korea. Now the virus is out of control in the US and the government still has the nerve to tell us that this is the fault of someone else.
You are talking about the regime that fired the previous official in charge of the chinese government's response to it's covid19 outbreak for aledgedly falsifying information, and afterwards inexplicably decided to officially leave out from their covid19 infection statistics any patient who was confirmed to carry the virus but subjectively wasn't experiencing major symptoms.
Meanwhile, the Chinese government's officials claim with a straight face that there are no new covid19 cases in China, and ridiculously stating that any new case happens to be foreigners importing an infection caught outside of China.
The reason why people think the Chinese governmemt is lying is because the Chinese government is patently and shamelesly lying.
"every commenter on reddit and HN were saying China was covering it up"
Stop using absolutes. There are plenty of people in the US that realize the US numbers have been suppressed by a lack of testing. There are plenty of people in the US who actually think China has handled the situation better than our own dysfunctional government.
The only absolute truth is that we don't have all of the information yet.
Who assumes Japan's numbers are honest? We all know they've been cooking their numbers for the stupid olympics.
At this point all I can believe are South Korea's numbers. They have transparent government unlike China, they have tested extensively unlike the US, Italy and Spain, and their climate is similar to the US unlike Taiwan and SE Asian countries.
i dont know if their data is bogus or not but I can tell you that japan has engineered their society with infection control in mind. if people are even the least bit under the weather they put on a face mask. Many people who directly interface with large numbers of people wear them all the time (clerks, train station employees etc). hygiene is very important. many more doors are automatic than in most of the world. anything that can be touchless usually is. when any authority even hints that they want their passengers/employees/citizens to do or not do something it is taken a command and followed rigorously.
I'm wondering if self-isolation along with a culture of face masks is why their numbers are so low. A single person wearing a cotton face mask wont move the numbers down at all but a majority of the population both retarding their exhalation of virulent loads and their inhalation of virulent loads will. Should also be emphasized that the philosophy behind face masks isnt "dont you make me sick" its "i really dont want to make you sick"
Meanwhile people just assume Japan's numbers must be honest because... why?
FWIW, Every report I read was skeptical of Japan's numbers, usually based in thinking they had an ulterior motive to downplay the outbreak because they didn't want to move the Olympics.
> Whatever China says must ALWAYS be a lie, on the other hand, the US numbers are not "suppressed," it's just incompetence. Meanwhile, nobody is batting an eye at Japan's ridiculously and obviously doctored numbers.
The phenomenon you're describing is called a reputation.
Lie, that involves intent so I don't know that I care to indicate or debate that.
Highly skeptical, yeah I am considering they've already a long track record of suppressing information / people related to the COVID situation.
Less free countries have little reason / a long track record of hiding bad news, from their own people, others, etc, and they have less free media, people to discover / tell anyone otherwise.
>Meanwhile people just assume Japan's numbers must be honest because... why?
Here we are discussing an article that doesn't assume that.
All of China's public numbers are fiction, all information is suppressed and censored, numbers created at the local and regional level are fabricated, credible accounting principles do not exist, Chinese companies cannot withstand any reasonable external audit.
There are many entities, specifically in the world of finance that perform their own measurements of Chinese economic numbers and they don't jibe with the official figures.
With respect to Coronavirus, China lied and suppressed information early on, including sanctioning and suppressing anyone who talked about it.
As far as Japan, why assume others are not suspicious?
Japan's numbers look 'off' there's been a lot of talk about that.
American numbers are transparent, created by different agencies, the issues are likely testing, the varying types of tests.
Edit: I would say that most nations are otherwise the same, the only reason Japan stands out in this round is because their numbers just look odd.
Edit: This is not a new phenomenon. During the Soviet era, we saw the same thing happening from the USSR et. al. Public numbers are politicized and an essential part of the propaganda effort. It's not even a controversial position. Consider that even right now, during the Huawei political crisis, nobody can discern who actually controls this major international conglomerate, as officially they are ostensibly 99% owned by employees, which is an obvious fallacy.
I don't think this sentiment is so wise as its popularity would lead you to believe. I have no reason to believe that most bad things that people do happen entirely by accident. Further, I suspect that the first line of defense for anyone who's been caught doing something naughty is to play dumb, and I see no reason to make it easier to get away with murder, so to speak.
I believe that is an accurate statement, only when applied to the actions of one individual.
When speaking of the actions of corporations, governments, or groups of people working in tandem in any fashion, malice needs to be your first assumption.
Stupidity can be dealt with after you fix the problem, but malice will not only make the problem worse, but will also make dealing with the problem more difficult.
now days hacker news is often filled non-tech related discussion, specially anything relate to China or politics; come on guys, this is Hacker news, not political news; if you are so passion about politics, go to some forum dedicate for those things;
[+] [-] reustle|6 years ago|reply
Our GitHub: https://github.com/reustle/covid19japan/
[+] [-] xrd|6 years ago|reply
[+] [-] Etheryte|6 years ago|reply
[1] https://coronavirus.jhu.edu/map.html
[+] [-] stevenwoo|6 years ago|reply
[+] [-] omneity|6 years ago|reply
I’d be happy to donate scraping capacity to this project. If this is relevant ping me at omar@ domain from my profile.
[+] [-] squaresmile|6 years ago|reply
[+] [-] robinduckett|6 years ago|reply
[+] [-] guardiangod|6 years ago|reply
The kicker is the most complex computer thing he ever did in his life was that Word presentation in university. HTML? Isn't that something you write by hand?
[+] [-] kenhwang|6 years ago|reply
[+] [-] reaperducer|6 years ago|reply
In a good number of very large (Fortune 100) companies, yes. Even in large tech companies.
Just as mechanical lathes and nail guns didn't put carpenters out of business, there are plenty of web sites that benefit from the attention of human hands, not just trusting the latest fad framework to concatenate several thousand dependencies pulled from random GitHub repositories.
[+] [-] glandium|6 years ago|reply
[+] [-] astrobe_|6 years ago|reply
[+] [-] cmehdy|6 years ago|reply
The reports seem to reflect the opacity of communication of Japan's situation about the Covid-19. It's good that the author put out actionable suggestions at the end, but I'm wondering if the opacity is a result of internal issues or intended opacity. Makes me glad to see Canada's stats[1] being so clear in comparison.
[1] https://www.canada.ca/en/public-health/services/diseases/201...
[+] [-] rwmj|6 years ago|reply
My friends run a mid-sized hair salon, about 8 workstations, on the west side of Tokyo [edit: Kichijoji] and are still operating. They see no reason to close down, haven't been told to close down by anyone, and although their business is down, it's still brisk.
Meanwhile the shops are full of both people and stock (except for loo roll).
Some young people in Tokyo have been returning by shinkansen to their parents' homes in the countryside (which is obviously a dreadfully bad idea). I persuaded my friends not to do that, but they were thinking about it.
[+] [-] jesuschroist|6 years ago|reply
[+] [-] btilly|6 years ago|reply
Given that honesty in testing consistently precedes the eventual slowdown in deaths by several weeks, I strongly suspect that Japan is maliciously incompetent and a lot of dead Japanese will be the result.
[+] [-] svara|6 years ago|reply
That's what really bugs me about those who love to point fingers at how this or that country is counting things wrong (often revealing their biases in the process).
Maybe every country has a random constant factor between actual cases and reported cases. But it doesn't matter - if it goes exponential, we'll know.
[+] [-] gwern|6 years ago|reply
[+] [-] nemetroid|6 years ago|reply
> Remember to not include death, because this is a quasi-hierarchy. The table tempts you to, but it's excluded. (The quasi-nature is because it is under the PCR tested positive umbrella on level 2.) So what about integrating level 3?
> That's simple. You don't. Because no matter how hard you try, the numbers won't add up. (e.g. Give it a try - you'll end up with 129 != 131 and 1147 != 1191. We'll need one of these numbers later.) Level 4 adds up though, so the plot thickens.
Note, however, that (131 - 129) + (1191 - 1147) = 46, i.e. the number of deaths.
So if you adjust the 131 and 1191 figures to match their subdivision, everything would sum up correctly, including the top level (pcr_tested_positive), which would then include deaths.
This would imply that 2 people without symptoms died, and 44 people with symptoms did.
Taking a guess, "deaths" used to be two separate columns: one under "with no symptoms" (2 deaths), and one under "with symptoms" (44 deaths). It was then decided that "deaths" should be its own top-level item, and someone decided that this was the best way to achieve that goal.
I can't find a way to make sense of the daily increases though, so this might be entirely wrong.
[+] [-] dataatemytime|6 years ago|reply
[+] [-] kevinventullo|6 years ago|reply
[+] [-] rurban|6 years ago|reply
The rest looks indeed cooked, on purpose. Pasting images instead of tables is the usual trick.
[+] [-] longtom|6 years ago|reply
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3551331
[+] [-] LorenPechtel|6 years ago|reply
[+] [-] noobermin|6 years ago|reply
[+] [-] lainga|6 years ago|reply
How's corporate Japan as a whole doing with remote work?
[+] [-] icoxfog417|6 years ago|reply
[+] [-] zadkey|6 years ago|reply
[+] [-] noobermin|6 years ago|reply
[+] [-] phoenixstrike|6 years ago|reply
Hey guys, maybe China was just having literally the same organizational problems the US is having now leading to underreporting of numbers, back in Jan-Feb when every commenter on reddit and HN were saying China was covering it up.
Meanwhile people just assume Japan's numbers must be honest because... why? Anyone who has the slightest awareness of how Japan works knows that this is par for the course. And no, it's not just the government brainwashing the people. The people are complacent in this and defend their government by rejecting criticism and continuing to vote them in office.
[+] [-] dang|6 years ago|reply
In fact, you should do the opposite: add flame retardant, i.e. use neutral language, narrow the scope of your claims to what the substance supports, edit out grand provocation.
[+] [-] usaar333|6 years ago|reply
Well, this jibes with reality.
I could care less about actual case counts - few countries are close to having the capacity to test everyone so who cares.
What is a huge problem with China is the death statistics. Looking at Lombardy with 6k+ deaths with the outbreak not yet peaked, am I really supposed to believe that only 2.5k died in Wuhan (similar population)? The CFR outside Hubei is ridiculously low - only 1 death out of 1200 cases in Zhejiang (!)?
During China's outbreak, there were additionally lots of accusations of case data being completely made up (insufficient variance in daily numbers) and looking at basically every other country's quite noisy reports today, that was almost certainly true.
The implications are pretty vast: China's reporting actually made the disease seem less urgent than it really was. Perhaps tens of thousands of lives would have been saved had they been more transparent.
[+] [-] keiferski|6 years ago|reply
I call it “crying wolf in reverse”: The problem with being a totalitarian state that falsely imprisons people on a regular basis is that you’ve lost all moral authority when it comes to arresting people for actual crimes. If you’ve falsified evidence, conducted show trials and made a mockery of rule of law before, why should anyone believe you when you say it’s a real crime this time?
We're seeing a similar situation with regards to their Covid-19 reporting. Is it necessarily inaccurate? No, but lying about everything else doesn't exactly lend us to trust you this time.
[+] [-] standardUser|6 years ago|reply
In the case of Japan, they do not appear to have high enough transmission rates to cause serious problems. I don't know if they are lying about any given fact or figure. But they are not covering up a widespread COVID-19 outbreak because we know what that type of outbreak looks like. It looks like Italy.
[+] [-] coliveira|6 years ago|reply
[+] [-] rumanator|6 years ago|reply
You are talking about the regime that fired the previous official in charge of the chinese government's response to it's covid19 outbreak for aledgedly falsifying information, and afterwards inexplicably decided to officially leave out from their covid19 infection statistics any patient who was confirmed to carry the virus but subjectively wasn't experiencing major symptoms.
Meanwhile, the Chinese government's officials claim with a straight face that there are no new covid19 cases in China, and ridiculously stating that any new case happens to be foreigners importing an infection caught outside of China.
The reason why people think the Chinese governmemt is lying is because the Chinese government is patently and shamelesly lying.
[+] [-] dtwest|6 years ago|reply
"every commenter on reddit and HN were saying China was covering it up"
Stop using absolutes. There are plenty of people in the US that realize the US numbers have been suppressed by a lack of testing. There are plenty of people in the US who actually think China has handled the situation better than our own dysfunctional government.
The only absolute truth is that we don't have all of the information yet.
[+] [-] maallooc|6 years ago|reply
At this point all I can believe are South Korea's numbers. They have transparent government unlike China, they have tested extensively unlike the US, Italy and Spain, and their climate is similar to the US unlike Taiwan and SE Asian countries.
[+] [-] intpx|6 years ago|reply
I'm wondering if self-isolation along with a culture of face masks is why their numbers are so low. A single person wearing a cotton face mask wont move the numbers down at all but a majority of the population both retarding their exhalation of virulent loads and their inhalation of virulent loads will. Should also be emphasized that the philosophy behind face masks isnt "dont you make me sick" its "i really dont want to make you sick"
[+] [-] dfxm12|6 years ago|reply
FWIW, Every report I read was skeptical of Japan's numbers, usually based in thinking they had an ulterior motive to downplay the outbreak because they didn't want to move the Olympics.
[+] [-] tengbretson|6 years ago|reply
The phenomenon you're describing is called a reputation.
[+] [-] wnevets|6 years ago|reply
Japan is notorious for faking numbers to save face like their murder rate [0] but americans never seem to doubt them. Very odd.
[0] https://www.latimes.com/archives/la-xpm-2007-nov-09-fg-autop...
[+] [-] duxup|6 years ago|reply
Lie, that involves intent so I don't know that I care to indicate or debate that.
Highly skeptical, yeah I am considering they've already a long track record of suppressing information / people related to the COVID situation.
Less free countries have little reason / a long track record of hiding bad news, from their own people, others, etc, and they have less free media, people to discover / tell anyone otherwise.
>Meanwhile people just assume Japan's numbers must be honest because... why?
Here we are discussing an article that doesn't assume that.
[+] [-] jariel|6 years ago|reply
There are many entities, specifically in the world of finance that perform their own measurements of Chinese economic numbers and they don't jibe with the official figures.
With respect to Coronavirus, China lied and suppressed information early on, including sanctioning and suppressing anyone who talked about it.
As far as Japan, why assume others are not suspicious?
Japan's numbers look 'off' there's been a lot of talk about that.
American numbers are transparent, created by different agencies, the issues are likely testing, the varying types of tests.
Edit: I would say that most nations are otherwise the same, the only reason Japan stands out in this round is because their numbers just look odd.
Edit: This is not a new phenomenon. During the Soviet era, we saw the same thing happening from the USSR et. al. Public numbers are politicized and an essential part of the propaganda effort. It's not even a controversial position. Consider that even right now, during the Huawei political crisis, nobody can discern who actually controls this major international conglomerate, as officially they are ostensibly 99% owned by employees, which is an obvious fallacy.
[+] [-] yufeng66|6 years ago|reply
[+] [-] DangitBobby|6 years ago|reply
[+] [-] Loughla|6 years ago|reply
When speaking of the actions of corporations, governments, or groups of people working in tandem in any fashion, malice needs to be your first assumption.
Stupidity can be dealt with after you fix the problem, but malice will not only make the problem worse, but will also make dealing with the problem more difficult.
[+] [-] objektif|6 years ago|reply
[+] [-] hujun|6 years ago|reply