[mathogre]

At my old company, Hadoop, JSON, Parquet were the solutions looking for a problem to solve. Yes we handled lots of data, but none of it was time critical. No matter, we had BIG DATA™. Bow when I say that. We weren't Google or Twitter. We were not surgically retrieving data, and while we had lots of data, we didn't have huge data centers; Dremel wasn't relevant to us.

It's funny! It would take our Hadoop team three weeks to get data together for our use. I often didn't have three weeks. In those times when I needed to use the data from the previous day, I'd just grab the raw data, organize it, process it, and be done with it in a few hours, using Unix/Linux tools and a bit of mathemagical wizardry.

"You're supposed to use Hadoop."

"You wanted to know what happened yesterday."

"I did!"

"If you want it from Hadoop, it will be ready in three weeks. Probably. That's if they have everything done."

<Crickets>

[AtlasBarfed]

A formal enterprise data group should probably

- be a clone/repo for disparate databases so you don't need to figure out access/security/location or impact production systems

- an interface to management types that aren't technical or don't have tech people to do these things

- should provide a "librarian" knowledge of the enterprise's data and data sources

- should have knowledge on how to analyze data using different tools

- be able to schedule movements/reports and manage that

If you don't need any of that, then ... yeah, don't use it. But those sets of requirements should be useful to anything that deems itself an "enterprise".