top | item 41529434

(no title)

Kydlaw | 1 year ago

I recently moved to data engineering role where everything uses GCP services (think BigQuery, DataProc, Cloud Storage, ...) and wondered is all that was really necessary?

What would be the simple yet robust infra for data eng? Not thought a lot about it for now, so I am curious if some of you have would have any insights.

discuss

ghomem|1 year ago

The same thing that happened to devops from 2017-2024 (see: https://logical.li/blog/devops/) is happening with dataops. Hype train and jargon based decisions are taking place.

In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake".

Decisions are sticky so take context into account.