ian-whitestone's comments

Gotcha. Thanks for the feedback! Happy to chat at any point, just shoot me a message. I'm fairly easy to find on Twitter/LinkedIn/my website.

Correct! thanks for calling out. Should I disclose this in the text next time? What's the common practice?

Snowflake has skyrocketed in popularity over the past 5 years and firmly planted itself at the center of many companies' data stacks. Snowflake came into existence in 2012 with a unique architecture, described in their seminal white paper as "the elastic data warehouse". Rather than have compute and storage coupled on the same machine like their competitors did, they proposed a new design that took advantage of the near-infinite resources available in cloud computing platforms like Amazon Web Services (AWS). In this post, we'll dive into the three layers of Snowflake's data warehouse architecture: cloud services, compute and storage.

Outside of being open source, how does ClickHouse differ from Snowflake/BigQuery? In what scenarios would I choose ClickHouse over those existing solutions?

Completely agree a DS shouldn't be responsible for all those things you mentioned (all the infrastructure, networking, security, etc.). In today's age, "data pipelines" can look like plugging into tools & frameworks that abstract away a lot of this complexity. For example - fivetran for extraction & dbt for data modelling. So once those engineering/devops heavy things you listed are abstracted away, I think it's fair for a DS to be responsible for building the data transformation logic in the pipeline. Of course, not all companies will have such tooling available, in which case it's not fair to place those expectations on a DS.

Agree with you that not all of these things will apply to every DS role - particularly research heavy ones. But my hope is the vast majority will.

Obsessed may have been overkill :)