(no title)
milesjag | 1 year ago
1 - I feel like more bespoke interfaces could be amazingly useful in the data world.
2 - Data engineering doesn’t really get me building creatively.
3 - Getting 3 languages to play ball (and that’s front-end only) sounds crazy challenging.
Data projects are nightmares but it is good exposure to the messiness of the real world. Have you had a chance to dig into DE much yet?
solardev|1 year ago
I never did proper "data engineering" per se, but we did have casual run-ins with geodatasets and GIS, and used them to build a home solar calculator that takes a ZIP code and estimates that area's available yearly sunlight, utility rates, electricity consumption, etc. and combine it into an estimated size for a home solar system. It was like a primitive version of Google Sunroof (https://sunroof.withgoogle.com/), which is much, much better.
I too love the intersection of data and visualization/usability, turning obscure spreadsheets into useful public interfaces :) The geospatial world is just one part of that. It used to be cool, but I guess nowadays it'd probably be more about turning data into LLM-digestible training sets so that they can more directly analyze questions and answer them in plain language.
Do you think traditional DE is still worth learning, with Skynet on the horizon? What's a good way to get started?
milesjag|1 year ago
There’s a lot to be said for data engineering when done at scale - systems design, devops, cloud engineering etc.
I’m not sure that’s going yet…
Building the individual units (reliable ETL pipelines) might be on the chopping block in the near future - I’m not convinced there’s 100 competing ways to build these.
Still - best way to get started is probably to get stuck in with running a data pipeline using a locally deployed Airflow instance. Read some data from some api and write it to a local database deployment (postgres?).
May I ask the same of web application development?