Ask HN: What are biggest data integration challenges not addressed today? Why?

[Eidamj1]

The market has generated several data integration and engineering solutions to assist in structuring and integrating disparate data sources (see: Fivetran, PreCog, dbt, Mozart Data, Palantir, y42, etc.).

IMO, these are somewhat limited in scope and generally based on a library pre-built connectors (or extremely expensive w/ large ramp-up in Palantir’s case).

What challenges remain in using these solutions? What sets them apart? Why?

From what I’ve seen, users want a common but configurable schema output they can query and one that’s generated from many disparate data sources… They want to be able to ingest data from structured and unstructured sources which might include web-based or local files… APIs, databases, flat files, websites, etc.

This is where I see limitations in existing SaaS solutions as they are not as open-ended in what data sources can be used as inputs for integration.

Thanks!