(no title)
ndementev | 3 years ago
May I ask you what do you mean by saying "all steps under 3"? Are you referring to https://docs.opendatadiscovery.org/use_cases/dq_visibility?
As for the
> How is the lineage generated or manually maintained
All lineage in the platform is generated and not manually handled by user in the UI. We are leveraging ODD Specification (https://github.com/opendatadiscovery/opendatadiscovery-speci...) and all ODD Collectors (agents that scrape metadata from your data sources) send payload to the ODD Platform in this specification's format. ODD Specification introduces something called ODDRN — OpenDataDiscovery Resource Names. These are basically strings, identifiers of specific data entities. All ODD Collectors generates same identifiers for same entities, allowing us automatically build a lineage graph in ODD Platform.
Not letting a user to manually change lineage in the UI is kinda our solution to one of the lineage problems. This way users can be sure that the lineage is correct, up to date and no one messed with it at least in the UI.
Of course if there's an described API endpoint, there's a way to change the lineage by sending a request on your own (e.g. via curl or custom script), but I wouldn't call it manual. This approach allows companies and users to write their own integrations, making the system open.
Cilvic|3 years ago
If lineage is as automatic as you say that's not clear to me after reading. Thanks for explaining!
ndementev|3 years ago
We are continue working on a documentation, thank you for bringing this up! We'll take a look how this can be improved.
> For example: all the steps under 3. are not part of ODD, or are they? Only step 1 is performed in ODD, yes?
Yes, that's correct. In this scenario ODD acts as a source of knowledge about the problem.