Skip to main content
Version: 0.12.1

Source Name

Certified Incubating Testing

Integration Details

Concept Mapping

This ingestion source maps the following Source System Concepts to DataHub Concepts:

Source ConceptDataHub ConceptNotes
Data Platform
Dataset
Data Job
Data Flow
Chart
Dashboard
User (a.k.a CorpUser)
CorpGroup
Domain
Container
Tag
GlossaryTerm
GlossaryNode
Assertion
DataProcess
MlFeature
MlFeatureTable
MlModel
MlModelDeployment
MlPrimaryKey
SchemaField
DataHubPolicy
DataHubIngestionSource
DataHubSecret
DataHubExecutionRequest
DataHubREtention

Supported Capabilities

CapabilityStatusNotes
Data ContainerEnabled by default
Detect Deleted EntitiesRequires recipe configuration
Data DomainRequires transformer
Dataset ProfilingRequires acryl-datahub[source-usage-name]
Dataset UsageRequires acryl-datahub[source-usage-name]
Extract DescriptionsEnabled by default
Extract LineageEnabled by default
Extract OwnershipEnabled by default
Extract TagsRequires transformer
Partition SupportNot applicable to source
Platform InstanceNot applicable to source
...

Metadata Ingestion Quickstart

Prerequisites

In order to ingest metadata from [Source Name], you will need:

  • eg. Python version, source version, source access requirements
  • eg. Steps to configure source access
  • ...

Install the Plugin(s)

Run the following commands to install the relevant plugin(s):

pip install 'acryl-datahub[source-name]'

pip install 'acryl-datahub[source-usage-name]'

Configure the Ingestion Recipe(s)

Use the following recipe(s) to get started with ingestion.

For general pointers on writing and running a recipe, see our main recipe guide.

'acryl-datahub[source-name]'

source:
type: source_name
config:
# Required fields
option1: value1

sink:
# sink configs
View All Recipe Configuartion Options
| Field | Required | Default | Description | | --- | :-: | :-: | --- | | `field1` | ✅ | `default_value` | A required field with a default value | | `field2` | ❌ | `default_value` | An optional field with a default value | | `field3` | ❌ | | An optional field without a default value | | ... | | |

'acryl-datahub[source-usage-name]'

source:
type: source-usage-name
config:
# Required Fields
option1: value1

# Options
top_n_queries: 10

sink:
# sink configs
View All Recipe Configuartion Options
| Field | Required | Default | Description | | --- | :-: | :-: | --- | | `field1` | ✅ | `default_value` | A required field with a default value | | `field2` | ❌ | `default_value` | An optional field with a default value | | `field3` | ❌ | | An optional field without a default value | | ... | | |

Troubleshooting

[Common Issue]

[Provide description of common issues with this integration and steps to resolve]