Merge branch 'datahub-project:master' into master

docs/cli.md

@@ -341,6 +341,24 @@ datahub timeline --urn "urn:li:dataset:(urn:li:dataPlatform:mysql,User.UserAccou
 
 ## Entity Specific Commands
 
+### dataset (Dataset Entity)
+
+The `dataset` command allows you to interact with the dataset entity.
+
+The `get` operation can be used to read in a dataset into a yaml file.
+```shell
+datahub dataset get --urn "$URN" --to-file "$FILE_NAME"
+```
+
+The `upsert` operation can be used to create a new user or update an existing one.
+
+```shell
+datahub dataset upsert -f dataset.yaml
+```
+
+An example of `dataset.yaml` would look like as in [dataset.yaml](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/cli_usage/dataset/dataset.yaml).
+
+
 ### user (User Entity)
 
 The `user` command allows you to interact with the User entity.
@@ -351,7 +369,7 @@ For detailed information, please refer to [Creating Users and Groups with Datahu
 datahub user upsert -f users.yaml
 ```
 
-An example of `users.yaml` would look like the following. You can refer to the [bar.user.dhub.yaml](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/cli_usage/user/bar.user.dhub.yaml) file for the complete code.
+An example of `users.yaml` would look like as in [bar.user.dhub.yaml](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/cli_usage/user/bar.user.dhub.yaml) file for the complete code.
 
 ```yaml
 - id: [email protected]
@@ -379,7 +397,7 @@ For more information, please refer to [Creating Users and Groups with Datahub CL
 datahub group upsert -f group.yaml
 ```
 
-An example of `group.yaml` would look like the following. You can refer to the [foo.group.dhub.yaml](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/cli_usage/group/foo.group.dhub.yaml) file for the complete code.
+An example of `group.yaml` would look like as in [foo.group.dhub.yaml](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/cli_usage/group/foo.group.dhub.yaml) file for the complete code.
 
 ```yaml
 id: [email protected]

docs/glossary/business-glossary.md

@@ -136,6 +136,10 @@ Glossary Terms and Term Groups abide by metadata policies like other entities. H
 - **Manage Direct Glossary Children**: If a user has this privilege on a Glossary Term Group, they will be able to create, edit, and delete Terms and Term Groups directly underneath the Term Group they have this privilege on.
 - **Manage All Glossary Children**: If a user has this privilege on a Glossary Term Group, they will be able to create, edit, and delete any Term or Term Group anywhere underneath the Term Group they have this privilege on. This applies to the children of a child Term Group as well (and so on).
 
+## Shift left and bring your glossary into Git
+
+You can use this [Github Action](https://github.com/acryldata/business-glossary-sync-action) and bring your Business Glossary into your git repositories. This can be the starting point to manage glossary in git.
+
 ## Managing Glossary with Git 
 
 In many cases, it may be preferable to manage the Business Glossary in a version-control system like git. This can make

metadata-ingestion/examples/cli_usage/dataset/dataset.yaml

@@ -0,0 +1,14 @@
+urn: urn:li:dataset:(urn:li:dataPlatform:snowflake,long_tail_companions.adoption.actuating,PROD)
+name: ACTUATING
+schema_metadata:
+  fields:
+    - id: pk
+      urn: urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:snowflake,long_tail_companions.adoption.actuating,PROD),pk)
+      type: "SchemaFieldDataTypeClass({'type': NumberTypeClass({})})"
+      nativeDataType: NUMBER(1,0)
+      nullable: true
+      recursive: false
+properties: {}
+subtypes:
+  - Table
+

metadata-ingestion/examples/data_product/dataproduct.yaml

@@ -22,6 +22,13 @@ tags:
 terms:
   - urn:li:glossaryTerm:ClientsAndAccounts.AccountBalance
 
+institutional_memory:
+  elements:
+    - title: URL for campaign
+      description: |-
+        Go here to see the campaign.
+      url: https://example.com/pet_of_the_week
+
 # Custom Properties
 properties:
   lifecycle: production

metadata-ingestion/src/datahub/api/entities/dataproduct/dataproduct.py

@@ -20,6 +20,8 @@
     GlobalTagsClass,
     GlossaryTermAssociationClass,
     GlossaryTermsClass,
+    InstitutionalMemoryClass,
+    InstitutionalMemoryMetadataClass,
     KafkaAuditHeaderClass,
     MetadataChangeProposalClass,
     OwnerClass,
@@ -74,13 +76,23 @@ def ownership_type_must_be_mappable_or_custom(cls, v: str) -> str:
         return v
 
 
+class InstitutionMemoryElement(ConfigModel):
+    url: str
+    description: str
+
+
+class InstitutionMemory(ConfigModel):
+    elements: Optional[List[InstitutionMemoryElement]] = None
+
+
 class DataProduct(ConfigModel):
     """This is a DataProduct class which represents a DataProduct
 
     Args:
         id (str): The id of the Data Product
         domain (str): The domain that the Data Product belongs to. Either as a name or a fully-qualified urn.
         owners (Optional[List[str, Ownership]]): A list of owners and their types.
+        institutional_memory (Optional[InstitutionMemory]): A list of institutional memory elements
         display_name (Optional[str]): The name of the Data Product to display in the UI
         description (Optional[str]): A documentation string for the Data Product
         tags (Optional[List[str]]): An array of tags (either bare ids or urns) for the Data Product
@@ -94,6 +106,7 @@ class DataProduct(ConfigModel):
     assets: Optional[List[str]] = None
     display_name: Optional[str] = None
     owners: Optional[List[Union[str, Ownership]]] = None
+    institutional_memory: Optional[InstitutionMemory] = None
     description: Optional[str] = None
     tags: Optional[List[str]] = None
     terms: Optional[List[str]] = None
@@ -251,6 +264,22 @@ def generate_mcp(
             )
             yield mcp
 
+        if self.institutional_memory and self.institutional_memory.elements:
+            mcp = MetadataChangeProposalWrapper(
+                entityUrn=self.urn,
+                aspect=InstitutionalMemoryClass(
+                    elements=[
+                        InstitutionalMemoryMetadataClass(
+                            url=element.url,
+                            description=element.description,
+                            createStamp=self._mint_auditstamp("yaml"),
+                        )
+                        for element in self.institutional_memory.elements
+                    ]
+                ),
+            )
+            yield mcp
+
         # Finally emit status
         yield MetadataChangeProposalWrapper(
             entityUrn=self.urn, aspect=StatusClass(removed=False)