Skip to content

Commit

Permalink
docs(relationship): update relationship docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@david-leifker
david-leifker committed Nov 7, 2024
1 parent b3d3b63 commit 3fef0de
Show file tree
Hide file tree
Showing 4 changed files with 50 additions and 140 deletions.
119 changes: 44 additions & 75 deletions docs/what/relationship.md
View file
Original file line number Diff line number Diff line change
@@ -1,107 +1,76 @@
# What is a relationship?

A relationship is a named associate between exactly two [entities](entity.md), a source and a destination.
A relationship is a named associate between exactly two [entities](entity.md), a source and a destination.


<p align="center">
<img width="70%" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/metadata-modeling.png"/>
</p>


From the above graph, a `Group` entity can be linked to a `User` entity via a `HasMember` relationship.
Note that the name of the relationship reflects the direction, i.e. pointing from `Group` to `User`.
This is due to the fact that the actual metadata aspect holding this information is associated with `Group`, rather than User.
Had the direction been reversed, the relationship would have been named `IsMemberOf` instead.
See [Direction of Relationships](#direction-of-relationships) for more discussions on relationship directionality.
A specific instance of a relationship, e.g. `urn:li:corpGroup:group1` has a member `urn:li:corpuser:user1`,
From the above graph, a `Group` entity can be linked to a `User` entity via a `HasMember` relationship.
Note that the name of the relationship reflects the direction, i.e. pointing from `Group` to `User`.
This is due to the fact that the actual metadata aspect holding this information is associated with `Group`, rather than
User.
Had the direction been reversed, the relationship would have been named `IsMemberOf` instead.
See [Direction of Relationships](#direction-of-relationships) for more discussions on relationship directionality.
A specific instance of a relationship, e.g. `urn:li:corpGroup:group1` has a member `urn:li:corpuser:user1`,
corresponds to an edge in the metadata graph.

Similar to an entity, a relationship can also be associated with optional attributes that are derived from the metadata.
For example, from the `Membership` metadata aspect shown below, we’re able to derive the `HasMember` relationship that links a specific `Group` to a specific `User`. We can also include additional attribute to the relationship, e.g. importance, which corresponds to the position of the specific member in the original membership array. This allows complex graph query that travel only relationships that match certain criteria, e.g. "returns only the top-5 most important members of this group."
Similar to the entity attributes, relationship attributes should only be added based on the expected query patterns to reduce the indexing cost.
Relationships are meant to be "entity-neutral". In other words, one would expect to use the same `OwnedBy` relationship
to link a `Dataset` to a `User` and to link a `Dashboard` to a `User`.
As Pegasus doesn’t allow typing a field using multiple URNs (because they’re all essentially strings), we resort to
using generic URN type for the source and destination.
We also introduce a `@Relationship` [annotation](../modeling/extending-the-metadata-model#relationship) to
limit the allowed source and destination URN types.

```
namespace: com.linkedin.group
import com.linkedin.common.AuditStamp
import com.linkedin.common.CorpuserUrn
/**
* The membership metadata for a group
*/
record Membership {
/** Audit stamp for the last change */
modified: AuditStamp
/** Admin of the group */
admin: CorpuserUrn
/** Members of the group, ordered in descending importance */
members: array[CorpuserUrn]
}
```

Relationships are meant to be "entity-neutral". In other words, one would expect to use the same `OwnedBy` relationship to link a `Dataset` to a `User` and to link a `Dashboard` to a `User`. As Pegasus doesn’t allow typing a field using multiple URNs (because they’re all essentially strings), we resort to using generic URN type for the source and destination.
We also introduce a `@pairings` [annotation](https://linkedin.github.io/rest.li/pdl_migration#shorthand-for-custom-properties) to limit the allowed source and destination URN types.

While it’s possible to model relationships in rest.li as [association resources](https://linkedin.github.io/rest.li/modeling/modeling#association), which often get stored as mapping tables, it is far more common to model them as "foreign keys" field in a metadata aspect. For instance, the `Ownership` aspect is likely to contain an array of owner’s corpuser URNs.
While it’s possible to model relationships in rest.li
as [association resources](https://linkedin.github.io/rest.li/modeling/modeling#association), which often get stored as
mapping tables, it is far more common to model them as "foreign keys" field in a metadata aspect. For instance,
the `Ownership` aspect is likely to contain an array of owner’s corpUser URNs.

Below is an example of how a relationship is modeled in PDL. Note that:
1. As the `source` and `destination` are of generic URN type, we’re able to factor them out to a common `BaseRelationship` model.
2. Each model is expected to have a `@pairings` annotation that is an array of all allowed source-destination URN pairs.
3. Unlike entity attributes, there’s no requirement on making all relationship attributes optional since relationships do not support partial updates.

1. This aspect, `nativeGroupMembership` would be associated with a `corpUser`
2. The `corpUser`'s aspect points to one or more parent entities of type `corpGroup`

```
namespace com.linkedin.metadata.relationship
namespace com.linkedin.identity
import com.linkedin.common.Urn
/**
* Common fields that apply to all relationships
* Carries information about the native CorpGroups a user is in.
*/
record BaseRelationship {
/**
* Urn for the source of the relationship
*/
source: Urn
/**
* Urn for the destination of the relationship
*/
destination: Urn
@Aspect = {
"name": "nativeGroupMembership"
}
```

```
namespace com.linkedin.metadata.relationship
/**
* Data model for a has-member relationship
*/
@pairings = [ {
"destination" : "com.linkedin.common.urn.CorpGroupUrn",
"source" : "com.linkedin.common.urn.CorpUserUrn"
} ]
record HasMembership includes BaseRelationship
{
/**
* The importance of the membership
*/
importance: int
record NativeGroupMembership {
@Relationship = {
"/*": {
"name": "IsMemberOfNativeGroup",
"entityTypes": [ "corpGroup" ]
}
}
nativeGroups: array[Urn]
}
```

## Direction of Relationships

As relationships are modeled as directed edges between nodes, it’s natural to ask which way should it be pointing,
or should there be edges going both ways? The answer is, "doesn’t really matter." It’s rather an aesthetic choice than technical one.
As relationships are modeled as directed edges between nodes, it’s natural to ask which way should it be pointing,
or should there be edges going both ways? The answer is, "doesn’t really matter." It’s rather an aesthetic choice than
technical one.

For one, the actual direction doesn’t really impact the execution of graph queries. Most graph DBs are fully capable of traversing edges in reverse direction efficiently.
For one, the actual direction doesn’t really impact the execution of graph queries. Most graph DBs are fully capable of
traversing edges in reverse direction efficiently.

That being said, generally there’s a more "natural way" to specify the direction of a relationship, which closely relate to how the metadata is stored. For example, the membership information for an LDAP group is generally stored as a list in group’s metadata. As a result, it’s more natural to model a `HasMember` relationship that points from a group to a member, instead of a `IsMemberOf` relationship pointing from member to group.
That being said, generally there’s a more "natural way" to specify the direction of a relationship, which closely relate
to how the metadata is stored. For example, the membership information for an LDAP group is generally stored as a list
in group’s metadata. As a result, it’s more natural to model a `HasMember` relationship that points from a group to a
member, instead of a `IsMemberOf` relationship pointing from member to group.

## High Cardinality Relationships

See [this doc](../advanced/high-cardinality.md) for suggestions on how to best model relationships with high cardinality.
See [this doc](../advanced/high-cardinality.md) for suggestions on how to best model relationships with high
cardinality.
57 changes: 0 additions & 57 deletions li-utils/src/main/java/com/datahub/util/validator/RelationshipValidator.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,6 @@ public static void validateRelationshipSchema(@Nonnull RecordDataSchema schema)
"Relationship '%s' contains a field '%s' that makes use of a disallowed type '%s'.",
className, field.getName(), field.getType().getType());
});

validatePairings(schema);
}

/**
Expand Down Expand Up @@ -109,59 +107,4 @@ public static void validateRelationshipUnionSchema(
relationshipClassName);
}
}

private static void validatePairings(@Nonnull RecordDataSchema schema) {

final String className = schema.getBindingName();

Map<String, Object> properties = schema.getProperties();
if (!properties.containsKey("pairings")) {
ValidationUtils.invalidSchema(
"Relationship '%s' must contain a 'pairings' property", className);
}

DataList pairings = (DataList) properties.get("pairings");
Set<Pair> registeredPairs = new HashSet<>();
pairings.stream()
.forEach(
obj -> {
DataMap map = (DataMap) obj;
if (!map.containsKey("source") || !map.containsKey("destination")) {
ValidationUtils.invalidSchema(
"Relationship '%s' contains an invalid 'pairings' item. "
+ "Each item must contain a 'source' and 'destination' properties.",
className);
}

String sourceUrn = map.getString("source");
if (!isValidUrnClass(sourceUrn)) {
ValidationUtils.invalidSchema(
"Relationship '%s' contains an invalid item in 'pairings'. %s is not a valid URN class name.",
className, sourceUrn);
}

String destinationUrn = map.getString("destination");
if (!isValidUrnClass(destinationUrn)) {
ValidationUtils.invalidSchema(
"Relationship '%s' contains an invalid item in 'pairings'. %s is not a valid URN class name.",
className, destinationUrn);
}

Pair pair = new Pair(sourceUrn, destinationUrn);
if (registeredPairs.contains(pair)) {
ValidationUtils.invalidSchema(
"Relationship '%s' contains a repeated 'pairings' item (%s, %s)",
className, sourceUrn, destinationUrn);
}
registeredPairs.add(pair);
});
}

private static boolean isValidUrnClass(String className) {
try {
return Urn.class.isAssignableFrom(Class.forName(className));
} catch (ClassNotFoundException e) {
throw new RuntimeException(e);
}
}
}
10 changes: 6 additions & 4 deletions test-models/src/main/pegasus/com/datahub/test/testing/RelationshipBar.pdl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,9 @@ import com.linkedin.common.Urn
/**
* For unit tests
*/
@pairings = [ {
"destination" : "com.linkedin.common.urn.Urn",
"source" : "com.linkedin.common.urn.Urn"
} ]
@Aspect = {
"name": "relationshipBar"
}
record RelationshipBar {

/**
Expand All @@ -19,5 +18,8 @@ record RelationshipBar {
/**
* For unit tests
*/
@Relationship = {
"name": "RelationshipBar"
}
destination: Urn
}
4 changes: 0 additions & 4 deletions test-models/src/main/pegasus/com/datahub/test/testing/RelationshipFoo.pdl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,6 @@ import com.linkedin.common.Urn
/**
* For unit tests
*/
@pairings = [ {
"destination" : "com.linkedin.common.urn.Urn",
"source" : "com.linkedin.common.urn.Urn"
} ]
record RelationshipFoo {

/**
Expand Down

0 comments on commit 3fef0de

Please sign in to comment.