Repository navigation
Support typed relationships and rich graph ingestion from Meteor #232
Description
Context
Compass currently stores lineage as directed upstream/downstream edges. As Meteor evolves to extract richer relationships, Compass needs to store and expose them. Additionally, entity resolution — matching observations from different sources into unified entities — belongs in Compass because it has the full graph context needed for resolution decisions.
Scope
- Schema evolution — Extend the Postgres schema to store edges with relationship types, not just lineage direction. Move from a lineage-specific table to a general-purpose edge table with type, source, target, and metadata.
- Ingestion API — Update the entity upsert endpoints to accept raw observations with typed relationships and cross-source hints from Meteor.
- Entity resolution — Build a resolution layer that matches incoming observations against the existing graph. When Meteor sends an observation from BigQuery and another from Tableau that reference the same logical entity, Compass recognizes them as one entity with multiple source facets. Resolution strategies:
- URN matching — exact or normalized URN overlap across sources
- Name + type matching — same name and compatible type across services
- Cross-source hint matching — shared identifiers annotated by Meteor
- Config-driven rules — explicit mapping rules for edge cases
- Entity identity storage — Store resolved entity identities. One logical entity, multiple source representations, queryable as either the unified entity or any individual facet.
- Backward compatibility — Existing clients sending only lineage data should continue to work.
Why
The relationship model and entity resolution are the foundation of the graph-native store. Every downstream feature — multi-hop traversal, graph-aware ranking, impact analysis — depends on having rich, typed relationships and unified entities stored in Compass.
Entity resolution belongs in Compass rather than Meteor because Compass holds the full graph state. It can compare incoming observations against everything it already knows, making resolution decisions with full context. This keeps Meteor stateless and simple — it delivers raw observations without needing to query existing state.
Dependencies
- Proto schema changes in
raystack/proton - Meteor relationship model and Compass sink protocol changes
Activity
ravisuhag commentedon Mar 29, 2026
Closing — core scope implemented in v2 entity system.
Typed, temporal, directed relationships are stored in the
edgestable with: source_urn, target_urn, type, properties (JSONB), valid_from/valid_to, and source provenance. Lineage is now just one edge type among many (reads_from, writes_to, owned_by, has_column, etc.). Graph traversal uses recursive CTEs with cycle detection.Entity upsert accepts upstreams/downstreams, and edges can be managed independently via
UpsertEdge/GetEdges/DeleteEdgeRPCs.Key files:
store/postgres/migrations/000001_init_schema.up.sql(edges table)store/postgres/edge_repository.go(Upsert, GetDownstream/GetUpstream with recursive CTE)core/entity/edge.go,core/entity/service.goUpsertEntity(with upstreams/downstreams),UpsertEdge,GetEdges,DeleteEdgeRPCsget_contextandimpacttools traverse the edge graphCross-source entity resolution beyond URN matching (name+type matching, hint-based matching) can be tracked as a separate issue if needed.