Schema Registry

The registry acts as a single source of truth for data schemas, storing them in a centralized repository. It supports immutable versioning, allowing systems to evolve their data formats while maintaining backward and forward compatibility. Key functions include:

• Schema IDs: Unique identifiers for each schema version.
• Version History: A complete audit trail of all schema changes.
• Retrieval API: Allows producers and consumers to fetch schemas by ID or subject.

The registry enforces data integrity by validating that messages conform to their registered schema before they are produced. It uses compatibility rules (e.g., BACKWARD, FORWARD, FULL) to check if a new schema version can safely read data written with older versions and vice-versa. This prevents breaking changes from disrupting downstream consumers.

Instead of sending raw data, producers serialize messages by embedding a compact schema ID reference. Consumers use this ID to fetch the schema from the registry and deserialize the message. This approach:

• Reduces Payload Size: Transmits an ID instead of the full schema.
• Decouples Systems: Producers and consumers only need to agree on the registry, not binary formats.
• Enables Evolution: Consumers can handle new data formats if they are compatible.

Provides tools for managing the schema lifecycle and enforcing organizational policies. Common features include:

• Ownership & Metadata: Assign schemas to teams or projects with descriptive metadata.
• Access Control Lists (ACLs): Restrict who can create, read, or update schemas.
• Lifecycle Management: Define rules for schema deprecation and deletion.
• Audit Logging: Track all schema-related operations for compliance.

Schema registries are typically deployed alongside message brokers like Apache Kafka or event streaming platforms. They integrate via serializers/deserializers (SerDes) plugins. For example, a Kafka producer using Avro serialization will automatically communicate with the registry to validate and tag outgoing messages with the correct schema ID.

These users produce and manage the schemas. They use the registry to:

• Enforce data contracts between services in an event-driven architecture (e.g., Apache Kafka).
• Validate that data produced to a topic adheres to the defined schema before it's written.
• Manage schema evolution (e.g., adding optional fields) without breaking downstream consumers.
• Centralize schema definition and versioning, replacing ad-hoc documentation.

These are the consumers of the schemas. They rely on the registry to:

• Automatically generate client code (e.g., Java, Python classes) from the schema definitions.
• Deserialize incoming data streams with confidence, knowing the structure is validated.
• Discover available data streams and their structures without manual coordination.
• Ensure their applications remain compatible as schemas evolve over time.

This group uses the registry for data discovery and understanding. It serves as a single source of truth for:

• Schema metadata, including field names, data types, and descriptions.
• Lineage information, showing where data originates and how it flows.
• Understanding the semantic meaning of fields before building models or running queries.
• This reduces time spent on data wrangling and prevents errors from misinterpreted data structures.

These users are responsible for the governance, security, and reliability of the data platform. They use the registry to:

• Implement access control policies (e.g., who can publish or read a schema).
• Audit schema changes and track compliance with data governance rules.
• Monitor schema usage and compatibility across the entire ecosystem.
• Integrate the registry with CI/CD pipelines to test schema changes before deployment.

Developers of ETL tools, BI platforms, and connectors (e.g., for databases like Snowflake or BigQuery) use schema registries to build dynamic, type-safe integrations. The registry allows their tools to:

• Auto-discover and adapt to new data sources and their schemas.
• Generate accurate target schemas for data transformation and loading.
• Provide users with real-time validation and schema previews within their UI.
• This is a key component for modern data stack interoperability.

An Attestation is a signed piece of data that conforms to a schema, issued by an attester about a subject. It is the primary data object created and verified using a Schema Registry.

• Key Components: A unique UID, the schema it references, the attester's signature, and the actual data payload.
• Example: A KYCVerified attestation containing a user's passport hash and verification date, signed by a licensed entity.

A Schema UID (Unique Identifier) is a deterministic hash that uniquely represents a registered schema on a specific registry. It is the immutable reference point for all related attestations.

• Calculation: Derived from the schema's string, resolver contract, and revocable flag.
• Critical Role: Enables off-chain systems to fetch and validate the schema definition before processing any attestation data.

A Resolver Contract is an optional, on-chain logic module attached to a schema during registration. It enforces custom validation or execution rules before an attestation is issued or revoked.

• Purpose: Adds business logic, like checking if the attester holds a specific NFT or if the subject has paid a fee.
• Example: A resolver for a LoanRepayment schema could verify the payment transaction exists before allowing the attestation.

The Attestation Graph is the interconnected network of all attestations linked by their schemas and subjects. It forms a verifiable web of trust and reputation data.

• Structure: Nodes are Ethereum addresses (subjects/attesters), and edges are attestations (signed data).
• Use Case: Enables complex queries like "Find all developers attested as SolidityAuditor by entities I trust."

A Schema Registry distinguishes between data stored on-chain versus referenced off-chain.

• On-chain: The schema definition (UID, fields), attestation metadata (timestamp, attester), and revocation status are stored on the blockchain.
• Off-chain: The actual data payload of an attestation (e.g., a PDF hash, a JSON object) is typically stored in decentralized storage (like IPFS or Arweave) or a private server, with only a reference hash stored on-chain.