Metadata Management

Context

In the current landscape of data-driven enterprises, the overwhelming surge in data volume has led to significant challenges in data management, understanding, and maintaining the integrity of critical information. Organizations are grappling with intricate data webs, hindering effective decision-making. The absence of a comprehensive solution for data flow, lineage, and governance impedes the harnessing of the full potential of data assets, posing critical issues in data quality assurance, compliance, and seamless integration. There is a pressing need for a Visual Data Lineage Tool to address these complexities.

Challenges at a Glance

1. Data Obscurity and Complexity:

   • Enterprises struggle to comprehend the origin, transformation, and destination of data, hindering decision-making.

2. Data Quality Assurance:

   • Organizations face ongoing challenges in ensuring data accuracy and reliability without a transparent view of data lineage.

3. Compliance and Security Concerns:

   • Regulatory requirements and data security are jeopardized without a tool to visualize and track data lineage, posing risks to compliance and privacy.

Why Choose Anansi?

Anansi excels in visualizing how systems are connected to one another through its robust data lineage capabilities. Leveraging the power of Neo4j as a Graph Database foundation, Anansi provides a clear and intuitive representation of the relationships and connections within your data ecosystem.

• Specialization and Customization:

  • Unique features and customization options catering to the specific needs of the target audience.
  • Anansi is easy to configure based on your use case. Create type-safe Tables and Catalogs per your convenience.

• Ease of Use

  • Anansi is designed to be intuitive to use. You should be able to get up and running very easily.

• Flexibility Across Industries:

  • Generic and adaptable to various industry segments, attracting a broader user base.

• Graph Database Foundation:

  • Anansi's utilization of the Neo4j Graph DB ensures a solid foundation for storing and managing relationships within data, enhancing the effectiveness of data lineage tracking.

• Visualization And Understanding:

  • The platform offers a visually appealing and easy-to-understand representation of the interconnected data landscape. Users can explore and navigate through the graph to gain insights into how different systems and components are linked.
  • Provides a clear and intuitive platform to map, visualize, and understand the complete journey of data within an organization.

• Data Quality Assurance:

  • Ensures transparency in data lineage, aiding in the identification and prompt rectification of data quality issues.

• Compliance and Security Support:

  • Enables organizations to meet regulatory requirements and ensures data security and privacy through visual tracking of data lineage.

• Use in Real Time:

  • Enable users to explore and analyze information in real-time, gaining insights into datasets as they interact with the data

• Run on the Cloud, Data Center OR on your Laptop:

  • Anansi can be deployed on a Cloud Service for a hassle-free experience, in a Data Center for managing or analyzing sensitive data, or on a laptop for personal projects.

• Assign Data Ownership:

  • Data ownership in Anansi, involves assigning responsibility for specific datasets within an organization.

Example Metadata Mangement in Anansi

We take an example bank. A bank's data is organized in a clear hierarchy, with different levels. Using tools like Anansi, the bank can efficiently manage metadata, ensuring data quality. Here is an example 5-level heirarchy, but the hierarchy can extend to any number of levels.

1. Level 1 - Bank:

   • At the top level is the overarching entity, the Bank. This represents the highest organizational unit.

2. Level 2 - Systems:

   • In the Bank level, there are multiple systems. These systems could represent different departments, divisions, or functional areas within the bank.

3. Level 3 - Microservices and Databases:

   • Each of the multiple systems can contain microservices and databases. This level introduces a finer granularity, where microservices and databases are components within individual systems.

4. Level 4 - Tables:

   • Within each database, there are multiple tables. Tables are fundamental units for organizing structured data.

5. Level 5 - Columns:

   • At the lowest level, each table contains multiple columns. Columns represent individual data attributes or fields within a table.

Attributes

We can assign any number of attributes to a metadata. Anansi forces you to assign a type to each attribute that you created.

• Example Metadata: Context, Bank, System, Micro Services, DB, DB table, DB column, ...
• Example Attributes: Name, Description, Owner, Tags, Type of System, ...
• Example Types: String, Integer, Float, Date-Time, ...

Metadata Lineage

Data lineage is a visual representation that shows the movement and transformation of data from its origin (source) to its final destination (target), illustrating how data flows through different processes and systems in an organization.

In the image below, we use the node 'Summit' to check its lineage.

Owner

Assign Metadata Entities to an Owner

In the image below, we select the Account System Entity and add Bob as the owner for it.

Assign an Owner to Metadata Entities

In the image below:
We select Bob as the owner → Choose a catalog name → Choose a Node Name → Finally add entities like System1 and System3 to Bob.

Tag

Tags are like the hashtags (#) that you see on social media. They are used to search for similar data and group together similar data.

Assign Metadata Entities to a Tag

In the image below:
Choose a Catalog Name → Select the required Node Name → Add entities like System3 and System4 to the selected Tag.

Assign Tags to a Metadata Entity

In the image below, we select the Account System Entity and add the tags 'Summit' and 'Testingjan' to it.