Metadata enrichment is about scaling the onboarding of new data into a governed data landscape by taking data and applying the appropriate business terms, data classes and quality assessments so it can be discovered, governed and utilized effectively. This feature significantly increases the productivity of the data stewards who provide business context to data by ensuring data quality, usefulness and protection for broader consumption.

To take full advantage metadata enrichments, you’ll want to publish data assets, enriched with metadata‚ to a catalog where users can quickly find and access the right information. Ensuring the right business terms are assigned to data is the cornerstone for governance, as it defines what the business nature of the data is, and which policies and rules apply to its management and protection. Since business users want to search for data using business terminology with which they are familiar, it’s also key to the findability of data. If data is not well classified, this will impact the precision and recall of search results as well as the application of appropriate policies.

But how do your ensure that when a business user searches for data, the search goes beyond data silos to encompass your end-to-end data landscape? A data fabric is an architectural approach to simplify data access in an organization to facilitate self-service data consumption.  Agnostic to data environments, processes, utility and geography while integrating end-to-end data management capabilities, a data fabric architecture can automate data discovery, governance and consumption, enabling enterprises to manage data as a product. With a data fabric, enterprises elevate the value of their data by providing the right data, at the right time, regardless of where it resides.

At the heart of our data fabric solution is a data catalog that powers intelligent, self-service discovery of data models and more in a cloud-based enterprise metadata repository, called IBM Watson Knowledge Catalog. By bringing together infrastructure, technical and business metadata using common workflows, users can apply necessary governance and related policies. Ultimately this tool help users know their data, trust their data, protect their data and consume their data through metadata enrichment.

The new metadata enrichment feature of Watson Knowledge Catalog allows users to enrich their data with data quality analysis, profiling and automatic assignment with one click, and to deliver quality data faster, kick start governance and scale business understanding in an integrated experience. Using standard Watson Knowledge Catalog connectivity that’ss seamlessly integrated into IBM Cloud Pak for Data projects, data’s term assignment is powered by machine learning that isn’t constrained to a single global model, and can vary at the project level. This approach supports a setup where teams can easily test and use new models without impacting other teams, allowing for organizations to scale business understanding of data and, when needed, manually adjust the outcome of automated jobs.

Some of the new capabilities of Watson Knowledge Catalog include:

Entirely revisited workflow and user experience

The workflow and user experience for discovering, enriching and publishing a large number of data assets from a source to a catalog has been entirely revisited.  By separating the metadata import and the metadata enrichment processes, the new workflow better reflects the different user roles who typically perform these different steps.

Scalability and elasticity

Meta data enrichment can handle thousands of assets and their columns. With search and filters applied, users can now easily drill down find, enrich or curate their assets faster and kickstart the overall governance and quality programs.

Broad list of supported connections

Supported by an ever-growing list of natively supported connections, users can now connect to dozens of data sources.

More control, scheduling and insight into the runs

Thanks to the use of the common jobs framework of Cloud Pak for Data, users can now schedule runs, have insight into all the current and past runs, get information on key statistics and be notified about the progress in a way which is consistent with other platform services.

You can now not only schedule discovery process, but also define data scope of re-run so it is limited only to new or modified assets.

Support for BI reporting

With the reporting capability of Watson Knowledge Catalog, you can get insights from data, like seeing metrics related to the evolution of data dimensions, and export it to the external database.

Public API

With public API you can now manage metadata enrichment from external tools and workflows. See the official public APIs.

Project level settings

With the settings on the project level, there’s now a single-entry point to apply settings to multiple metadata enrichment assets at once. Users can set thresholds for the machine learning supported business term assignment and set categories that provide the scope of the business terms and data classes used.

Improved filters usability and bulk actions

With the broad set of filters, you can find relevant assets and columns much faster, than before and when needed bulk edit multiple columns or assets.

Read this article to see how to use metadata enrichment to implement data governance policies, and if you are interested in trying out the metadata enrichment feature of Watson Knowledge Catalog, check out the free trial.

The post Metadata enrichment – highly scalable data classification and data discovery appeared first on Journey to AI Blog.

Today, most enterprises use services from more than one Cloud Service Provider (CSP). Getting operational visibility across all vendors is a common pain point for clients. Further, modern architecture such as a microservices architecture introduces additional operational complexity.

Figure 1 Hybrid Multicloud and Complexity Evolution

Traditionally this calls for more manpower. But this traditional approach introduces more challenges. As shown in the following diagram, an issue in the environment triggers several events across the full stack of the business solution. This results in an unmanageable event flood. Moreover, there are often duplicate events due to full-stack level observability and these events result in data silos.

Figure 2 IT Service Management Complexity

IT is a critical part of every enterprise today, and even a small service outage directly affects the top line. Consequently, it is not uncommon for clients to ask for a 30-minute resolution commitment when something goes wrong. This is usually not enough time for a human to resolve an issue.

What is the solution?

This is where AIOps comes to the rescue, preventing these issues before they occur. AIOps is the application of artificial intelligence (AI) to enhance IT operations. Specifically, AIOps uses big data, analytics, and machine learning capabilities to do the following:

• Collect and aggregate the huge and ever-increasing volumes of operations data generated by multiple IT infrastructure components, applications and performance-monitoring tools
• Identify significant events and patterns related to system performance and availability issues
• Diagnose root causes and report them to IT for rapid response and remediation, or automatically resolve these issues without human intervention

By replacing multiple manual IT operations tools with an intelligent, automated platform, AIOps enables IT operations teams to respond more quickly and proactively to slowdowns and outages, with less effort. It bridges the gap between an increasingly difficult-to-monitor IT landscape and user expectations for little to no interruption in application performance and availability. Most experts consider AIOps the future of IT operations management.

How could we reimagine cloud service management and operations with AI?

Refer to the lower part of the diagram below (box 3: Environment), which represents the environments where the workloads run. Continuous releases and deployments of these applications are typically achieved through the continuous delivery process and tooling that is shown on the left side of the diagram (box 2: Continuous Delivery).

Figure 3 AI Infused DevSecOps and IT Control Tower

The applications continuously send telemetry information into the operational management tooling (box 4: Continuous Operations). Both the continuous delivery tooling and the continuous operations tooling ingest all the data into the AIOps engine shown at the top (box 7: AIOps Engine). The AIOps engine is focused on addressing four key things:

1. Descriptive analytics to show what happened in an environment
2. Diagnostics to show why it happened
3. Predictive analytics to show what will happen next
4. Prescriptive analytics to show how to achieve or prevent the prediction

In addition to this, enterprise-specific data sources such as a shift roster, SME skill matrix or knowledge repository enrich the AIOps engine (box 1: Enterprise specific data).

Additionally, the AIOps engine consumes public domain data such as open-source communities, product documentations and sentiments from social networks (box 6: Public domain content). ChatOps and Runbook Automation ingest the insights and the automation that the AI system produces and leverage it to establish the new day in the life of an incident (box 5: Continuous Operations). ChatOps brings humans and chatbots for conversation-driven collaboration or conversation-driven DevOps. Additionally, the AIOps engine also dynamically reconfigures the DevSecOps tools, providing continuous delivery and continuous operations through AI-derived policy ingestion.

Several products in the marketplace have already evolved to provide AIOps capabilities such as an anomaly detection feature. This framework consumes the outcomes provided by these AIOps engines (denoted as edge analytics in Figure 3) and combines multiple sources to provide an enterprise-level view.

IT processes such as incident/problem-resolution processes are ad hoc in nature. They differ greatly from structured business processes such as loan approval processes or claim settlement processes. IT processes have stringent SLAs due to the high cost of outage to the business, and the persona involved collaborate intensely and interact with disparate tools to accomplish their goals. Applying business process automation technologies to IT processes will not yield high productivity benefits. ChatOps have transformed the way ITOps teams collaborate to resolve IT incidents. AIOps and ChatOps are the appropriate tools to drive productivity in IT processes. ChatOps enhances the collaboration experience of SRE with other personas participating in IT processes. AIOps delivers insights for SRE to accelerate incident resolution process.

In a nutshell, as clients undertake large digital transformation programs based on a hybrid cloud (or multicloud) architecture, IT Operations needs to be reimagined. With ever increasing complexity, AIOps is an indispensible. To know more about AI for IT Operations and IBM PoV, refer to IBM Consulting.

The post AIOps reimagines hybrid multicloud platform operations appeared first on Journey to AI Blog.