Apache Atlas Update: Have you been watching?

It has been awhile since I’ve written anything.  Time to “catch up!”

A lot has been happening in the world of metadata management and governance.   We are now seeing many real life use cases, as machine learning, intelligent data classifications, graph database technology and more are being applied to the information governance domain.    Efforts for standardization in the metadata and governance space are moving forward also.  For this post, let’s take a look at Apache Atlas.

Apache Atlas continues to mature, celebrating several major milestones in 2017.  Shortly after its second birthday (Apache Atlas was launched as an incubator project in May of 2015), Apache Atlas graduated to a top level project status signifying that the project’s community and products have been well-governed under the Apache Software Foundation’s (ASF) meritocratic process and principles.  This is evidence of the hard work performed by the collective Apache Atlas team that Apache Atlas is increasingly ready for real world implementations.  Of course, that milestone, while worthy of recognition, is just one of the many steps Atlas is taking, and continues to make, going forward.  Here are other significant developments for Apache Atlas this year:

• Introduction of OMRS and its other complementary APIs.  OMRS is a key part of the Open Metadata framework that introduces the notion of repository metadata sharing and access.  In the true spirit of Apache communities, Apache Atlas is not alone in the world of enabling information governance; sharing of metadata between diverse metadata repositories can now be realized, in addition to simpler federation of metadata across multiple Atlas repositories.
• New common models for critical types of metadata.  To facilitate metadata sharing via OMRS, and to establish a more widely adaptable set of asset definitions, it was agreed by the Atlas team that a common definition for data structures, processes, and other data asset attributes.  This helps facilitate metadata sharing by increasing the likelihood that integrators building interfaces to Atlas will choose a common type definition for their content instead of designing their own custom types while providing extension points if needed.
• New Glossary Model.  A detailed new glossary model was designed (and API implemented) for a stronger semantic layer.  Business concepts and their relationships are the cornerstone of disciplined information governance.
• Streamlining of the Apache Atlas infrastructure.   The underlying graph database implementation was upgraded to take maximum advantage of JanusGraph, itself becoming the leading standard for open source graph engines.
• Continued/ongoing clean-up of the install and build procedures.  Considering the wider adoption of Apache Atlas throughout the governance community, Atlas team has enhanced test suites to assure that the new functionality added is well tested and the build and install processes are more streamlined..  For example, packaging and building Apache Atlas within Docker containers.
• The number of new Committers!  Apache, as everyone knows (or should know), is a meritocracy.  This means that recognition and influence is determined by an acknowledged investment of time, effort, and contributions.  Formal recognition as a committer requires many months of hard work to moving a project forward.  Congratulations to all the new Committers this year!    Even more important, the increase in Committers and contributors overall is yet another illustration of how Apache Atlas is growing in importance and general industry awareness.
• The Virtual Data Connector use case.  Self service data exploration environments need to provide an integrated view of data from many different systems and organizations.  Access is needed in order to discover new uses and interesting patterns in the data.   The VDC project aims to provide a single endpoint for accessing data that presents a virtualized view of the data assets with the appropriate data security.  This is accomplished by extending the integration of Apache Atlas with Apache Ranger via the tag-based security access introduced in Apache Atlas in 2016, in order to provide security access based on both the classification tags (eg PII and SPI tags, subject area of the data etc.)  An additional plug-in is added to Apache Atlas to control access to metadata based on whether an end-user is allowed to discover a data sources’ metadata.

So….it’s been a very busy year for Apache Atlas.  While most of these capabilities have already been developed and are being tested, they will become generally available in the upcoming Apache Atlas v1.0 which will be a huge milestone release for the community. The project is maturing, and gaining increased attention across the industry, in the information governance space, and beyond.   The code continues to mature, with increase in adoption and variety of applications every week.   The critical mass of industry expertise contributing to Apache Atlas continues to grow.    Start watching!   Start playing!  Join in and help Apache Atlas reach its next set of milestones!

–Ernie

 

Main Apache Atlas web site
Atlas Wiki

Links to specific Apache Atlas Topics

Open Metadata and Governance
Link to more details on OMRS
Building Out the Open Metadata Typesystem
Virtual Data Connector