New Horizons for Lineage!

Hi Everyone,

It has been awhile since I’ve written any observations about lineage and our world of data integration and metadata. I’ve been spending time these past four years listening and learning. During that time I have been able to gather more insights about lineage, data governance and management, the evolution of open-source (for lineage and metadata), and hot new trends like “Observability” and “Active Metadata.”

With that in mind, it’s time to take this blog to places it hasn’t been before — to new technologies and methods for defining, qualifying, and establishing lineage. But also to new use cases and new ways of thinking about meeting the challenges being faced while understanding your data pipelines.

I have gained a deeper appreciation for tracking lineage (not only for Classic DataStage as covered for many years in this blog) but also for a myriad of other Extract-Transform-Load (ETL) solutions, known and unknown relational databases, a vast array of long outdated on-premises tools, sparkling new cloud-based data shaping solutions, as well as the next generation of DataStage!

Stay tuned! It’s been a fun journey and fulfilling learning experience for me, and I look forward to sharing it with you, starting with a series of posts that look closely at the need for something called Descriptive Lineage.

In the meantime, have some fun with this recent experiment in presenting lineage in Virtual Reality!

• Futuristic demo of lineage in three dimensions!  Zoom session VR lineage demo. Watch this lab session where you can see the end result while also watching the speaker wearing a VR headset.
• Short teaser video for Lineage VR:  The Future of Lineage?
• Try it yourself! Move your mouse to look around in this active 360 degree view! https://www.youtube.com/watch?v=7kajf8tT-HY

-Ernie