Posted on

Generating #datavault models & Issues to address | Accelerated Business Intelligence

Generating #datavault models & Issues to address | Accelerated Business Intelligence

I agree with Dan regarding his post on tools that generate DataVault models. Most of the tools that I have evaluated don’t go any further than generating a source DataVault as both he and Ronald Damhof have stated in posts long ago.

Without the necessary metadata and mathematical and technical stuff that Dan refers to, automatic model generation is not adding much value. And I wonder if we will ever get there. Even today – with the knowledge that we need lots of metadata – source systems get built without the badly needed metadata that would support automatic model generation from them. Most database systems don’t support storing all that metadata either as far as I know.

However I do see some added value with these (model generation) tools when we – the data modelers – create the needed metadata or semantic layer between the source system and the DataVault. Some of the tools even implicitly rely on that already. This semantic layer is the actual business view, the translation of the source model. It is data modeling by hand, just not necessarily with the usual data modeling tools. The model generation tools – when properly created – could support the data model governance lifecycle.

Posted on

First thoughts on using a graph database for metadata

It’s been a while since I briefly looked into graph databases for the purpose of using them for metadata. As I have a metadata “problem” at hand, I decided to look into it again and just as last time, opted for Neo4j.

The problem I have is relatively simple. The BI solution at a client has many documents describing business data domains, data models (well, a sort of data models), mapping documents, mapping rules etc. These documents are stored on a file server in several directories, often arranged by change request. There is no good procedure in place yet – but in the works – to make sure there is a central place to store the documentation that actually describes the production state.

To make things worse, documents exist in different versions that were sometimes incorrectly “forked” into a newer version, resulting in changes getting lost from other versions. On top of the cake, the intend is to move all documentation over to Microsoft SharePoint (may the force be with us…).

In addition we have DDL in files and ETL in Informatica PowerDesigner. Due to the fact that naming conventions were sometimes a bit forgotten, it is pretty difficult to tie all documentation together. There are mapping rules/functions described in Word documents that are referenced in mapping specs in Excel sheets, but nobody knows which ones etc. etc. You get the drift.

So as we work our way in making a decent inventory of this metadata (at least that is how I see these documents, DDL and ETL code) I thought it would be a good time to make the relationships between this metadata explicit (as the way SharePoint is going to be used won’t help us here).

And that’s exactly where I see a good use of a graph database, simply because these are very strong in describing the relationships between things.

My first rough thoughts, without really thinking it all through, is a graph model like the following: