By now, you’ve most probably heard about data warehouse automation. Trends, data and development styles in a data-driven environment change fast and often. Automation in any form can help you develop your solutions faster. One tool and platform that can help you with this is the WhereScape tech stack. If you’ve been thinking over the idea of data warehouse automation, you’ve probably heard of WhereScape. But if not, here’s a brief introduction.
WhereScape is a data warehouse automation platform that allows for the development of agile data warehouses. With the use of technology and control systems, data warehouse automation achieves economies of scale while maintaining a predictable quality level. It enables you to design, document, implement, monitor and manage the complete data warehouse with a single tool, rather than several.
WhereScape RED and WhereScape 3D are mostly used for the mentioned automation. RED is focused on working with your metadata for the warehouse and its usage. RED keeps all the information about structures, scripts, templates, routines, and so on, and can be used as a stand-alone tool. Using RED, you can design, deploy, and manage your whole data warehouse with ease, effectivity and, most importantly, governance. It’s a very powerful tool when used efficiently. It covers the DWH use case from start to finish and since you can do almost anything you are able to script, it can deliver a lot of complexity. In the end, you can build your whole warehouse simply with this tool and create and schedule jobs that will operate it.
We’ve talked about the more automated way of doing things. This is where Wherescape 3D comes into play. 3D is a modeling tool that leverages automated generation based on defined design. It doesn’t matter if you use 3NF, DataVault or another design pattern. 3D can speed up your repetitive tasks by automating the generation. This modeler tool is focused on customizable workflows that generate your objects. Then you simply push them to RED and work with them there. This is probably most useful when doing DataVault designs, where WhereScape has a lot of already created workflows, but it can help in any automation case.
In my eyes, using only WhereScape RED is a huge waste of automation potential, because the synergy that 3D and RED bring together really speeds things up. As mentioned, the stack can get very complex and can be a bit confusing.
Both WhereScape RED and 3D are very powerful and configurable tools. With the option of running scripts from both 3D and RED, you can achieve almost anything. As I like to add, if you wish, you can even play your favorite YouTube video in a browser at the end of deployment. WhereScape will support almost any technical feature you can think of. Most are included or they are available through enablement packs and so forth, but be aware of possible complexity if you bring custom-developed items into your stack.
This advice goes with the core recommendation for deploying the product. Since it is a highly customizable tool, it will bring major complexity if you start pursuing custom paths from the very beginning. In most cases, you will, further down the road, find that the feature you sought was there and that you just overengineered at the beginning. Start easy, use most of the predefined templates and workflows and when you see something does not work for you, change it. But do not reinvent the wheel, because someone has probably already done it for you.
Wherescape 3D will help you quickly create your designs by leveraging automation and deploying fast. Use it as often as you can. With 3D and RED combinations, you can get your design into your warehouse in minutes. You will then be able to easily check whether it is working as you intended. By this, I don’t mean rushing blindly with your development, but utilizing the perfect option to iterate and master your solution with incremental and testable steps.
During my work for different clients, I tend to encounter an ambition to put all things in one place. This is not in some cases the best approach. 3D is good at some things, while other things are much faster in RED (custom transformations, more advanced rules, and so on). Don’t push it all into one place simply to have it all there. With the latest 3D version there is the useful option of using your RED objects as sources for your next designs. If you find out that something is much easier for you in RED, do that thing there and then use it as an input for further designs. For example, we can provide the creation of your business vault. You generate a data vault in 3D, then in RED you create your custom business transformations and use the transformations as sources for the generation of a predefined “data mart” layer.
Now and again, I see teams that get the urge to automatize everything. It usually happens in a few instances that occur during irregular activities. Examples often relate to templates for initial loads and fixes from bad design choices. In both such circumstances, the option of a workaround, then the editing of all your workflows and templates for each specific use can be the better option. I encourage you to think about how often an adjustment would really help you and if it would be beneficial to automate it.
The last point on the list is governance. The tools themselves bring some degree of governance to the system itself, but things can get messy and confusing when not approached properly. I encourage everyone working with a WhereScape stack to create good naming conventions for the 3D model structure, so that everyone knows what is where even after a few hundred models have been created and stored in the repository. The same applies to RED groups and projects. Create for yourself a hierarchy that will make sense for the various cases of use. There could be even more angles to this. You can have one structure based on your development cycle (JIRA IDs, for example) and a second based on your business meaning. Do things in such a way that people working can easily navigate inside your repository as effortlessly as possible. It will save you a lot of time.
This was an introduction to the WhereScape stack with some useful tips that will help you to get started.
Stay tuned for the next episode!
