I have been doing data warehousing for as long as I can remember. In many forms – sometimes “operational”, sometimes “analytical”, sometimes Kimball, sometimes Inmon. In most cases a hybrid. I pride myself on being technology agnostic and really not concerning myself too much with all the bells and whistles that a particular tool can provide, be that an ELT/ETL tool, a particular modelling tool, a report writer or analytical cube tool. For me, the reason I build a data warehouse is to satisfy a particular requirement for a particular user, group or department – in 99% of the cases the end user actually does not care how fancy your pattern is, or that you have “factless facts” or “hot swappable dimensions” they just want the data. THEY REALLY DO NOT CARE ABOUT HOW YOU GOT THERE, just that you did get there.
As of late, this beast called Agile has been sprouting up on most projects, and has definitely started gaining impetus in Data Warehousing. I think the transactional folks can consider themselves lucky, they have a ten year head-start over us warehousing folks. I will try not to re-hash what you can Google in this post, but I will add some really cool links that you can follow. If you need to brush up on your Agile, then head off to this article – Agile Methodology, those folks should explain in one paragraph what it is as well as what the scrum is. There are many variants of the methodology, this article provides good insight into Scaled Agile Framework, SAFe.
The purpose of this blog is to look at Agile in Data Warehousing. Specifically this post looks at a methodology called “The Scrum”. It is the first in a series of posts – each one will focus on the values of agile (as laid out in the agile manifesto). What do these values bring to my data warehouse? How can I apply these values? And what mindset change do I need to go through to get there. Following on directly from this, I will be unpacking the 12 Principles of Agile.
I am loving my current project (which shall not be named), as I finally have a leader who gets Agile, and who is forever trying to change this old hat (yours truly) from a waterfall guy to a leaner and meaner agile guy. Getting there, but the journey is quite an epic story……
On our particular project, we are using Scrum. Scrum what? Scrum is a method which, at the heart of it all, empowers your team to be autonomous. If you are used to having “developers” and “testers” and “analysts”, think of this as the catalyst to cross-skilling everyone to do everything. So of course, you will have blowback – no developer enjoys documentation and analysts refer to developer work as “the geeky stuff” the whole time.
Similarities to rugby
So let’s say we look at rugby (the best sport that has ever been created). In rugby the objective of the game is to get the ball over the line and to prevent your opponent from doing the same. Very similarly in agile, the objective is to deliver a piece of quality work with the least amount of rework in the fastest amount of time.
In rugby you have structure – you have props, you have your loose forwards, you have a back line and you have this brilliant guy bringing everything together called your fly half. In agile, you would liken props to your analytical minds (the first who see the ball). You would liken your loose forwards to your more hard-core developers (those guys who can do analytical stuff but who you really do need to focus on development first and foremost). You would liken your back line to your testing minds (those who see the ball last, those guys who you use within development but who also quality assure). And then there is the fly half – the scrum master. In rugby, the fly half directs play, he ultimately decides if the forwards should bash it up or he should let the back line run, if he should run himself or if he should kick it out. In agile, the scrum master has a similar role – he controls the tasks within a user story and decides which “player” needs to be called on and when.
You also have something called the coach, the guy who makes sure the forwards know how to drive and ruck and maul, they guy who makes sure the inside centre knows his channel and the outside centre knows how to cut a gap. In agile, you would liken the coach to the product owner. He doesn’t essentially need to be in the game to make an impact. But he does need to know how much effort is required to produce a particular piece of work. He needs to work out from his available plays, which is the best one to play at which time. He also needs to know when to call on the “reserves” (architecture) how to block out the “noise of the press” (things that don’t matter to the product) and when it is time to bolster up the squad.
Each rugby player has a predefined role to play, but if you have ever watched a game you will know that in broken play the loose forwards sometimes join the back line and the back line sometimes joins the rucks and mauls. The same goes for an agile scrum – the roles are very flat. In fact, you only have 3 actual “titles” – developers, scrum master and product owner. This scares some folks, but it leads to beautiful team cohesion. In rugby, the team with the most cohesion wins. In agile scrum, the team with the most cohesion delivers.
In my next blog, I will go into a whole lot more detail about the scrum. I would love to hear from you, please feel free to comment.