Data Lake Vs Data warehouses
Today, my post is not technical. I would rather like to reflect on a new concept that I have been reading about Data Lakes in comparison to Data warehouses.
Data warehouse has been around for a long time and it has proven to be effective and therefore widely used to fulfill information needs of organizations. On the other hand, Data Lake is a new phenomenon which came to existence following the boom of high data generation aka big data flood. I have read that 90% of today’s data is generated in the last 2 years. This is massive and obviously traditional data warehouses are not able to handle this. Today’s data is not just large in size. It also comes in various forms and disseminates at a very high speed. These characteristics are famously described in 3V’s (Volume – to refer to its large size, Variety – to refer to its various forms, and Velocity – to refer to the speed at which today’s data disseminates). There are other 3V’s that I came across at InsideBigdata blog as Veracity – to refer to the fact that big data is susceptible to biases and is full of noise and abnormality; Validity – meaning is the data correct and accurate for the intended use; and Volatility – refers to how long is data valid and how long should it be stored. (For more detail, see the link: http://insidebigdata.com/2013/09/12/beyond-volume-variety-velocity-issue-big-data-veracity/). Now that Data Lakes are designed in order to handle today’s large and complex data, the question is “how is it different from Data warehouse and is it intended to replace Data warehouses?”. With this in mind, I tried to investigate different views and compiled as follows.
To start with the definitions:
A data lake is defined as a storage repository that holds a vast amount of raw data in its native format until it is needed, and
A Data warehouse is defined as a subject-oriented, integrated, time-variant and non-volatile collection of data in support of the organisation’s decision making processes. The purpose of data warehouse is to integrate data from many transnational and operational systems, in order to present the resulting information as a “single integrated version of the truth” to decision makers on all levels of the organisation. The design of the data warehouse, if done properly, allows them to access, report and analyse that information from all the relevant and possible angles; in order to get consistent and accurate information as a result. Usually, different data marts to serve as the access layer of the data warehouse environment that are used to get data out to the users are built. A data mart is a subset of the data warehouse that is usually oriented to a specific business line or team. Data marts are small slices of the data warehouse.
However, there are many critiques against the data warehouse and data mart approaches. One of thse is that its careful design and subsequent implementation takes time and effort. A related critique to this is that the dimensional structures used in data warehouse and data mart are inflexible and limiting.
The proponents of Data lake argue that these problems can be avoided easily. James Dixon, the founder and CTO of Pentaho, who coined the term for the first time describes a data lake as follows.
“If you think of a data mart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.”. Many data scientists agree today that a data lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured, and unstructured data. The data structure and requirements are not defined until the data is needed. This provides flexibility for since the requirements can be defined anytime later by the business and yet the same data lake can be used to satisfy the requirement.
Due to these facts, some say “Yes” loudly to data lake and some say “No” loudly to it and in favour of data warehous. The similarity they share is simple: they both serve as storage. In order to take side, i believe it is important to understand the differences. The following table taken from kdnuggets (http://www.kdnuggets.com/) summarizes the differences.
Data:- A data warehouse only stores data that has been modeled/structured, while a data lake is no does not care about structure. It stores all—structured, semi-structured, and unstructured.
Processing:- Before we can load data into a data warehouse, we first need to give it some shape and structure i.e., we need to model it. That’s called schema-on-write. With a data lake, you just load in the raw data, as-is, and then when you’re ready to use the data, that’s when you give it shape and structure. That’s called schema-on-read. Two very different approaches.
Storage:- One of the primary features of big data technologies like Hadoop is that the cost of storing data is relatively low as compared to the data warehouse. There are two key reasons for this: First, Hadoop is open source software, so the licensing and community support is free. And second, Hadoop is designed to be installed on low-cost commodity hardware.
Agility:- A data warehouse is a highly-structured repository, by definition. It’s not technically hard to change the structure, but it can be very time-consuming given all the business processes that are tied to it. A data lake, on the other hand, lacks the structure of a data warehouse—which gives developers and data scientists the ability to easily configure and reconfigure their models, queries, and apps on-the-fly.
Security:- Data warehouse technologies have been around for decades, while big data technologies are relatively new. Thus, the ability to secure data in a data warehouse is much more mature than securing data in a data lake. It should be noted, however, that there’s a significant effort being placed on security right now in the big data industry. It’s not a question of if, but when.
Users:- The motto of Data Warehouse and data mart intensive environment is self-service, i.e, developing the structure and inviting users to consume information. On the other hand, data lake is not by any means easy and matured for business users to come and get useful information. It has to be still manipulated by data scientists.
As a conclusive remark, many recommend to use each according to their needs. While data warehouse is more matured, secured suitable for self service BI, data lake is suitable for storing big data characterized by the 3V’s mentioned above with low cost.