Designing warehouse data flow

kuonirat — Mon, 06 Feb 2012 11:25:35 GMT

Making people happy

So, the first thing to know is, data warehouse is supposed to make the end users happy. One could feel bad about being man in the middle, or even worse - the warehouse technical support, but this doesn't matter. It's the end user who has to be happy. So, happy about what? Well, if he gets answers he needs, we assume, he's happy. Nevertheless I would really like to make an opposition here. For example, "can I have some data about users that satisfy these conditions?" is not a good question. "Some data" is not a concrete and not a kind of question You can get answer for.

In summary, why querying on the production servers is a bad idea?

analytical queries performance is very poor
analytical queries could slow down the entire production system
additional potential crash reason
joins between databases on different servers are costy in OLTP solutions
data is usually sharded
data is not cleaned
data is cryptic
security breach

Filling analytical databases with data

To give an answer, we have to have knowledge. Knowledge comes from data You can analyze in a reasonable time with a reasonable effort. That's why it should be stored in some database. Whether it will be YesSQL or NoSQL database is a matter of data type, not a matter of which side of this holly war You want to be on. So another purpose of this system is to fill appropriate databases with appropriate data in a reasonable time.

High throughoutput

If you've got a lot of data to analyze, it means that a lot of data has to be loaded into the warehouse (Thanks, Captain Obvious!). So this has to work fast.

Pipe or stage?

In a sequenced process the speed is on the opposite side of the fast fault recovery ability, because when you push the data stream through some mechanism, that can either go entirely well or entirely fail, you cannot take advantage of a part of the process being already done. You have to retry the whole process. You can divide the process into smaller, persistable stages (using storage areas), but this impacts performance. Thus You have to reach compromise.

A general rule:

use piping where you have to stream a lot of data, but the failure is rare, because the data is narrow and homogenic. An example of this may be a very long fact table (billions of rows), which doesn't have much columns and the columns are of basic types (e. g. integers)
store your data in every stage otherwise, because this improves the ability to recover fast from a failure

Use ETL tool?

I recommend using dedicated ETL tools for small (up to several GB) amounts of data and complicated transformations. If You want to reach high troughoutput, get rid of them. Code Your ETL manually, use pure SQL. You can use high level language for the ETL (like Java, or even PHP if this is a requirement), but this will be ok only if You can fire it up in a clustered environment (like Hadoop) or it will be a rare case, so one of Your server cores can handle it entirely while not consuming much of other resources needed by other parallelly running transformations.

MySQL as one of the storage areas

It's ok, but use easily loadable storage engine (like MyISAM). Avoid using indexes. The processes should be designed in a "one time full scan" pattern, avoid per row lookups.

Loading from various sources

The system has to be able to load from various sources. But believe me, maintaining many connectors may be very inconvenient. So what to do? Create only one for each data source and use them as gates used by your internal system mechanisms. Don't connect to external sources directly from various points (task processing stages) of Your system. The better idea is to plug those points into Your very few connectors (through some well defined interface), which in turn do the external communication thing. Failure in communication will be easier to handle if You've got, simplifying, only one potential place in code, You have to look at. But remember, that every new external connection needed during the workflow is a very risky thing that You really want to avoid, because it increases the dependance on some external system to actually be working. And hope is not what an engineer should rely in the first place on.

This is actually the main disadvantage of ready to use ETL systems. Almost every time You create a transformation or a job, You define Your connection. This is bad. I know there are systems, which handle centralized data sources (Pentaho BI Server may be an example), but, ironically, they don't easily integrate with the data integration designing tools. They're more dedicated to charting and reporting (but maybe You can find something better?). Anyway, I think for bigger systems, multipoint connections to the same data source are an antipattern.

High query performance

You have to use the most responsive and fast analytical database you can afford. Use VectorWise for less than 1 TB of data to reload every day (data from production databases). Seriously, don't believe that any single machine solution can be faster. At least not for now (the beginning of 2012). Use some distributed, unstructured solution for more behavioral data (such as logging http requests), that requires more than several dozens of TB to store.

Load tables entirely or incrementally? Use SCD?

If Your business users know what they want and they tell You this, You're in heaven. Why? Because You can design Your architecture to meet those needs. You count costs and decide, whether it is worth to implement SCD principles. If you've got a lot of data, millions of users, this cost may be to high. You can select the most needed data, design facts and dimensions and do beautiful incremental loading solutions... Ok, now back to earth. If Your business users don't have a clue what they want and they work in a more "on demand" way, just load the production tables entirely doing only basic ETL before. Decode mysterious "type_id" columns, get rid of meaningless columns, maybe clean some data and do some helper snapshot tables, that will increase queries performance. Just for the biggest ones (or those extracted e. g. daily from some exotic data source) consider doing some incremental loading mechanisms. Don't be afraid to break the rules, because afterall, it's the end users' happiness what matters, not Yours (the engineer). This is not that bad, as it sounds.

Moreover, using SCDs assumes that people want to analyze the consequences of their decisions far after they're made. This assumption should be correct, but in practice most of the questions are about "What's going on now? I want a number to show, we're doing well!". To analyze the past, one should be really self-conscious and able to admit, he has made some mistakes maybe (surely, he hasn't seen this). This is not convinient in many cases, so the cost of implementing SCDs maybe to high comparing to the actual requirements (those which come from practice, not those specified). This is not only a critisism, it's just how people function.

Availability

Tasks

To increase availability, use concept of tasks. If You granulate Your one big process, You increase availability. Moreover, You increase performance, because many of the tasks can run in parallel fashion, taking advantage of multicore machines.

Tasks mutual independence

To achieve good availability, failure of one task should not affect failure of other. As long as two tasks do not depend on each other, also hangup of one task should not cause other task not to start eventually. This pushes You to design Your system in a more task oriented, than batch oriented architecture. Should You use maybe some already written tasks management system? Well, if You find one, that fits Your needs... but beware of limitations. It's really unlucky, if one day You come to a conclusion, that because of the limitations, You cannot make a very much required feature or optimization... especially, when it's far too late to change Your mind.

Tasks ordering

Tasks should be ordered. Ordered like in a restaurant, not like in sorting. It means You should divide Your system into ordering part, and the making the dirty work part, which updates the task statuses accordingly during the workflow. You should make a new task for every job, not reuse old, already done tasks. So create a list of tasks every day or so, pick them, execute and set as finished with a success or an error. If You don't follow this pattern, You'll create a batch processing oriented architecture, which is bad, because it is not failure resistant and it is slower (does not take much advantage of parallel processing).

Tasks granularity

Choose Your tasks granularity wisely. Should it be per table? Per shard? Maybe even per partition? Maybe You should use multiple granularity levels? Think it over. The more granularity You achieve, the more failure resistant and parallelized system You create. But this is almost always at the cost of development, administrating and You can easily exaggerate and the throughoutput performance will suffer too much. Experiment.

Tasks parallelization

Beware of deadlocks. Remember, that You create a parallel system. If You don't know what mutex or semaphore is, critical section and so on, better know it before. For advanced programming I recommend reading "The Art of Multiprocessor Programming" by Herlihy Maurice and Shavit Nir.

Tasks statuses

Your tasks should have meaningful statuses, that can indicate both progress and potential errors. The higher granularity, the more monitoring You can apply, but at the cost of development and You sacrifice the generic nature of the statuses. So choose wisely according to Your needs. Perhaps You have to handle different types of tasks, with different workflows (thus maybe different statuses sets).

Double buffering

If You can afford Vertica, at least in theory (never tested it acutally) You don't have to worry about core hours, when Your warehouse has to work and reserve some night hours for data loading. Otherwise consider the double buffering solution:

create views that for Your queries are the only access to Your underlying tables
load data into tables that are not being queried at the time You're loading
switch the view to the new tables instantly, so no one will notice
repeat this every hour, day or week - depending on how often You reload Your data

VectorWise can handle this double (or more) buffering solutions quite ok, but the transactions isolations do not always work as expected. This is not an issue, if You can just repeat a query in a new transaction (connection). Otherwise You can be quite astonished, You don't see any rows in Your current view anymore ;-)

Maintainability

The system should be at least remotely maintainable in a way, that adding new processes, transformations, etc. would be as harmless as possible. One way to achieve this is to code the system in a way that... doesn't require much coding. The less source code, the less possibility to make a mistake or to commit a crime, such as copy-paste antipattern. So remember one, very important paradigm: use convention over configuration. Don't reinvent the wheel every time a new transformation appears. This also has another plus side: if You design Your system with some layering in mind, You can plug various new transformations at various points of Your system, making good external interfaces and reusable code.

For example, let's say You load Your data from some SQL dumps, that arrive to Your data warehouse backup server from production servers. Now, let's say someone says: "We wan't also load some data from a couple of additional servers." This is the time when You say: "Ok, let's do that, but make Your administrators to upload the dumps in the same way I receive the old ones. This way I don't have to change much in my system and it can follow highly tested workflow." Argue, if anyone disagrees, because afterall it's the end user happiness what matters and he won't be happy, if he sees too many errors too often.

Monitoring

I can confirm Kimball's words, which I heard from him with my own ears: You have to monitor Your ETL. You have to know, if anything went bad and fix it. ETL is not something that just works, because the input changes constantly. Because You alter it constantly to meet new needs.

Task statuses monitoring

Your tasks must provide an easy way to read their current statuses. If they're stored in a database table, it should have a column, which indicates the status and there should be a limited number of possible status values (use some enum for example). The status should be easily interpretable and human readable (for example it should contain "Error" suffix, if it indicates any error). Doing this will allow You to create easy queries that can summerize the data flow processing in a nice, instant report. The goal is to quickly know, if the whole processing is going fine and if it is not, which tasks have failed. It's very important (thus the verbosity of this simple paragraph).

Logging

Make Your workflow verbose, do heavy logging, especially when You encounter error. Check te processes statuses after exit, don't swallow exceptions. Believe me, it will pay back! At the cost of initial errors waterfall, You get some serious peace later...

Implementing a buzzword

For business users a data warehouse is a black box they get answers from. They may even treat You as a part of this black box... Nevertheless, they don't understand the limits and possibilities. Not even now I'm sure whether it's an issue or not. If You have to "make a data warehouse", this means not only, You've got no idea what exactly You have to create. It is also a possibility to experiment, to adapt to the needs on demand. Am I talking about agile? I don't know, I really don't like buzzwords... You can educate, if You're a believer - the World needs such people. For others I suggest not to ask too many questions. They confuse. Just do what You think will be best and try to know for others.

How to deal with "Out of memory" problem in VectorWise

kuonirat — Mon, 11 Oct 2010 07:38:32 GMT

Until now, VectorWise's temporary queries results have to fit memory. Not always there is a possibility to rewrite the query in a "better" way (otherwise one could optimize infinitely to reach zero query time and no memory usage ;) ). Consider an example, where one would like to count distinct users, which visit your website, but per day. Following the example, let's say Your website has been running for more than four years and You've got approximately from 7 to 13 millions of unique users per day... But You want the exact numbers. I don't know, how much RAM You have, but believe me, hundrends of gigabates would be needed.

One approach is to write a script, that would generate many sql queries with highly narrowing constraints. It's not hard to code (for example in ruby ;) ) a simple script, that in loop would generate the aforementioned queries with constraints per day. Now to get the result in the exact same way, as it would come with the original query, just put these statements at the end of the generated queries:

Code:

\silent
\script result.csv
\notitles
\vdelim TAB
\trim
\g

Of course there should be only one '\g' command preceeded by the queries separated only by semicolons.

But what if we need some temporary aggregate, that don't fit the memory? Could we use "create table as" syntax to create such table? The answer is "no", because before being written to the temporary table, the result also has to fit the memory. But there is another solution. Assuming, that rows are generated independently, one could create a temporary table, and make many "insert select" statements into this table, with each one of them fitting the memory. Performing the reducing query on such table will be vector-wise-fast.

Migrating large tables from MySQL to VectorWise

kuonirat — Fri, 02 Jul 2010 10:08:16 GMT

One can face a problem when migrating from MySQL to VectorWise, when a fast and appropriate method is required. Using multiple inserts is not the way VectorWise likes to see it, but using additional dumps can have a high storage cost.

I shall give here a hint, how to migrate large tables from MySQL to VectorWise using low disk space and memory usage. We'll use linux named pipes. First we have to create a pipe:

$ mknod my_pipe p

Now we make a query to mysql:

$ mysql --quick -h my_host -umy_user my_dbname -B --default-character-set=utf8 -e "select col1, col2, col3 from \`my_table\`;" > pipe &

This will make mysql to hang on pipe in the background (notice the ampersand), while collecting results, that are pushed down by MySQL server.

Notice the --quick option which is the key to the whole memory saving. It will make mysql not wait until the result is returned from the server. Instead, it will directly output to the pipe. Someone has to immediately start to read from the other side of the pipe, so we use a COPY FROM statement to do that:

$ echo "COPY TABLE my_table (
col1 = 'c0tab',
col2 = 'c0tab',
col3 = 'c0nl'
) FROM 'pipe' WITH ON_ERROR = CONTINUE \g" | sql vw_dbname

This method is ok, as soon as You have a very simple data (not necessarily small of course). However we'll need some more conversion if we have NULL columns in MySQL or timestamps like '0000-00-00 00:00' which are not supported by VectorWise. We could use more complicated query to MySQL server (which is probably better choice) or use sed to alter some data. For example:

$ mysql --quick -h my_host -umy_user my_dbname -B --default-character-set=utf8 -e "select col1, col2, col3 from \`my_table\`;" | sed 's/NULL//g;s/\n//g;s/0000-00-00/0001-01-01/g' > pipe &

Which will on the fly remove "NULL" strings, newline from within character fields and replace the problematic timestamp with more "eatable" for VectorWise value. Note that sed will be, surprisingly, the bottleneck of this procedure, so should be avoided in favor of more sophisticated query to MySQL server. This method would also cause problems, when the character fields contains some "dangerous" characters, for example tabs.

Konrad Procak

Actian Community Forums - Blogs - kuonirat

Designing warehouse data flow

How to deal with "Out of memory" problem in VectorWise

Migrating large tables from MySQL to VectorWise