Untitled Document

Transactional, dynamic and/or relational Databases. Evolution and perspectives.

:: THE DATABASE

The Beginning

The transactional model for databases came out in the 1970s. The initial purpose was reducing costs associated with the storage of data of most systems.

The main issue here was that programmers (who were very expensive), had to rewrite large amounts of software related to the storage of data. Applications then, knew in detail how its data was stored in the physical layout of the disk. Maintenance, reorganization, additions or deletion of data forced large amounts of programming efforts.

The creation of Databases, as we know them today solved this problem, as freed programmers of the interaction with the physical organization of data and created a common interface (i.e. SQL language) for operational purposes of the database. This way the ‘use’ of the database was separated with the administration of the database.

This model up to date, has been very successful. Most business in the world, no matter how big or small, make use of some kind of database, and in most cases is supported by one of the various Databases brands existing today.

Databases evolved accordingly to their success. As technology became faster and cheaper, Databases were basically the same model of the 70’s but they increased functionality and complementary tools. Today Databases have such an array of tools and functionality that a Database Specialist is required to build an application of any type. Moreover, most of the functionality of Databases are rarely used by a single customer.

The rationale behind this is that Databases manufacturers chose this ‘multipurpose approach’ was to have a single ‘box’ capable of addressing all different needs customers would come across worldwide. This is very intelligent, and contributes to the very prosperous business of Databases, however the world is changing rapidly and a new approach may soon be needed.

Everyday we come across a new device capable of generating computer data. Not anymore desktop computers alone and data centers are data generators. PDAs, mobiles, large networks, etc. generate an exponential output of data everyday that requires to be stored and retrieved safely.

And just how is that to be carried out? Will we have to install a Database into our cell phones to be able to put up with these new needs or is there a better alternative?

Today

Database researchers are rapidly concluding that database management systems need to be redesigned.

Is not possible any longer to pretend satisfy data management needs of a bank, a data warehouse, a telecommunications company or a seven/eleven with EXACTLY THE SAME PRODUCT, only varying the number of users, processors and price of licence. A more in depth analysis of needs is needed and the solution should come in the form of a simpler configurable product, capable of handling emergent problems traditional Databases are not prepared for.

Lets mention a few examples: the datawarehouse of a large Telco. Storing CDRs alone, lets say some 100 million records a day on a traditional Database can easily turn into a multi-million dollar project. Depending on the application, a DW is basically updated in bulks and read (queried) very often. This is not what traditional Databases are tailored for.
In case de DW is fed in ‘near real time’, stream processing capabilities would be required for the Database so that filtering and evaluation can be performed on data before it is definitely stored on the Database

Traditional database systems solve traditional problems; we need new-style databases to solve new-style problems. While the need for conventional database management systems isn’t going away, some of today’s problems require a configurable database system for handling VLDBs. Even without a crystal ball, it seems clear that tomorrow’s systems will also require a significant degree of configurability. As programmers and engineers, we learn to select the right tool to do a job; selecting a database is no exception. We need to operate in a mode where we recognize that there are options in data management, and we should select the right tool to get the job done as efficiently, robustly, and simply as possible.

The Concept

TeraManager (“TMGR”) is a Very Large Historical Data Base engine (VLHDB) with the following unique characteristics:

TeraManager requires only a small fraction of the space required by a traditional transactional database, being the cost savings quite relevant when handling massive amounts of data. Such space reductions can go up to (depending on the particular case) ~90% of the space required by a traditional Database, including indexes.

Performance in terms of indexing and retrieval speeds is very high, reducing bottle-necks processing issues and increasing over-all performance.

Being a fully Java developed engine; multi-platform Java enabled capability is assured. Provides the benefit of the capability to migrate historical data for Data Warehousing purposes (BI, regulation Compliance, etc) to “lower cost platforms” and not interfere with the “production” database.

Linear Scalability assures that access time for any given record is not affected by the size of the database.

Bulk appending and deleting large number of records (millions or more) is performed in a matter of instants.

Hot Swapping capability allows to have on line only the most requested clusters of data, leaving the rest “near on line” or “off line” for later use.

SQL (extended) operational interface allows developers with a variety of skills to enter the world of VLDB and Web services without having a large learning curve.

Enables migration of historical databases to less expensive storage platforms.

Energy Saving. This can be very obvious, but deserves to take a look at when talking about large Databases, since not only the space is a savings concern. At certain volumes, energy consumption can be a relevant issue as well.