Yige

HBase Series - Basic Concepts of HBase#

Content organized from:

• Github repository: God-Of-BigData/HBase Series

Basic Concepts#

The logical arrangement of data in HBase:

The physical arrangement of data in HBase:

Row Key#

• Row Key is the primary key used to retrieve records. Accessing data in HBase can only be done through Rowkey, Rowkey's range, or full table scans, with full table scans being very inefficient.

• Row Key can be any string and is sorted in dictionary order when stored. It is important to note:

  1. Because the dictionary order for Int sorting results in 1,10,100,11,18,19,2,20,21,…,9,91,...,99. If you use integer strings as row keys, to maintain the natural order of integers, row keys must be left-padded with 0.
  2. A read-write operation on a row is atomic (regardless of how many columns are read or written at once).

Column Family#

Each column in an HBase table belongs to a specific column family. Column families are part of the table's schema, so they need to be defined when creating the table. All columns in a column family are prefixed with the column family name, for example, info:ln, info:fn both belong to the info column family.

Column Qualifier#

Can be understood as the specific column name, for example, info:ln, info:fn both belong to the info column family, with their column qualifiers being ln and fn, respectively. It is important to note that column qualifiers are not part of the table schema; you can dynamically create columns during data insertion.

Column#

Columns in HBase consist of column families and column qualifiers, separated by : (colon). A complete column name should be expressed as column family name : column qualifier.

Cell#

A storage unit determined by row key and column in HBase is called a Cell, which is a combination of row, column family, and column qualifier, and contains a value and a timestamp.

Timestamp#

Each Cell stores multiple versions of the same data. Versions are indexed by timestamps, which are of type 64-bit integer. Timestamps can be automatically assigned by HBase during data writing or explicitly specified by the client. In each Cell, different versions of data are arranged in descending order by timestamp, with the latest data appearing first.

Rowkey Design Principles#

Reference: Things to Note When Designing HBase RowKey

Dispersing Rowkey
Rows in HBase are sorted in dictionary order by RowKey. This is very friendly for Scan operations, as rows with similar RowKeys are always stored in close proximity, making sequential reads more efficient than random reads. However, if a large number of read and write operations are concentrated in a specific RowKey range, it can create Region hotspots, degrading the performance of the RegionServer. Therefore, it is necessary to appropriately disperse RowKeys.

Controlling RowKey Length

• In HBase's underlying storage HFile, RowKey is a field in the KeyValue structure. Assuming a RowKey length of 100B, then in 10 million records, the RowKey alone would occupy nearly 1G of space, affecting the storage efficiency of HFile.
• HBase has MemStore and BlockCache, corresponding to write caches at the column family/store level and read caches at the RegionServer level, respectively. If RowKeys are too long, the density of stored data in the cache will decrease, affecting data landing or query efficiency.

Ensuring RowKey Uniqueness

Storage Structure#

Regions#

• All rows in an HBase Table are arranged in dictionary order by Row Key. HBase Tables are horizontally split into multiple Regions based on the range of row keys (row key range), with each Region containing all rows between the start key and end key.

• Each table initially has only one Region. As data continues to increase, the Region will grow. When it reaches a threshold, the Region will split into two new Regions. As the number of rows in the Table increases, there will be more and more Regions.

• A Region is the smallest unit of distributed storage and load balancing in HBase. This means that different Regions can be distributed across different Region Servers. However, a single Region will not be split across multiple Servers.

Region Server#

Region Server runs on the DataNode of HDFS. It has the following components:

• WAL (Write Ahead Log): Used to store data records that have not yet been persisted, allowing for recovery in case of failure.
• BlockCache: Read cache. It stores frequently read data in memory, and if storage is insufficient, it will evict excess data based on the Least Recently Used principle.
• MemStore: Write cache. It stores new data that has not yet been written to disk and sorts it before writing to disk. Each column family on each Region has a MemStore.
• StoreFile: The underlying StoreFile is HFile, which is Hadoop's binary format file, storing row data in the form of Key\Values in the file system.

When a Region Server accesses a sub-table, it creates a Region object and then creates a Store instance for each column family of the table. Each Store may have 0 or more corresponding StoreFiles, and each StoreFile corresponds to one HFile, which is the actual file stored on HDFS.

System Architecture#

The HBase system follows a Master/Slave architecture, consisting of three different types of components:

Zookeeper

1. Ensures that there is only one Master in the cluster at any time;
2. Stores the addressing entry for all Regions;
3. Monitors the status of Region Servers in real-time and notifies the Master of Region Server's online and offline information;
4. Stores HBase's schema, including which Tables exist and what Column Families each Table has.

Master

1. Allocates Regions to Region Servers;
2. Responsible for load balancing of Region Servers;
3. Detects failed Region Servers and reallocates their Regions;
4. Garbage collection of junk files on GFS;
5. Handles schema update requests.

Region Server

1. The Region Server is responsible for maintaining the Regions assigned to it by the Master and handling IO requests sent to the Regions;
2. The Region Server is responsible for splitting Regions that become too large during operation.

Component Collaboration#

HBase uses ZooKeeper as a distributed coordination service to maintain the status of servers in the cluster. ZooKeeper is responsible for maintaining the list of available services and providing services such as service failure notifications:

• Each Region Server creates a temporary node on ZooKeeper, and the Master monitors the nodes through ZooKeeper's Watcher mechanism, allowing it to detect newly added Region Servers or Region Servers that have failed;
• All Masters competitively create the same temporary node on ZooKeeper. Since ZooKeeper can only have one node with the same name, only one Master can successfully create it, making that Master the primary Master, which periodically sends heartbeats to ZooKeeper. Backup Masters listen to the node where the primary HMaster resides through the Watcher mechanism;
• If the primary Master fails to send a heartbeat in time, its ZooKeeper session will expire, and the corresponding temporary node will be deleted, triggering the Watcher event defined on that node, notifying the backup Master Servers. Upon receiving the notification, all backup Master Servers will compete to create a temporary node again, completing the election of the primary Master.