HBase, a hadoop NoSQL database – Backup Process Explained

Hi Readers,

I am happy to post another blog post on how to take HBase backup of a HBase data/ tables in hortonworks sandbox 2.5.

Important:  You may get the following error in hbase shell.
ERROR: Can’t get master address from ZooKeeper; znode data == null
Just open ambari(Hortonworks Sandbox) / cloudera manager(Cloudera Sandbox) start the zookeeper service (or) HBase master to get rid of the above error.

I will help you to understand the different ways the backup can be taken for a HBase datasets.

Type 1 :

Export

HBase has a utility called Export, which is used to export the data of the HBase table to plain sequence files in the HDFS folder. A map reduce job gets created which will call the hbase cluster to get the data from specified table and write it to the hdfs folder.

You can export data from HBase table to

  • Local File System: It just copies into my hdfs rather than my local file system. So, from hdfs i need to move the part-m-00000 to local file system.
  • HDFS- It just copies to my hdfs.
[root@sandbox ~]# hbase org.apache.hadoop.hbase.mapreduce.Export "person" "/jthiyaga/"

where person is the table name in hbase and jose is local file system directory.

2017-03-22 17:12:53,304 INFO [main] util.RegionSizeCalculator: Calculating region sizes for table "person".
2017-03-22 17:12:54,034 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2017-03-22 17:12:54,034 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15af6c30f1e0017
2017-03-22 17:12:54,041 INFO [main] zookeeper.ZooKeeper: Session: 0x15af6c30f1e0017 closed
2017-03-22 17:12:54,041 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-03-22 17:12:54,639 INFO [main] mapreduce.JobSubmitter: number of splits:1
2017-03-22 17:12:54,679 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-03-22 17:12:55,331 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1490198811520_0001
2017-03-22 17:12:59,111 INFO [main] impl.YarnClientImpl: Submitted application application_1490198811520_0001
2017-03-22 17:12:59,311 INFO [main] mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1490198811520_0001/
2017-03-22 17:12:59,316 INFO [main] mapreduce.Job: Running job: job_1490198811520_0001
2017-03-22 17:14:17,273 INFO [main] mapreduce.Job: Job job_1490198811520_0001 running in uber mode : false

Import

You can also import the data from local filesystem. A map reduce job gets created which will call the import the data from hdfs folder to the existing hbase table.

[root@sandbox ~]# hbase org.apache.hadoop.hbase.mapreduce.Import "person" "/jthiyaga/"
2017-03-22 17:27:56,304 INFO [main] impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
2017-03-22 17:27:56,434 INFO [main] client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/172.17.0.2:8050
2017-03-22 17:27:56,776 INFO [main] client.AHSProxy: Connecting to Application History server at sandbox.hortonworks.com/172.17.0.2:10200
2017-03-22 17:28:06,379 INFO [main] input.FileInputFormat: Total input paths to process : 1
2017-03-22 17:28:06,605 INFO [main] mapreduce.JobSubmitter: number of splits:1
2017-03-22 17:28:07,507 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1490198811520_0002
2017-03-22 17:28:09,156 INFO [main] impl.YarnClientImpl: Submitted application application_1490198811520_0002
2017-03-22 17:28:09,546 INFO [main] mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1490198811520_0002/
2017-03-22 17:28:09,551 INFO [main] mapreduce.Job: Running job: job_1490198811520_0002
2017-03-22 17:28:47,656 INFO [main] mapreduce.Job: Job job_1490198811520_0002 running in uber mode : false
2017-03-22 17:28:47,878 INFO [main] mapreduce.Job: map 0% reduce 0%
2017-03-22 17:29:11,169 INFO [main] mapreduce.Job: map 100% reduce 0

Type 2:

CopyTable

The copytable will directly write the data into an existing HBase table. This can be done in same cluster or a remote cluster.

Both cluster should be online in order to make this done.

[root@sandbox jthiyaga]# hbase org.apache.hadoop.hbase.mapreduce.CopyTable --new.name=personNew person

where person is existing table (with data) and personNew is a table which i have created. The copytable will copy data from person to personNew table. Behind the scenes a mapreduce job will run.

Type 3:

Snapshots

The above types needs the following:

  • a fast reliable network connection to connect the remote cluster.

You can also use Snapshots to have less performance impact on your cluster.

Type 4:

Cluster Shutdown

Simple, easy but not recommended type of HBase data backup. As you need to shutdown the HBase cluster to stop data write to the cluster.

Just open hbase-site.xml. The following shows your table and data are being saved at this location ‘hdfs dfs -ls /apps/hbase/data

<property>
 <name>hbase.rootdir</name>
 <value>hdfs://sandbox.hortonworks.com:8020/apps/hbase/data</value>
 </property>

The following shows the personNew table which i have imported using Import option.

[root@sandbox conf]# hdfs dfs -ls /apps/hbase/data/data/default
Found 6 items
drwxr-xr-x - hbase hdfs 0 2016-09-13 10:59 /apps/hbase/data/data/default/ATLAS_ENTITY_AUDIT_EVENTS
drwxr-xr-x - hbase hdfs 0 2017-03-20 16:16 /apps/hbase/data/data/default/MyStream
drwxr-xr-x - hbase hdfs 0 2016-09-13 10:59 /apps/hbase/data/data/default/atlas_titan
drwxr-xr-x - hbase hdfs 0 2016-09-13 11:09 /apps/hbase/data/data/default/iemployee
drwxr-xr-x - hbase hdfs 0 2017-03-20 17:27 /apps/hbase/data/data/default/person
drwxr-xr-x - hbase hdfs 0 2017-03-22 17:38 /apps/hbase/data/data/default/personNew
[root@sandbox conf]# hdfs dfs -ls /apps/hbase/data/data/default/personNew
Found 3 items
drwxr-xr-x - hbase hdfs 0 2017-03-22 17:38 /apps/hbase/data/data/default/personNew/.tabledesc
drwxr-xr-x - hbase hdfs 0 2017-03-22 17:38 /apps/hbase/data/data/default/personNew/.tmp
drwxr-xr-x - hbase hdfs 0 2017-03-22 17:38 /apps/hbase/data/data/default/personNew/59acfc94838d84a3f1810efaacda0da8

Thanks for reading this post. Subscribe my blog for  some awesome posts.

Hbase, a hadoop NoSQL database-snapshot explained

Hi Readers,

I am happy to post another blog post on how to take HBase snapshot of a table in hortonworks sandbox 2.5.

I have created a table named ‘person’ in HBase.

hbase(main):032:0> list
TABLE
person
1 row(s) in 0.080 seconds.

I use the scan command to show the data inside the person table.

hbase(main):033:0> scan 'person'
ROW COLUMN+CELL
 1 column=location:, timestamp=1490028012845, value=Chennai
 1 column=name:, timestamp=1490027975127, value=Raj
 1 column=pincode:, timestamp=1490028023696, value=12341
 2 column=location:, timestamp=1490028131440, value=London
 2 column=name:, timestamp=1490028040307, value=Ajay
 2 column=pincode:, timestamp=1490028098868, value=9039
 3 column=location:, timestamp=1490028123508, value=Qwerty
 3 column=name:, timestamp=1490028074518, value=Anad
 3 column=pincode:, timestamp=1490028104978, value=329039
3 row(s) in 0.1230 seconds.

Now I will show how to take a snapshot.

In hbase-site.xml, hbase.snapshot.enabled value should be true.

   
 hbase.snapshot.enabled  
 true 

In HBase shell type the following command  : snapshot ‘table_name’, ‘snapshot_name’

hbase(main):034:0> snapshot 'person', 'personSnapshot20032017'

Use the following command to list the snapshots:

hbase(main):034:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
 MyStreamSnapshot-03202017 MyStream (Mon Mar 20 16:16:52 +0000 2017)
 personSnapshot20032017 person (Mon Mar 20 16:44:21 +0000 2017)
2 row(s) in 0.1680 seconds

=> ["MyStreamSnapshot-03202017", "personSnapshot20032017"]

You can also delete a snapshot using the following command: delete_snapshot ‘snapshot_name’

You can also clone a table from a snapshot: clone_snapshot ‘snapshot_name’, ‘table_name’.

The snapshot details can be found under this hdfs location

/apps/hbase/data/.hbase-snapshot/personSnapshot20032017/data.manifest

/apps/hbase/data/.hbase-snapshot/personSnapshot20032017/.snapshotinfo

Scenario:

You have deleted a table in HBase (say) person table. You have the snapshot file.(type list_snapshots in hbase shell)

Just use the following command: restore_snapshot ‘snapshot_name’.

restore_snapshot 'personSnapshot20032017'. It will automatically restore the table_schema at that time.

What is a Snapshot?

A snapshot is a set of metadata information that allows an admin to get back to a previous state of the table. A snapshot is not a copy of the table; it’s just a list of file names and doesn’t copy the data.

Transfer snapshot from one cluster to another cluster:

The org.apache.hadoop.hbase.snapshot.ExportSnapshot tool copies all the data related to a snapshot (HFiles, logs, and snapshot metadata) to another cluster:

hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot01082014 -copy-to hdfs://127.0.0.1:9000/hbase -mappers 8 -bandwidth 100

The above command has eight map jobs which will run to export all snapshots to another cluster with a limiting bandwidth of 100 MB/s.

HBase CheatSheet:

Create a table – create ‘table_name’,’column_family’

  • List a table – list
  • View a table – scan table_name
  • Insert data – put ‘table_name’,’row1′,’column_name:’,’column_value’
  • Update data – use the above command but change the column value.
  • Select /Read data – get ‘table_name’,’row1′
  • delete a cell – delete ‘table_name’, ‘row1’, ‘column_name ’, ‘time stamp’
  • disable a table – disable ‘table_name’
  • drop a table – drop ‘table_name’
  • enable a table – enable ‘table_name’
  • describe a table – describe ‘table_name’
  • exists a table – exists ‘table_name’
  • count no.of rows in a table – count ‘table_name’
  • truncate a table – truncate ‘table_name’

Thanks for reading this post. Subscribe my blog for  some awesome posts.

How can I check version of Hive, Pig, Hbase, Sqoop in Cloudera 4.3.0?

How can I check version of Hive, Pig, Hbase, Sqoop in Cloudera 4.3.0? by Jose Praveen

Answer by Jose Praveen:

I think you are using cloudera sandbox.

My solution would be

Open a terminal in the cloudera sandbox and type

cat /usr/lib/hadoop/cloudera/cdh_version.properties

You will get the cloudera package release version.

Check the release version in this link. By this you can get all the tools version details of your sandbox.

Hope this helps !!!!

How can I check version of Hive, Pig, Hbase, Sqoop in Cloudera 4.3.0?

Packtpub offering Mapt pro – 4000 courses, 10 days entire ebook library free

Hi Readers,

I am happy to post another blog post on how to get Mapt pro for 10 days entire ebook library free.

Just click the below link

https://www.packtpub.com/packt/offers/free-learning (link)

Before Packtpub was offering everyday free ebook titles (link) on different topics. For this month, the entire ebook library(Pro edition) is free(10 days only) to access where you can access more than 1,000 technologies & topics.

Bonus Topic – How to enable Bash on Ubuntu on Windows application in a windows 10 device and check which version is it running.

Hi Readers,

I am happy to post another blog post on how to enable bash on ubuntu on windows application in a windows 10 device and check which version is it running.

If you have done this step (link) then proceed the below steps.

Open Settings –> Update & Security –> For Developers

Enable Developer Mode and restart the system.

bash-on-windows-10-josepraveen

After restarting your system just search for bash on ubuntu for windows 10. A command prompt opens.

To check which Ubuntu release has been installed. Type

cat /etc/lsb-release

josept@LENOVO-PC:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

My system is running Ubuntu 14.04.4 LTS.

Happy Learning !!

Schedule a task in linux using crontab

Hi Readers,

I am happy to post another blog post on how to schedule a task in linux using crontab.

cron enables users to schedule jobs to run periodically at certain times or dates. It is commonly used to automate system maintenance or administration. cron checks for scheduled jobs every minute but not every second.

 

* * * * * command or shell script to be executed
┬ ┬ ┬ ┬ ┬
│ │ │ │ │
│ │ │ │ │
│ │ │ │ └───── day of week (0 - 6) (0 is Sunday, or use names)
│ │ │ └────────── month (1 - 12)
│ │ └─────────────── day of month (1 - 31)
│ └──────────────────── hour (0 - 23)
└───────────────────────── min (0 - 59)

Use the below command to check existing cron that has been scheduled in the linux box.

$ crontab -l

Use the below command to edit the cron entries.

$ crontab -e

 

Schedule a job to run everyday at 12.10  am.

10 0 * * * /usr/jpraveen/testautomate.sh

Schedule a job to run every sunday at 12.10  am.

10 0 * * 7  /usr/jpraveen/testautomate.sh

Schedule a job to run only on weekends at 12.10  am.

10 0 * * 6,7  /usr/jpraveen/testautomate.sh

Schedule a job to run everyday  at 9  am and 6 pm.

0 9,18 * * *  /usr/jpraveen/testautomate.sh

 

Thanks for reading this post. Subscribe my blog for  some awesome posts.

Happy Learning !!

Integrate Swagger with Spring boot application for RESTful API documentation-Lab 4

Hi Readers,

I am happy to post another blog post on how to integrate swagger with Spring boot application for RESTful API documentation.

You have created a RESTful project(say) and like to integrate swagger for representing your RESTful API then this post is for you.

In pom.xml add the below dependency

 io.springfox
 springfox-swagger2
 2.2.2
 io.springfox
 springfox-swagger-ui
 2.2.2

Create a swagger config class similar to this.

Inside the predicate method, add your request mapping path.

Take a maven build and run as spring boot app.

Open a browser and type the below url.

http://localhost:8080/swagger-ui.html

swaggerprojectsjosepraveen1.PNG

From the above diagram the highlighted ones points to my @RequestMapping value of my controller.

Let me know your thoughts and experiences on how to integrate swagger with Spring boot application for RESTful API documentation.

Happy Learning !!