Big data

HBase after seven years of development, and finally at the end of February this year, released 1.0.0 version. This version provides some exciting features, and, without sacrificing stability, the introduction of a new API. Although 1.0.0 compatible with the old version of the API, but still should be familiar with the new API as soon as possible. And understand how to combine with the current red Spark, the data written and read. In view of the domestic and foreign information on the new API HBase 1.0.0 little, so this article. This article will be introduced in two parts, the first part of the new version of the API to explain the use of CRUD basic operation; the second part of the Spark to explain how to write RDDs HBase table, on the contrary, HBase table is how to RDDs loaded Spark inside. Environment configuration In order to avoid unnecessary inconsistencies in the version of the trouble, API and HBase environment are 1.0.0 version. HBase is a stand-a...

Spark reads HDFS into HBase (1.0.0 new API) HBase after seven years of development, and finally at the end of February this year, released 1.0.0 version. This version provides some exciting features, and, without sacrificing stability, the introduction of a new API. Although 1.0.0 compatible with the old version of the API, but still should be familiar with the new API as soon as possible. And understand how to combine with the current red Spark, the data written and read. In view of the domestic and foreign information on the new API HBase 1.0.0 little, so this article. This article will be introduced in two parts, the first part of the new version of the API to explain the use of CRUD basic operation; the second part of the Spark to explain how to write RDDs HBase table, on the contrary, HBase table is how to RDDs loaded Spark inside. Environment configuration In order to avoid unnecessary inconsistencies in the version of the trouble, API and HBase env...

I. Overview Program running environment is very important, this test is based on: Hadoop-2.6.5 Spark-1.6.2 Hbase-1.2.4 Zookeeper-3.4.6 Jdk-1.8 Nonsense not much to say, directly on demand Andy column=baseINFO:age, value = 21 Andy column=baseINFO:gender, value = 0 Andy column=baseINFO:telphone_number, value = 110110110 Tom column=baseINFO:age, value = 18 Tom column=baseINFO:gender, value = 1 Tom column=baseINFO:telphone_number, value = 120120120   As shown in the table above, will be used to group the spark, to achieve this effect: [Andy, (21,0,110110110)] [Tom, (18,1,120120120)] Demand is relatively simple, mainly familiar with the process of running the program Second, the specific code package com .union .bigdata .spark .hbase ; import org .apache .hadoop .hbase .HBaseConfiguration ; import org .apache .hadoop .hbase .mapreduce .TableSplit ; import org .apache .hadoop .hbase .util .Base 64 ; import org .apache .hadoop .hbase .util .Bytes ; import org .apac...

package com.hun.scala      import java.text.SimpleDateFormat   import java.util.Date      import org.apache.commons.codec.digest.DigestUtils   import org.apache.hadoop.fs.Path   import org.apache.hadoop.hbase.client.{HTable, Table, _}   import org.apache.hadoop.hbase.io.ImmutableBytesWritable   import org.apache.hadoop.hbase.mapreduce.{HFileOutputFormat2, LoadIncrementalHFiles}   import org.apache.hadoop.hbase.util.Bytes   import org.apache.hadoop.hbase.{HBaseConfiguration, KeyValue, TableName}   import org.apache.hadoop.mapreduce.Job   import org.apache.spark.{SparkConf, SparkContext}      /**     * Created by hun on 2016/10/17.     *     * 终于...

Copyright statement: This article is the bloggers original article, without the blog owner may not be reproduced. Try to achieve their own class, The role of this class: call the internal method Select the columns to be displayed based on the entered table name, column family, column name According to the input of the column family, column name, column value, filter conditions, is to filter the data with a filter Return Value: SQLContext Has completed the registration table, you can directly operate sql method, the use of sql language query processing The code is as follows: package  cn.deppon.Tool      import  java.util      import  scala.collection.JavaConverters._   import  org.apache.hadoop.hbase.HBaseConfiguration   import  org.apache.hadoop.hbase.client.{HTable, ResultScanner, Scan}   import  org.apache.hadoop.hbase.filter._   import  org.apache.h...