How to build and use parquet-tools to read parquet files

Goal:

How to build and use parquet-tools to read parquet files.

Solution:

1. Download and Install maven.

Follow below link:
http://maven.apache.org/download.cgi

2. Download the parquet source code

1
git clone https://github.com/Parquet/parquet-mr.git

3. Build the parquet-tools.

1
2
cd parquet-mr/parquet-tools/
mvn clean package -Plocal
The resulting jar is target/parquet-tools.jar.

Note, you may meet error such as below:
Failure to find com.twitter:parquet-hadoop:jar:1.6.0rc3-SNAPSHOT in https://oss.sonatype.org/content/repositories/snapshots was cached in the local repository
It is because the pom.xml is pointing to version 1.6.0rc3-SNAPSHO, however that version does not exist in https://oss.sonatype.org/content/repositories/snapshots/com/twitter/parquet-hadoop/ .
The fix is to modify parquet-mr/pom.xml and also parquet-mr/parquet-tools/pom.xml to one valid version, for example:
<version>1.6.1-SNAPSHOT</version>

4. Show help manual

1
2
cd target
java -jar parquet-tools-1.6.1-SNAPSHOT.jar --help

 5. Dump the schema

Take sample nation.parquet file for example.
1
2
3
4
5
6
7
# java -jar parquet-tools-1.6.1-SNAPSHOT.jar schema /tmp/nation.parquet
message root {
  required int64 N_NATIONKEY;
  required binary N_NAME (UTF8);
  required int64 N_REGIONKEY;
  required binary N_COMMENT (UTF8);
}

6. Read the data


1
2
3
4
5
6
7
# java -jar parquet-tools-1.6.1-SNAPSHOT.jar cat /tmp/nation.parquet
N_NATIONKEY = 0
N_NAME = ALGERIA
N_REGIONKEY = 0
N_COMMENT =  haggle. carefully f
 
(... ...)

7. Read first n records

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# java -jar parquet-tools-1.6.1-SNAPSHOT.jar head -n3 /tmp/nation.parquet
N_NATIONKEY = 0
N_NAME = ALGERIA
N_REGIONKEY = 0
N_COMMENT =  haggle. carefully f
 
N_NATIONKEY = 1
N_NAME = ARGENTINA
N_REGIONKEY = 1
N_COMMENT = al foxes promise sly
 
N_NATIONKEY = 2
N_NAME = BRAZIL
N_REGIONKEY = 1
N_COMMENT = y alongside of the p

8. Show meta info


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# java -jar parquet-tools-1.6.1-SNAPSHOT.jar meta /tmp/nation.parquet
file:        file:/tmp/nation.parquet
creator:     parquet-mr
 
file schema: root
--------------------------------------------------------------------------------
N_NATIONKEY: REQUIRED INT64 R:0 D:0
N_NAME:      REQUIRED BINARY O:UTF8 R:0 D:0
N_REGIONKEY: REQUIRED INT64 R:0 D:0
N_COMMENT:   REQUIRED BINARY O:UTF8 R:0 D:0
 
row group 1: RC:25 TS:1352 OFFSET:4
--------------------------------------------------------------------------------
N_NATIONKEY:  INT64 SNAPPY DO:0 FPO:4 SZ:130/219/1.68 VC:25 ENC:PLAIN,BIT_PACKED
N_NAME:       BINARY SNAPPY DO:0 FPO:134 SZ:267/296/1.11 VC:25 ENC:PLAIN,BIT_PACKED
N_REGIONKEY:  INT64 SNAPPY DO:0 FPO:401 SZ:79/218/2.76 VC:25 ENC:PLAIN,BIT_PACKED
N_COMMENT:    BINARY SNAPPY DO:0 FPO:480 SZ:468/619/1.32 VC:25 ENC:PLAIN,BIT_PACKED

9. Dump all data

Note: Values are in column format.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# java -jar parquet-tools-1.6.1-SNAPSHOT.jar dump --disable-meta  /tmp/nation.parquet
INT64 N_NATIONKEY
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 25 ***
value 1:  R:0 D:0 V:0
value 2:  R:0 D:0 V:1
value 3:  R:0 D:0 V:2
(...)
 
BINARY N_NAME
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 25 ***
value 1:  R:0 D:0 V:ALGERIA
value 2:  R:0 D:0 V:ARGENTINA
value 3:  R:0 D:0 V:BRAZIL
(...)
 
INT64 N_REGIONKEY
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 25 ***
value 1:  R:0 D:0 V:0
value 2:  R:0 D:0 V:1
value 3:  R:0 D:0 V:1
(...)
 
BINARY N_COMMENT
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 25 ***
value 1:  R:0 D:0 V: haggle. carefully f
value 2:  R:0 D:0 V:al foxes promise sly
value 3:  R:0 D:0 V:y alongside of the p
(...)

Commentaires

Enregistrer un commentaire

Posts les plus consultés de ce blog

Controlling Parallelism in Spark by controlling the input partitions by controlling the input partitions

Image
It is useful to be able to control the degree of parallelism when using Spark. Spark provides a very convenient method to increase the degree of parallelism which should be adequate in practice. This blog entry of the "Inside Spark" series describes the knobs Spark uses to control the degree of parallelism. Controlling the number of partitions in the " local " mode Spark is designed ground up to enable unit testing. To control the degree of parallelism in the local mode simply utilize the function JavaRDD<String> org.apache.spark.api.java.JavaSparkContext.parallelize(List<String> list, int numSlices) The following snippet of code demonstrates this usage- public class SparkControlPartitionSizeLocally { public static void main ( String [] args ) { JavaSparkContext sc = new JavaSparkContext ( "local" , "localpartitionsizecontrol" ); String input = "four score and sev...
Lire la suite

Spark optimization

Image
One operation and maintenance 1. Master hang up, standby restart is also invalid Master defaults to 512M of memory, when the task in the cluster is particularly high, it will hang, because the master will read each task event log log to generate spark ui, the memory will naturally OOM, you can run the log See that the master of the start through the HA will naturally fail for this reason. solve Increase the Master's memory spark-env.sh , set in the master node spark-env.sh : export SPARK_DAEMON_MEMORY 10g # 根据你的实际情况 Reduce the job information stored in the Master memory spark.ui.retainedJobs 500 # 默认都是1000 spark.ui.retainedStages 500 Hang up or suspend Sometimes we will see the web node in the web ui disappear or in the dead state, the task of running the node will report a variety of lost worker errors, causing the same reasons and the above, worker memory to save a lot of ui The information leads to gc when the heartbeat is lost...
Lire la suite

Spark performance optimization: shuffle tuning

Image
Article directory 1 shuffle tuning 1.1 Summary of tuning 1.2 Overview of ShuffleManager Development 1.3 HashShuffleManager operating principle 1.3.1 Unoptured HashShuffleManager 1.3.2 Optimized HashShuffleManager 1.4 SortShuffleManager operating principle 1.4.1 General operating mechanism 1.4.2 bypass running mechanism 1.5 shuffle related parameters tuning 1.5.1 spark.shuffle.file.buffer 1.5.2 spark.reducer.maxSizeInFlight 1.5.3 spark.shuffle.io.maxRetries 1.5.4 spark.shuffle.io.retryWait 1.5.5 spark.shuffle.memoryFraction 1.5.6 spark.shuffle.manager 1.5.7 spark.shuffle.sort.bypassMergeThreshold 1.5.8 spark.shuffle.consolidateFiles 2 write in the last words Shuffle Summary of tuning Most of the performance of Spark operations is mainly consumed in the shuffle link, because the link contains a large number of disk IO, serialization, network data transmission and othe...
Lire la suite