Manage files in HDFS using WEBHDFS REST APIs

Web services have become indispensable, in the current trend of development of applications, for exchange of data across the applications / web applications. Various application programming interfaces (APIs) are emerging to expose Web services. Representational state transfer (REST), used by browsers, is logically the choice for building APIs.
Web services have become indispensable, in the current trend of development of applications, for exchange of data across the applications / web applications. Various application programming interfaces (APIs) are emerging to expose Web services. Representational state transfer (REST), used by browsers, is logically the choice for building APIs.
I share my understanding and experience in using WEBHDFS REST API.
What is WebHDFS?

Hadoop provides a Java native API to support file system operations such as create, rename or delete files and directories, open, read or write files, set permissions, etc.
This is perfectly useful for applications running within the Hadoop cluster, in cases where an external application needs to exchange data/files in HDFS and need to perform operations like create directories and write files to that directory or read the content of a file stored on HDFS, a special API is required
Hortonworks developed an additional API to support these requirements based on standard REST functionalities
WebHDFS concept is based on HTTP operations like GET, PUT, POST and DELETE.

Authentication can be based on user.name query parameter (as part of the HTTP query string) or if security is turned on then it relies on Kerberos.
Not much configuration needed to enable WebHDFS. Only we have to add the property shown below in the hdfs-site.xml

<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property

<name>dfs.webhdfs.enabled</name>

</property

Regular operations like Create a directory, List all directories, Open a directory/file, delete a directory/file and etc.. are straight forward operations
We have to give appropriate operator for op=<operation_type> in the WebHDFS URL
create/upload a file to HDFS is a little complex. So let us see how to upload a file into HDFS using WebHDFS Rest API

Upload a file into HDFS using WEBHDFS Rest API in Java Jersey Application
Jersey RESTful Web Services framework is open source for developing RESTful Web Services in Java and provides support for JAX-RS APIs and serves as a JAX-RS
Stack I used here is:

Java version 1.7
Jersey version 1.16
Hadoop version 2.6.0
Jetty version 9.3.11

Step 1:

Please create a Java web project. It should be Jersey rest application

Step 2:

Write a controller createFile/uploadFile()
java

// package
//imports
@Path("/work")
public class WebhdfsDemoController {
 String baseUrl = "http://52.200.193.76:50070/webhdfs/v1/";
 String userName = "hduser";

 @Path("/createFile")
 @POST
 @Consumes(MediaType.MULTIPART_FORM_DATA)
 @Produces(MediaType.APPLICATION_JSON)
 public void createFile(@HeaderParam("file_name") String fileName, 
@FormDataParam("file") InputStreaminputStream) {
  System.out.println("Entered into WebhdfsDemoController.createFile() controller method");
  String output = null;

  try {
   String baseUrl = "http://<datanode_host>:<datanode_ip>/webhdfs/v1/";

   String mkdirUrl = fileName + "?user.name=" + userName + "&op=CREATE"
        
+"&namenoderpcaddress=localhost:54310&overwrite=true";

   mkdirUrl = baseUrl + mkdirUrl;

   System.out.println("createDirectory() hitting the url is: " + mkdirUrl);

   WebHDFSFileUploadService service = new WebHDFSFileUploadService();
   
   File sourceFile = //Write a core java logic to convert inputstream object to File object
   
   try {
    output = service.uploadFile(mkdirUrl, sourceFile);
   } catch (ParseException e) {
    e.printStackTrace();
   }

   System.out.println("createFile output: " + output);
  } catch (MalformedURLException e) {
   e.printStackTrace();
  } catch (IOExceptionioe) {
   ioe.printStackTrace();
  }

  System.out.println("Exiting from WebhdfsDemoController.createFile() controller method");
 }}

// package

//imports

@Path("/work")

public class WebhdfsDemoController {

String baseUrl = "http://52.200.193.76:50070/webhdfs/v1/";

String userName = "hduser";

@Path("/createFile")

@POST

@Consumes(MediaType.MULTIPART_FORM_DATA)

@Produces(MediaType.APPLICATION_JSON)

public void createFile(@HeaderParam("file_name") String fileName,

@FormDataParam("file") InputStreaminputStream) {

System.out.println("Entered into WebhdfsDemoController.createFile() controller method");

String output = null;

try {

String baseUrl = "http://<datanode_host>:<datanode_ip>/webhdfs/v1/";

String mkdirUrl = fileName + "?user.name=" + userName + "&op=CREATE"

+"&namenoderpcaddress=localhost:54310&overwrite=true";

mkdirUrl = baseUrl + mkdirUrl;

System.out.println("createDirectory() hitting the url is: " + mkdirUrl);

WebHDFSFileUploadService service = new WebHDFSFileUploadService();

File sourceFile = //Write a core java logic to convert inputstream object to File object

try {

output = service.uploadFile(mkdirUrl, sourceFile);

} catch (ParseException e) {

e.printStackTrace();

}

System.out.println("createFile output: " + output);

} catch (MalformedURLException e) {

e.printStackTrace();

} catch (IOExceptionioe) {

ioe.printStackTrace();

}

System.out.println("Exiting from WebhdfsDemoController.createFile() controller method");

}}

Step 3:
• Write a Service method to upload file to HDFS using HttpUrlConnection
• WebHDFSFileUploadService.java

// package
// imports
public class WebHDFSFileUploadService {
 public String uploadFile(String urlTo, File file) throws ParseException, IOException {
  System.out.println("Entered into WebHDFSFileUploadService.uploadFile() controller method");
  HttpURLConnection connection = null;
  DataOutputStreamoutputStream = null;
  InputStreaminputStream = null;

  String twoHyphens = "--";
  String boundary = "*****" + Long.toString(System.currentTimeMillis()) + "*****";
  String lineEnd = "\r\n";

  String result = "";

  intbytesRead, bytesAvailable, bufferSize;
  byte[] buffer;
  intmaxBufferSize = 1 * 1024 * 1024;

  try {
   FileInputStreamfileInputStream = new FileInputStream(file);

   URL url = new URL(urlTo);
   connection = (HttpURLConnection) url.openConnection();

   connection.setDoInput(true);
   connection.setDoOutput(true);
   connection.setUseCaches(false);

   connection.setRequestMethod("PUT");
   connection.setRequestProperty("Connection", "Keep-Alive");
   connection.setRequestProperty("User-Agent", "Mozilla/5.0");
   connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);

   outputStream = new DataOutputStream(connection.getOutputStream());
   outputStream.writeBytes(twoHyphens + boundary + lineEnd);
   outputStream.writeBytes("Content-Type: multipart/form-data" + lineEnd);
   outputStream.writeBytes(lineEnd);

   bytesAvailable = fileInputStream.available();
   bufferSize = Math.min(bytesAvailable, maxBufferSize);
   buffer = new byte[bufferSize];

   bytesRead = fileInputStream.read(buffer, 0, bufferSize);
   while (bytesRead> 0) {
    outputStream.write(buffer, 0, bufferSize);
    bytesAvailable = fileInputStream.available();
    bufferSize = Math.min(bytesAvailable, maxBufferSize);
    bytesRead = fileInputStream.read(buffer, 0, bufferSize);
   }

   outputStream.writeBytes(lineEnd);

   outputStream.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);

   inputStream = connection.getInputStream();
   result = this.convertStreamToString(inputStream);

   fileInputStream.close();
   inputStream.close();
   outputStream.flush();
   outputStream.close();

   System.out.println("Exiting from WebHDFSFileUploadService.uploadFile() controller method");
   return result;
  } catch (Exception e) {
   System.out.println("Multipart Form Upload Error");
   e.printStackTrace();
   System.out.println("Exiting from WebHDFSFileUploadService.uploadFile() controller method");
   return "error";
  }
 }

 private String convertStreamToString(InputStream is) {
  BufferedReader reader = new BufferedReader(new InputStreamReader(is));
  StringBuildersb = new StringBuilder();

  String line = null;
  try {
   while ((line = reader.readLine()) != null) {
    sb.append(line);
   }
  } catch (IOException e) {
   e.printStackTrace();
  } finally {
   try {
    is.close();
   } catch (IOException e) {
    e.printStackTrace();
   }
  }
  returnsb.toString();
 }

// package

// imports

public class WebHDFSFileUploadService {

public String uploadFile(String urlTo, File file) throws ParseException, IOException {

System.out.println("Entered into WebHDFSFileUploadService.uploadFile() controller method");

HttpURLConnection connection = null;

DataOutputStreamoutputStream = null;

InputStreaminputStream = null;

String twoHyphens = "--";

String boundary = "*****" + Long.toString(System.currentTimeMillis()) + "*****";

String lineEnd = "\r\n";

String result = "";

intbytesRead, bytesAvailable, bufferSize;

byte[] buffer;

intmaxBufferSize = 1 * 1024 * 1024;

try {

FileInputStreamfileInputStream = new FileInputStream(file);

URL url = new URL(urlTo);

connection = (HttpURLConnection) url.openConnection();

connection.setDoInput(true);

connection.setDoOutput(true);

connection.setUseCaches(false);

connection.setRequestMethod("PUT");

connection.setRequestProperty("Connection", "Keep-Alive");

connection.setRequestProperty("User-Agent", "Mozilla/5.0");

connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);

outputStream = new DataOutputStream(connection.getOutputStream());

outputStream.writeBytes(twoHyphens + boundary + lineEnd);

outputStream.writeBytes("Content-Type: multipart/form-data" + lineEnd);

outputStream.writeBytes(lineEnd);

bytesAvailable = fileInputStream.available();

bufferSize = Math.min(bytesAvailable, maxBufferSize);

buffer = new byte[bufferSize];

bytesRead = fileInputStream.read(buffer, 0, bufferSize);

while (bytesRead> 0) {

outputStream.write(buffer, 0, bufferSize);

bytesAvailable = fileInputStream.available();

bufferSize = Math.min(bytesAvailable, maxBufferSize);

bytesRead = fileInputStream.read(buffer, 0, bufferSize);

}

outputStream.writeBytes(lineEnd);

outputStream.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);

inputStream = connection.getInputStream();

result = this.convertStreamToString(inputStream);

fileInputStream.close();

inputStream.close();

outputStream.flush();

outputStream.close();

System.out.println("Exiting from WebHDFSFileUploadService.uploadFile() controller method");

return result;

} catch (Exception e) {

System.out.println("Multipart Form Upload Error");

e.printStackTrace();

System.out.println("Exiting from WebHDFSFileUploadService.uploadFile() controller method");

return "error";

}

private String convertStreamToString(InputStream is) {

BufferedReader reader = new BufferedReader(new InputStreamReader(is));

StringBuildersb = new StringBuilder();

String line = null;

try {

while ((line = reader.readLine()) != null) {

sb.append(line);

}

} catch (IOException e) {

e.printStackTrace();

} finally {

try {

is.close();

} catch (IOException e) {

e.printStackTrace();

}

returnsb.toString();

}

Step 4:
Test your application
• Make a war file of your application and deploy your application into Jetty server
• Open any rest console and provide below URL
o http://://work/createFile
o Inputs:
 Header Param: . file_name : //Uploaded file with be saved with this name
 Choose a file from rest console

Example: Input I gave here is:
- file_name : test1/files/sample-test.txt
- file : test-file.txt (Selecting this file from local directory)
I have chosen the file test-file.txt from local directory. This file will be saved to HDFS in the given path: /test1/files/sample-test.txt
Then check whether the file is uploaded in the given path, using Hadoop fs command
Login to putty and give the command:

hadoopfs –ls /test1/files/

1	hadoopfs –ls /test1/files/

• You will see sample-test.txt file in the given path
• To view the content of the file,

hadoopfs –cat /test1/files/sample-test.txt

1	hadoopfs –cat /test1/files/sample-test.txt

• It will display the file content on the screen
Thus upload of the file to HDFS using WebHDFS API from a Java Jersey Application is successful.
Here I used HttpURLConnection to upload my files to HDFS

Rechercher dans ce blog

Big data

Manage files in HDFS using WEBHDFS REST APIs

Commentaires

Enregistrer un commentaire

Posts les plus consultés de ce blog

Controlling Parallelism in Spark by controlling the input partitions by controlling the input partitions

Spark performance optimization: shuffle tuning

Spark optimization