Using Apache VFS for SFTP

Apache VFS(http://commons.apache.org/vfs/) can be used for accessing different file systems. In this post I am describing how we used VFS to get “.gz” files from a remote server to our server. These files are data dumps for one of our projects and we have used external tables to put the data into our database. This post is just to give you an idea on how to approach similar problem/scenario and it is not meant to be a 100% solution.

 

First we need to download all the required jar files for using VFS: -

http://commons.apache.org/vfs/download.html

 For our project we have the following files in our library;

1. apache-ant-1.6.5.jar
2. commons-httpclient-2.0.jar
3. commons-logging-1.1.1.jar
4. commons-logging-adapters-1.1.1.jar
5. commons-logging-api-1.1.1.jar
6. commons-net-1.4.1.jar
7. commons-vfs-20070730.jar
8. jakarta-oro-2.0.8.jar
9. jcifs-1.1.11.jar
10. jsch-0.1.40.jar
11. mail-1.4.1.jar
12. ojdbc14.jar

 

Now, if you are going to put this program on a server (which is the case most of the time), make sure that you can make connection from a command line or by using a SFTP client to the remote server. This will let you know if you are able to make the connection from the server to begin with (or if there is any security setting stopping from making a SFTP connection).

 

Explanation on creating a VFS manager provided at http://commons.apache.org/vfs/api.html might be more helpful than you think. It has described three ways to get you started.

 

-         Simplest Way - VFS.getManager()

-         To configure Commons VFS programmatically.

-         To configure Commons VFS from xml file.

 

A Simple Example: -

 

If you have a “.gz” file you can create a uri as follows. Notice that .txt in the end represents the extension of the file inside the archive file. If you take this option then you have to repeat your SFTP process for every file inside “.gz” archive.

 

String uri = "gz:sftp://" + userName + ":" + password + "@" + serverName + ":" +

port + fileAtLocationServer + "/" + filename+”.” + “.gz” + "!/" + fileInsideGzFile + ".txt"

 

Then you can set the host key. In our case we are just passing a “no” which indicates that no strict host key is used. This is usually the step where lots of things go wrong depending on the security options of a server.

 

FileSystemOptions fsopt = new FileSystemOptions();

SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fsopt,                                                                          "no");

 

fileSystemManager = (StandardFileSystemManager)VFS.getManager();

 

Once you have the File system manager then you pass the actual uri to get your content as a FileObject. Once you have this file object, you can use input/output stream to write the bytes on your machine.

 

fileObject = fileSystemManager.resolveFile(uri, fsopt);

 

Don’t forget to close your file manager.

 

Check out this link for detailed code http://nileshbansal.blogspot.com/2007/07/accessing-files-over-sftp-in-java.html . It was very helpful to me, but remember you can directly get files inside your “.gz” file by using “gz:sftp://” in your uri. This saves you an extra step to unzip them.

 

For help with other file systems see http://commons.apache.org/vfs/filesystems.html

 

Hope it helps a little with your work.