org.apache.storm.hdfs.blobstore

Class HdfsBlobStoreImpl

java.lang.Object

org.apache.storm.hdfs.blobstore.HdfsBlobStoreImpl

public class HdfsBlobStoreImpl
extends Object

HDFS blob store impl.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	HdfsBlobStoreImpl.KeyInHashDirIterator

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.permission.FsPermission	BLOBSTORE_DIR_PERMISSION

Constructor Summary

Constructors

Constructor and Description
HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path, Map<String,Object> conf)
HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path, Map<String,Object> conf, org.apache.hadoop.conf.Configuration hconf)

Method Summary

Modifier and Type	Method and Description
protected void	delete(org.apache.hadoop.fs.Path path)
void	deleteKey(String key) Delete a key from the blob store.
boolean	exists(String key) Check if the key exists in the blob store.
void	fullCleanup(long age)
protected int	getBlobReplication(String key)
protected org.apache.hadoop.fs.Path	getKeyDir(String key)
long	getLastBlobUpdateTime() Get the last update time of any blob.
protected Iterator<BlobStoreFile>	listBlobStoreFiles(org.apache.hadoop.fs.Path path)
Iterator<String>	listKeys() List relevant keys.
protected Iterator<String>	listKeys(org.apache.hadoop.fs.Path path)
BlobStoreFile	read(String key) Get an input stream for reading a part.
void	shutdown()
protected int	updateBlobReplication(String key, int replication)
void	updateLastBlobUpdateTime() Updates the last updated time of existing blobstores to the current time.
void	validateBlobUpdateTime() Validates that the last updated blob time of the blobstore is up to date with the current existing blobs.
BlobStoreFile	write(String key, boolean create) Get an object tied to writing the data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BLOBSTORE_DIR_PERMISSION

public static final org.apache.hadoop.fs.permission.FsPermission BLOBSTORE_DIR_PERMISSION

Constructor Detail

HdfsBlobStoreImpl

public HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path,
                         Map<String,Object> conf)
                  throws IOException

Throws:

IOException

HdfsBlobStoreImpl

public HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path,
                         Map<String,Object> conf,
                         org.apache.hadoop.conf.Configuration hconf)
                  throws IOException

Throws:

IOException

Method Detail

listKeys

public Iterator<String> listKeys()
                          throws IOException

List relevant keys.

Returns:

all keys that are available for reading

Throws:

IOException - on any error

listKeys

protected Iterator<String> listKeys(org.apache.hadoop.fs.Path path)
                             throws IOException

Throws:

IOException

read

public BlobStoreFile read(String key)
                   throws IOException

Get an input stream for reading a part.

Parameters:

key - the key of the part to read

Returns:

the where to read the data from

Throws:

IOException - on any error

write

public BlobStoreFile write(String key,
                           boolean create)
                    throws IOException

Get an object tied to writing the data.

Parameters:

key - the key of the part to write to.

create - whether the file needs to be new or not.

Returns:

an object that can be used to both write to, but also commit/cancel the operation.

Throws:

IOException - on any error

exists

public boolean exists(String key)

Check if the key exists in the blob store.

Parameters:

key - the key to check for

Returns:

true if it exists else false.

deleteKey

public void deleteKey(String key)
               throws IOException

Delete a key from the blob store.

Parameters:

key - the key to delete

Throws:

IOException - on any error

getKeyDir

protected org.apache.hadoop.fs.Path getKeyDir(String key)

fullCleanup

public void fullCleanup(long age)
                 throws IOException

Throws:

IOException

listBlobStoreFiles

protected Iterator<BlobStoreFile> listBlobStoreFiles(org.apache.hadoop.fs.Path path)
                                              throws IOException

Throws:

IOException

getBlobReplication

protected int getBlobReplication(String key)
                          throws IOException

Throws:

IOException

updateBlobReplication

protected int updateBlobReplication(String key,
                                    int replication)
                             throws IOException

Throws:

IOException

delete

protected void delete(org.apache.hadoop.fs.Path path)
               throws IOException

Throws:

IOException

shutdown

public void shutdown()

getLastBlobUpdateTime

public long getLastBlobUpdateTime()
                           throws IOException

Get the last update time of any blob.

Returns:

the last updated time of blobs within the blobstore.

Throws:

IOException - on any error

updateLastBlobUpdateTime

public void updateLastBlobUpdateTime()
                              throws IOException

Updates the last updated time of existing blobstores to the current time.

Throws:

IOException - on any error

validateBlobUpdateTime

public void validateBlobUpdateTime()
                            throws IOException

Validates that the last updated blob time of the blobstore is up to date with the current existing blobs.

Throws:

IOException - on any error