Class HdfsBlobStore
- All Implemented Interfaces:
AutoCloseable
,Shutdownable
We currently have NIMBUS_ADMINS and SUPERVISOR_ADMINS configuration. NIMBUS_ADMINS are given READ, WRITE and ADMIN access whereas the SUPERVISOR_ADMINS are given READ access in order to read and download the blobs form the nimbus.
The ACLs for the blob store are validated against whether the subject is a NIMBUS_ADMIN, SUPERVISOR_ADMIN or USER who has read, write or admin privileges in order to perform respective operations on the blob.
For hdfs blob store 1. The USER interacts with nimbus to upload and access blobs through NimbusBlobStore Client API. Here, unlike local blob store which stores the blobs locally, the nimbus talks to HDFS to upload the blobs. 2. The USER sets the ACLs, and the blob access is validated against these ACLs. 3. The SUPERVISOR interacts with nimbus through HdfsClientBlobStore to download the blobs. Here, unlike local blob store the supervisor interacts with HDFS directly to download the blobs. The call to HdfsBlobStore is made as a "null" subject. The blobstore gets the hadoop user and validates permissions for the supervisor.
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.storm.blobstore.BlobStore
BlobStore.BlobStoreFileInputStream, BlobStore.BlobStoreFileOutputStream, BlobStore.KeyTranslationIterator
-
Field Summary
Fields inherited from class org.apache.storm.blobstore.BlobStore
BASE_BLOBS_DIR_NAME
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionboolean
blobExists
(String key, Subject who) Checks if a blob exists.createBlob
(String key, SettableBlobMeta meta, Subject who) Creates the blob.void
deleteBlob
(String key, Subject who) Deletes the blob data and metadata.void
fullCleanup
(long age) Gets the InputStream to read the blob details.getBlobMeta
(String key, Subject who) Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.int
getBlobReplication
(String key, Subject who) Gets the replication factor of the blob.long
listKeys()
Returns an iterator with all the list of keys currently available on the blob store.void
prepare
(Map<String, Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Allows us to initialize the blob store.protected void
prepareInternal
(Map<String, Object> conf, String overrideBase, org.apache.hadoop.conf.Configuration hadoopConf) Allow a Hadoop Configuration to be passed for testing.void
setBlobMeta
(String key, SettableBlobMeta meta, Subject who) Sets the metadata with renewed acls for the blob.void
setLeaderElector
(ILeaderElector leaderElector) Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.void
shutdown()
updateBlob
(String key, Subject who) Updates the blob data.int
updateBlobReplication
(String key, int replication, Subject who) Modifies the replication factor of the blob.void
Updates the last update time of existing blobs in the blobstore to the current time.void
Validates that the blob update time of the blobstore is up to date with the current existing blobs.void
writeMetadata
(String key, SettableBlobMeta meta) Methods inherited from class org.apache.storm.blobstore.BlobStore
close, createBlob, createBlob, filterAndListKeys, readBlob, readBlobTo, startSyncBlobs, storedTopoIds, updateBlob, validateKey
-
Constructor Details
-
HdfsBlobStore
public HdfsBlobStore()
-
-
Method Details
-
prepare
public void prepare(Map<String, Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Description copied from class:BlobStore
Allows us to initialize the blob store. -
prepareInternal
protected void prepareInternal(Map<String, Object> conf, String overrideBase, org.apache.hadoop.conf.Configuration hadoopConf) Allow a Hadoop Configuration to be passed for testing. If it's null then the hadoop configs must be in your classpath. -
createBlob
public AtomicOutputStream createBlob(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyAlreadyExistsException Description copied from class:BlobStore
Creates the blob.- Specified by:
createBlob
in classBlobStore
- Parameters:
key
- Key for the blobmeta
- Metadata which contains the acls informationwho
- Is the subject creating the blob- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
AuthorizationException
KeyAlreadyExistsException
-
updateBlob
public AtomicOutputStream updateBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStore
Updates the blob data.- Specified by:
updateBlob
in classBlobStore
- Parameters:
key
- Key for the blobwho
- Is the subject having the write privilege for the blob- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
AuthorizationException
KeyNotFoundException
-
getBlobMeta
public ReadableBlobMeta getBlobMeta(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStore
Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.- Specified by:
getBlobMeta
in classBlobStore
- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
AuthorizationException
KeyNotFoundException
-
setLeaderElector
Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.- Specified by:
setLeaderElector
in classBlobStore
- Parameters:
leaderElector
- the leader elector
-
setBlobMeta
public void setBlobMeta(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStore
Sets the metadata with renewed acls for the blob.- Specified by:
setBlobMeta
in classBlobStore
- Parameters:
key
- Key for the blobmeta
- Metadata which contains the updated acls informationwho
- Is the subject having the write privilege for the blob- Throws:
AuthorizationException
KeyNotFoundException
-
deleteBlob
Description copied from class:BlobStore
Deletes the blob data and metadata.- Specified by:
deleteBlob
in classBlobStore
- Parameters:
key
- Key for the blobwho
- Is the subject having write privilege for the blob- Throws:
AuthorizationException
KeyNotFoundException
-
getBlob
public InputStreamWithMeta getBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStore
Gets the InputStream to read the blob details.- Specified by:
getBlob
in classBlobStore
- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Returns:
- InputStreamWithMeta has the additional file length and version information
- Throws:
AuthorizationException
KeyNotFoundException
-
blobExists
Checks if a blob exists.- Parameters:
key
- blobstore keywho
- subject- Throws:
AuthorizationException
- if authorization is failed
-
listKeys
Description copied from class:BlobStore
Returns an iterator with all the list of keys currently available on the blob store. -
shutdown
public void shutdown() -
getBlobReplication
public int getBlobReplication(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStore
Gets the replication factor of the blob.- Specified by:
getBlobReplication
in classBlobStore
- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Returns:
- BlobReplication object containing the replication factor for the blob
- Throws:
AuthorizationException
KeyNotFoundException
-
updateBlobReplication
public int updateBlobReplication(String key, int replication, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStore
Modifies the replication factor of the blob.- Specified by:
updateBlobReplication
in classBlobStore
- Parameters:
key
- Key for the blobreplication
- The replication factor the blob has to be setwho
- Is the subject having the update privilege for the blob- Returns:
- BlobReplication object containing the updated replication factor for the blob
- Throws:
AuthorizationException
KeyNotFoundException
-
writeMetadata
public void writeMetadata(String key, SettableBlobMeta meta) throws AuthorizationException, KeyNotFoundException -
fullCleanup
- Throws:
IOException
-
getLastBlobUpdateTime
- Throws:
IOException
-
updateLastBlobUpdateTime
Description copied from class:BlobStore
Updates the last update time of existing blobs in the blobstore to the current time.- Overrides:
updateLastBlobUpdateTime
in classBlobStore
- Throws:
IOException
- on any error
-
validateBlobUpdateTime
Description copied from class:BlobStore
Validates that the blob update time of the blobstore is up to date with the current existing blobs.- Overrides:
validateBlobUpdateTime
in classBlobStore
- Throws:
IOException
- on any error
-