Package org.apache.storm.blobstore
Class BlobStore
java.lang.Object
org.apache.storm.blobstore.BlobStore
- All Implemented Interfaces:
AutoCloseable
,Shutdownable
- Direct Known Subclasses:
HdfsBlobStore
,LocalFsBlobStore
Provides a way to store blobs that can be downloaded. Blobs must be able to be uploaded and listed from Nimbus, and
downloaded from the Supervisors. It is a key value based store. Key being a string and value being the blob data.
ACL checking must take place against the provided subject. If the blob store does not support Security it must validate that all ACLs set are always WORLD, everything.
The users can upload their blobs through the blob store command line. The command line also allows us to update and delete blobs.
Modifying the replication factor only works for HdfsBlobStore as for the LocalFsBlobStore the replication is dependent on the number of Nimbodes available.
-
Nested Class Summary
Modifier and TypeClassDescriptionprotected class
Input stream implementation used for writing both the metadata containing the acl information and the blob data.protected class
Output stream implementation used for reading the metadata and data information.static class
Blob store implements its own version of iterator to list the blobs. -
Field Summary
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
void
createBlob
(String key, byte[] data, SettableBlobMeta meta, Subject who) Wrapper called to create the blob which contains the byte data.void
createBlob
(String key, InputStream in, SettableBlobMeta meta, Subject who) Wrapper called to create the blob which contains the byte data.abstract AtomicOutputStream
createBlob
(String key, SettableBlobMeta meta, Subject who) Creates the blob.abstract void
deleteBlob
(String key, Subject who) Deletes the blob data and metadata.<R> Set<R>
filterAndListKeys
(KeyFilter<R> filter) Filters keys based on the KeyFilter passed as the argument.abstract InputStreamWithMeta
Gets the InputStream to read the blob details.abstract ReadableBlobMeta
getBlobMeta
(String key, Subject who) Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.abstract int
getBlobReplication
(String key, Subject who) Gets the replication factor of the blob.listKeys()
Returns an iterator with all the list of keys currently available on the blob store.abstract void
prepare
(Map<String, Object> conf, String baseDir, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Allows us to initialize the blob store.byte[]
Wrapper around readBlobTo which returns a ByteArray output stream.void
readBlobTo
(String key, OutputStream out, Subject who) Reads the blob from the blob store and writes it into the output stream.abstract void
setBlobMeta
(String key, SettableBlobMeta meta, Subject who) Sets the metadata with renewed acls for the blob.abstract void
setLeaderElector
(ILeaderElector leaderElector) Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.void
Start the syncing blobs between the local running instance of the BlobStore and others.Get IDs stored in blob store.void
updateBlob
(String key, byte[] data, Subject who) Wrapper called to create the blob which contains the byte data.abstract AtomicOutputStream
updateBlob
(String key, Subject who) Updates the blob data.abstract int
updateBlobReplication
(String key, int replication, Subject who) Modifies the replication factor of the blob.void
Updates the last update time of existing blobs in the blobstore to the current time.void
Validates that the blob update time of the blobstore is up to date with the current existing blobs.static final void
validateKey
(String key) Validates key checking for potentially harmful patterns.Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.storm.daemon.Shutdownable
shutdown
-
Field Details
-
BASE_BLOBS_DIR_NAME
- See Also:
-
-
Constructor Details
-
BlobStore
public BlobStore()
-
-
Method Details
-
validateKey
Validates key checking for potentially harmful patterns.- Parameters:
key
- Key for the blob- Throws:
IllegalArgumentException
-
prepare
public abstract void prepare(Map<String, Object> conf, String baseDir, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Allows us to initialize the blob store.- Parameters:
conf
- The storm configurationbaseDir
- The directory path to store the blobsnimbusInfo
- Contains the nimbus host, port and leadership information
-
startSyncBlobs
Start the syncing blobs between the local running instance of the BlobStore and others. A no-op for the HdfsBlobStore where HDFS itself does the syncing but for the LocalFsBlobStore ZK state updates are run periodically here. -
createBlob
public abstract AtomicOutputStream createBlob(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyAlreadyExistsException Creates the blob.- Parameters:
key
- Key for the blobmeta
- Metadata which contains the acls informationwho
- Is the subject creating the blob- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
AuthorizationException
KeyAlreadyExistsException
-
createBlob
public void createBlob(String key, byte[] data, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyAlreadyExistsException, IOException Wrapper called to create the blob which contains the byte data.- Parameters:
key
- Key for the blobdata
- Byte data that needs to be uploadedmeta
- Metadata which contains the acls informationwho
- Is the subject creating the blob- Throws:
AuthorizationException
KeyAlreadyExistsException
IOException
-
createBlob
public void createBlob(String key, InputStream in, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyAlreadyExistsException, IOException Wrapper called to create the blob which contains the byte data.- Parameters:
key
- Key for the blobin
- InputStream from which the data is read to be written as a part of the blobmeta
- Metadata which contains the acls informationwho
- Is the subject creating the blob- Throws:
AuthorizationException
KeyAlreadyExistsException
IOException
-
updateBlob
public abstract AtomicOutputStream updateBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Updates the blob data.- Parameters:
key
- Key for the blobwho
- Is the subject having the write privilege for the blob- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
AuthorizationException
KeyNotFoundException
-
updateBlob
public void updateBlob(String key, byte[] data, Subject who) throws AuthorizationException, IOException, KeyNotFoundException Wrapper called to create the blob which contains the byte data.- Parameters:
key
- Key for the blobdata
- Byte data that needs to be uploadedwho
- Is the subject creating the blob- Throws:
AuthorizationException
IOException
KeyNotFoundException
-
getBlobMeta
public abstract ReadableBlobMeta getBlobMeta(String key, Subject who) throws AuthorizationException, KeyNotFoundException Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
AuthorizationException
KeyNotFoundException
-
setLeaderElector
Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi. -
setBlobMeta
public abstract void setBlobMeta(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyNotFoundException Sets the metadata with renewed acls for the blob.- Parameters:
key
- Key for the blobmeta
- Metadata which contains the updated acls informationwho
- Is the subject having the write privilege for the blob- Throws:
AuthorizationException
KeyNotFoundException
-
deleteBlob
public abstract void deleteBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Deletes the blob data and metadata.- Parameters:
key
- Key for the blobwho
- Is the subject having write privilege for the blob- Throws:
AuthorizationException
KeyNotFoundException
-
getBlob
public abstract InputStreamWithMeta getBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Gets the InputStream to read the blob details.- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Returns:
- InputStreamWithMeta has the additional file length and version information
- Throws:
AuthorizationException
KeyNotFoundException
-
listKeys
Returns an iterator with all the list of keys currently available on the blob store.- Returns:
Iterator<String>
-
getBlobReplication
Gets the replication factor of the blob.- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Returns:
- BlobReplication object containing the replication factor for the blob
- Throws:
Exception
-
updateBlobReplication
public abstract int updateBlobReplication(String key, int replication, Subject who) throws AuthorizationException, KeyNotFoundException, IOException Modifies the replication factor of the blob.- Parameters:
key
- Key for the blobreplication
- The replication factor the blob has to be setwho
- Is the subject having the update privilege for the blob- Returns:
- BlobReplication object containing the updated replication factor for the blob
- Throws:
AuthorizationException
KeyNotFoundException
IOException
-
close
public void close()- Specified by:
close
in interfaceAutoCloseable
-
filterAndListKeys
Filters keys based on the KeyFilter passed as the argument.- Parameters:
filter
- KeyFilter- Returns:
- Set of filtered keys
-
readBlobTo
public void readBlobTo(String key, OutputStream out, Subject who) throws IOException, KeyNotFoundException, AuthorizationException Reads the blob from the blob store and writes it into the output stream.- Parameters:
key
- Key for the blobout
- Output streamwho
- Is the subject having read privilege for the blob- Throws:
IOException
KeyNotFoundException
AuthorizationException
-
readBlob
public byte[] readBlob(String key, Subject who) throws IOException, KeyNotFoundException, AuthorizationException Wrapper around readBlobTo which returns a ByteArray output stream.- Parameters:
key
- Key for the blobwho
- Is the subject having the read privilege for the blob- Throws:
IOException
KeyNotFoundException
AuthorizationException
-
storedTopoIds
Get IDs stored in blob store.- Returns:
- a set of all of the topology ids with special data stored in the blob store.
-
updateLastBlobUpdateTime
Updates the last update time of existing blobs in the blobstore to the current time.- Throws:
IOException
- on any error
-
validateBlobUpdateTime
Validates that the blob update time of the blobstore is up to date with the current existing blobs.- Throws:
IOException
- on any error
-