org.apache.storm.utils

Class ServerUtils

java.lang.Object

org.apache.storm.utils.ServerUtils

public class ServerUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	IS_ON_WINDOWS
static org.slf4j.Logger	LOG
static int	SIGKILL
static int	SIGTERM

Constructor Summary

Constructors

Constructor and Description
ServerUtils()

Method Summary

Modifier and Type	Method and Description
static boolean	areAllProcessesDead(Map<String,Object> conf, String user, String workerId, Set<Long> pids) Find if all processes for the user on workId are dead.
static boolean	canUserReadBlob(ReadableBlobMeta meta, String user, Map<String,Object> conf)
static String	containerFilePath(String dir)
static String	currentClasspath() Returns the value of java.class.path System property.
String	currentClasspathImpl()
static int	execCommand(String... command)
static void	extractZipFile(ZipFile zipFile, File toDir, String prefix) Extracts the given file to the given directory.
static void	forceKillProcess(String pid)
static ClientBlobStore	getClientBlobStoreForSupervisor(Map<String,Object> conf)
static int	getComponentParallelism(Map<String,Object> topoConf, Object component)
static Map<String,Integer>	getComponentParallelism(Map<String,Object> topoConf, StormTopology topology)
static long	getDiskUsage(File dir) Takes an input dir or file and returns the disk usage on that local directory.
static double	getEstimatedTotalHeapMemoryRequiredByTopo(Map<String,Object> topoConf, StormTopology topology)
static int	getEstimatedWorkerCountForRasTopo(Map<String,Object> topoConf, StormTopology topology)
static String	getFileOwner(String path)
static long	getMemInfoFreeMb() Get system free memory in megabytes.
static BlobStore	getNimbusBlobStore(Map<String,Object> conf, NimbusInfo nimbusInfo, ILeaderElector leaderElector)
static BlobStore	getNimbusBlobStore(Map<String,Object> conf, String baseDir, NimbusInfo nimbusInfo, ILeaderElector leaderElector)
static int	getPathOwnerUid(String fpath) Get the userId of the onwer of the path by running “ls -dn path” command.
static URL	getResourceFromClassloader(String name) Returns the current thread classloader.
URL	getResourceFromClassloaderImpl(String name)
static int	getUserId(String user) Get the userId for a user name.
static <T> List<T>	interleaveAll(List<List<T>> nodeList)
static boolean	isAbsolutePath(String path)
static boolean	isAnyPosixProcessPidDirAlive(Collection<Long> pids, String user) Find if the process is alive using the existence of /proc/<pid> directory owned by the supplied user.
static boolean	isAnyPosixProcessPidDirAlive(Collection<Long> pids, String expectedUser, boolean mockFileOwnerToUid) Find if the process is alive using the existence of /proc/<pid> directory owned by the supplied expectedUser.
static boolean	isAnyProcessAlive(Collection<Long> pids, int uid) Are any of the processes alive and running for the specified userId.
static boolean	isAnyProcessAlive(Collection<Long> pids, String user) Are any of the processes alive and running for the specified user.
static boolean	isProcessAlive(long pid, String user) Is a process alive and running?.
static boolean	isRas(Map<String,Object> conf) Check if the scheduler is resource aware or not.
static void	killProcessWithSigTerm(String pid)
static long	nimbusVersionOfBlob(String key, ClientBlobStore cb)
static Subject	principalNameToSubject(String name)
static String	scriptFilePath(String dir)
static void	sendSignalToProcess(long lpid, int signum)
static ServerUtils	setInstance(ServerUtils u) Provide an instance of this class for delegates to use.
static String	shellCmd(List<String> command) Returns the combined string, escaped for posix shell.
static void	unJar(File jarFile, File toDir) Unpack matching files from a jar.
static void	unpack(File localrsrc, File dst, boolean symLinksDisabled)
static void	unTar(File inFile, File untarDir, boolean symlinksDisabled) Given a Tar File as input it will untar the file in a the untar directory passed as the second parameter This utility will untar “.tar” files and “.tar.gz”,“tgz” files.
static void	unZip(File inFile, File toDir) Given a File input it will unzip the file in a the unzip directory passed as the second parameter.
static void	validateTopologyAckerBundleResource(Map<String,Object> topoConf, StormTopology topology, String topoName) RAS scheduler will try to distribute ackers evenly over workers by adding some ackers to each newly launched worker.
static void	validateTopologyWorkerMaxHeapSizeConfigs(Map<String,Object> stormConf, StormTopology topology, double defaultWorkerMaxHeapSizeMb)
static String	writeScript(String dir, List<String> command, Map<String,String> environment) Writes a posix shell script file to be executed in its own process.
static String	writeScript(String dir, List<String> command, Map<String,String> environment, String umask) Writes a posix shell script file to be executed in its own process.
static boolean	zipDoesContainDir(String zipfile, String target) Determines if a zip archive contains a particular directory.
static long	zipFileSize(File myFile) Given a zip File input it will return its size Only works for zip files whose uncompressed size is less than 4 GB, otherwise returns the size module 2^32, per gzip specifications.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

IS_ON_WINDOWS

public static final boolean IS_ON_WINDOWS

SIGKILL

public static final int SIGKILL

See Also:

Constant Field Values

SIGTERM

public static final int SIGTERM

See Also:

Constant Field Values

Constructor Detail

ServerUtils

public ServerUtils()

Method Detail

setInstance

public static ServerUtils setInstance(ServerUtils u)

Provide an instance of this class for delegates to use. To mock out delegated methods, provide an instance of a subclass that overrides the implementation of the delegated method.

Parameters:

u - a ServerUtils instance

Returns:

the previously set instance

interleaveAll

public static <T> List<T> interleaveAll(List<List<T>> nodeList)

getNimbusBlobStore

public static BlobStore getNimbusBlobStore(Map<String,Object> conf,
                                           NimbusInfo nimbusInfo,
                                           ILeaderElector leaderElector)

getNimbusBlobStore

public static BlobStore getNimbusBlobStore(Map<String,Object> conf,
                                           String baseDir,
                                           NimbusInfo nimbusInfo,
                                           ILeaderElector leaderElector)

isAbsolutePath

public static boolean isAbsolutePath(String path)

shellCmd

public static String shellCmd(List<String> command)

Returns the combined string, escaped for posix shell.

Parameters:

command - the list of strings to be combined

Returns:

the resulting command string

getDiskUsage

public static long getDiskUsage(File dir)

Takes an input dir or file and returns the disk usage on that local directory. Very basic implementation.

Parameters:

dir - The input dir to get the disk space of this local dir

Returns:

The total disk space of the input local directory

getClientBlobStoreForSupervisor

public static ClientBlobStore getClientBlobStoreForSupervisor(Map<String,Object> conf)

currentClasspath

public static String currentClasspath()

Returns the value of java.class.path System property. Kept separate for testing.

Returns:

the classpath

getResourceFromClassloader

public static URL getResourceFromClassloader(String name)

Returns the current thread classloader.

zipDoesContainDir

public static boolean zipDoesContainDir(String zipfile,
                                        String target)
                                 throws IOException

Determines if a zip archive contains a particular directory.

Parameters:

zipfile - path to the zipped file

target - directory being looked for in the zip.

Returns:

boolean whether or not the directory exists in the zip.

Throws:

IOException

getFileOwner

public static String getFileOwner(String path)
                           throws IOException

Throws:

IOException

containerFilePath

public static String containerFilePath(String dir)

scriptFilePath

public static String scriptFilePath(String dir)

writeScript

public static String writeScript(String dir,
                                 List<String> command,
                                 Map<String,String> environment)
                          throws IOException

Writes a posix shell script file to be executed in its own process.

Parameters:

dir - the directory under which the script is to be written

command - the command the script is to execute

environment - optional environment variables to set before running the script’s command. May be null.

Returns:

the path to the script that has been written

Throws:

IOException

writeScript

public static String writeScript(String dir,
                                 List<String> command,
                                 Map<String,String> environment,
                                 String umask)
                          throws IOException

Writes a posix shell script file to be executed in its own process.

Parameters:

dir - the directory under which the script is to be written

command - the command the script is to execute

environment - optional environment variables to set before running the script’s command. May be null.

umask - umask to be set. It can be null.

Returns:

the path to the script that has been written

Throws:

IOException

execCommand

public static int execCommand(String... command)
                       throws org.apache.commons.exec.ExecuteException,
                              IOException

Throws:

org.apache.commons.exec.ExecuteException

IOException

sendSignalToProcess

public static void sendSignalToProcess(long lpid,
                                       int signum)
                                throws IOException

Throws:

IOException

killProcessWithSigTerm

public static void killProcessWithSigTerm(String pid)
                                   throws IOException

Throws:

IOException

forceKillProcess

public static void forceKillProcess(String pid)
                             throws IOException

Throws:

IOException

nimbusVersionOfBlob

public static long nimbusVersionOfBlob(String key,
                                       ClientBlobStore cb)
                                throws AuthorizationException,
                                       KeyNotFoundException

Throws:

AuthorizationException

KeyNotFoundException

canUserReadBlob

public static boolean canUserReadBlob(ReadableBlobMeta meta,
                                      String user,
                                      Map<String,Object> conf)

unJar

public static void unJar(File jarFile,
                         File toDir)
                  throws IOException

Unpack matching files from a jar. Entries inside the jar that do not match the given pattern will be skipped.

Parameters:

jarFile - the .jar file to unpack

toDir - the destination directory into which to unpack the jar

Throws:

IOException

unTar

public static void unTar(File inFile,
                         File untarDir,
                         boolean symlinksDisabled)
                  throws IOException

Given a Tar File as input it will untar the file in a the untar directory passed as the second parameter

This utility will untar “.tar” files and “.tar.gz”,“tgz” files.

Parameters:

inFile - The tar file as input

untarDir - The untar directory where to untar the tar file

symlinksDisabled - true if symlinks should be disabled, else false

Throws:

IOException

unpack

public static void unpack(File localrsrc,
                          File dst,
                          boolean symLinksDisabled)
                   throws IOException

Throws:

IOException

extractZipFile

public static void extractZipFile(ZipFile zipFile,
                                  File toDir,
                                  String prefix)
                           throws IOException

Extracts the given file to the given directory. Only zip entries starting with the given prefix are extracted. The prefix is stripped off entry names before extraction.

Parameters:

zipFile - The zip file to extract

toDir - The directory to extract to

prefix - The prefix to look for in the zip file. If not null only paths starting with the prefix will be extracted

Throws:

IOException

unZip

public static void unZip(File inFile,
                         File toDir)
                  throws IOException

Given a File input it will unzip the file in a the unzip directory passed as the second parameter.

Parameters:

inFile - The zip file as input

toDir - The unzip directory where to unzip the zip file

Throws:

IOException

zipFileSize

public static long zipFileSize(File myFile)
                        throws IOException

Given a zip File input it will return its size Only works for zip files whose uncompressed size is less than 4 GB, otherwise returns the size module 2^32, per gzip specifications.

Parameters:

myFile - The zip file as input

Returns:

zip file size as a long

Throws:

IOException

isRas

public static boolean isRas(Map<String,Object> conf)

Check if the scheduler is resource aware or not.

Parameters:

conf - The configuration

Returns:

True if it’s resource aware; false otherwise

getEstimatedWorkerCountForRasTopo

public static int getEstimatedWorkerCountForRasTopo(Map<String,Object> topoConf,
                                                    StormTopology topology)
                                             throws InvalidTopologyException

Throws:

InvalidTopologyException

getEstimatedTotalHeapMemoryRequiredByTopo

public static double getEstimatedTotalHeapMemoryRequiredByTopo(Map<String,Object> topoConf,
                                                               StormTopology topology)
                                                        throws InvalidTopologyException

Throws:

InvalidTopologyException

getComponentParallelism

public static Map<String,Integer> getComponentParallelism(Map<String,Object> topoConf,
                                                          StormTopology topology)
                                                   throws InvalidTopologyException

Throws:

InvalidTopologyException

getComponentParallelism

public static int getComponentParallelism(Map<String,Object> topoConf,
                                          Object component)
                                   throws InvalidTopologyException

Throws:

InvalidTopologyException

principalNameToSubject

public static Subject principalNameToSubject(String name)

currentClasspathImpl

public String currentClasspathImpl()

getResourceFromClassloaderImpl

public URL getResourceFromClassloaderImpl(String name)

getMemInfoFreeMb

public static long getMemInfoFreeMb()
                             throws IOException

Get system free memory in megabytes.

Returns:

system free memory in megabytes

Throws:

IOException - on I/O exception

isProcessAlive

public static boolean isProcessAlive(long pid,
                                     String user)
                              throws IOException

Is a process alive and running?.

Parameters:

pid - the PID of the running process

user - the user that is expected to own that process

Returns:

true if it is, else false

Throws:

IOException - on any error

isAnyProcessAlive

public static boolean isAnyProcessAlive(Collection<Long> pids,
                                        String user)
                                 throws IOException

Are any of the processes alive and running for the specified user. If collection is empty or null then the return value is trivially false.

Parameters:

pids - the PIDs of the running processes

user - the user that is expected to own that process

Returns:

true if any one of the processes is owned by user and alive, else false

Throws:

IOException - on I/O exception

isAnyProcessAlive

public static boolean isAnyProcessAlive(Collection<Long> pids,
                                        int uid)
                                 throws IOException

Are any of the processes alive and running for the specified userId. If collection is empty or null then the return value is trivially false.

Parameters:

pids - the PIDs of the running processes

uid - the user that is expected to own that process

Returns:

true if any one of the processes is owned by user and alive, else false

Throws:

IOException - on I/O exception

getUserId

public static int getUserId(String user)

Get the userId for a user name. This works on Posix systems by using “id -u” command. Throw IllegalArgumentException on Windows.

Parameters:

user - username to be converted to UID. This is optional, in which case current user is returned.

Returns:

UID for the specified user (if supplied), else UID of current user, -1 upon Exception.

getPathOwnerUid

public static int getPathOwnerUid(String fpath)

Get the userId of the onwer of the path by running “ls -dn path” command. This command works on Posix systems only.

Parameters:

fpath - full path to the file or directory.

Returns:

UID for the specified if successful, -1 upon failure.

areAllProcessesDead

public static boolean areAllProcessesDead(Map<String,Object> conf,
                                          String user,
                                          String workerId,
                                          Set<Long> pids)
                                   throws IOException

Find if all processes for the user on workId are dead. This method attempts to optimize the calls by:

checking a collection of ProcessIds at once using userId one Posix systems instead of user

Returns:

true if all processes for the user are dead on the worker

Throws:

IOException - if external commands have exception.

isAnyPosixProcessPidDirAlive

public static boolean isAnyPosixProcessPidDirAlive(Collection<Long> pids,
                                                   String user)
                                            throws IOException

Find if the process is alive using the existence of /proc/<pid> directory owned by the supplied user. This is an alternative to “ps -p pid -u uid” command used in isAnyPosixProcessAlive(Collection, int)

Processes are tracked using the existence of the directory "/proc/<pid> For each of the supplied PIDs, their PID directory is checked for existence and ownership by the specified uid.

Parameters:

pids - Process IDs that need to be monitored for liveness

user - the userId that is expected to own that process

Returns:

true if any one of the processes is owned by user and alive, else false

Throws:

IOException - on I/O exception

isAnyPosixProcessPidDirAlive

public static boolean isAnyPosixProcessPidDirAlive(Collection<Long> pids,
                                                   String expectedUser,
                                                   boolean mockFileOwnerToUid)
                                            throws IOException

Find if the process is alive using the existence of /proc/<pid> directory owned by the supplied expectedUser. This is an alternative to “ps -p pid -u uid” command used in isAnyPosixProcessAlive(Collection, int)

Processes are tracked using the existence of the directory "/proc/<pid> For each of the supplied PIDs, their PID directory is checked for existence and ownership by the specified uid.

Parameters:

pids - Process IDs that need to be monitored for liveness

expectedUser - the userId that is expected to own that process

mockFileOwnerToUid - if true (used for testing), then convert File.owner to UID

Returns:

true if any one of the processes is owned by expectedUser and alive, else false

Throws:

IOException - on I/O exception

validateTopologyWorkerMaxHeapSizeConfigs

public static void validateTopologyWorkerMaxHeapSizeConfigs(Map<String,Object> stormConf,
                                                            StormTopology topology,
                                                            double defaultWorkerMaxHeapSizeMb)
                                                     throws InvalidTopologyException

Throws:

InvalidTopologyException

validateTopologyAckerBundleResource

public static void validateTopologyAckerBundleResource(Map<String,Object> topoConf,
                                                       StormTopology topology,
                                                       String topoName)
                                                throws InvalidTopologyException

RAS scheduler will try to distribute ackers evenly over workers by adding some ackers to each newly launched worker. Validations are performed here: (Config.TOPOLOGY_RAS_ACKER_EXECUTORS_PER_WORKER * memory for an acker + memory for the biggest topo executor) < max worker heap memory. When RAS tries to schedule an executor to a new worker, it will put Config.TOPOLOGY_RAS_ACKER_EXECUTORS_PER_WORKER ackers into the worker first. So Config.TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB need to be able to accommodate this.

Parameters:

topoConf - Topology conf

topology - Topology (not system topology)

topoName - The name of the topology

Throws:

InvalidTopologyException