weka.knowledgeflow

Interface StepManager

All Known Implementing Classes:

StepManagerImpl

public interface StepManager

Client public interface for the StepManager. Step implementations should only use this interface

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CON_AUX_DATA_BATCH_ASSOCIATION_RULES
static java.lang.String	CON_AUX_DATA_CHART_DATA_POINT
static java.lang.String	CON_AUX_DATA_CHART_LEGEND
static java.lang.String	CON_AUX_DATA_CHART_MAX
static java.lang.String	CON_AUX_DATA_CHART_MIN
static java.lang.String	CON_AUX_DATA_CLASS_ATTRIBUTE
static java.lang.String	CON_AUX_DATA_ENVIRONMENT_PROPERTIES
static java.lang.String	CON_AUX_DATA_ENVIRONMENT_RESULTS
static java.lang.String	CON_AUX_DATA_ENVIRONMENT_VARIABLES
static java.lang.String	CON_AUX_DATA_GRAPH_TITLE
static java.lang.String	CON_AUX_DATA_GRAPH_TYPE
static java.lang.String	CON_AUX_DATA_INCREMENTAL_STREAM_END
static java.lang.String	CON_AUX_DATA_INSTANCE
static java.lang.String	CON_AUX_DATA_IS_INCREMENTAL
static java.lang.String	CON_AUX_DATA_LABEL
static java.lang.String	CON_AUX_DATA_MAX_SET_NUM
static java.lang.String	CON_AUX_DATA_PRIMARY_PAYLOAD_NOT_THREAD_SAFE
static java.lang.String	CON_AUX_DATA_SET_NUM
static java.lang.String	CON_AUX_DATA_TEST_INSTANCE
static java.lang.String	CON_AUX_DATA_TESTSET
static java.lang.String	CON_AUX_DATA_TEXT_TITLE
static java.lang.String	CON_AUX_DATA_TRAININGSET
static java.lang.String	CON_BATCH_ASSOCIATOR
static java.lang.String	CON_BATCH_CLASSIFIER
static java.lang.String	CON_BATCH_CLUSTERER
static java.lang.String	CON_CHART
static java.lang.String	CON_DATASET
static java.lang.String	CON_ENVIRONMENT
static java.lang.String	CON_GRAPH
static java.lang.String	CON_IMAGE
static java.lang.String	CON_INCREMENTAL_CLASSIFIER
static java.lang.String	CON_INCREMENTAL_CLUSTERER
static java.lang.String	CON_INFO
static java.lang.String	CON_INSTANCE
static java.lang.String	CON_JOB_FAILURE
static java.lang.String	CON_JOB_SUCCESS
static java.lang.String	CON_TESTSET
static java.lang.String	CON_TEXT
static java.lang.String	CON_THRESHOLD_DATA
static java.lang.String	CON_TRAININGSET
static java.lang.String	CON_VISUALIZABLE_ERROR

Method Summary

Modifier and Type	Method and Description
java.lang.String	environmentSubstitute(java.lang.String source) Substitute all known environment variables in the given string
StepManager	findStepInFlow(java.lang.String stepNameToFind) Finds a named step in the current flow.
void	finished() Step implementations processing batch data should call this to indicate that they have finished all processing.
ExecutionEnvironment	getExecutionEnvironment() Get the executing environment.
java.util.List<StepManager>	getIncomingConnectedStepsOfConnectionType(java.lang.String connectionName) Get a list of steps that are the source of incoming connections of the given type
StepManager	getIncomingConnectedStepWithName(java.lang.String stepName) Get the named step that is connected with an incoming connection.
java.util.Map<java.lang.String,java.util.List<StepManager>>	getIncomingConnections() Get a Map of all incoming connections.
Instances	getIncomingStructureForConnectionType(java.lang.String connectionName) Attempt to retrieve the structure (as a header-only set of instances) for the named incoming connection type.
Instances	getIncomingStructureForConnectionType(java.lang.String connectionName, Environment env) Attempt to retrieve the structure (as a header-only set of instances) for the named incoming connection type.
Instances	getIncomingStructureFromStep(StepManager sourceStep, java.lang.String connectionName) Attempt to get the incoming structure (as a header-only set of instances) from the given managed step for the given connection type.
Step	getInfoStep() Returns a reference to the step being managed if it has one or more outgoing CON_INFO connections.
Step	getInfoStep(java.lang.Class stepClass) Returns a reference to the step being managed if it has one or more outgoing CON_INFO connections and the managed step is of the supplied class
Logger	getLog() Get the log
LoggingLevel	getLoggingLevel() Get the currently set logging level
Step	getManagedStep() Get the actual step managed by this step manager
java.lang.String	getName() Get the name of the step managed by this StepManager
java.util.List<StepManager>	getOutgoingConnectedStepsOfConnectionType(java.lang.String connectionName) Get a list of downstream steps connected to this step with the given connection type.
StepManager	getOutgoingConnectedStepWithName(java.lang.String stepName) Get a named step connected to this step with an outgoing connection
java.util.Map<java.lang.String,java.util.List<StepManager>>	getOutgoingConnections() Get a Map of all outgoing connections.
Settings	getSettings() Get the knowledge flow settings
boolean	getStepMustRunSingleThreaded() Returns true if the step managed by this step manager has been marked as one that must run single-threaded.
void	interrupted() Step implementations processing batch data should call this as soon as they have finished processing after a stop has been requested.
boolean	isStepBusy() Returns true if, at this time, the step managed by this step manager is currently busy with processing
boolean	isStepFinished() Return true if the current step is finished.
boolean	isStopRequested() Return true if a stop has been requested by the runtime environment
boolean	isStreamFinished(Data data) Returns true if this data object marks the end of an incremental stream.
void	log(java.lang.String message, LoggingLevel level) Write a message to the log at the given logging level
void	logBasic(java.lang.String message) Log a message at the "basic" level
void	logDebug(java.lang.String message) Log a message at the "debug" level
void	logDetailed(java.lang.String message) Log a message at the "detailed" level
void	logError(java.lang.String message, java.lang.Throwable cause) Log an error message.
void	logLow(java.lang.String message) Log a message at the "low" level
void	logWarning(java.lang.String message) Log a warning message.
int	numIncomingConnections() Get the number of steps that are connected with incoming connections
int	numIncomingConnectionsOfType(java.lang.String connectionName) Get the number of steps that are connected with the given incoming connection type
int	numOutgoingConnections() Get the number of steps that are connected with outgoing connections
int	numOutgoingConnectionsOfType(java.lang.String connectionName) Get the number of steps that are connected with the given outgoing connection type
void	outputData(Data... data) Output one or more Data objects to all relevant steps.
void	outputData(java.lang.String outgoingConnectionName, Data data) Output data to all steps connected with the supplied outgoing connection type.
void	outputData(java.lang.String outgoingConnectionName, java.lang.String stepName, Data data) Output a single Data object to the named step with the supplied outgoing connection type
void	processing() Step implementations processing batch data should call this to indicate that they have started some processing.
void	setStepIsResourceIntensive(boolean isResourceIntensive) Mark the step managed by this step manager as resource intensive
void	setStepMustRunSingleThreaded(boolean mustRunSingleThreaded) Marked the step managed by this step manager as one that must run single-threaded.
void	statusMessage(java.lang.String message) Write a status message
boolean	stepIsResourceIntensive() Returns true if the step managed by this step manager has been marked as being resource (cpu/memory) intensive.
void	throughputFinished(Data... data) Signal that throughput measurement has finished.
void	throughputUpdateEnd() End a throughput measurement.
void	throughputUpdateStart() Start a throughput measurement.

Field Detail

CON_DATASET

static final java.lang.String CON_DATASET

See Also:

Constant Field Values

CON_INSTANCE

static final java.lang.String CON_INSTANCE

See Also:

Constant Field Values

CON_TRAININGSET

static final java.lang.String CON_TRAININGSET

See Also:

Constant Field Values

CON_TESTSET

static final java.lang.String CON_TESTSET

See Also:

Constant Field Values

CON_BATCH_CLASSIFIER

static final java.lang.String CON_BATCH_CLASSIFIER

See Also:

Constant Field Values

CON_INCREMENTAL_CLASSIFIER

static final java.lang.String CON_INCREMENTAL_CLASSIFIER

See Also:

Constant Field Values

CON_INCREMENTAL_CLUSTERER

static final java.lang.String CON_INCREMENTAL_CLUSTERER

See Also:

Constant Field Values

CON_BATCH_CLUSTERER

static final java.lang.String CON_BATCH_CLUSTERER

See Also:

Constant Field Values

CON_BATCH_ASSOCIATOR

static final java.lang.String CON_BATCH_ASSOCIATOR

See Also:

Constant Field Values

CON_VISUALIZABLE_ERROR

static final java.lang.String CON_VISUALIZABLE_ERROR

See Also:

Constant Field Values

CON_THRESHOLD_DATA

static final java.lang.String CON_THRESHOLD_DATA

See Also:

Constant Field Values

CON_TEXT

static final java.lang.String CON_TEXT

See Also:

Constant Field Values

CON_IMAGE

static final java.lang.String CON_IMAGE

See Also:

Constant Field Values

CON_GRAPH

static final java.lang.String CON_GRAPH

See Also:

Constant Field Values

CON_CHART

static final java.lang.String CON_CHART

See Also:

Constant Field Values

CON_INFO

static final java.lang.String CON_INFO

See Also:

Constant Field Values

CON_ENVIRONMENT

static final java.lang.String CON_ENVIRONMENT

See Also:

Constant Field Values

CON_JOB_SUCCESS

static final java.lang.String CON_JOB_SUCCESS

See Also:

Constant Field Values

CON_JOB_FAILURE

static final java.lang.String CON_JOB_FAILURE

See Also:

Constant Field Values

CON_AUX_DATA_SET_NUM

static final java.lang.String CON_AUX_DATA_SET_NUM

See Also:

Constant Field Values

CON_AUX_DATA_MAX_SET_NUM

static final java.lang.String CON_AUX_DATA_MAX_SET_NUM

See Also:

Constant Field Values

CON_AUX_DATA_TEST_INSTANCE

static final java.lang.String CON_AUX_DATA_TEST_INSTANCE

See Also:

Constant Field Values

CON_AUX_DATA_TESTSET

static final java.lang.String CON_AUX_DATA_TESTSET

See Also:

Constant Field Values

CON_AUX_DATA_TRAININGSET

static final java.lang.String CON_AUX_DATA_TRAININGSET

See Also:

Constant Field Values

CON_AUX_DATA_INSTANCE

static final java.lang.String CON_AUX_DATA_INSTANCE

See Also:

Constant Field Values

CON_AUX_DATA_TEXT_TITLE

static final java.lang.String CON_AUX_DATA_TEXT_TITLE

See Also:

Constant Field Values

CON_AUX_DATA_LABEL

static final java.lang.String CON_AUX_DATA_LABEL

See Also:

Constant Field Values

CON_AUX_DATA_CLASS_ATTRIBUTE

static final java.lang.String CON_AUX_DATA_CLASS_ATTRIBUTE

See Also:

Constant Field Values

CON_AUX_DATA_GRAPH_TITLE

static final java.lang.String CON_AUX_DATA_GRAPH_TITLE

See Also:

Constant Field Values

CON_AUX_DATA_GRAPH_TYPE

static final java.lang.String CON_AUX_DATA_GRAPH_TYPE

See Also:

Constant Field Values

CON_AUX_DATA_CHART_MAX

static final java.lang.String CON_AUX_DATA_CHART_MAX

See Also:

Constant Field Values

CON_AUX_DATA_CHART_MIN

static final java.lang.String CON_AUX_DATA_CHART_MIN

See Also:

Constant Field Values

CON_AUX_DATA_CHART_DATA_POINT

static final java.lang.String CON_AUX_DATA_CHART_DATA_POINT

See Also:

Constant Field Values

CON_AUX_DATA_CHART_LEGEND

static final java.lang.String CON_AUX_DATA_CHART_LEGEND

See Also:

Constant Field Values

CON_AUX_DATA_ENVIRONMENT_VARIABLES

static final java.lang.String CON_AUX_DATA_ENVIRONMENT_VARIABLES

See Also:

Constant Field Values

CON_AUX_DATA_ENVIRONMENT_PROPERTIES

static final java.lang.String CON_AUX_DATA_ENVIRONMENT_PROPERTIES

See Also:

Constant Field Values

CON_AUX_DATA_ENVIRONMENT_RESULTS

static final java.lang.String CON_AUX_DATA_ENVIRONMENT_RESULTS

See Also:

Constant Field Values

CON_AUX_DATA_BATCH_ASSOCIATION_RULES

static final java.lang.String CON_AUX_DATA_BATCH_ASSOCIATION_RULES

See Also:

Constant Field Values

CON_AUX_DATA_INCREMENTAL_STREAM_END

static final java.lang.String CON_AUX_DATA_INCREMENTAL_STREAM_END

See Also:

Constant Field Values

CON_AUX_DATA_IS_INCREMENTAL

static final java.lang.String CON_AUX_DATA_IS_INCREMENTAL

See Also:

Constant Field Values

CON_AUX_DATA_PRIMARY_PAYLOAD_NOT_THREAD_SAFE

static final java.lang.String CON_AUX_DATA_PRIMARY_PAYLOAD_NOT_THREAD_SAFE

See Also:

Constant Field Values

Method Detail

getName

java.lang.String getName()

Get the name of the step managed by this StepManager

Returns:

the name of the managed step

getManagedStep

Step getManagedStep()

Get the actual step managed by this step manager

Returns:

the Step managed by this step manager

getExecutionEnvironment

ExecutionEnvironment getExecutionEnvironment()

Get the executing environment. This contains information such as whether the flow is running in headless environment, what environment variables are available and methods to execute units of work in parallel.

Returns:

the execution environment

getSettings

Settings getSettings()

Get the knowledge flow settings

Returns:

the knowledge flow settings

numIncomingConnections

int numIncomingConnections()

Get the number of steps that are connected with incoming connections

Returns:

the number of incoming connections

numOutgoingConnections

int numOutgoingConnections()

Get the number of steps that are connected with outgoing connections

Returns:

the number of outgoing connections

numIncomingConnectionsOfType

int numIncomingConnectionsOfType(java.lang.String connectionName)

Get the number of steps that are connected with the given incoming connection type

Parameters:

connectionName - the type of the incoming connection

Returns:

the number of steps connected with the specified incoming connection type

numOutgoingConnectionsOfType

int numOutgoingConnectionsOfType(java.lang.String connectionName)

Get the number of steps that are connected with the given outgoing connection type

Parameters:

connectionName - the type of the outgoing connection

Returns:

the number of steps connected with the specified outgoing connection type

getIncomingConnectedStepsOfConnectionType

java.util.List<StepManager> getIncomingConnectedStepsOfConnectionType(java.lang.String connectionName)

Get a list of steps that are the source of incoming connections of the given type

Parameters:

connectionName - the name of the incoming connection to get a list of steps for

Returns:

a list of steps that are the source of incoming connections of the given type

getIncomingConnectedStepWithName

StepManager getIncomingConnectedStepWithName(java.lang.String stepName)

Get the named step that is connected with an incoming connection.

Parameters:

stepName - the name of the step to get

Returns:

the step connected with an incoming connection or null if the named step is not connected

getOutgoingConnectedStepWithName

StepManager getOutgoingConnectedStepWithName(java.lang.String stepName)

Get a named step connected to this step with an outgoing connection

Parameters:

stepName - the name of the step to look for

Returns:

the connected step

getOutgoingConnectedStepsOfConnectionType

java.util.List<StepManager> getOutgoingConnectedStepsOfConnectionType(java.lang.String connectionName)

Get a list of downstream steps connected to this step with the given connection type.

Parameters:

connectionName - the name of the outgoing connection

Returns:

a list of downstream steps connected to this one with the named connection type

getIncomingConnections

java.util.Map<java.lang.String,java.util.List<StepManager>> getIncomingConnections()

Get a Map of all incoming connections. Map is keyed by connection type; values are lists of steps

Returns:

a Map of incoming connections

getOutgoingConnections

java.util.Map<java.lang.String,java.util.List<StepManager>> getOutgoingConnections()

Get a Map of all outgoing connections. Map is keyed by connection type; values are lists of steps

Returns:

a Map of outgoing connections

outputData

void outputData(java.lang.String outgoingConnectionName,
                Data data)
         throws WekaException

Output data to all steps connected with the supplied outgoing connection type. Populates the source and connection name in the supplied Data object for the client

Parameters:

outgoingConnectionName - the type of the outgoing connection to send data to

data - a single Data object to send

Throws:

WekaException - if a problem occurs

outputData

void outputData(Data... data)
         throws WekaException

Output one or more Data objects to all relevant steps. Populates the source in each Data object for the client, HOWEVER, the client must have populated the connection type in each Data object to be output so that the StepManager knows which connected steps to send the data to. Also notifies any registered StepOutputListeners. Note that the downstream step(s)' processIncoming() method is called in a separate thread for batch connections. Furthermore, if multiple Data objects are supplied via the varargs argument, and a target step will receive more than one of the Data objects, then they will be passed on to the step in question sequentially within the same thread of execution.

Parameters:

data - one or more Data objects to be sent

Throws:

WekaException - if a problem occurs

outputData

void outputData(java.lang.String outgoingConnectionName,
                java.lang.String stepName,
                Data data)
         throws WekaException

Output a single Data object to the named step with the supplied outgoing connection type

Parameters:

outgoingConnectionName - the name of the outgoing connection

stepName - the name of the step to send the data to

data - the data to send

Throws:

WekaException - if a problem occurs

getIncomingStructureForConnectionType

Instances getIncomingStructureForConnectionType(java.lang.String connectionName)
                                         throws WekaException

Attempt to retrieve the structure (as a header-only set of instances) for the named incoming connection type. Assumes that there is only one step connected with the supplied incoming connection type.

Parameters:

connectionName - the type of the incoming connection to get the structure for

Returns:

the structure of the data for the specified incoming connection, or null if the structure can't be determined (or represented as an Instances object)

Throws:

WekaException - if a problem occurs

getIncomingStructureForConnectionType

Instances getIncomingStructureForConnectionType(java.lang.String connectionName,
                                                Environment env)
                                         throws WekaException

Attempt to retrieve the structure (as a header-only set of instances) for the named incoming connection type. Assumes that there is only one step connected with the supplied incoming connection type.

Parameters:

connectionName - the type of the incoming connection to get the structure for

env - the Environment to use

Returns:

the structure of the data for the specified incoming connection, or null if the structure can't be determined (or represented as an Instances object)

Throws:

WekaException - if a problem occurs

getIncomingStructureFromStep

Instances getIncomingStructureFromStep(StepManager sourceStep,
                                       java.lang.String connectionName)
                                throws WekaException

Attempt to get the incoming structure (as a header-only set of instances) from the given managed step for the given connection type.

Parameters:

sourceStep - the step manager managing the source step

connectionName - the name of the connection to attempt to get the structure for

Returns:

the structure as a header-only set of instances, or null if the source step can't determine this at present or if it can't be represented as a set of instances.

Throws:

WekaException - if a problem occurs

isStepBusy

boolean isStepBusy()

Returns true if, at this time, the step managed by this step manager is currently busy with processing

Returns:

true if the step managed by this step manager is busy

isStopRequested

boolean isStopRequested()

Return true if a stop has been requested by the runtime environment

Returns:

true if a stop has been requested

isStepFinished

boolean isStepFinished()

Return true if the current step is finished.

Returns:

true if the current step is finished

processing

void processing()

Step implementations processing batch data should call this to indicate that they have started some processing. Calling this should set the busy flag to true.

finished

void finished()

Step implementations processing batch data should call this to indicate that they have finished all processing. Calling this should set the busy flag to false.

interrupted

void interrupted()

Step implementations processing batch data should call this as soon as they have finished processing after a stop has been requested. Calling this should set the busy flag to false.

isStreamFinished

boolean isStreamFinished(Data data)

Returns true if this data object marks the end of an incremental stream. Note - does not check that the data object is actually an incremental one of some sort! Just checks to see if the CON_AUX_DATA_INCREMENTAL_STREAM_END flag is set to true or not;

Parameters:

data - the data element to check

Returns:

true if the data element is flagged as end of stream

throughputUpdateStart

void throughputUpdateStart()

Start a throughput measurement. Should only be used by steps that are processing instance streams. Call just before performing a unit of work for an incoming instance.

throughputUpdateEnd

void throughputUpdateEnd()

End a throughput measurement. Should only be used by steps that are processing instance streams. Call just after finishing a unit of work for an incoming instance

throughputFinished

void throughputFinished(Data... data)
                 throws WekaException

Signal that throughput measurement has finished. Should only be used by steps that are emitting incremental data. Call as the completion of an data stream.

Parameters:

data - one or more Data events (with appropriate connection type set) to pass on to downstream connected steps. These are used to carry any final data and to inform the downstream step(s) that the stream has ended

Throws:

WekaException - if a problem occurs

logLow

void logLow(java.lang.String message)

Log a message at the "low" level

Parameters:

message - the message to log

logBasic

void logBasic(java.lang.String message)

Log a message at the "basic" level

Parameters:

message - the message to log

logDetailed

void logDetailed(java.lang.String message)

Log a message at the "detailed" level

Parameters:

message - the message to log

logDebug

void logDebug(java.lang.String message)

Log a message at the "debug" level

Parameters:

message - the message to log

logWarning

void logWarning(java.lang.String message)

Log a warning message. Always makes it into the log regardless of what logging level the user has specified.

Parameters:

message - the message to log

logError

void logError(java.lang.String message,
              java.lang.Throwable cause)

Log an error message. Always makes it into the log regardless of what logging level the user has specified. Causes all flow execution to halt. Prints an exception to the log if supplied.

Parameters:

message - the message to log

cause - the optional Throwable to log

log

void log(java.lang.String message,
         LoggingLevel level)

Write a message to the log at the given logging level

Parameters:

message - the message to write

level - the level for the message

statusMessage

void statusMessage(java.lang.String message)

Write a status message

Parameters:

message - the message

getLog

Logger getLog()

Get the log

Returns:

the log object

getLoggingLevel

LoggingLevel getLoggingLevel()

Get the currently set logging level

Returns:

the currently set logging level

environmentSubstitute

java.lang.String environmentSubstitute(java.lang.String source)

Substitute all known environment variables in the given string

Parameters:

source - the source string

Returns:

the source string with all known variables resolved

getInfoStep

Step getInfoStep(java.lang.Class stepClass)
          throws WekaException

Returns a reference to the step being managed if it has one or more outgoing CON_INFO connections and the managed step is of the supplied class

Parameters:

stepClass - the expected class of the step

Returns:

the step being managed if outgoing CON_INFO connections are present and the step is of the supplied class

Throws:

WekaException - if there are no outgoing CON_INFO connections or the managed step is the wrong type

getInfoStep

Step getInfoStep()
          throws WekaException

Returns a reference to the step being managed if it has one or more outgoing CON_INFO connections.

Returns:

the step being managed if outgoing CON_INFO connections are present

Throws:

WekaException - if there are no outgoing CON_INFO connections

findStepInFlow

StepManager findStepInFlow(java.lang.String stepNameToFind)

Finds a named step in the current flow. Returns null if the named step is not present in the flow

Parameters:

stepNameToFind - the name of the step to find

Returns:

the StepManager of the named step, or null if the step does not exist in the current flow.

stepIsResourceIntensive

boolean stepIsResourceIntensive()

Returns true if the step managed by this step manager has been marked as being resource (cpu/memory) intensive.

Returns:

true if the managed step is resource intensive

setStepIsResourceIntensive

void setStepIsResourceIntensive(boolean isResourceIntensive)

Mark the step managed by this step manager as resource intensive

Parameters:

isResourceIntensive - true if the step managed by this step manager is resource intensive

setStepMustRunSingleThreaded

void setStepMustRunSingleThreaded(boolean mustRunSingleThreaded)

Marked the step managed by this step manager as one that must run single-threaded. I.e. in an executor service with one worker thread, thus effectively preventing more than one copy of the step from executing at any one point in time

Parameters:

mustRunSingleThreaded - true if the managed step must run single-threaded

getStepMustRunSingleThreaded

boolean getStepMustRunSingleThreaded()

Returns true if the step managed by this step manager has been marked as one that must run single-threaded. I.e. in an executor service with one worker thread, thus effectively preventing more than one copy of the step from executing at any one point in time

Parameters:

mustRunSingleThreaded - true if the managed step must run single-threaded