I have been writing on a small white paper on our Reakt language. There’s a first draft posted on this blog a while ago and here’s a second draft. Comments are most welcome.

Reakt™ – The ruleCore CEP language

A White Paper from ruleCore

DRAFT rev2

The Rulecore CEP Language – Reakt! – When you need to react immediately.

Reakt is a language for complex event processing, or CEP for short. It is designed to provide a powerful and productive tool for developers, allowing them to quickly create solutions for detecting critical business situations in real-time.

Reakt is not a traditional programming language. Reakt is a business situation description language. By using a declarative approach it allows the developer to concentrate on what kind of situations to detect and not how

The design goal of Reakt is to provide a way for organisations to react immediately to critical situations. It achieves this goal by finding related events from streams of inbound events. What events a critical situation consists of, and how the events are related, are defined by the user using Reakt. Reakt allows the user to specify conditions and temporal relationships which must hold in order for the events to be considered a noteworthy situation. When the situation is detected by Reakt, an user defined real-time report is created which provides actionable information about the situation. Allowing the enterprise to react immediately.

This short white paper gives an informal description of the language and explains the different language elements using examples instead of formal definitions.

Rulecore Reakt is a small domain specific complex event processing language designed for detecting situations by observing streams of events. An event is a description of something that happens. For example ‘engine started’, ‘order created’, ‘invoice sent’ or other types of events which are relevant to your business. The events originate normally from your internal business systems, from your suppliers or other partners. You can also get events from public data sources like Facebook, Twitter or events extracted from news feeds or weather and traffic data feeds.

A situation is a formal description of a combination, a pattern, of events as they happen over time along with optional restrictions among the events. A simple example of situations that you might want to detect:

"A vehicle has crossed River Thames in London more than 10 times during the last three hours"

or

"ParcelDelivery event came before Order Event unless delivery is of type priority one"

Main Reakt language features:

• Continuous Evaluation – Designed to operate on live streams of events as they happen.
• Declarative – Rules and other items are specified using a high-level declarative approach.
• Modular – Rules are defined using modular and re-usable building blocks.
• Temporal – Handling of time is an integral part of Reakt. Timing restrictions and other temporal conditions can be stated using a number of declarations.
• Location Aware – Reakt includes built in support to understand location information and other geographical objects such as zones.
• Implementation Independent – The elements of Reakt are defined using standard XML and are implementation independent. Different implementations can be created in order to provide specialized execution environments.
• Automatic Life Cycle Management – Rule instances are created and deleted automatically in response to events. Rule instances are analyzed and deleted if their situation is determined to be undetectable.
• Dynamic Context Management – Each rule instance maintains its own virtual view into the streams of inbound events. The view provides a window into the incoming event streams in order to create a dynamic context for rule evaluation.
• Separation of Concerns – Reakt uses reusable blocks for;
  • Selection of events from the inbound stream
  • Rule life cycle management
  • Situation definition
  • Rule trigger reporting

The design goals of Reakt is to provide a language which allows the user to describe complex situations consisting of multiple events with non trivial temporal and geospatial relationships. The language is declarative. Lacking any procedural elements provides a base for formal verification and automated optimization and analysis of the language.

Machine Friendly. A design assumption made early in the language design was that it is likely that computer systems will use Reakt to create and modify rules more frequently than humans. The rate at which new rules need to be created and old ones modified is assumed to be high. Thus the language was designed to be easily understandable by machines and at the same time being reasonably human readable for system developers.

Friendly XML. The current version of Reakt uses XML for its syntax. XML was chosen to take advantage of existing knowledge among developers, known standards and existing tools for generating and parsing. It would also be possible to create alternative syntaxes for Reakt. For example to create a more compact human friendly representation for business users.

EDA Building block. The Reakt language was designed to support execution environments which fits into an architectural style called Event Driven Architecture (EDA). In order be a natural building block in an EDA, the ruleCore Reakt language and its supporting execution environment, the ruleCore CEP Server, was designed to be completely event driven. This purely event driven approach allows all systems in a distributed messaging architecture to integrate with ruleCore using simple events as their only communication mechanism.

The System Information Model, or SIM for short, is both the name of the information model used by ruleCore CEP Server and the name of the run-time data structure modeled after the familiar DOM model, commonly used to represent information originating from XML. The SIM contains all information about different Reakt items such as rules, situations and other types of run-time information.

The SIM is part of the ruleCore CEP Server implementation and not the language definition. But as all the language elements are executed against the SIM, it is described here briefly. Different implementations can implement varying information models. The appearance of the SIM can also vary  in ruleCore CEP Server depending on which dynamically loadable modules are present in the system.

The SIM is used for various purposes, for example:

• Listing available definitions – All definitions of items such as rules, views, zones, entities, situations and actions can be found in the SIM. Definitions are created using the systems events, for example AddRuleDef, AddSituationDef and AddActionDef.
• Listing user defined events – Before an event can be sent into ruleCore CEP Server, it must be defined using the AddEventDef system event. The SIM contains all user defined events.
• List of rule instances – When a rule instance is created, it’s state and internal data structures can be monitored by querying the SIM. The ruleCore CEP Server provides a query mechanism using XPath. But other implementations might query the SIM using other methods.
• Monitoring situation progress – As a situation develops, its progress can be monitored by querying the SIM.
• Examine view contents – Each rule instance is executed in an unique context of events called the event stream view. The view contains a window into the inbound stream of events and contains only semantically related events.
• Referencing items – When an item such as a rule needs to reference other items it is done using references through the SIM.
• Component Specific Data – Each plug-in component that extends the engine creates an entry in the SIM and may add any component specific information.

The SIM consists internally of dynamic data structures which are updated as needed when the internal state of the ruleCore CEP Server changes in response to inbound events. These data structures have an external representation which is the user’s view of the SIM.

The external representation of the SIM can be treated as a DOM document which can be queried using XPath and transformed using XSLT. XPath queries can be executed against the SIM using the SysInfoQuery system event.

If is also useful to note that the ruleCore CEP Server uses events for all tasks. Special pre-defined system events are used to create rules and other items in ruleCore CEP Server in addition to monitor the run-time state of the rule evaluation.

The main language element in Reakt is the rule. Reakt uses a class of rules called reactive rules to provide its situation detection capabilities. Reactive rules should not to be confused with production rules or other types of rules which must be explicitly evaluated. A rule in Reakt is never explicitly evaluated, its evaluation is automatically managed by the execution environment in response to inbound events.

A rule consists of a number of elements, the main ones being:

• Initialization directives - Specifies which events can lead to the creation of a rule instance
• Termination directives – Specifies what causes a rule instance to be deleted.
• The rule level – Specifies on what level in the rule hierarchy instances should be created.
• Trigger Directives – Controls how and when to report Rule triggering.
  
• The View – A reference to a view, providing execution context for the rule.
• The Situation – A reference to a situation to track and detect situations.
• The Action – A reference to an action, which will generate an event when
  • the situation is detected, or
  • if the situation is determined to be undetectable.

Each rule has a name and a attribute which can be used limit the number of rule instances that can be created of a particular rule. A common usage of the limit is to create a singleton rule, of which only one single instance can exist.

Rules at every level can publish their trigger events externally so that they are visible outside of the ruleCore CEP Server.

All rule declarations includes a level specification. Every rule instance is created on a specific level in a hierarchy of rules. Level zero is the level on which inbound events arrive. Thus the lowest level for a rule is 1 (one). A rule will only see events from a level directly below it.

For example, three levels of rules where the rule at level one sees the inbound events:

<Rule name="New Location" limit="1000" level="1"> … </Rule>

<Rule name="Zone Entry" limit="1000" level="2"> … </Rule>
<Rule name="Zone Exit" limit="1000" level="2"> … </Rule>

<Rule name="Zone Visit" limit="1000" level="3"> … </Rule>

Rule instances are automatically created and deleted based on directives in the rule declaration section.

Create. The <Initialize> declaration controls when a new rule instance can be created. All rule instances are created in response to events. The <Initialize> section lists those events which can lead to new rule instances being created.

Create a rule instance when an event of type GPSPosition arrives.

<Initialize>
  <Assert>
   <Event>
    <base:XPath>sim:EventDef[@eventType="GPSPosition"]</base:XPath>
   </Event>
  </Assert>
</Initialize>

The <Key> declaration is the unique key for each rule instance. The key consists of one or multiple properties of the init events.

<Key>
  <Property name="Vehicle"/>
  <Property name="Zone"/>
</Key>

The key declaration allows for easy detection of situations per something. For example, the above key declaration creates rule instances which are per vehicle and zone. So there would be rule instances created for each unique combination of vehicle and zone.

Terminate. The <Terminate> section determines when a rule instance is deleted. The <Terminate> section consists of a number of directives which allows for creation of flexible termination strategies. A rule could for example be terminated when it has triggered three times, when it’s view is empty, on a specific date and time or when a particular event arrives.
<Terminate>
  <Triggered count="3"/>

  <ViewEmpty name="ZoneEntry"/>

  <Time>2010-01-31T00:00:00Z</Time>
  <Assert>
   <Event>
    <base:XPath>sim:EventDef[@eventType="Location"]</base:XPath>
   </Event>
  </Assert>
</Terminate>

A rule may trigger when one of two things happen; the situation is detected or it is automatically determined that the situation is undetectable in the future. Depending on the terminate declarations a rule can fire one or multiple times.

<Trigger>
  <SituationDetected/>
  <UndetectableSituation/>
</Trigger>

When a rule triggers, its action part is executed. The action part creates an event and thus the visible result when a rule triggers is a new event, created according to your specification.

Related to rule triggering is the reporting of the rule triggering. The triggering of the rule and reporting that triggering is separated in two stages. Thus it is possible to postpone the report of a rule triggering. For example, it is possible to report rule triggering, or multiple triggerings, each minute or when the rule has triggered five times or whichever comes first. It is also possible to limit the trigger reports so that only a specified number of reports are created. The most common case is to create rules whose triggering is reported only once.

<Report>

  <Every trigCount="10" timeframe="00:10:00">
  <Max count="234"/>
  <AtTermination/>

</Report>

The rule also contains a reference to the rule’s view, situation and action. All references in Reakt are specified using XPath and executed against the SIM.

The view reference contains a reference to a view declaration. The view provides the context in which the rule is executed

<Views>
  <ViewRef name="ZoneEntry">
   <base:XPath>sim:ViewDef[@name='ZoneEntry']</base:XPath>
  </ViewRef>
</Views>

The situation reference contains a reference to the situation which is evaluated in the context of the view.

<Situations>
  <SituationRef name="ZoneEntry">
   <base:XPath>sim:SituationDef[@name="ZoneEntry"]</base:XPath>
  </SituationRef>
</Situations>

The action reference contains a reference to the action part which generates a new event when the rule triggers.

<Actions>
  <SituationDetected situationName="ZoneEntry">
  <ActionRef name="ZoneEntry" eventVisibility="external">
   <base:XPath>sim:ActionDef[@name="ZoneEntry"]</base:XPath>
  </ActionRef>
  </SituationDetected>
</Actions>

Here is a complete example of a rule.

<Rule name="ZoneEntry" limit="1000" level="1">

<!– Create rule instance per Vehicle and Zone –>

<Key>
<Property name="Vehicle"/>
<Property name="Zone"/>
</Key>

<!– Create a rule instance then the Location event arrives,

if one does not exists per the key above –>

<Initialize>
<Assert>
<Event>
<XPath>EventDef[@eventType="Location"]</XPath>
</Event>
</Assert>
</Initialize>

<!– Delete rule instance when rule has triggered 300 times, the view is

empty of at the specified time or when the Reset event arrives –>

<Terminate>
<Triggered count="300"/>

<ViewEmpty name="ZoneEntry"/>

<Time>2010-01-31T00:00:00Z</Time>

<Assert>
<Event>
<XPath>EventDef[@eventType="Reset"]</XPath>
</Event>
</Assert>
</Terminate>

<!– Report rule triggerings every 10th triggering of every tenth minute

or at rule termination, report a maximum of 234 triggerings –>

<Report>

<Every trigCount="10" timeframe="00:10:00">
<Max count="234"/>
<AtTermination/>
</Report>

<!– Trigger rule when situation is detected –>

<Trigger>
<SituationDetected/>
</Trigger>

<Views>
<ViewRef name="ZoneEntry">
<XPath>ViewDef[@name='ZoneEntry']</XPath>
</ViewRef>
</Views>

<Situations>
<SituationRef name="ZoneEntry">
<XPath>SituationDef[@name="ZoneEntry"]</XPath>
</SituationRef>
</Situations>

<Actions>
<SituationDetected situationName="ZoneEntry">
<ActionRef name="ZoneEntry" eventVisibility="external">
<XPath>ActionDef[@name="ZoneEntry"]</XPath>
</ActionRef>
</SituationDetected>
</Actions>

</Rule>

The rule declaration contains a reference to a view. The view provides the context in which the rule is evaluated. The view keeps related events together and selects only interesting events from the inbound stream of events.

Each rule instance contains its own private view instance. The view contains a collection of events. The view is kept updated based the stream of inbound events. The view provides a dynamically changing execution environment for the rule as each rule instance is executed in the context of its view.

The view consists of a number of declarations, each specifying a property of an event that must be true in order for the event to be present in the view.

All these declarations are evaluated to determine which events are present in the view of each rule instance.

Example of properties available:

• The type view property is used to include events into the view only if they are of the specified type.
• The maxage view property defines the maximum age for each event in the view. You could for example use the age property to let each rule instance see only events from the past 10 minutes.
• The maxcount property defines the maximum number of events in the view. You could for example use this property to limit the view to contain only the last 100 received events.
• The match property defines which event properties must match in order for them to be included in the same view. You could for example require that the property CustomerID matches on all events in the view, thereby creating a view which contains events about the same customer.
• The assert property defines an XPath expression that must evaluate to true for all instances of a specified event type. The assert property could for example be used to compare a certain element in the event body with a constant.

<ViewDef name="ZoneEntry">

<Properties>

<!– All events in the view must be of types InsideZone or OutsideZone –>

<Type>

<Event>

<XPath>EventDef[@eventType="InsideZone"]</XPath>

</Event>

<Event>

<XPath>EventDef[@eventType="OutsideZone"]</XPath>

</Event>

</Type>

<!– All events in the view must be from the same vehicle –>

<Match name="vehicle">

<Value>

<Event>

<XPath>EventDef[@eventType="OutsideZone"]</XPath>

</Event>

<Property name="Vehicle"/>

</Value>

<Value>

<Event>

<XPath>EventDef[@eventType="InsideZone"]</XPath>

</Event>

<Property name="Vehicle"/>

</Value>

</Match>

<!– All events in the view must be from the same zone –>

<Match name="zone">

<Value>

<Event>

<XPath>EventDef[@eventType="InsideZone"]</XPath>

</Event>

<Property name="Zone"/>

</Value>

<Value>

<Event>

<XPath>EventDef[@eventType="OutsideZone"]</XPath>

</Event>

<Property name="Zone"/>

</Value>

</Match>

<!– Keep only the last 100 events in the view –>

<MaxCount>100</MaxCount>

</Properties>

</ViewDef>

A situation describes an interesting combination of multiple events as they occur over time.

The situation detection capability of Reakt have the following features.

1. Declarative – Complex situations are described using a high level declarative approach, avoiding complex procedural solutions using traditional programming languages.
2. Temporal support – Support for describing temporal relationships between events. Timing restrictions such as deadlines for detection can easily be added.
3. Extensible – The situation detector is designed to be extended with new task specific detector nodes.
4. Monitoring – The detector is designed to be monitored and thus the progress of situation detection can easily be tracked.

As a situation describes an interesting combination of multiple events as they occur over time, it must start somewhere. A situation starts developing when a specific initiator event occurs. This is the first event in what could possibly be a fully developed situation consisting of multiple events. This also means that a situation has an exact point in time where is starts developing, namely the timestamp of the initiator event.

The situation develops over time and can either be detected at some point in time or automatically determined to be undetectable. A situation can develop over long periods of time and consist of a large number of contributing events. The ruleCore CEP Server provides a persistent rule state allowing rule instances and their situations to survive restarts and unexpected crashes.

The progress of a situation, as it develops, can be followed by submitting a query event to the ruleCore CEP Server about the state of a particular rule’s situation detector. The degree of completion of a situation can be monitored to provide an early warning about a situation which could possibly be detected soon.

For example, detect when a vehicle has been outside a zone for at least five seconds:

<SituationDef name="OutsideZone5s">

<Detector>

<After timeframe="00:00:05">

<Not>

<EventPickup>

<XPath>Views/View[@default='true']/Events/Event[@eventType="InsideZone"]<XPath>

</EventPickup>

</Not>

</After>

</Detector>

</SituationDef>

The above situation example is detected when the innermost event pickup does not pick up any events from the view five seconds after the rule instance was created. If InsideZone events have arrived into the view during those five seconds the situation is not detected.

A situation is defined using a tree with detector nodes. The <Detector> node is the root of the situation detector. Each node can be either true or false depending on its semantics.

One of the central detector nodes is the <EventPickup> node which selects events from the view. It selects events by executing a XPath query against the view. If the query returns any events from the view, the <EventPickup> node becomes true.

The detector is evaluated each time it is possible that changes in the view or temporal conditions in detector nodes can affect the state of the detector.

Actions are executed as the last part of a rule’s evaluation in response to a detected situation, or optionally when it is determined that a partially detected situation can’t be detected at any point in the future.

An action is used to create a new event. The new event is called the reaction event and is used to notify external systems about a detected situation. Different events can be created for a detected situation and for undetectable situations.

The outbound event created by the action is created using a XSLT stylesheet. The stylesheet is executed in the context of the rule and can access all the events that contributed to the detection of the situation. The stylesheet can also access the events present in the view at the time the action is evaluated.

Reakt is a trademark of Rulecore KB of Sweden. Rulecore is a registered trademark of Rulecore KB of Sweden

"And now," cried Max, "let the wild rumpus start!"

— Maurice Sendak: Where the Wild Things Are

It’s not just equities and futures markets where strange stuff happens! An “algo gone wild” was spotted in the oil market (it actually happened earlier this year) and intrepid Reuters journalists got to the bottom of it.

High frequency trading firm Infinium Capital Management is at the center of a six-month probe by CME Group (and reportedly the CFTC) into why its brand new trading program malfunctioned and racked up a million-dollar loss in about a second, just before markets closed on Feb. 3. The algorithm, which was brand new, went live 4 minutes before the end of trading. It fired in 2000-3000 orders per second before being shut off. The oil price surged $1 then slid $5 over the course of the next two days. Read about the full story here:

http://www.reuters.com/article/idUSTRE67O2QQ20100825

I know the CEO of Infinium Chuck Whitman from the CFTC technology advisory committee – he’s a good guy and very knowledgeable. I believe him when he says his wild algos had no malicious intent – the algos were just broken and shouldn’t have been put live.

With algorithms and HFT comes the possibility of mistakes. Many more firms outside of the equities world are embracing HFT and their inexperience can cause market disruptions such as the Feb 3rd CME issue. A flash crash in oil or other commodities - or even foreign exchange - is not to be scoffed at. In fact, many commodities markets are much less liquid and homogenous than equities, and can be even more vulnerable to mistakes or manipulation. In the case of Infinium, the algo caused a spike in trading volumes by nearly eight times in less than a minute. It was a classic case of the algo running wild until it faltered and 'choked'. This is not how HFT strategies are supposed to work.

There are a number of best practices that can be used to mitigate against algos going wild:

The first best practice is diligent backtesting – using historic data and realistic simulation to ensure many possible scenarios have been accounted for. What does the algo do in a bull market, a bear market, at the open, at the close, when unexpected spikes occur, during a flash crash, when non-farm payrolls or other economic news is released etc. etc.? Of course there’s always the possibility of a “black swan” scenario – but then there’s always the possibility of an earthquake in London – but I bet the buildings aren’t built to withstand one – it’s a matter of covering likely possibilities as best you can. A backtesting process needs to be streamlined of course – as short time to market of new algos is key.

A second best practice is building a real-time risk firewall into your algo environment. Just like a network firewall stops anomalous network packets reaching your computer, so the risk firewall should stop anomalous trades getting to trading venues. These anomalous trades might be human or computer generated – such as “fat finger” errors, risk exposures (for a trader, a desk or an institution) being breached, or even algos gone wild (e.g. entering spurious loops and spitting out anomalous orders). Real-time risk monitoring is a second level protection for those problems you don’t catch in backtesting.

A third best practice is to use real-time market surveillance in your algo environment. Even if trades do not breach risk parameters, they may breach compliance rules, regulations or may be perceived by a regulator as market manipulation (by accident if not design). Detecting these patterns as they happen enables good internal policing by trading firms, rather than investigation or prosecution by regulators.

An algorithm is a tool in a trader's toolkit, and it needs to be taken care of as such. If it is well-oiled and the trader or quant or risk manager monitors its progress then the algo will do its job quickly and nicely. If the trader/quant/risk manager doesn’t properly prepare the algo or ignores the algo and lets it get rusty, so to speak, it could lose its edge and run amok. Algorithms must be monitored constantly for performance and for errors, and sometimes tweaked on-the-fly to ensure best results. A good algorithmic trading platform will enable trading firms to do just that.

Trading firms are not the only ones who need to be on guard for possible algos gone wild. In the case of Infinium, the regulators and the exchange were also slow on the uptake. This shows that everyone needs to be proactive in using the correct tools to monitor algorithmic trading. Sensing and responding to market patterns before the aberrations or errors have a chance to move prices is the right thing to do - in all asset classes. Be like Max and tame the wild things!

In a blog post about CEP being considered a supporting act for BPM and other technologies, Raimer von Ammon of CITT (a spinoff from University of Regensburg, Germany) commented about the EU proposal for a “U-CEP” (Ubiquitous CEP) research programme. In that comment Rainer mentioned the connection between CEP (an IT paradigm around events) and Biology (covering event-driven neural and evolutionary systems - Epigenetics and Brain Research) which form part of the U-CEP EU research programme proposal. Rainer has kindly elaborated some more on this topic, which IMHO is worthy of a separate post on this blog … and I note has parallels to one of James Odell’s posts on the “inside of an agent”…

Two or three recently emerging disciplines yield to a similar paradigm in order to explain how the environment (actually the universe of global event clouds) and their processing by sensors (respectively receptors and the activation of effectors) drive our daily life, but also the whole evolutionary process of life on our earth since the Big Bang.

The discipline of the New Biology or Epigentics has detected - on the basis of the nanotechnology, microbiology and cell biology – within the last years that the life of cells is determined by its physical and energetic environment and not by their genes. Genes are only the molecular “blue print”, the design pattern, on which the structure or architecture of cells, tissue and organs is based. But the environment as a global cloud of signals, energies or so called events are at the end responsible for the way of life of a cell. In each single cell the mechanisms of life are triggered by the processing of the “events” of its environment and not by its genes.

Since “quantum physics” and the “Heisenberg uncertainty relation” we know today that such events are at last energy as eddies of quarks and photons, but we can define different types of events, also on the physical level. Around 3.5 billion years ago the first monads as bacilli and algae lived on the earth, 650 millions of years ago the multicellular organisms as trees and other plants were created as a more complex and more intelligent life, and since around only 100,000 years the today’s human is a united cell structure of 50 trillions of single cells where each cell is doing event processing on the base of its receptors of its cell membrane. So, a human being can allegedly process 120,000 events per second unconsciously (by the right side of the brain), but logically a human can only process around (I guess) 7 or a few information units at the same time (by the left side of the brain).

Against this background CEP is based on a very similar model. EPAs (event processing agents) are actually cells where event adaptors are the receptors of a cell membrane and the event processing logic based on an EPL (event processing language) are the effectors of a cell. EPNs (event processing networks) are actually a united multicellular structure and so on.

The Epigenetics explains how the environmental signals (events) control the activity of the genes. The primacy of the DNA is no longer valid, and the new found information flow is now called the “primacy of the environment”. Recent experiments of The Epigeneitcs have proved that our beliefs and thinking is energy in the sense of environmental signals. The Epigenetics found that all these kinds of environmental signals influence the regulating proteins which control the activity of the genes and that the global event cloud as environmental signals influences and changes the DNS (so called reverse transcriptase).

What is an individual?
According to the Epigenetics an individual is determined by the sum of its receptors in the cell membrane of all its 50 trillions of (specialized) cells. Each individual is unique because it is controlled or driven by the protein machines of its cells as a result of the received events (signals) of the global event cloud of the universe. The global event cloud does always exist and each event is an undestroyable energy which exists always and forever (see Unus Mundus and space-time continuum).

The Epigenetics explains an individual as an analogy of a broadcasting of television programs where the adjustment buttons are the receptors of the cell membranes which determine which program we receive in which manner. If we add or switch off receptors, we receive a different program or we see the same program differently. This can be understood as the incarnation of an individual. On the basis of such a model we can explain phenonemas like reincarnation or time travel as well.

Results of the recent Brain Research
On the basis of recent experiments of the Brain Research, researchers like Wolf Singer, Gerhard Roth et al. claim that a free will does not exist. They believe that their experiments have proven that the brain region which is responsible for a deliberate decision is only later activated when a signal or event is received while the protein machinery as the activity activator was already started “long” before. This cognition that there is no free will could be supported by the Epigenetics and the functionality of a cell membrane and its receptors and effectors/event adaptors and EPA’s as the basis of the whole protein machinery and as the motor of life.

“If you think you think you only think you think” versus “Cogito ergo sum” (Descartes) …”

For the U-CEP course/textbook I’m gathering some materials since some time which I’ll add to the U-CEP doc within the next days; here are some nice video lectures as a basis for U-CEP enhancements/projects, e.g. from Bruce Lipton, I like him as a great guy and great lecturer:

http://epigenetics.uni-saarland.de/en/home/ (German language only)
http://www.tonyb.freeyellow.com/id68.html
http://www.veoh.com/collection/AgriculturalNews/watch/v378752sbRFQa3F# (video)

Great to hear that TIBCO might support such U-CEP ideas, perhaps Larry Ellison or also Bill Gates would be interested in supporting such things … as an investment in the future

Related posts:

1. EPTS4: Research, an academic perspective
2. CEP topics at Microsoft Research…
3. EPTS5-4: CEP in network mgmt and CEP research

A recent CEP RFP requested marketshare data for the “CEP market“, worldwide and in specific territories. Presumably the contracts departments for large enterprises are hoping that software development tool purchases are tracked by some industry body (Nielsens maybe) just like Cornflake packets and Persil washing powder. Unfortunately, this is not the case - and of course the definition of “customer” is itself open to interpretation (someone who uses the software? someone who paid money for the software? a licensee?). Some companies also embed “CEP software” in larger categories of software (e.g. messaging or BAM); others sell only stacks that might include CEP licenses, without necessarily any use of said licenses.

Nonetheless, it is interesting to try and find some data points here. Software companies generally make “customer numbers” claims via press releases and in analyst reports, as well as SEC etc reports, and these can be collated and cross-referenced to try and come to a “big picture”. Doing so finds some interesting details such as:

• the most recent available data (for 2008-2009) gives TIBCO BusinessEvents the leading market share figure of 28% (by number of customers)
• this market share number has pretty much remained constant (except for one anomoly year, when it was 10 points higher!) for the previous 4 years
• other established CEP vendor shares are 21%, 18% and 14%  - a pretty even spread overall, if the data is accurate
• the cutoff date for available data (early 2009) excludes any major effects from the large companies getting involved in CEP (IBM, Oracle, Microsoft) which, along with CEP market growth and the addition of many new startups, will likely make future numbers difficult to interpret
• vendors 2, 3 and 4 (in order of market share) report being almost exclusively (>80%) in financial / capital markets - so probably ~50% of business in CEP (to-date) is in the financial area
• overall market growth, year on year, was up to 60% initially, but dropped to just over 30% in the latest numbers. Something to do with the financial market meltdown, perhaps?

Related posts:

1. The CEP Market in mid 2010: more consolidation
2. TIBCO end-of-year CEP results
3. Another vendor buys their way into CEP…

According to the FT, the Financial Industry Regulatory Association is undertaking a “sweep” of broker-dealers that offer direct market access to high-frequency traders to find out if they have pre-trade risk management controls in place for their algorithmic customers. (Read the full story here: http://tinyurl.com/28rg287). Here at Progress Software we have been advocating the use of pre-trade risk management tools for some time. The prospect of High Frequency Trading (HFT) firms running trades directly through broker algorithms (naked access) to exchanges/ECNs without adequate supervision has always concerned us. Brokerage firms simply give their clients an exchange ID number, which the clients then enter into their algorithms so that they can trade directly with the exchange.

Regulators are right to be concerned. Unfiltered access to trading destinations can end up causing trading errors or worse. Anything can happen with naked access; from fat fingered errors, trading limits being breached, even fraud - all of which can cost brokers, traders and regulators big money.

Although the SEC proposed banning naked access in January, and the Committee of European Securities Regulators (CESR) is likely to follow in its footsteps, there has been considerable pushback from brokers and trading firms. They say that adding a pre-trade risk management step would cause a "latency hop" in the trading process, claiming it would make them uncompetitive.  A layer of risk management often adds a third party into the mix, and - consequently - a delay.

But if everyone is required to add real-time, pre-trade risk management, then the competitive disadvantage is moot. The benefits of being able to pro-actively monitor trades before they hit an exchange or ECN far outweigh any microscopic latency hops in any event. The naked access playing field is hardly level. There are trading systems out there that claim to throughput a trade in 16 microseconds from outside an exchange's firewall, while exchange and ECN systems range anywhere from 250 to 700 microseconds (according to measurements by latency management firm Corvil).

Latency is clearly an important issue and complex event processing offers a solution. Brokers can achieve ultra-low-latency pre-trade risk management without compromising speed of access.  An option is a low latency "risk firewall" utilizing complex event processing as its core, which can be benchmarked in the low microseconds.  With a real-time risk solution in place, a message can enter through an order management system, be run through the risk hurdles and checks, and leave for the exchange a few microseconds later.

It is the ideal solution to a tricky question - how do I manage risk before my trade hits the exchange without compromising speed? The benefits are clear - a latency hop of a few microseconds not only saves money, it can also prepare your firm for the regulatory future.

Already some years ago, we gave a – very “applied” – university course about edBPM together with colleagues from Audi, Deutsche Post, Hamburger Sparkasse and TeamBank:
http://www.citt-online.com/index.php?id=veranstaltungen&id3=industrieaufgaben&id4=more
http://www.citt-online.com/index.php?id=veranstaltungen&id3=exp06&id4=more

Together with Audi, we started a student project about NextGeneration Navigation Systems / Intelligent Cars based on CEP:
http://www.citt-online.com/downloads/myAudiNavigation.pdf

This first idea was actually born together with the showcase of my former BEA Systems colleague Paco Gomez as an automotive application combining BEA’s event server with location-based services.
The first student team of Katja Borschert, Robert Gottanka, Martin Schober and Christian Silberbauer adapted the showcase for the Audi use case. Later some PhD students like Thomas Ertlmair, Thomas Paulus, Christoph Emmerberger and Florian Springer added some use cases and transferred the showcase according to the new CEP solution from Oracle after the acquisition of BEA. We presented these project ideas a lot of times, internationally at the edBPM Expert Meetings in Regensburg, at the ServiceWave 2008 in Madrid, at our Haifa workshop/visit in Israel February 2009 and so on. Unfortunately, the crisis stopped a lot of projects worlwide. Now, at least Germany is booming again allegedly, especially the Automotive domain, so let’s start again…

In the meantime, we are enhancing the idea by ed(B)PM and U-CEP aspects with some more use cases. These use cases are based on the reference model and the first sketch of a taxonomy in http://www.citt-online.com/downloads/62750370.pdf as a determinstic approach based on in advance defined event patterns and reactions as a “concert” of collaborating processes, but also as a non-deterministic approach for so far unknown, but potentially interesting or critical event patterns based on additional AI- or neural network components. This would be our submission e.g. to the 3rd Workshop on Multimodal Interaction for Automotive Applications (MIAA) in conjunction with ACM IUI in Palo Alto next spring 2011. We will do it in cooperation with a large telco carrier probably.

Those ideas are an enhancement of such research project about driverless cars like Audi’s TTS “Shelley” http://www.silicon.de/hardware/server-desktops/0,39038998,41529810,00/audis+geisterhafter+tts+_shelley.htm

Interesting question to be discussed e.g. at our next edBPM/U-CEP expert meetings is also who would be the responsible of such enhanced intelligent machines or cyborgs or robots etc. in the case of problems or damages? CEP/ed(B)PM providers, EPL programmers, …?
http://www.silicon.de/lifestyle/interview/0,39038973,41530460,00/welche+rechte+und+pflichten+hat+ein+roboter.htm
(I’m sorry, but for 2 links you have to try your translation system if you are not a German speaker)

Cheers

Rainer

Just some quick thoughts of what I think will emerge in CEP in the next couple of years:

• A small tailor made language. Maybe looking something like Erlang or F#? 
• A formally defined model specially developed for event processing, think of the relational model as comparison, which provides a solid foundation.
• Small light weight libraries which contains the essentials needed for event processing. Used as a base for complete stand alone server of to implement CEP functionality in various kínds of applications.
• Services for processing live data targetting various domains. Like finance (already happening?), logistics, sensor networks, SCADA and that kind of things.
• More specialised CEP tools which solve one particular problem in the CEP domain. For example event enrichment or aggregation. One size does not fit all, as the database community already knows.
• Better GUI concepts, the ones today are too low level.

Looking for all the world like someone yelled "fire" in a crowded nightclub, prop and quant traders are stampeding out of investment banks and headed for the hedge fund world. Some, mainly the prop traders, are being pushed gently out the door as banks prepare for the Volcker Rule (http://tinyurl.com/39ap28d). Others, like the quants (http://tinyurl.com/23c5h6d), are in search of the mega-bonuses that their prop trader or hedge fund manager compatriots are (or were) getting.

Impending changes in regulation are prompting banks to spin off proprietary trading activities, many by expanding their operations overseas where Messieurs Dodd and Frank cannot reach them. I’m very concerned about this “regulatory arbitrage” in which firms may move away from the US to find less strict regulatory regions. We don’t want to lose the lead in this important area of the economy.

Spin offs and regulatory arbitrage may well leave a herd of US traders looking for work and many may end up working at - or starting - hedge funds. The quants, having slaved over hot computers for the last few years to line bankers' pockets, are forming their own trading companies or joining prop trading firms with a profit-sharing deal.

Most of these traders will be in for a rude awakening when they sit down to work. Prop traders joining hedge funds will find that the technology budgets may not be as generous as they were at their last bulge bracket employer's firm. The quants, who are essentially programmers, will face huge challenges in finding firms that have the kind of low latency, scalable architecture that they need to design, tweak and trade with their algorithms. The level of trading freedom is different, too. Hedge fund managers will have something to say about a trader's profits - or lack thereof. Quants may find that designing an algorithm and handing it over to the trading desk is not quite the same as being responsible for the profits that the algo makes - or doesn't make.

Make no mistake, these prop traders and quants are highly intelligent and adaptable people. There will be many challenges to face going forward, but technology need not be one of them. There are instantly useable, scalable platforms that quants and hedge funds can use to build and deploy algorithms. These platforms, such as Progress Apama's Complex Event Processing Platform, offer a robust technology infrastructure to successfully create, test, deploy and manage their algorithmic strategies.

Algorithmic trading software is constantly transforming. As the volume of real-time market data continues to increase, algorithmic trading solutions demand an infrastructure that can respond to market data with near zero latency. To trade effectively in competitive markets requires rapid, opportunistic response to changing market conditions before one's competition can seize those opportunities. The people that are running for the doors and into the arms of hedge funds or other trading firms, will need this advantage. Competition is fierce, and their previous employers already have the technology advantage.

Opher Etzion wrote an interesting blog responding to analyst Phil Howard, which in turn provoked another response by Rainer Ammon. Phil had commented that the recent developments in event processing pointed to a convergence with “BPM” (and - per his own prediction and with somewhat less evidence - Data Warehousing).

• Phil says:
  …Then there was IBM’s acquisition of AptSoft in 2008, …with an emphasis on integration with business process management. And, of course, Oracle and TIBCO are doing much the same thing…
  I cannot speak for Oracle (who AFAIK have continued to develop the BEA event server as a part of the Oracle EDA and BAM, not BPM,  suite), but in TIBCO, CEP technology is considered an adjunct to BPM. Or course there are a number of patterns we see using CEP and BPM together, but in no way can TIBCO CEP be considered emphasising integation with TIBCO BPM. Indeed, more TIBCO customers use CEP together with SOA (AMX BusinessWorks) than with TIBCO BPM (iProcess or AMX BPM)…
• Phil adds:
  …So, the clear trend is towards integrating complex event processing with other types of process management, though these may not necessarily be with business process management per se…
  So, what process management is not business process management? Indeed we do see “business processes” implemented using CEP: this is the event/pattern - decision - action cycle that CEP tools provide. Instead of “integrating with”, think “more agile or dynamic business processes”. And although events provide great integration mechanisms, the implication that CEP is a mere “supporting act” is not seen so much in practice.
• Phil concludes:
  Indeed, we will have to wait to see if complex event processing becomes completely subsumed into other technology areas and, if so, what new acronyms the industry can come up with: what is the acronym for a convergence of complex event processing and business process management?
  Conveniently for Phil, Rainer has already coined “edBPM” for event driven BPM.
• Opher gives the wise man’s response:
  This is a similar situation to databases; database can be used for various reasons, and also be embedded with various other technologies and products…
  Which makes perfect sense: CEP technology can be used standalone and in supporting roles, and different vendors will see different markets and focus different technologies for each…
• Rainer adds a comment containing an interesting quote from a European solutions provider (who is not, AFAIK, working in the CEP space):
  CEP a lot was indeed now written all over the world and spun very much, and unfortunately it is also an unusual amount of charlatanism… A machine for example runs with very continuous transitions through the phase space of their operating states. The prediction of a complex event would be in this case e.g. the simple need for maintenance, even for conditions that were never previously in this exact combination … Even this simple example could solved only in a combination of methods.
  As it happens I have a nice successful CEP-based counterexample here.

From the last point, there is still seemingly a lot of education required for systems integrators and their architects and software designers - especially if they want to be involved in “charlatan event processing applications”! Luckily CEP vendors like TIBCO have partners folk who are more than happy to arrange webinars and talks about CEP and event processing to these folk. Just drop your TIBCO rep a line!

Related posts:

1. Adaptive Process Models and CEP

Forbes published an article suggesting some “high frequency trading” programmers have cut deals to earn them more of the profits from operating the trading applications they write. I wonder if CEP tool vendors - especially the ESP vendors usually involved in algo trading solutions - can do the same? I suspect this model is restricted to algo trading though - there aren’t many other apps that are so directly “profit generating”…

Related posts:

1. EPTS4 - 4th EPTS Event Processing Symposium
2. Gartner Event Processing Summit 2008: highlights
3. Nick Leeson (Part Deux)