How to aggregate scores in a multi-heuristic detection system : A comparison between WOWA and Neural Networks

Apr 24, 2020 by Thibault Debatty | 758 views

APT Detection

https://cylab.be/blog/75/how-to-aggregate-scores-in-a-multi-heuristic-detection-system-a-comparison-between-wowa-and-neural-networks

Cyber-attacks are becoming increasingly complex and therefore require more sophisticated detection systems. A lot of these are actually combine multiple detection algorithms. A crucial step is then to aggregate all detection scores correctly.

Today we released a short paper where we compare two aggregation approaches:

  1. train a Weighted Ordered Weighted Average (WOWA) operator using a genetic algorithm and
  2. train a Neural Network using backpropagation.

Download the paper: How to aggregate scores in a multi-heuristic detection system : A comparison between WOWA and Neural Networks [PDF]

MARk: Visualizations with D3.js
Detecting suspicious or malicious activity in a network is not a trivial task. In recent years the attacks perpetrated have grown in sophistication and frequency. For this reason a new detection tool was developed, in the form of the Multi Agent Ranking framework (MARk). MARk sets the groundwork for the implementation of large scale detection and ranking systems through the implementation of a distributed storage in conjuncture with highly specialized, stand-alone detector agents. The detector agents are responsible for analyzing specific predefined characteristics and producing a report of any suspicious activity encountered.
Fixing "[circuit_breaking_exception] [parent] Data too large, data for [<http_request>]" ELK Stack error
Recently I have encountered an error I wasn't too familiar with how to resolve, working with the ELK Stack. This specific error is the "[circuit_breaking_exception] [parent] Data too large, data for [<http_request>]". It is not directly visible where the error originates from, but with some sleuthing I discovered that it is caused by Elasticsearch preventing some requests from executing to avoid possible out of memory errors, as detailed in Elasticsearch Circuit Breaker documentation.
Collecting data with Filebeat
In modern network infrastructures, there are a lot of sources of data, that can be of interest for collection and analysis, to see if possible suspicious activity is present in the network. More often than not, this data is collected and send to a Security Information and Event Management (SIEM) tool, running on the network, where it can be processed and reviewed by domain specialists.