A Kolmogorov-Smirnov Test to Detect Changes in Stationarity in Big Data
Additional information
Authors
Zhao D.,
Bu L.,
Alippi C.,
Wei Q.
Type
Journal Article
Year
2017
Language
English
Abstract
The paper proposes an effective change detection test for online monitoring data streams by inspecting the least squares density difference (LSDD) features extracted from two non-overlapped windows. The first window contains samples associated with the pre-change probability distribution function (pdf) and the second one with the post-change one (that differs from the former if a change in stationarity occurs). This method can detect changes by also controlling the false positive rate. However, since the window sizes is fixed after the test has been configured (it has to be small to reduce the execution time), the method may fail to detect changes with small magnitude which need more samples to reach the requested level of confidence. In this paper, we extend our work to the Big Data framework by applying the Kolmogorov-Smirnov test (KS test) to infer changes. Experiments show that the proposed method is effective in detecting changes.
Journal
IFAC-PapersOnLine
Volume
50
Start page number
14260
End page number
14265
ISSN
2405-8963
Keywords
change detection test, KS test, LSDD
