On cheap entropy-sparsified regression learning
Additional information
Authors
Type
Journal Article
Year
2022
Language
English
Abstract
Regression learning is one of the long-standing problems in statistics, machine learning and deep learning (DL). We show that writing this problem as a probabilistic expectation over (unknown) feature probabilities, increasing the number of unknown parameters and seemingly making the problem more complex - actually leads to a simplification, allowing to incorporate the physical principle of entropy maximization. It helps decomposing a very general setting of this learning problem (including discretization, feature selection and learning multiple piecewise-linear regressions) into an iterative sequence of simple sub-steps, which are either analytically solvable or cheaply computable through an efficient second-order numerical solver with a sub-
linear cost scaling. This leads to the computationally cheap and robust non-DL second-order Sparse Probabilistic Approximation for Regression Task ANalysis (SPARTAn) algorithm, that can be efficiently applied to problems with millions of feature dimensions on a commodity laptop, when the state-of-the-art learning tools would require supercomputers. SPARTAn is compared to a range of commonly used regression learning tools on synthetic problems and on the prediction of the El Niño Southern Oscillation, the dominant inter-annual mode of tropical climate variability. The obtained SPARTAn learners provide more predictive, sparse, and physically explainable data descriptions, clearly discerning the important role of ocean temperature variability at the thermocline in the equatorial Pacific. SPARTAn provides an easily interpretable description of the timescales by which these thermocline temperature features evolve and eventually ex-
press at the surface, thereby enabling enhanced predictability of the key drivers of the
inter-annual climate.
Journal
Proceedings of the National Academy of Sciences
Month
December
Start page number
1
End page number
12
