Task 3.4 Big Data Optimisation and Analysis

Partners involved

The partners involved in Task 3.4 are BRUNEL, ICCS, LIBRA, and SUBTERRA for Task 3.9. Moreover, TITANIA has also been involved as end user, along with LA PARRILLA for Task 3.9.

Objectives and outcomes

The main goal of Task 3.4 is to develop a human assistance tool for early warning hazard prediction in a sustainable digital mine of the future. The tool is based on data fusion and Long Short-Term Memory Recurrent Neural Network (LSTM-RNN), which stems from the big data optimisation concept which involves complex applications with elements such as predictive models, statistical algorithms, and what-if analysis in a mine operation practice.


A Human Decision Tool within the Dig_IT Decision Support System (DSS) is developed to alert the mining operator if any slope instability occurs within the open pit mine. The tool will be able to predict at least 16 minutes before the real occurrence to give some time for the mining operator to make strategic operation decision-making.

The outcome of this task will incorporate real-time data exchange between Brunel (Data Model) and ICCS (Data Warehouse). Then, the data will be translated to the Decision Support System (LIBRA) for displaying it in the interactive Human Decision Tool.  

The Human Decision Tool, which is part of the Dig_IT IoT Middleware layer, will: 
  • Construct a residual signal and compare it with a predefined threshold in generating alarms in the presence of bad data resulting from limited communication capacity. 
  • Perform fault detection and fault-tolerant control (FTC) under communication constraints feature/event detection, classification, and prediction in large-scale data environments in supervised and unsupervised modes.
  • Perform efficient evolutionary methods in preprocessing to seamlessly incorporate data denoising and data fusion across the entire early warning system, thereby augmenting the system’s overall robustness, observability, and accuracy.
  • Perform feature engineering techniques to understand the nature of the data and use short time series analysis to establish a model for detecting, identifying and predicting the events/failures.
  • Develop an early warning event prediction model that combines statistical time series modelling, relevant domain knowledge and intelligent search technique.
  • Apply the hazard prediction model using the real-time Dig_IT Decision Support System dashboard.

Software and tools used

The Fault Tolerant Control software package is built on Python 3 with standard libraries such as NumPy, Pandas, Matplotlib, Seaborn, Keras, and Sklearn. A decent machine with a good CPU/GPU with the installed Python libraries can be used to train the model. The model output from the prediction model will be updated in the Decision Support System (DSS) via the MQTT protocol that has been established between the CPU and the Dig_IT Platform.