At Sentiance, we use machine learning to extract intelligence from smartphone sensor data such as accelerometer, gyroscope and location. We’ve been doing this for quite a while now, and are very proud on our state-of-the-art results regarding sensor based activity detection, map matching, driving behavior, venue mapping and more.

The obvious next step is to go from simply detecting what you are doing, to predicting what you will be doing in the future. Knowing your near-term future allows us the explain the intent of your current situation. For example, if we detect you are currently running, and we can predict that you will be on a train followed by a visit to your work location, then we can immediately explain why you are running; you’re obviously not just being sportive today!

Deep learning to the rescue

To be able to come up with long-term predictions, we started out with a simple Markov Chain like approach and ended up turning to deep learning. We trained a Long Short-Term Memory (LSTM) recurrent neural network on several thousands of event timelines. The network learns to encode general human behavior and surprisingly is able to quickly adapt to specific user habits. The following figure illustrates the architecture of our deep learning pipeline:

LSTM architecture used for event prediction. The input consists of a sequence of your last 128 events, while the output is a prediction of your next event, together with a duration estimate of the current event. The network is implemented using TensorFlow

I’m extremely proud of the results we achieve, mainly because of the following:

1. The network is often able to predict events that a human observer would not even think of. This sometimes feels like magic, even for the geekiest of our data scientists – and we all like magic!
2. We started out with simpler Bayesian models (Markov based) and gradually moved to more advanced solutions
3. Deep learning in this case actually solves a real problem instead of just following the hype
4. Our deep learning pipeline actually runs in production for millions of users. We put a lot of effort in making it scalable, reproducible and maintainable.

Check out our technical blog post that outlines the details on how we trained and tested our models, and how they actually work: http://www.sentiance.com/2017/04/25/predictive-analytics-applying-deep-learning-on-mobile-sensor-data/

Check out the videos of our prediction results (Links to the original blog post)

The post Deep learning for long-term predictions appeared first on Computer vision for dummies.