Machine learning notes

To follow:
Sean Owen>
in-memory column data (and on disk) concept of pre-fetching, cache locality

Exploit and Explore problem

CAP theorem (+GPUs +spark +word2vec ) by

Chaos Monkey Army
Common architecture circuitbreakers
Latency monkey (chaos monkey)

Google whitepaper on Beam

Akka streams
comes from AirBnB job scheduler

2 read:


Data provenance

Alluxio (formerly Tachyon):
(like Redis)


Decouple producers from consumers

Hive Metastore

Holden Karau and Rachel Warren. High Performance Spark: Best Practices for Scaling and Optimizing Apache Spark


Spark on Yarn: HA Spark

Brendan Gregg. Systems Performance: Enterprise and the Cloud

cool visualisation
monitoring framework

Hystrix dashboard

Video Data
Youtube tutorials:

Apache Spark partial

Data profiling Python:

To check types of EC2 instances
g2 are GPU instances
and p2 are bad boys

Nick Pentreath. Machine Learning with Spark

Automating Tinder
13,000+ face images database

Spark + Stanford CoreNLP (Sentiment)
Neural network creates vector representation from words, no pre-processing (or some)

On chat bots and enterprise concerns

“Bots are new apps”.

It is a bold statement but it’s supported by Microsoft, Facebook, Slack and Telegram.
So you can almost say it’s a new reality. Which means new opportunities for all of us. Let’s dissect the issue of chat bots so you come prepared as the land grab is starting very soon.

chat bot

Continue reading On chat bots and enterprise concerns

New era of space exploration: commercial

Yay U.S. Congress says yes to space mining!

This is very important. Average person doesn’t give a shit about space. Cold war rockets race is in the past so governments invest near to nothing. Legislation for commercial exploitation motivates businesses to start new era of exploration.

So you know what this means – mankind has just improved its scalability!

Commercial interest = more R&D investment = better propulsion, robotics etc for space travel = colonies on Moon, Mars etc so humanity truly becomes multi-planetary species = much improved chances for humanity to spread across Universe, keep evolving and doing good things (hopefully) as an intelligent race.

Super excited about this space 🙂

Messaging OS

So wanted to share some thoughts on the latest trend, the “messaging OS”.

Not sure if you noticed or not but as my business currently heavily involves  OTT communication, I’m pretty sensitive to this stuff, also feels weird when something you’ve been telling your team and discussing internally suddenly becomes a topic of discussion on TechCrunch, in Mary Meeker reports etc.

Continue reading Messaging OS