Expert Apache Spark consulting, developing, and engineering

Task:

Create a reference implementation of Spark MLLib in churn modeling project.

Solution:

We used Mahout Random Forest Classification and Spark MLLib Random Forest Regressions to predict the probability of churn of customers for online-shop.

Task:

Develop Prediction Model for webspam and hyperlink analysis designed and trained (with provided data) to achieve certain prediction goals.

Solution:

We have built model for Spam\No-spam prediction for links analysis company. We have used Big Data methods for data size of 70+ Gb. There were a lot of text features, which were preprocessed by using TF-IDF, Word2Vec and Features Selecting methods. Various data cleaning and ETL methods were applied. As result we have built classifier model, which was deployed as a RESTful web service

Task:

1. to calculate the number of Wikipedia pages views per last year and per last 90 days;
2. create a General table with the rank of each page;
3. pages with the same number of views should have the same rank.

Solution:

Our main steps were to establish proper big data pipeline:

1. collecting data;
2. punctuation removing;
3. date format recogniting;
4. language filtering;
5. aggregating by needed pages;
6. table building;
7. rank calculating.

Task:

Develop a prediction algorithm optimization in Python, including algorithm benchmarking before and after tailoring the random forest algorithm.

Solution:

We have completed these main steps: features engineering and normalization, different models  (Logistic Regression, SVM, Decision Trees, Ensemble models etc), model optimizations (via grid search, features importances searching, pearson correlation calculating, cross validation, graph analysis, overfitting detecting), dataset balancing and model scoring.

Task:

To run Monte-Carlo simulations by using Big Data methods and Apache Spark.

Solution:

We have built Monte-Carlo simulator with high speed, running on the big cluster (100 nodes) and processing in parallel way with 1000 outputs.

Task:

The goal is to create the read \ write system that takes the data from US data network provide to the HDFS using Spark Streaming.

Solution:

The input data represent a set of files that contain logs in certain form. Based on the content of each log it is possible to determine many parameters characterizing the log, particularly, subscriber_id. These impressions are then aggregated by Spark Streaming into a data warehouse. Spark Streaming can be preceded by Apache Kafka Streaming if it is demanded to accelerate and/or improve processing, transformation and further data transfer. At this stage (aggregation using Spark) the log data are joining on subscriber id. During this, all the files collect in a 15 minute interval, which is controlled by config file. The Spark Streaming application create the files in a new directory on each batch window. The aggregated data write to HDFS and copied to the OSP as gzipped files (performing a multi-threaded writing process). The most demand method to write and copy result data to HDFS & OSP is the using of SFTP or SCP, but it should be tested.

Task:

To identify sequences of events that are more predictive than their part worth components.

Solution:

We was be able to determine that a model that incorporates sequence (L-C->X) is more predictive than just including L, C and X in the model. As result we have get the patterns with the highest frequency and found the persent how often they appiers in datasets.

Task:

1) to write the programm for Extreme Machine Learning algorithms;

2) create Scala API for Apache Spark with Classification and Regression for Exreme ML.

As resut we have two API in Scala for apache Spark with Regression and Classification algorithms by using Extreme Machine Learning. The main advantages are:

- high speed of processing;

- high accuracy;

- unique script.