The one stop shop for all your (big) data science and engineering needs

Visualization using d3.js and DataMaps to visualize attack vector within given network. Application takes data from IBM QRadar and with help of RESTful API sent query into d3.js, where it geocodes to lat-ling and places  into the map. Lines are colorized and filtered on front-end for the different level of attack points.

For the current visualization we’ve used:

• HTML5 for general page layout,
• OpenStreetMap for basic map layer,
• GeoJSON for target the province borders on representing map,
• CSS3 for visual color and general styling,
• JavaScript for hover effects.

With the world map we’ve created additional layer with GeoJSON encoding China provinces (including Taiwan). Next, we’ve created additional layer with data, associated with country/province from the base map layer. Then applied the chloroplet to get the visual color distribution across the map.

The main goal is to get similar functionality with Vizabi library in NVD3. For front-end purpose we’ve used Angular.js and modified original NVD3 code to deliver such features:

• Develop selects that changes the input parameters of the axes for graph.
• Develop changing time scale.
• Play button on the timeline.
• Approaching similarity with Vizabi in displaying data on a chart Bar chart.
• Ability to define the thickness of the lines on the chart Line chart.

Task:

Create a reference implementation of Spark MLLib in churn modeling project.

Solution:

We used Mahout Random Forest Classification and Spark MLLib Random Forest Regressions to predict the probability of churn of customers for online-shop.

Task:

Develop d3 graph to display paid / unpaid work for men / women depending on the country / countries selected.

Solution:

We used NWD3 collection of components for d3.js, added responsiveness and cross-browser support.

Task:

Build simple web solution that displays realtime data using Cubism.js with Google Big Query database as datasource.

Solution:

We have built simple authorization, API for working with BigQuery, front-end controller for managing multiple graphs and simple static cubism.js graphs.

Task:

Develop Prediction Model for webspam and hyperlink analysis designed and trained (with provided data) to achieve certain prediction goals.

Solution:

We have built model for Spam\No-spam prediction for links analysis company. We have used Big Data methods for data size of 70+ Gb. There were a lot of text features, which were preprocessed by using TF-IDF, Word2Vec and Features Selecting methods. Various data cleaning and ETL methods were applied. As result we have built classifier model, which was deployed as a RESTful web service

Task:

The task was to connect to customer’s ticket database through the Zendesk API and generate a dataset for a two-series line chart widget in Geckoboard. The first series of the chart will be tickets created by week. The second series will be tickets solved per week. Data needs to be generated for a rolling 52-week period if possible

Solution:

To solve this task we got data from Zendesk API about amount of solved and created tickets per each week . Then processed data in Python, created push-widget in Geckoboard and pushed data to widget via API.

Task:

1. to calculate the number of Wikipedia pages views per last year and per last 90 days;
2. create a General table with the rank of each page;
3. pages with the same number of views should have the same rank.

Solution:

Our main steps were to establish proper big data pipeline:

1. collecting data;
2. punctuation removing;
3. date format recogniting;
4. language filtering;
5. aggregating by needed pages;
6. table building;
7. rank calculating.

Task:

Automatic scoring of the MMA fights according to the draftkings.com rules.

Solution:

We used Python to develop script with algorithm for scoring calculation.

Task:

Сonnect to VMware vCenter, collect virtual machine data and send it to a specified URL in JSON or XML file.

Solution:

We used pyVmomi library and Python scripts to complete task.

Task:

The main goal was to create web app for optimizing products prices from Shopify store data. User can sign up using his shopify account, choose products and optimize their prices. And have access to profit/revenue charts.

Solution:

We used authentification via Shopify API (user gave access to his account on shopify.com), Then our app imported all products and offers from Shopify and optimized products prices. App reoptimizes prices every day using Celery and displayes revenue/profit line-chart using d3.js for each product. It also saves yesterday data in MySQL DB and track product visitors using tracking pixel and saves them in DB.