Latest revision as of 13:07, 13 September 2016

1 Overview

Data Warehousing and Business Intelligence is a growing area in Industry. How does bxp fit into this area and what can it do to facilitate Data Mining and other "Big Data" processes.

Firstly it is important to recognise the area as three stages. Input, Process and Output.

• Input: Data acquisition and attaining raw data.
• Process: Business Intelligence (BI) tools and their application
• Output: The presentation of data, results and ability to go exploring.

bxp can deliver solutions at all levels of the Information Pyramid.

Posted by Business Intelegence	Michael A. Urmeneta, MS

The different levels are often referred to as:

• EIS: Executive Information Systems. Highly summaries data with extensive drill down. Provides strategic decision support.
• MIS: Management Information Systems. A broader focus with a week to week focus to organise, evaluate and efficiently manage departments within an organization.
• DSS: Decision Support System. Programs required to assist with analysis and decision-making within an organization.
• BIS: Business Information Systems. Tactical information with a day to day focus. More data than information.

2 Examples of clients using / have used this solution

3 Input: Data acquisition

The data acquisition is done initially through the MetaData module. MetaData_-_Overview

The MetaData module would be one of the more technical modules so we tend to work closely on the projects and their implementation, in order to guide our clients through their various data sources.

Discussions about how data gets from a client into bxp can be found here. MetaData_-_Retrieve

Specifically discussing the famous Extract - Transform and Load process of other system ("ETL") approach, our development documentation is available from File:Module - MetaData v2-0.pdf

Essentially these are the functions that allow you to transform the data. A quick summative list is available from MetaData_Process_Rules

White papers which are usually completely worked solutions demonstrating for a specific client. For security reasons All n One do not go too much into the specific security details of how our clients interactions work (for obvious reasons). Upon completion of a Non-Disclosure Agreement various low security examples can be discussed.

A demonstration of a complex MetaData process of disceting a Nortel Agent by Application report is available from MetaData_-_Worked_Example_Nortel_Report

Where appropriate / similar data warehousing projects have been implement All n One will attempt to facilitate a discussion between parties to go through the real world pros and cons of data warehouse implementation solutions.

4 Process: High Volume management

This is an area of enormous interest for All n One. Our development team have many years experience in working with database and their optimisation including Master papers and delivering lectures in the Institute of Technology, Tallaght on the subject.

It is important to note that All n One are not Data Scientists, nor are we a big data / data analysis specialist house. We have extensive knowledge in the area and the capabilities but we don't proclaim to know everything nor have perfect tools for your expert data. Datasets of 10M+ tuples are easily managed in bxp though the solution has been rated to 1000M+ tuples per table.

Data size and processing power is reviewed with the client to ensure appropriateness of solution provided.

bxp operates an nTier structure separating web and database. The_bxp_Infrastructure

We have a public statement on capacity management. Bxp_Infrastructure_Capacity. Further specific technical details are available with Non-Disclosure Agreement and contractual agreements in place.

As Oracle MySQL is the primary database engine using MyISAM tables for high read capability, our OLAP solution of choice is Mondrain http://community.pentaho.com/projects/mondrian/

Here is an example of how you set about configuring your own Mondrain on top of a MySQL instance. http://mysqluc.com/presentations/mysql06/hyde_final.ppt

MySQL supports aggregate Group by capabilities which facilitates Mondrain output. https://dev.mysql.com/doc/refman/5.6/en/group-by-modifiers.html

All n One are working towards making the Mondrian interface options available through a future Data Mining module in bxp targeted for delivery in 2017.

Real time exploration of output would be facilitated by providing the raw MyISAM data read to process on a machine of your own choosing.

If you have specific rollups / cubes of data for management to explore, these would be facilitated through MetaData and built as a "KeyStat" report. ( Detail continued into the next section)

5 Output: Front-End Reporting

KeyStats wallboards provide our clients are real-time interfaces to data. Module_-_KeyStats A drill down is as simple as providing an A HREF tag on the report to click through to a further detail report.

Reports already pre-built and demonstrable are available in the Data Profiling report gallery Data_Profiling_-_Report_Gallery

Prebuilt reports will go a long way to allowing you to prototype and try different outputs for most data sets before getting in to build completely custom solutions.

KeyStats provides complete accessibility to the database. So all tables, all data. It allows you to design and implement complete custom reporting with the power of HTML and the underlying database speed of MySQL. Module_-_KeyStats

Whilst the data layout is anything you can imagine in terms of tables and drill downs, the biggest questions we get asked is about our ability to visualise data.

We use FusionCharts, Specifically FusionChartsXT FusionCharts These charts give bxp the capability to render any amount of 2D and 3D representations of data with an almost infinite amount of configuration options.

As a worked example, bxp would render your output data to screen as xml, then you'd be able to configure as follows http://www.fusioncharts.com/dev/getting-started/using-xml-as-data-format.html

Creating interactive KeyStats as an example is covered here Key_Stats_-_Visualisations

6 Development Approach

To facilitate discussions and review options a number of steps for any project would be put in place.

1. Put an NDA (Non-Disclosure Agreement) in place
2. Share the sample outputs of each of the data sources so they can be scoped
3. Build into a dedicated test system the data import projects on a one by one basis to demonstrate capability
4. Sketch and prototype reports using KeyStats to provide idea of final output
5. Quote the work
6. Upon agreement, developed the solutions

All n One have a structured development approach. All_n_One_Project_Development_Approach

7 Next steps

So why not get in touch and see how bxp and the All n One team can help you realise your data warehousing and BI needs. Just call Nick on +353 1 429 4000 or email sales@bxpsoftware.com