Analysis

Data Ingestion

Data Ingestion is the process of reading data from a local or remote source, transferring the data from the source to centralised storage locations and transforming the data into more useful formats before storage. It's an easy concept to understand but the reality is deceptively complex.

The best sources of data have to be considered and evaluated. Part of the evaluation is to make sure that irrelevant, duplicated or poor quality data is not transferred and then ingested. Remote data sources also have security considerations because they may not reside in the same security domain as the destination.

Lastly, the transformation process (known as ETL) is the first opportunity to enrich the data. It is a process called Feature Engineering, which basically deconstructs complex data into formats that are easier to analyse (especially for Machine Learning). The main purpose is to store data in a way that further analysis is not only enabled but rather as the next primary objective.

We offer services to help identify all the relevant data sources available to our clients, how to efficiently ingest the data and how to extract the hidden features in the data at the correct point in the life-cycle.

Data Storage

The new field of Data Engineering exists because of the technical problems encountered when storing and managing large volumes of data. It covers the mechanical storage challenges which are solved by big data architecture, selection of the right tools for the job and hosting in the correct cloud environment. The cloud has been the enabler of modern Data Science because of the cheaper storage capability, but also the agile availability of processing and memory.

The storage designs also extend the Feature Engineering work done during data ingestion. Specific decisions need to be made if and how to store enriched data or to calculate features on the fly. These decisions can have a great impact later during analysis, so there is a need to constantly revisit design decisions and fine tune the designs.

We are able to create highly performant and reliable architecture designs, but also engage with the internal teams of our clients to provide guidance and advice if they would like to build the storage systems themselves.

Machine Learning and Deep Learning Analysis

There have been huge advances made in the fields of Machine Learning (ML) and especially Deep Learning (DL). These algorithms extend far beyond the capabilities of statistical analysis that most companies employ for BI and MI reports.

Deep Learning in particular can be used to construct services that are able to perform predictive and prescriptive analysis. Using such services, we can help our clients to answer deep business critical questions, such as probability based predictions of what might happen next under current conditions, and then what are the best next actions to take.

Our offerings include:

• Matching the available data with the possible types of analysis methods
• Advising what additional data might improve the analytical capabilities, so that better training data can be created
• Selecting the most appropriate ML or DL algorithm with the problem case
• Optimising the algorithm through a process called Hyper-parameter Tuning, which determines the best configuration settings

Visualisation and Actionable Insights

As part of the introduction of advanced analysis, there is always a component of change management required, to ensure that people who are most dependant on reliable answers become comfortable with the new methods. A focus on very effective visualisations is therefore key, because the consumers of the analytics need to understand the results provided, but also the interpretation of the confidence scores of the results.

Choosing the correct presentation technology and methods play a big role in winning the trust of decision makers. Equally important, trust creates the opportunity for automation and cost savings once specific predictive and prescriptive results are consistently accurate and with high confidence scores.

It is worth mentioning that the production of insights without taking action, invalidates the investment in Data Science. Actionable insights and decision support is the holy grail of advanced analytics.

We are able to create visualisations, using various technologies, that will present results in a clear and insightful way, so that decisions can be taken. As the quality of the data and training improves, these systems become capable of making recommendations of the action to take. We can assist in this improvement process and also engage with leaders and decision makers to ensure they become confident in the results.