A recent study by Mckinsey & IDC found that 20% of a knowledge worker’s time was used up by searching for market information. On average, they calculate that for every 1,000 knowledge workers employed in an organization, this represents approximately 3 million USD / year wasted looking for market insights. In total, this costs the Fortune 1000 companies 2.5 billion USD a year in searching for data that cannot be found.
IDC refers to this as the “Knowledge Deficit”. When considering that for 2019, ESOMAR estimated 45 $B in Market Research Data and Services, this deficit represents a major issue for companies to overcome.
How does AI fill the insight gap?
AI applied to Market Intelligence is more than ever being recognised as key to tackling the ever growing insight gap, with AI being pushed to the forefront of strategic recommendations.
AI is a powerful technology that can read thousands of articles per minute, understand important concepts, link topics together, highlight contradictions, in other words an AI properly trained is able to takeover knowledge workers’ repetitive, time consuming and complex tasks an complete it in couple minutes where it would typically take us hours, when not days.
The table below illustrates the performance difference between deep learning algorithm vs human intervention, emphasizing a x30,000 factor improvement when using an algorithmic approach.
|2 articles / min||1,000 articles / s.|
How does this Machine Learning process work?
Deep learning, which is a facet of machine learning, is capable of self- learning from data that is unstructured or without metadata. The advantage of using this type of technology is that it learns on its own from a large amount of training data – in ReportLinker’s case training data is made of economic articles, analysis and reports – and it is capable of reiterating through this process as many times as it is required to achieve acceptable precision.
What is ReportLinker’s method to leverage Machine Learning for insights detection?
Despite the advantages machine learning offers, effective & robust insight retrieval requires rigorous methodology when it comes to implementation, in order to avoid common pitfalls such as over-fitting.
We have modeled the tasks undertaken by our data scientists. Below is an example of the steps they follow to detect industry & sector specific insights:
- Annotated corpus
We prepared an annotated corpus of 100,000+ sentences providing typical insights from a range of 20+ categories, spanning from Market forecast, to Merger and Acquisitions, to consumer insights.
- Feature engineering
We prepared a list of potential features the model learns from, such as word embedding, article topics, named entities found in the text etc … All together we ended up with a list of 400 features
We built a data pipeline based on Tensorflow, using python and libraries such as Keras, Pandas… and we conducted the learning task on a set of 80,000 sentences.
The 20,000 remaining annotated sentences were kept safely outside of the learning process as a seperate test corpus. This way we were able to quickly assess precision and recall of each iteration and adapt the features accordingly. This allowed us to eliminate the risk of overfitting our model. We finally came up with a model providing the required performance we needed.
Once implemented into production, results obtained with this method were much better than manual approaches, such as Boolean research techniques.
AI enables us to detect emerging trends and early signals thanks to our trained learning algorithm.
The graph below illustrates the performance difference between our trained MI solution vs human intervention in surfacing insights.
How can Reportlinker help your Market Intelligence team gain in efficiency?
This type of technology offers enormous potential to facilitate the role of MI professionals including an increased productivity and accuracy.
It is becoming the new norm because of the benefits it represents such as:
- Freeing up time for more value-added activities (reducing manual labor intensive search tasks)
- Reducing the risks in missing critical data
- Better managing the ever increasing data flow available to analyse.
Fill in the form to watch our one minute demo: