(first published in 2015 at http://www.dataswft.com . Updated 7 Dec 2017)
What drives data driven cultures…besides coffee?
How do businesses deal with intuitive insights and machine generated insights? In a conversation with a brand consultant and travelista about my product Dataswft, sifting through #realtime #bigdata #analytics, he asked “where are the warmer human things that drives AI and ML technologies.” To be ‘data driven’ he pointed out, is a culture, unaware of the Tableau sponsored report from Economist “Fostering a data-driven culture.” To quote the report “IT security is indeed a job for experts, but data are everyone’s business.” I still struggle with the plural nature of data!
“Is Dataswft a technical thingy for data driven, or is it enabling data driven cultures?” the brand expert enquired. Time to act is a key metric for data-driven I explained. And so, to be data driven is an everyday matter as long as it provides value. But, how does that differ from a data driven culture? Then is the constant posing questions, small questions constitute a data driven culture?
Consider this scenario. For an online ad campaign the frequency of tweaks need balancing between regular and not at all. Regular requires minimal amount of data to analyze for metrics such as reach, clicks, CPC so on. Something like a weeks worth of data is good. But that is a heuristic that applies to a human scale of attention and processing. Or, compress it to a day, such that the human manager can take a look end of the day or beginning of the day so on. Also, these tweaks are post fact ie. historical data analysis using heuristic approaches.
Add machine learning, and it opens up two opportunities. First, unlike humans its not limited by fatigue, attention. Of course, we will never discount human creativity and imagination. Especially when dealing with limited information, limited time or limited capacity to process. These constraints are best obviated by humans than machines. Machine learning can give us the speed and capacity to deal with large data sets. The second opportunity for machine learning is the capactity to predict, by utilizing well defined mathematical models or algorithms.
With artifical intelligence the same campaign can now run with greater efficiency, more frequent tweaks, instead of weekly or daily windows. It can be real time, though more relevant to IT security and fraud management. Data driven coupled with such system intelligence gives us the opportunity to ask several ‘small questions’ that you can liken to ‘infinetesimal element’ in decomposing physical forces.
This finer abstraction, I conjecture, will allow for more accurate sampling of data and analysis by the machine. For the campaign manager, these frequent ‘small question’ analyses can present a visualization that is richer, provide better trending on the data and lead to better decisions.
If data driven represents our ability not limited to capture and store of data, but to process it continuously, and asking of it ‘small questions’ that are well-modelled, then our ability to connect with the output of a data driven process, coupled with human intuition stands for a data driven culture.
Providing answers instantly is what technology does well. With the technology, its the culture that realizes potential and pushes the envelope.
Consider an investment bank that needs to run value at risk calculations covering a host of financial products invested in by clients, touples of market price data for months, hundreds of sophisticated risk models designed to predict risk against different scenarios. To this person, its important to know how much money is to be set aside against the dynamic risk and how much capital can be unlocked to earn. Calculations here can run into over 15 billion and to execute in under 30 seconds can make a huge difference to these money managers. This scenario is possible only in a data driven culture and keep a handle on risk in a volatile market that involves many asset classes.
Data driven cultures are those we see at the top of the ‘culture pyramid’, crunching all their data by the second, minute and hour and not some end of the day or end of the week event. But as one moves higher up the pyramid, the response times that the culture will accept reduces from minutes and hours to mere seconds. That is not to say every industry out there needs to optimize at nano second level, real time analytics, but each industry should choose based on where the opportunity lies and where demand lies. Security industry can only survive in a real time processing of events. Social media marketing may find untapped opportunity in an hourly cycle. Education and learning industry may find it suitable by moving to end of day, from end of term, so on. In all these, the data driven culture consists of small questions we ask of the data.
Put another way, data driven cultures are those complementing their solely heuristic decision making process (read gut feel) with a data driven approach and thus; what do the data say! That is not just about the quantity of it, but also a quality that heuristic rules and human cognition are likely to be overwhelmed given the volume, variety, velocity.
In a digital world, even when its the same question asked few minutes back, or yesterday and last quarter, so on, paradoxically, the answer is never the same, instead gets better or is likely different.