Back Into Things…
As a follow-up on the “cliffhanger” from Big Data is a big deal because it can help answer questions fast, there are three top limitations facing Big Data efforts right now: Data Quality, People and Process, and Technical Access to Information. It is important to understand these limitations first so that we can properly address the concept of Big Data. So, lets jump right in and explain each limitation.
Number One And By Far The Biggest Issue – Data Quality
We are not exaggerating. Data quality is without a doubt the biggest, most important limitation facing Big Data at the moment. If you do not believe us, then let us take a moment to explain things with a very controversial topic: climate change. We will start by getting one big fact out of the way. Climate Change is not a myth. However, it is the first science presented on a data premise. And in doing so, climate scientists prematurely presented models that didn’t take into account the driving climate variables. Consequently, scientific models have changed over and over again throughout time as the resolution of source data has increased. And from these data have come simulations on top of simulations. And these simulations prove countless theories of various models only demonstrable by Hollywood blockbusters.
Point being, we are dealing with inferior data for a world scale problem. And what is more, we have jumped into apolitical, emotionally driven world with a data report. We have pretty much set ourselves up for a lot of unnecessary headaches. We have become the frog in slowly warming water, and we will hit that boiling point late. All because we started with a data justification approach using low quality data. Are they right the world is warming? Yes. Do they have enough data to prove the right mitigation, mediation, or policy adjustments? No, and that will remain the case until either we increase the data quality or take a non-data tact.
People And Processes Are A Generation Away.
Our processes in IT have been driven by Department of Defense (DoD) and General Services Administration (GSA) business models from the 1950’s. It was simple, but a bit insulting honestly. The concept was to put anyone managing 0s and 1s technology in the back of the organization. Because they are nerds. They look goofy, cannot talk, do not understand what we actually do here and they smell funny. That has been the approach to IT since the 1950’s.
Unfortunately, nothing has changed. Well, except that there are a few bakers dozen of the hoodie-wearing, mountain-dew-drinking, late-night-owls who happen to be loaded now. Consequently, there is a pseudo culture of geek chic. Our investment in people has not matured yet. We cannot balance the maturity of service, data, governance, design, and product life cycle in order to embrace that engine culture as core to the business. This means we need more effective information-sharing processes to get the right information to the right people.
Access To Environments
If you asked about this limitation pre-hosting environments or pre-cloud, the answer would have been different. They would have been limited to massive corporations, defense, intel, and some of the academia co-investing with those groups. But we live in the cloud era now. Consequently, the answer has only expanded, leading to something of a strain in shifting to a BigData infrastructure like the cloud.
If you can manage the strain of shifting to this BigData infrastructure, this barrier should be the least of your problems. If you can allow your staff to get the data they need at the speed they need so they can process in parallelization without long wait times, you are looking good. Get a credit card. Or if you are part of the Government, buy off a Cloud GWAC, and get your governance and policies moving, as they are likely behind and not ready. Likely they will prolong the siloed information phenomenon. Focus on the I in IT, and let the CTO respond to the technology stack. Just remember to stay on top of the shift, and this limitation will not slow you down.
In Conclusion: Focus On Data Quality, Have A Workforce Investment Plan, And Continue Working Your Information Access Policies
Then we reach a tipping point that moves you into BigData. This is where they are required to combine to help you deal with the complicated enormity at speeds that answer questions for you, MIS and its reports. If you can focus on those limitations in the aforementioned order (likely solving in reverse), you will be able to implement parallelization of data discovery.
This will shorten the distance from A to B and create new economies, new networks, and enable your customer or user base to do things they could not before. It is the train, plane, and automobile factor all over again. If data is all about being smart and being efficient, then we need to be able to tackle the limitations to data smartly and efficiently. This means shortening that distance from A to B, enhancing the quality of the data, and building better “automobiles” to take people to point B.
And now, to throw the shameless plug in, this is what we do. This is Why we focus on spatial data science and also answers the question: Why is change so fundamental?