BigData is a big deal because it can help answer questions fast

Blog post
added by
Wiki Admin

BigData is not just size and speed of complex data – it is moving us from information to knowledge

 

As our Why we focus on spatial data science article discusses, the progress of knowledge fields – history to math to engineering to science and to philosophy – or the individual pursuit of knowledge is based on moving from experiments to hypotheses to computation to now The Fourth Paradigm: Data-Intensive Scientific Discovery. This progression has happened over the course of human history and is now abstracting itself on the internet.

The early 90s web was about content, history, and experiments. The late 90s web was about transactions, security and eCommerce. The 2000s web was about engineering entities breaking silos – within companies, organizations, sectors, and communities. The 2010s web has been about increasing collaborating of communication, work production, and entering into knowledge collaboration. The internet progression is just emulating human history capability development.

When you are ready to move into BigData, it means you are wanting to Answer new questions.

That said, The BigData phenomenom is not about the input of all the raw data and explosion that the Internet of Things is being touted as. The resource sells, and the end product is the consumed byproduct. So lets focus on that by-product – its knowledge. Its not the speed of massive amounts of new complex and various quality data as our discussion on IBM’s 4 V’s focus on.

Its about what we can do with the technology on the cheap that before required supercomputer clusters that only the big boys had. Now with cloud, internet, and enough standards, if we have good and improving data, we ALL now have the environment to be answering complicated questions while sifting through the noise. Its about the enablement of the initial phase of knowledge discovery that everyone is complaining about the “web” right now “too much information” or “drowning in data”.

The article on Throwing a Lifeline to Scientists Drowning in Data discusses how we need to be able to “sift through the noise” and make search faster. That is the roadblock, the tall pole in the tent, the showstopper.

Parallelizing the search is the killer app – this is the Big Deal, we should call it BigSearch

If you have to search billions of records and map them to another billion records, doing that in sequence is the problem. You need to shorten the time it takes to sift through the noise. That is why Google became an amazing success out of nowhere. They did and are currently doing it better than anyone else – sifting through the noise.

The United States amazing growth is because of two things – we have resources and we found out how to get to them faster. Each growth phase of the United states was based on that fact alone, and a bit of stopping the barbarians at the gates our ourselves from implosion. You could say civilization. Some softball examples out of hundreds

  • Expanding West dramatically exploded after trains, which allowed for regional foraging and mining
  • Manufacturing dramatically exploded production output, which allowed for city growth
  • Engines shortened time between towns and cities, which allowed for job explosion
  • Highway systems shortened time between large cities, which allowed for regional economies
  • Airplanes shorten time between the legacy railroad time zones, which allowed for national economies
  • Internet shortened access to national resources internationally, which allowed for international economies
  • Computing shortened processing time of information, which allows for micro-targetted economies worldwide

Each “age” resulted in shortening the distance from A to B.  But, Google is sifting through data. Scientists are trying to sift as well through defined data sensors, link them together and ask very targetted simulated or modeled questions. We need to address the barriers limiting entities success to do this. 

 

So what is the point of this metaphoric drivel

Blog post
added by
Wiki Admin

So what is the point of this metaphoric drivel about cowpaths, space shuttles, and chariots?

Yes, fair enough. Aside from being a fun story, there should be a point.

I think there are 3, not unlike the Goldilocks story.

Change Agents can’t come in too hot to put in new technology and abandoned the old as there are consequences

Change Agents can’t come in too cold and put in new technologies just putting it in the footprint and same design footprint of the old.

Change Agents need to find the transition balance between the old and new that allows the new ecosystems to be adopted and the old ecosystem to adapt.

To get this balance, there are three factors standing in the way of introducing a disruption such as this:

  • Scaling – Scaling Research Readiness for solution expansion, adoption, and architecture qualities
  • Legacy – Legacy investment stakeholders agendas
  • Transition – Patterns for new investment that benefits the new solution and addresses legacy investment stakeholders

Read the next blog post for considering the disruption factors on an example topic – advancing our global network keeping up with the Computers to make the internet truly 21st century.

More to come.

When you are ready to move into BigData, it means you are wanting to Answer new questions.

Blog post
added by
Wiki Admin

The article on Throwing a Lifeline to Scientists Drowning in Data discusses how we need to be able to “sift through the noise” to be able this faster and faster deluge of sensors and feeds. Its not amount information management models of fast, large retail or defense data. It is about finding the signals you need to know to take advantage.

In controlled environments, like retail and business, this has been done for years on end to guide business analytics and targeted micro-actions. 

For instance, gambling industries have been doing this for 15 plus years taking in all the transnational data of each pull of a slot machine from all their machines from all their hotels linked with your loyalty card you entered and time of year and when you go, your profile, your trip patterns, then laws allowing, they adjust the looseness of the slots, the coupons provided, the trip rewards all to make sure they do what they are supposed to do in capitalism – be profitable. 

Even in uncontrolled environments such as intelligence, defense or internet search, the model is build analytics on analytics to improve the data quality and lifecycle so that the end analytics can improve. Its sound equalizers on top of the sound board.

Do go for the neat tech for your MIS. Go because your users are asking more of you in the data information knowledge chain. 

Continue on to read more on our follow-on article: BigData is a big deal because it can help answer questions fast

What cow paths, space shuttles, and chariots have in common

Blog post
edited by
Wiki Admin

A colleague recently sent me a chain email (they do still exist) about the old adage on how new technology is driven by thousand year old standards. I had seen it before. I remember then I liked it. But, my new habit on chain emails or viral urban legends was to poke around. Being childlike, I hope for fun new ways too see things, but being a problem-solver as well, I am skeptical of these amazing discovery of trivial connections. Regardless, its still a fun story where one can mine some good nuggets.

The anecdote essentially notes how historical inventions are connected and a moral. Reading it backwards, it connotes how the width of the space shuttle rocket boosters are due to width of railroad tunnel. And how railroad tracks width are due to the carriage wheel width. And how that width is tied to chariot width because of the width of two horses. Point being, the boosters width is derived due to width of two horses rear-ends.

Like I said, it is fun, but the tangents are more loosely coupled and coincidental than the “seven degrees of Kevin Bacon” concept. Snopes nicely walks us through how while this is true, but only through generalities – not unlike how someone could say the clothes we wear now is because of a medieval tailor sized it that way. Snopes can be a party pooper some time, but they did also note a few things about people and change (insert my agenda HERE). This is why I do like stories like this as I can tie my own tangential take-aways from it.

Snopes points out humans presets on change:

Although we humans can be remarkably inventive, we are also often resistant to change and can be persistently stubborn (or perhaps practical) in trying to apply old solutions to new conditions. When confronted with a new idea such as a “rail,” why go to the expense and effort of designing a new vehicle for it rather than simply adapting ones already in abundant use on roadways? If someone comes along with an invention known as an “iron horse,” wouldn’t it make sense to put the same type of conveyance pulled by “regular” horses behind it?

It goes on for several more examples noting how new innovations leverage the blueprints of previous generation inventions, regardless of their direct influence. The tone felt a bit down when noting this, but I felt this continuity is not wholly a bad thing.

As a physical society that build infrastructure to share, this compatibility is needed to limit the impact of disruption while progressing towards addressing societal challenges of Maslo’s Hierarchy of Needs globally.

For example, lets say there is a future decision to stop using dams for hydroelectric power and go into a series of nano-electric generators that works off river flow that would impede water less and generators more power. This is great as we have a lower cost, simpler, more efficient solution that also does not disrupt the ecosystem such as riparian development, fiash spawning, etc. like dams have for decades.

How do we transition to the new nano solution. The railroad story says we would use the previous footprint of the dam, and once ready, slowly migrate to the new solution to allow the water flow to slowly come back in place. This would allow the wetlands and riparian ecosystem to grow back at natures pace, and allow for fish and river life to adapt generationally.

Yet, the new solution does not require the same footprint. We could build it anywhere along the river. It could even be setup in a series of micro generators, and once the level of energy put into the grid matches the dams, in theory, the dam could just be exploded, and we could progress on without anyone in the future anthropocene historic footprint to be aware that a dam was ever there.

But, removing the previous infrastructure in a responsible way will be key. Blowing up a dam means the water release would cause major sediment displacement, kill the dam-resulted adapted riparians and wetland ecosystems, and generations of fish and river life would actually die as a result. The dismantling process, though not required for the new direct energy human need, is very critical to consider the indirect impact of the evolved ecosystem. 

If still interested, check out the follow-up blog post So what is the point of this metaphoric drivel

Some favorite TED talks

Blog post
edited by
Wiki Admin
– “Migrated to Confluence 5.3”

A business partner last night said “I don’t wake up and turn on my phone, or watch TV, or check email right away. I try to keep it simple… ” he said as several of us waxed rhapsodic of the pre-pocket tech and internet days and how teenagers patterns know no other world. But yet he continued, “OK, well that’s not true, I do get my morning dose of TED for inspiration”. 

Its just one more to add to the many morning intake mediums. People seeking personal philosophical guidance in the morning through religion, scripture, reading a story, meditation, prayer, mind-body engagement or quiet time. People seeking temporal context in morning news TV, newspaper, internet and feeds, websurfing (can I still use that term?), tablet time. People seeking social engagement with morning coffeee at the diner with the guys/gals, spouse or/and kid quality time, the facebook rise-and-shiner, or other social media digests. People seeking inspiration in either of the above

Personally, I have yet to ever find my morning ritual and I bounce in different mediums. Sometimes, its playing trains or toys or some activity with the family when we get a good rhythm going that morning, sometimes it is tablet browsing when feeling curious on various news or video feeds, sometimes it is mindless TV news digestion, and probably more rare than I should, sometimes it is outside quiet time in a run, bike, walk, or reading or whanot. Other times, the day gets going to fast, and there is no interstitial time, and an east coast call to this mountain time zone starts right up.

Though, I haven’t found my rhythm, but over the last partial decade here are a few of the greatest TED hits I’ve tweeted out as greatest hits and found inspirational :

Hans Rosling: Stats that reshape your world-view (Jun 2007)

Geoffrey West: The surprising math of cities and corporations (July 2011)

TEDxUofM – Jameson Toole – Big Data for Tomorrow (May 2011)

Eli Pariser: Beware online “filter bubbles” (Mar 2011)

Sugata Mitra: Build a School in the Cloud (Feb 2013)

Deb Roy: The birth of a word (Mar 2011)

-mt