Thursday, 31 May 2012

The Journey from Data Warehouses to Big Data

The Journey from Data Warehouses to Big Data
Data Warehouses were driven by the explosion of data within corporate ERP and CRM systems and the need for better management reports. Generally speaking, the standard reports from these packaged systems did not fit the needs of most corporate so initially there was a huge demand to download data to Excel or Access. This approach spawned a huge industry out of end user computing with the report development process in the hands of power users within each business group. While at the time it certainly added value, if left unchecked it created huge internal disconnects for management reporting processes. The advent of better replication and BI tools helped corral some of the wild west attitude that prevailed, while in parallel the management reporting processes, integrated ERP/CRM user friendly reports improved but most importantly, the deployment of Data Warehouses aimed at delivering really useful end user reporting started to gain traction.

If you were not around to see the buzz and hype that surrounded the planning and deployment of Data Warehouses then you might think this all happened really quickly and industry got real value from most deployments. The real truth is that most Data Warehouse deployments in the early years struggled to gain traction due to several factors and for the first 10 years there were many costly failures.

The data stored in Data Warehouses is primarily extracted from the ERP and CRM systems so thus comes from known sources that are updated at regular intervals and where the ERD is under tight control. Big Data deployments typically have several new data sources that are coming from outside the control of the corporate IT group. The data structures, data provenance, data quality, timing of updates and several other factors are not within the control of the consumers in most cases. These data sources can provide data in several formats which include even unstructured data.

If we are to believe in the vision that “data is the new oil” for service enabled businesses of the future, then businesses must develop staff, technology and processes that will assist in unlocking the value beneath the surface. The challenge is a very different one to deploying Data Warehouses so the staff and approach need to be adjusted. The future vision of business and government functions powered by Big Data enabled services disrupts the existing paradigms of static processes that are occasionally reviewed and changed. The new world business order will have even more nimble business processes where we understand not just what happened so we can report on that historical event or what even might be a feasible scenario that could play out but we will better understand the factors influencing these events so we can do further models and react closer to real time. The organisational ability to react to change will be key, organisations that are better informed with nimble processes will be better positioned to react to downturns or take advantage of upturns as the opportunities arise.This is where the FuturICT Flagship hopes to play an important role in the development of this new Big Science.

Wednesday, 23 May 2012

FuturICT and what it means for the future of Data Business

FuturICT and what it means for the future of Data Business

Over the next several weeks I will publish a series of articles on the FuturICT blog on what FuturICT means for the future of Data Business. Yes I am explicitly using the term Data Business as that is the portion of the project I am closely related to and these two words highlight the potential impact of the project for our business partners as well as broad segments of society. 

From the beginning we have always said the project is about connecting Science with Policy with Technology. The planned synthesis of these three items is not as easy as it seems so the FuturICT goal is a real “Grand Challenge”. Within the project there is a focus on leveraging ICT, Social Science and Complexity Science in order to deliver the goals of FuturICT. I like to use the slogan “Big Society, Big Problems, Big Data, Big Complexity so hence we need this new multidisciplinary approach from Big Science to help us”.

In order to understand the broad ranging concepts that are involved in FuturICT I intend to elaborate on the topics below in separate posts. Hopefully they will give you some context as to what are the most important components of the projects and how they relate to business and the jobs, growth and increased competitiveness agenda that is so badly required in the EU.
  • ·         The journey from Data Warehouses to Big Data
  • ·         Planetary Nervous System (PNS)
  • ·         Living Earth Simulator (LES)
  • ·         Global Participatory Platform (GPP)
  • ·         Innovation Accelerator (IA)
  • ·         The Service Economy, Knowledge Economy, Innovation Economy,....
  • ·         Business Engagement and Business Impact
  • ·         Building a scientific community as the first step to starting a movement that will revolutionise how we use data
·         ....and more topics to come.....

Wednesday, 16 May 2012

Dirk Helbing: New Ways to Promote Sustainability and Social Well-Being in a Complex, Strongly Interdependent World

This is the text of the speech that I gave this morning in Maribor, Slovenia at the Out of the Box Conference 2012- Special session with World Thinkers

Since the origin of human civilization, there is a continuous struggle between chaos and order. While chaos may stimulate creativity and innovation, order is needed to coordinate human action to create synergy effects, more efficiency, and common goods such as our transportation infrastructures, universities, schools, and theaters, institutions (like parliaments and courts), but also language and culture.

According to Hobbes, civilization started with everyone fighting against everybody else (“homo hominis lupus”). Even today, civilization is highly vulnerable, as the outbreak of war in former Yugoslavia has shown, but also the situation in many countries today, particularly after natural disasters.

On the one hand, we are struggling with conflicts, which are the result of suppression and lack of participation, and of sanctioning people who are not part of the mainstream culture. On the other hand, we are suffering of many social dilemmas, such as the exploitation and destruction of our environment, global warming, overfishing, tax evasion, exploitation of social benefit systems, and other tragedies of the commons.

In order to mitigate problems like these, we need to learn how to understand and manage complexity in our techno-socio-economic-environmental systems. For this, we need to think out of the box, because complex systems work differently from what our intuition suggests.

Complex systems are often hard to predict and hard to manage. And they behave in surprising ways. Their behavior is NOT well understood from the properties of their COMPONENTS. It’s rather the INTERACTIONS between these components, which we need to focus on, because they are the basis of the self-organization of complex systems and of new, so-called emergent properties, which cannot be understood from the component properties. Society is more than the sum of its parts.

The change of perspective from a component-oriented to an interaction-oriented view may be AS hard and revolutionary as the transition from the geo-centric to the helio-centric worldview, which made modern physics possible, and many of the benefits that came with it. This new systemic view will enable us to find new solutions to old problems such as social conflicts and tragedies of the commons.

Let me stress that economic value generation would not be possible without many kinds of social capital, such as trust, solidarity, social values, norms, and culture. While absolutely crucial for social well-being and the functioning of society, social capital is largely invisible, and hardly understood. We know, however, that it is the result of our social network interactions. But social capital may be damaged or exploited, as the environment has been damaged and exploited. Hence, we need to learn how to value and protect social capital. The evaporation of trust during the financial crisis may serve as a warning example. It caused the evaporation of thousands of billions of dollars in the stock markets and elsewhere.

In order to learn how to protect social capital, we need, first of all, a global systems science, which allows us to gain a holistic understanding of our world, of systemic risks, and how integrated systems design can create more resilient and sustainable systems. Changing interactions in the system can mitigate problems, since the current kinds of interactions are causing them. In fact, unstable interactions cause problems such as tragedies of the commons, segregation, conflict, revolutions, wars, and cascading effects causing extreme events and disasters. Moreover, our global markets are constructed in a way that ethical behavior tends to have a competitive disadvantage – that’s why ethical behavior has so hard times to survive and spread.

We may understand such systemic instabilities as situations, in which a situation gets out of control even, if everyone is trying to do his/her best. Take dense, but continuous traffic flow on a circular road as an example. Sooner or later, the traffic flow will break down, creating a traffic jam, although everyone tries to avoid it. In other words, systemic instabilities makes systems largely uncontrollable. But to some extent, this can be changed!

Societies have found many ways to create social order. The most archaic one is the creation of families and tribes, which builds on the mechanism of genetic inheritance. Neighborhood interactions (even with strangers) and direct reciprocity based on repeated interactions (I help you and you help me) is a more modern mechanism to promote social cooperation. Furthermore, humans have created sanctioning institutions (including the police). And last but not least, cooperation and social order may be promoted by reputation mechanisms. To counteract global destabilization, we are currently seeing a growth in sanctioning efforts, but reputations systems are also quickly spreading, e.g. in the internet.

To stabilize our ever more complex systems, we will have to integrate decentralized management elements into our management approaches, utilizing the principle of self-organization. New information and communication technologies make it possible to overcome barriers to social, economic and political participation. They could also support fair ways of sharing. Again, the interaction rules are crucial here. If someone who cuts a cake can take the first piece, it will often be the biggest one. If the person, who cuts the cake takes the last piece, he/she will be very careful to cut it in a fair way, with all pieces having an equal size.

New information and communication technologies can also promote an inter-cultural understanding (via an intercultural translator), facilitate social money (with a memory and reputation), or value-sensitive action. They can furthermore promote accountability and awareness.

Awareness helps to avoid many mistakes one would otherwise make. The FuturICT project plans to create a Planetary Nervous System, which will support such awareness of the state of the world, including the value of social capital. FuturICT’s Living Earth Simulator will help to anticipate possible scenarios (as we have done it by mental simulation in the past for much simpler situations). This can warn us of systemic risks, but also point us to new opportunities. And finally, FuturICT’s Global Participatory Platform will make these new instruments, which serve to gain insights into our complex world, accessible to everyone.

Remember, most people DO want to appear attractive and beautiful. If we manage to create awareness of the implications of their decisions and actions, they will change their behavior. I am deeply convinced that we CAN create a better world – if we create suitable instruments, gain better insights, and do it together.

Thank you!