Thursday, 31 May 2012

The Journey from Data Warehouses to Big Data


The Journey from Data Warehouses to Big Data
Data Warehouses were driven by the explosion of data within corporate ERP and CRM systems and the need for better management reports. Generally speaking, the standard reports from these packaged systems did not fit the needs of most corporate so initially there was a huge demand to download data to Excel or Access. This approach spawned a huge industry out of end user computing with the report development process in the hands of power users within each business group. While at the time it certainly added value, if left unchecked it created huge internal disconnects for management reporting processes. The advent of better replication and BI tools helped corral some of the wild west attitude that prevailed, while in parallel the management reporting processes, integrated ERP/CRM user friendly reports improved but most importantly, the deployment of Data Warehouses aimed at delivering really useful end user reporting started to gain traction.

If you were not around to see the buzz and hype that surrounded the planning and deployment of Data Warehouses then you might think this all happened really quickly and industry got real value from most deployments. The real truth is that most Data Warehouse deployments in the early years struggled to gain traction due to several factors and for the first 10 years there were many costly failures.

The data stored in Data Warehouses is primarily extracted from the ERP and CRM systems so thus comes from known sources that are updated at regular intervals and where the ERD is under tight control. Big Data deployments typically have several new data sources that are coming from outside the control of the corporate IT group. The data structures, data provenance, data quality, timing of updates and several other factors are not within the control of the consumers in most cases. These data sources can provide data in several formats which include even unstructured data.

If we are to believe in the vision that “data is the new oil” for service enabled businesses of the future, then businesses must develop staff, technology and processes that will assist in unlocking the value beneath the surface. The challenge is a very different one to deploying Data Warehouses so the staff and approach need to be adjusted. The future vision of business and government functions powered by Big Data enabled services disrupts the existing paradigms of static processes that are occasionally reviewed and changed. The new world business order will have even more nimble business processes where we understand not just what happened so we can report on that historical event or what even might be a feasible scenario that could play out but we will better understand the factors influencing these events so we can do further models and react closer to real time. The organisational ability to react to change will be key, organisations that are better informed with nimble processes will be better positioned to react to downturns or take advantage of upturns as the opportunities arise.This is where the FuturICT Flagship www.futurict.eu hopes to play an important role in the development of this new Big Science.

Wednesday, 23 May 2012

FuturICT and what it means for the future of Data Business


FuturICT and what it means for the future of Data Business

Over the next several weeks I will publish a series of articles on the FuturICT blog on what FuturICT means for the future of Data Business. Yes I am explicitly using the term Data Business as that is the portion of the project I am closely related to and these two words highlight the potential impact of the project for our business partners as well as broad segments of society. 

From the beginning we have always said the project is about connecting Science with Policy with Technology. The planned synthesis of these three items is not as easy as it seems so the FuturICT goal is a real “Grand Challenge”. Within the project there is a focus on leveraging ICT, Social Science and Complexity Science in order to deliver the goals of FuturICT. I like to use the slogan “Big Society, Big Problems, Big Data, Big Complexity so hence we need this new multidisciplinary approach from Big Science to help us”.

In order to understand the broad ranging concepts that are involved in FuturICT I intend to elaborate on the topics below in separate posts. Hopefully they will give you some context as to what are the most important components of the projects and how they relate to business and the jobs, growth and increased competitiveness agenda that is so badly required in the EU.
  • ·         The journey from Data Warehouses to Big Data
  • ·         Planetary Nervous System (PNS)
  • ·         Living Earth Simulator (LES)
  • ·         Global Participatory Platform (GPP)
  • ·         Innovation Accelerator (IA)
  • ·         The Service Economy, Knowledge Economy, Innovation Economy,....
  • ·         Business Engagement and Business Impact
  • ·         Building a scientific community as the first step to starting a movement that will revolutionise how we use data
·         ....and more topics to come.....

Wednesday, 16 May 2012

Dirk Helbing: New Ways to Promote Sustainability and Social Well-Being in a Complex, Strongly Interdependent World


This is the text of the speech that I gave this morning in Maribor, Slovenia at the Out of the Box Conference 2012- Special session with World Thinkers


Since the origin of human civilization, there is a continuous struggle between chaos and order. While chaos may stimulate creativity and innovation, order is needed to coordinate human action to create synergy effects, more efficiency, and common goods such as our transportation infrastructures, universities, schools, and theaters, institutions (like parliaments and courts), but also language and culture.

According to Hobbes, civilization started with everyone fighting against everybody else (“homo hominis lupus”). Even today, civilization is highly vulnerable, as the outbreak of war in former Yugoslavia has shown, but also the situation in many countries today, particularly after natural disasters.


On the one hand, we are struggling with conflicts, which are the result of suppression and lack of participation, and of sanctioning people who are not part of the mainstream culture. On the other hand, we are suffering of many social dilemmas, such as the exploitation and destruction of our environment, global warming, overfishing, tax evasion, exploitation of social benefit systems, and other tragedies of the commons.

In order to mitigate problems like these, we need to learn how to understand and manage complexity in our techno-socio-economic-environmental systems. For this, we need to think out of the box, because complex systems work differently from what our intuition suggests.

Complex systems are often hard to predict and hard to manage. And they behave in surprising ways. Their behavior is NOT well understood from the properties of their COMPONENTS. It’s rather the INTERACTIONS between these components, which we need to focus on, because they are the basis of the self-organization of complex systems and of new, so-called emergent properties, which cannot be understood from the component properties. Society is more than the sum of its parts.

The change of perspective from a component-oriented to an interaction-oriented view may be AS hard and revolutionary as the transition from the geo-centric to the helio-centric worldview, which made modern physics possible, and many of the benefits that came with it. This new systemic view will enable us to find new solutions to old problems such as social conflicts and tragedies of the commons.

Let me stress that economic value generation would not be possible without many kinds of social capital, such as trust, solidarity, social values, norms, and culture. While absolutely crucial for social well-being and the functioning of society, social capital is largely invisible, and hardly understood. We know, however, that it is the result of our social network interactions. But social capital may be damaged or exploited, as the environment has been damaged and exploited. Hence, we need to learn how to value and protect social capital. The evaporation of trust during the financial crisis may serve as a warning example. It caused the evaporation of thousands of billions of dollars in the stock markets and elsewhere.

In order to learn how to protect social capital, we need, first of all, a global systems science, which allows us to gain a holistic understanding of our world, of systemic risks, and how integrated systems design can create more resilient and sustainable systems. Changing interactions in the system can mitigate problems, since the current kinds of interactions are causing them. In fact, unstable interactions cause problems such as tragedies of the commons, segregation, conflict, revolutions, wars, and cascading effects causing extreme events and disasters. Moreover, our global markets are constructed in a way that ethical behavior tends to have a competitive disadvantage – that’s why ethical behavior has so hard times to survive and spread.

We may understand such systemic instabilities as situations, in which a situation gets out of control even, if everyone is trying to do his/her best. Take dense, but continuous traffic flow on a circular road as an example. Sooner or later, the traffic flow will break down, creating a traffic jam, although everyone tries to avoid it. In other words, systemic instabilities makes systems largely uncontrollable. But to some extent, this can be changed!

Societies have found many ways to create social order. The most archaic one is the creation of families and tribes, which builds on the mechanism of genetic inheritance. Neighborhood interactions (even with strangers) and direct reciprocity based on repeated interactions (I help you and you help me) is a more modern mechanism to promote social cooperation. Furthermore, humans have created sanctioning institutions (including the police). And last but not least, cooperation and social order may be promoted by reputation mechanisms. To counteract global destabilization, we are currently seeing a growth in sanctioning efforts, but reputations systems are also quickly spreading, e.g. in the internet.

To stabilize our ever more complex systems, we will have to integrate decentralized management elements into our management approaches, utilizing the principle of self-organization. New information and communication technologies make it possible to overcome barriers to social, economic and political participation. They could also support fair ways of sharing. Again, the interaction rules are crucial here. If someone who cuts a cake can take the first piece, it will often be the biggest one. If the person, who cuts the cake takes the last piece, he/she will be very careful to cut it in a fair way, with all pieces having an equal size.

New information and communication technologies can also promote an inter-cultural understanding (via an intercultural translator), facilitate social money (with a memory and reputation), or value-sensitive action. They can furthermore promote accountability and awareness.

Awareness helps to avoid many mistakes one would otherwise make. The FuturICT project plans to create a Planetary Nervous System, which will support such awareness of the state of the world, including the value of social capital. FuturICT’s Living Earth Simulator will help to anticipate possible scenarios (as we have done it by mental simulation in the past for much simpler situations). This can warn us of systemic risks, but also point us to new opportunities. And finally, FuturICT’s Global Participatory Platform will make these new instruments, which serve to gain insights into our complex world, accessible to everyone.

Remember, most people DO want to appear attractive and beautiful. If we manage to create awareness of the implications of their decisions and actions, they will change their behavior. I am deeply convinced that we CAN create a better world – if we create suitable instruments, gain better insights, and do it together.

Thank you!  

Tuesday, 27 March 2012

Ethics and Social Behaviour in ICT today

Two cases in the news today which highlight ethical issues in ICT and changes in social behaviour. First the US case of the celebrity hacker who failed to realise that private meant private and the other in the UK was the case of the twitter user jailed for offensive comments.

Thursday, 22 March 2012

UCL Energy Institute professors announced as government advisors



Professors Tadj Oreszczyn and Paul Ekins, UCL Energy Institute, on their appointment as advisors to the Department of Energy and Climate Change’s new Energy Efficiency Deployment Office (EEDO). The Office will be working with leading industry experts to ensure that they have the best possible evidence, analysis and policy response to a challenging agenda in an area which has previously been fragmented across stakeholder groups and government departments.

Full article: http://www.bartlett.ucl.ac.uk/energy/news/eedo-advisers

Wednesday, 21 March 2012

FuturICT – Participatory Computing for Our Complex World at the Swiss Academy of Arts and Sciences





The FuturICT Project on 
“Global Participatory Computing for Our Complex World”

Questions and Answers by Dirk Helbing


Q: What will the FuturICT project do?
A: The FuturICT project is about creating new information and communication systems that are beneficial for individuals and society. Our aim is to learn how to better understand and manage complex global systems, with a focus on sustainability and resilience.


Q: What is the progress of the FuturICT project in building a Crystal Ball to tell the future of the world?

A: In contrast to the Crystal Ball projects in the United States, FuturICT is rather developing a global systems science to understand our world and systemic risks better. It also creates new instruments to explore our options to act, and to manage our future in a participatory way.


Q: Can you be more specific?

A: Yes, of course. We are now entering an age of Big Data, which creates many new opportunities. Consequently, the FuturICT project is about creating new information and communication systems to stimulate human imagination and responsible action by bringing together data, models, and people.


Q: What kinds of systems are you talking about?

A: New platforms such as a “Planetary Nervous System” to measure the state of the world, a Living Earth Simulator to study “What … if…” scenarios, and a Global Participatory Platform to support communication and collaboration, furthermore an Innovation Accelerator to address burning questions more efficiently and to create new business opportunities, and Exploratories to identify emerging risks and opportunities.


Q: Why do you talk about a Planetary Nervous System?

A: Because it will measure the state of the world, including social and economic activity patterns in real-time and to turn these data into useful information to create a picture of the world that makes sense.

Q: Isn’t it dangerous for society to collect so much data?
A: If it is done in the wrong way, yes. At the moment, many bad things are happening with our data: sensitive data is stolen and used in ways we would never agree to. It is important to find ways that allow one to do business, but protect individuals and society. FuturICT will develop privacy and value-sensitive ICT designs. It will also study how personalized search and recommender systems manipulate individuals and impact society.


Q: Wouldn't FuturICT itself create a Big Brother watching us?

A: No, in contrast to companies and secret services, FuturICT is not interested in tracking individuals. The goal is rather to create awareness for the implications of our decisions and actions, and to highlight problems and opportunities. The goal is not to collect all the data one can get, but to create devices that allow tailored measurements, e.g. of social capital and social impact.


Q: Can this lead to more sustainability?

A: Yes, this is the idea. Social well-being depends on many factors, not just economic productivity. Everyone knows today that our environment must be protected from destruction, but the same holds also for social capital such as solidarity, trust, and safety.

Q: Do you think this will make a difference?
A: Yes, absolutely. Current success indicators such as the global domestic product per capita and also common risk measures do not consider social capital. This is why social capital is exploited and damaged, and why too high risks are taken. In the end, this produces high societal costs.

Q: What is the Living Earth Simulator about? Do you want to make Isaac Asimov’s psychohistory come true?
A: The purpose of the Living Earth Simulator is not to predict the history, but to explore our future options. Considering interdependencies in our world, the Living Earth Simulator will study "what if scenarios" to give a better picture of our options and their possible side effects. Putting it differently, it serves to turn information into knowledge.

Q: How will it be possible to get data of everything that is happening in the world and feed it into a gigantic computer simulation?
A: FuturICT is not planning to create an exact model of the world. Scientific modelling is the art of approximation, i.e. the art to distinguish factors that matter from those which have little influence.


Q: What kinds of questions could the Living Earth Simulator answer?

A: As we are using supercomputers to design cars, planes, medical drugs, and almost everything, we would eventually get decision support for difficult economic, social, and political questions.


Q: How can this be done? Isn't the behaviour of humans completely unpredictable?

A: FuturICT does not aim at predicting the behaviour of individual people. The required models become less complicated when zooming out to focus on societal patterns rather than on individuals.

Q: But wouldn't this nevertheless require one to forecast the future?
A: FuturICT is not planning to do long-term forecasts, it is not futurology. One may imagine the principle more like weather forecasting, which works only short-term and is probabilistic rather than exact. However, using real-time measurements of a network of sensors and computational models, one can still give useful advice that generates much more value than one needs to invest.

Q: Isn't it much simpler to simulate the weather than to simulate society?
A: Indeed! This is why it took so long until computational social science took off. Models and simulation techniques for social and economic processes and complex systems are steadily improving. Powerful new empirical and experimental tools are developing quickly. This includes data mining in the internet, online polls, web experiments, and crowd sourcing techniques of all kinds.

Q: People react to forecasts while the weather doesn't. Wouldn't this make simulation results useless?
A: The response of people to information can be considered by opinion formation models. Online polls, news forums, blogs, tweets, prediction markets and search trends provide the data needed to apply them.


Q: But surprises are always possible!

A: Yes, society does not work like a clock. Take the Club of Rome's Limit to Growth study, for example. Its main merit was probably not to predict the future, but to create awareness. This has changed people's minds towards protecting our environment and saving our natural resources. Society is a complex system, and besides trends, one must consider spatial and network effects, diversity, history, context, and singular events. One book or idea can sometimes change everything!

Q: If randomness is important, why then are models useful?
A: Not everything is random. Models can still help us to be more successful on average. They can help us to better understand interdependencies in the system and reduce problems resulting from them. Short-term forecasts enable us to better adapt to the temporal evolution of the system. This can, for example, largely support the coordination between people, companies etc.


Q: Aren't our economy and our society too complex for a meaningful analysis?

A: To simulate fluid flows in a computer, we do not need to know the motion of all its molecules. Similarly, cultural habits, social conventions and norms, trends, laws, and contracts (serve to) make social and economic interactions more predictable. In fact, everyone is making plans. That's why people marry, study at the university, build homes, etc.


Q: But people also get divorced, or may not find a job after their studies.

A: Complex systems such as society are, in fact, characterized by a low degree of predictability, and they are hard to manage. This is because of their tendency to self-organize and develop new, so-called "emergent" properties.


Q: What kind of problems does this produce?

A: The strongly coupled, complex systems we have created them in the past are prone to cascading effects. These can cause extreme events. Examples are blackouts of electrical power systems, financial crises, political instabilities (as we have seen it in the Arab spring), conflict, or disease spreading.


Q: So, what do you want to do about it?

A: FuturICT is watching out for advance warning signs of systemic instabilities, which can cause large-scale crises. The goal is to reduce these and create more resilient systems, which can absorb shocks.


Q: Can you give an example?

A: Yes, the financial crisis. What started with a bubble in the US real estate market caused a global financial crash, and an economic and public spending crisis in many countries. Now, politicians are even worried that Europe may break apart.


Q: How could that have been avoided?

A: The financial architecture was lacking suitable firewalls to stop the spreading of the problem. One would have to engineer the system like our electric circuits, which have fuses to avoid that short circuits cause our house to burn down.


Q: That's all? Why do you need a Living Earth Simulator for this?

A: It would help to analyse more difficult problems. For example, what would happen, if Greece left the Eurozone? Or: What would be the implications of transaction fees in financial markets? When is short-selling good, when is it harmful? What is the impact of high-frequency trading? Should leverage effects be limited?


Q: So, would the computer tell decision-makers what to do and sooner or later control the world?

A: No, the Living Earth Simulator is not intended to replace human decisions. But FuturICT will create instruments to explore the implications of human decisions and actions from multiple perspectives, as we have created telescopes in the past. People will still choose between various options based on their respective goals, priorities and values.


Q: But wouldn't the Living Earth Simulator mainly help the rich and mighty?

A: No, because the FuturICT project is going to build a Global Participatory Platform, which shall democratize the use of data and models and make them accessible to non-experts. It serves to turn knowledge into wisdom and to overcome barriers to the participation in social, economic and political affairs.


Q: So, everyone will be able use it?

A: Yes, this is why we talk about participatory computing. The challenge, however, will be to design the system in a way that promotes responsible use. Transparency, accountability and the use of reputation systems will be important, here.


Q: And what about privacy? Isn’t it endangered when mining Big Data?

A: We are aware that today's handing of human activity data is potentially harmful to individuals and society, which needs to be changed. This is why FuturICT is investing a lot into ethical research. We want to develop privacy-respecting data mining techniques and value-sensitive information and communication systems. 


Q: Will FuturICT be good or bad for business?

A: We believe that value-sensitive design will be appreciated by customers. Moreover, FuturICT wants to create open platforms that will catalyse many new services and business. FuturICT's Innovation Accelerator wants to create an innovation ecosystem, which can spark off spin-offs and opportunities for everyone.


Q: FuturICT has many ambitious goals. How do you want to manage all this?

A: FuturICT can build on the leading academic powerhouses, many supercomputer centres, 1000 supporters in 4 continents, and multi-disciplinary communities in many countries. The vision of this project excites many people. Some even want to work for the project for free. We believe the idea is timely and convinces people. That is why it will move things ahead.


Q: What makes you so optimistic?

A: We receive an enormous support. FuturICT is backed up by the best academic institutions like Oxford, University College London, Imperial College, ETH Zurich, EPFL, CNR, CNRS, Inria, Fraunhofer, DFKI and many more. We have more than 100 letters of support by university heads. Similarly, we find great interest among industry representatives.


Q: You are talking about a Planetary Nervous System, Living Earth Simulator, and Global Participatory Platform. Isn't this project just too ambitious and destined to fail?

A: The project goals are very logical, and one needs to have a vision, in which direction to go. Technology is quickly progressing, and many ideas will be picked up and promoted by companies. Just remember how quickly information and communication systems evolve. Five years ago, we basically did not have Twitter, Facebook, and iPhones. What will happen in the next 10 years is hard to imagine.


Q: What will be the biggest challenge for FuturICT?

A: To change the common way of thinking about complex systems. Traditionally, we have a component-oriented view of the world. However, the behaviour of complex systems is determined by their interactions. An interaction-oriented view will therefore lead to a new understanding of our problems and novel solutions. These will be based on improved interactions, using real-time measurements and adaptive response. It’s almost like the paradigm shift from the geo-centric to the helio-centric world view, which made modern physics and most of our modern technologies possible. 

Wednesday, 8 February 2012

FuturICT Industry Alignment Events


The FuturICT flagship project will align the research of hundreds of the best scientists in Europe through a 10 year, €1 billion research programme to develop new methods which integrate different scientific models, data and concepts. It is worth noting that FuturICT will be the largest Big Data research project globally and is well positioned to play a very influential role in the development of academic research and commercial applications across the world.

FuturICT have scheduled two Industry Alignment events and are interested in engaging with suitable delegates from commercial organisations, industry associations, policy makers, regulators. Suitable delegates will be invited to attend one of the FuturICT Industry Alignment events that are scheduled for:

The goal of these events is for the FuturICT leaders (Prof. Dirk Helbing – Scientific Coordinator, Prof. Steven Bishop – Management Coordinator and Prof. Paul Lukowicz – ICT Coordinator) to meet and engage with representatives from industry who are interested in FuturICT activities and explore how FuturICT and industry can align their research agendas as well as discuss in-kind / cash or other contributions to support the research agenda. We already have attendees from several large technology and service companies as well as industry associations and others.

If you are interested in attending one of these events, please email jb.mccarthy@ucc.ie as soon as possible as space is limited. If help is needed to cover travel costs, please let JB know as we have some capacity to help with flight/hotel costs for delegates.

We look forward to seeing you there.
Regards,
JB McCarthy, FuturICT Dissemination and Stakeholder Liaison (jb.mccarthy@ucc.ie)
Dirk Helbing, FuturICT Scientific Coordinator, Zurich (dirk.helbing@gess.ethz.ch)
Steven Bishop, FuturICT Management Coordinator, London (s.bishop@ucl.ac.uk)