Tuesday, June 30, 2015

Strong Artificial Intelligence is Emerging as we Talk

The “return of Artificial Intelligence” is an impressive trend of the blogosphere. I spent quite some time and pleasure reading the two great posts from Tim Urban’s blog WaitButWhy entitled “The AI Revolution: The Road to Superintelligence” and “The AI Revolution : Immortality or Extinction”  (part 1 and part 2). The core of these posts is the difference between ANI (Artificial Narrow Intelligence, what used to be called weak AI), AGI (Artificial General Intelligence, or strong AI) and ASI (Artificial Superintelligence). These posts are strongly influenced by Ray Kurzweil and his numerous books, but do a great job of collecting and sorting out conflicting opinions. They show a consensus in favor of the emergence of AGI between 2040 and 2060. I strongly recommend reading these posts because they are entertaining, actually quite deep and offer a very good introduction to the concepts that I will develop later on. On the other hand, they miss the importance of perception, emotions and consciousness, which I will address in this post.

The diversity of opinions is striking. On the one hand, we have very enthusiastic advocates, such as Ray Kurzweil and his book “How to Create a Mind” which I will refer to later on, or Kevin Kelly with this great Wired article: “The Three Breakthroughs that have Finally Unleashed AI on the world”. For this camp, strong AI is feasible, it’s coming soon and it’s good. Obviously Larry Page is in that camp. On the other side, we find either people who simply don’t believe in the feasibility of strong AI, such as Gerard Berry, famous for saying that “computers are stupid”, or people who are very worried of what it could mean for mankind such as Stephen Hawking, Bill Gates or Elon Musk, to name a few. One of the reason this topic is so hot in the Web is that the investment race has started. Each major software company is investing massively into AI, as is explained by Kevin Kelly in his paper. IBM and Watson have started the race, while Google was acquiring massively companies in the fields of AI and robotics. Facebook has a massive AI program that has attracted a lot of attention. Kevin Kelly quotes Yahoo, Twitter, LinkedIn or Pinterest for having invested into AI capabilities recently. There is no debate about the tidal wave of ANI (weak form of AI), which is both depicted by Kevin Kelly or Tim Urban. It’s already here, it’s working quite well and it’s improving rapidly. The big race (and those who invest believe that there is a game changer ahead) is to get first to the next generation of Artificial Intelligence.

I decided to write my own opinion a couple of months ago, first because I got tired to hear the same old arguments about why strong AI was impossible, and also because I found while reading Tim Urban or Kevin Kelly (to name a few) that some of the key ingredients to make it happen were missing. For instance, there is too much emphasis on computing power, which is a key factor but is not enough, in my opinion, to produce AGI, even though I have read and appreciated Ray Kurzweil’s books. I must say that I have been away too far from computer science to qualify as an expert in any form. Let say that I am an educated amateur, because I started my career and my PhD in the fields of knowledge representation, rule-based systems and so-called expert systems. I have worked a long time ago on machine learning applied to algorithm generation and then more recently on intelligent agent learning with the GTES framework. There is some irony for me when writing these pages, since one of my first lecture when I was a student at the Ecole Normale Supérieure in 1984 was on the topic of AI replacing today’s workforce (to be compared with one of my posts on the same topic last year).
In this post I will explore four ideas, which seems to be, in my opinion, missing from what I have read during the past few months:
  • Speculating about AI algorithms today as a way to achieve strong AI is hazardous since these algorithms will be synthesized.
  • True intelligence requires senses, it requires to perceive and experience the world. This is one of the key lesson from biology in general and neuroscience in particular from the last decades, I do not see why computer AI would escape this fate.
  • A similar case may be made about the need for computer emotions. Contrary to what I have heard, artificial emotions are no more complex to embed than computer reasoning.
  • Self-consciousness may be hard to code, but it will likely emerge as a property of the next generation complex systems. We are not taking about giving a “soul to a computer” but letting free will and consciousness of oneself in relation to time and environment  become a key perceived feature of tomorrow smart autonomous systems, in the sense of the Turing test.

1. Artificial Intelligence is Grown, Not Designed

This is not a new idea. I have made Kevin Kelly’s book “Out of Control” a major reference for this blog. The central idea of his book is that in order to create really intelligent systems, you must relinquish control. This is true for weak and strong AI alike. What makes this idea more relevant today is the combined availability of massive computing power, massive storage and massive amounts of data. As I explained in my Big Data post where I quoted Thomas Hofmann, “Big Data is becoming at the core of computer science”, the new way of designing algorithms is to grow them from massive amounts of data. These new algorithms are usually “simple” (parts or whole are sub-linear) in order to absorb really massive amounts of data (peta-bytes today, much more tomorrow). One thing that we have learned from the past years is that simpler algorithms trained on really huge corpus of evidence do better than more complex algorithms trained on smaller samples. This has been shown in machine translation, grammar checking and other machine learning domains.

One of the key AI algorithmic technology of the moment is Convolutional Neural Networks (CNN) and the emphasis is on “deep learning”. CNN are a family of neural networks – trying to replicate the brain mechanism for leaning from layers of neurons – characterized by the control of back-propagating the information from the training set into the neuron network. For instance, you may read Mark Montgomery entry on “recent trends in artificial intelligence algorithms”. Deep Learning has received a lot of media attention thanks to the success of Deep Mind and his founder Demis Hassabis. Feedforward neural networks are a good example of systems that are grown, not designed.

If you read carefully about the best methods for speech recognition and language generation, you will see that you need more than CPU power and large training sets, you actually need lots of memory to keep that information “alive”. I borrow from Mark Montgomery the citation from Sepp Hochreiter because he makes a very important point: “The advent of Big Data together with advanced and parallel hardware architectures gave these old nets a boost such that they currently revolutionize speech and vision under the brand Deep Learning. In particular the “long short-term memory” (LSTM) network, developed by us 25 years ago, is now one of the most successful speech recognition and language generation methods”. Kevin Kelly attributes the “long awaited arrival of AI” to three factors : cheap parallel computation, big data and better algorithms. I obviously agree with those three, but his vision of “big data” as the availability of large training set is too narrow.

I also do not believe that the current algorithms of 2015 are indicative of what we will grow in 2040 when we have massively superior computing and storage capabilities. History shows that the mind follows the tool and that scientists adapt continuously to the new capacities of their tools. We are still in the infancy stage, because our computing capabilities are really very limited (more on this in the next section). Among the skeptics in the computer science community are people who think – and I need to agree – that being able to play old arcade games at a “genius level” is still very far from a true step towards AGI. The ability to devise and explore search and game strategies has been around for a long time in the AI community (i.e., playing a game without the rules). Many of the critics about the possibility of AI quote the difficulty to create, to produce art or to invent new concepts. Here I tend to think the opposite, based on the last decade of seeing computers used in music or mathematics (as a hint, I would like to quote Henri Poincaré : “Mathematics is the art of giving the same name to different things”). Creation is not difficult to express as a program, it is actually surprisingly easy and effective to write a program that explores a huge abstract space that represent new ideas, new images or new music. The hard part is obviously to recognize value in the creation, but computers are getting better at it.

2. A Truly Smart Artificial Intelligence Must Experience the World

The most common argument against strong AI and “true” natural language processing, when I was still close to the scientific AI community, was the “semantic problem”, that is, the difficulty to associate a meaning to words in a computer program. What we have learned in the last decades is that natural language cannot be understood through formal methods. Grammar, syntactical rules, lexicography cannot help you much without a “semantic reference" which is necessary to understand, even to disambiguate, many sentences that make our everyday life. Somehow, one needs a phenomenology foundation to understand humans and to be able to discuss convincingly.

The true revolution that is happening gradually is that the Web may be used as this “phenomenology foundation”. This was explained to me many years ago by Claude Kirchner during a talk at the NATF : if you are a computer and need to think “from experience” about a dog, why not used the network of millions of documents returned by a Google search with the query “dog” as the phenomenology reference ? It requires massive amounts of computing and storage, but it is more and more feasible. In all its richness, diversity and links with other experiences, this cloud of documents (text / image / video / ..) makes a solid foundation to answer  common-knowledge-questions about dogs. This is a departure from previous approaches where the huge amount of sources available on the web is used to produce “abstractions” (concepts that are represented by bit-vectors produced by techniques such as Latent Semantic Indexing from my departed friend Thomas Landauer). The idea here is to keep the whole network of document in memory as a substitute for experiencing a dog. I am a little heavy here –one could say that it is lazy deep learning -, because it is a key point when one wants to understand when we may get strong AI widely available : it is not the same thing to have the whole set of documents stored in your computer brain or to build a model through training. This is, to me, a key point since we have learned from other scientists that it is very hard to separate perception and thinking, as it is hard to separate body and mind. An obvious reference that comes to mind is Alain Berthoz and his work on sight (for instance, you may read his book on decision).

As a first hint that having access to huge amount of data builds the capability to understand texts, we have started to see significant progress in natural language processing (NLP) and we are bound to see much more when more storage and more processing power become available. NLP is one of the key priority for the Facebook AI program that I mentioned earlier. It is also a key priority for Google, Apple and many, many others. There are already a number of exciting signs that we are making progress. For instance, computers can now play with words games, such as the one that make IQ tests, better than most humans. This is not yet an example of keeping all “experience knowledge in memory”, but a sign that deep learning applied to massive amount of data can work pretty well. Another sign that race towards NLP is raging is the apparition of services that are mostly based on answering questions. The obvious reference here is IBM Watson, but there are many other innovative services that are popping up, such as texting services on top of WeChat. Many of these texting/concierge services are using a hybrid of human/robot assistance, waiting for technology to become fully sufficient. I also hear a lot of frustration in my close circle about the shortcomings of Google translate or Apple Siri, but the progress rate is very impressive. If you are not convinced, read this fascinating article about IBM Watson’s training.  During a lecture which I attended last month, Andrew McAffee used the graph (Figure 9) where you see the level of coverage/precision reached by Watson versions after versions, as a great illustration of the power of exponential technology growth.

This being said, one of the reason I am emphasizing the need for memory is that the slowing down of DRAM capacity increase may happen faster than the suspected decline of Moore’s Law. It turns out that there are many ways to continue increasing the processing power, even if speed is closed to its limit and if integration (reducing the transistor dimension) is also, in its two dimension version, not so far from hitting hard limits. On the other hand, DRAM performance seems to progress slower and with fewer routes to continue its growth. You may take a look at the table or the following chart to see that computer memory is progressing slower than processors, who are progressing slower than disks (this last part is very well explained in “The Innovator Dilemma”). Another way to look at it is as follows. I have been waiting of 1 Pb (peta-byte) of memory on my PC for many years … in the early 90s, I had a couple of megabytes, today I have a couple of gigabytes. Even at the previous CAGR of 35%, it may take 50 years to get there, which is why I am more with the group of thinkers who predict of AGI occurrence in 2060, compared to the optimistic group (2040). On the one hand, you could say that asking for one Pb is asking a lot (there are many ways to get this number, mine was simply 100K experiences time 10 Gbyte of real-life data), but clearly considering that memory will continue to grow at the same rate is too optimistic.

Linking a computer to a very large set of “experiences” in one step, the next approach is to build autonomous robots with their own senses. I often talk about the robotic arm from the University of Tokyo which is about to catch an egg that is launched towards it at full speed, and which is also able to play baseball with the accuracy of a professional player. The reason for this engineering feast is not an incredible algorithm, it is the incredible speed at which the robot sees the world, at 50 thousands images per second. At that speed, the ball or the egg moves very slowly and the control algorithm for the arm has a much easier job to perform. Because of the importance of senses, experiences and perception, it may be the case that we see faster progress from autonomous robots than cloud AI as far as reaching AGI is concerned. One could say that the best way to train an artificial intelligence is to let it learn by doing, by acting and exploring with a full feedback circle (which is precisely what happens with the DeepMind arcade games experiments). This may mean that autonomous robots, which will clearly be fitted with exceptional perception senses – one may think of Google autonomous car as an example – will be in the best situation to grow an emergent strong form of artificial intelligence.

3. Learning and Decisions Require Emotions

To continue on what we can learn from biology and neurosciences, it seems clear that computers needs to balance different types of thinking to reach decisions on a large range of topics, in a way which will appear « intelligent » to us humans. A lot of my thinking for this section has been influenced by Michio Kaku’s book “The Future of the Mind”, but many other references could be quoted here, starting from Damasio’s bestseller “Descartes’ error”. The key insight from neuroscience is that we need both rational thinking from the cortex and emotional thinking to take decisions. Emotions seem mostly triggered by “pattern-recognition” low level circuitry of the brain and the nervous system. This distinction is also related to the system 1 / system 2 description of Kahneman. We seem to be designed to mix inductive and deductive logic.

Michio Kaku has a very elegant way of looking at the role of emotions in the process of thinking. Emotions are a “cost / evaluation” function that is hard-wired (through DNA) and has evolved slowly through evolution (through DNA), to play two key roles. On the one hand, emotions are a valuation function that is used as a meta-strategy to search and to learn when we us the deductive, rational way thinking. For people trained in optimization problems, emotions define the first level of the “objective function”. However, as evolved creatures, we build our own goals, our own desires and our own cost functions for new situations, that is, how we value new experiences. The second role of emotions is to be the foundation (one could say, the anchors) for the cost function that we grow through experience.

This is closely related to a key cycle in biology which we could call the “learning cycle for living beings”: pleasure leads to desire, desire to planning, planning to action, actions lead to experiencing emotions, such as pleasure, fear, pain, etc. I heard about this cycle a few years ago while attending a complex systems conference. It seems to describe the learning loop for a large set of living beings, from very simple to us humans. Emotions, both positive such as pleasure and negative such as fear, play a key role in this cycle, from evaluating situations and formulating plans. We can see that a similar design is relevant to the goal of generating strong artificial intelligence.  It is clear that a truly smart system must be able to generate its own goals, which is actually easy, as explained earlier. Simulating “free will” from randomness is a simple task (very debatable from a philosophy standpoint but efficient from a pragmatic one). However, intelligence in goal generation requires to use an objective function that may evolved as the smart system is learning. Computer emotions may be used as seeds (anchors) of this objective function. For Michio Kaku, emotions are case-based heuristics that have been finely tuned through Darwinian evolution to make us a more adaptive species. Mixing emotions and reasoning is not really a new concept in AI.  It is a way of mixing case-based reasoning, in a “compiled form” that has been learned previously by previous generation of software instances with logic deductive reasoning that is “interpreted” and unique to each instance. This is clearly a multi-agent model (system1 vs sysytem2) that reminds us of “The Society of Mind” proposed by Marvin Minsky in 1986.

A great illustration of this idea proposed by Mikio Kaku is the sense of humor, which may be described as our ability to appreciate the difference between what we expect (the outcome of our own world model simulation) and what happens. This is how magic tricks and jokes work. Because we value this difference, we are playful creatures: we love to explore, to be surprised, to play game. Kaku makes a convincing argument that the sense of humor is a  key evolution trait that favors our learning ability as a living species. It is also very natural to think that smart AIs, with a similar ability to plan ahead and simulate constantly what they expect to happen, should be given a similar “sense of humor” (e.g., affinity for the unexpected) as a search “meta-strategy”. This remark also brings back to the need for “emotions” to avoid danger (i.e., how we learn not to play with fire). Kaku also sees the use of free will, in the sense of exploiting some form of randomness – with the same debate whether it is “true” freedom or a trick to use some form of biological pseudo-random generator -, as a meta-strategy evolved as a Darwinian advantage for species competition. He takes the hare as an example, which needed to develop random paths to avoid the fox. But a more general case can be made from game theory where we know that mixed strategies (that combine some form of choice or “free will”) fare better in a competition that pure (deterministic) strategies. A similar and more technical point could be made about the use of randomization in search algorithms, which has been proven in the past decade to be an effective meta-strategy.

I strongly recommend reading Michio Kaku’s book, which has a much larger scope than what is discussed here. For instance, the pages about experiments at Berkeley to read thoughts are very interesting. His insights about the role of emotions are quite fascinating, and make a nice complement to Kurweil’s book which I’ll discuss in the next section. To summarize and conclude this section, designing computer emotions is probably the best way to introduce some form of control into an emergent reasoning autonomous system. Emotions are both a bootstrap and a scaffolding mechanism for growing free will. They constitute our first level of objective function, hard-wired together with the more primitive senses signals such as pain. As we learn to derive more complex goals, plans and ambitions, emotions are a control mechanism to keep the new objective function within stable bounds. Emotions are somehow a simpler information processing mechanism than the cortex deductive thinking (which is why they work faster in our bodies) and they evolve at the species level, much more than the individual level (we learn to control them, not to change them). This makes computer emotions a mechanism that is far easier to control than emerging intelligence. My intuition is that this will become a key area for autonomous smart robots.

4. Consciousness is an Emerging Property of Complex Thinking Systems

Another classical argument of skeptics about the possibility of strong AI is that computers, contrary to humans, will never be aware of their thinking, therefore not consciously aware of their actions. I disagree with this statement since I think that consciousness will emerge gradually as we build more complex AI systems with deeper reasoning and perceiving abilities (cf. Section 2: perceiving is as important as reasoning). I am aware (pun intended) that there are many ways to understand this statement and that the precise definition is where the hot debate stays. Here, my own thinking has been influenced by Ray Kurzweil’s book “How to create a mind”. Even if I do not subscribe with the complete story (i.e., that everything you need to create a mind is explained in this book), I found this book a great read for two reasons: it contains a lot of insights and substance about the story of NLP and AI, and it proposes a model for conscious reasoning which is both practical and convincing. As you may have guessed, my main concern with the approach proposed by Kurzweil is the weak role played by senses and emotions is his mind design.

What I envision is a progressive path towards consciousness:

  • Self versus environment: the robot, or autonomous AI, is able to understand its environment, to see and recognize itself as part of the world (the famous “mirror test”).
  • Awareness of thoughts:  the robot can tell what it’s doing, why and how – it can explain its processing/ reasoning steps
  • Time awareness : the robot can think about its past, its present and its future. It is able to formulate scenarios, to define goals and to learn from what actually happens compared to its prediction
  • Choice consciousness: the robot is aware of its capability to make choices and creates a narrative  (about its goals, its aspirations, its emotions and its experiences) that is a foundation for these choices. “Narrative” (story) is a vague term, which I use to encompass deductive/inductive/causal reasoning.

Although I see a progression, this is not a step-by-step hierarchy. It is an embedded set of capabilities that emerge when sensing, modeling and reasoning skills grow. Emergence of consciousness is a key element of Kurzweil’s book, as shown by this quote: “My own view, which is perhaps a subschool of panprotopsychism, is that consciousness is an emergent property of a complex physical system. In this view the dog is also conscious but somewhat less than a human”. The emergent characteristic also implies that it is difficult to characterize, and even more difficult to understand how it comes to be. However, once an AI has reached the four levels of conscious abilities that I just described, it is able to talk to us about self-awareness in a very convincing manner.  One could object that this is a narrow, practical definition of consciousness, but I would say that it is the one that matters practically, for strong AI and autonomous robot applications. I will not touch in this post the key question of knowing if human consciousness is of the same nature (an emerging property of ourselves as a complex system, an essentially different characteristic of our species, or an attribute of our immortal soul). One of the hot questions about strong AI is the “hard problem of consciousness” defined by David Chambers. The “easy problems of consciousness” are self-awareness capabilities that Chambers and many others see as easily accessible to robots. “Hard problems” qualifies reflective thoughts about one’s experiences that seem harder to capture with a computer program/ Without trying to answer this hard question, it is clear to me that consciousness requires experience, hence the emphasis I have put on senses, perceptions and emotions. I also believe that, given sufficient complexity, sensing and reasoning capabilities, emergence may grow “artificial consciences” that will come close the “hard level of consciousness”. It is also clear that this will open a number of ethical issues about what we can and cannot do when we experiment with this type of strong AI program. For lack of time, I refer you to James Hughes’s book “Citizen Cyborg”, where the rights of emerging conscious beings are discussed.

5. Concluding Thoughts

There is much more that needs to be said, especially on the philosophical level about consciousness and the political level about the societal risks. So I will not risk a “conclusion”, I will conclude with a few thoughts. My previous post on this topic is almost 10 years old, but I have a keen intuition that many will follow sooner than 2025 :)

  • First, it is clear now that weak AI, or ANI, is already there in our lives, and has been progressing for the last twenty years making these lives easier. The two articles from Tim Urban and Kevin Kelly that I mentioned in that post give a detailed account with plenty of evidence. I can also point out James Haight post “What’s next for artificial intelligence in the enterprise?”. Kevin Kelly emphasizes the advent of “AI as a service”, delivered from the cloud by a small set of world leaders. I think he has a fair point, there is clearly a first move/scale advantage that will favor IBM, Google and a few other large players.
  • However, there are more opportunities than “smart thinking in the cloud”, (weak) AI is everywhere and will continue to be ubiquitous. Machine learning is already here in our smartphone and the next decades of Moore’s Law mean that connected objects and smart devices will be really smart.
  • The race towards strong (or at least stronger) AI is on, as illustrated by the massive investments made by large players in that field. The next target is NLP (natural language processing) which is within our reach because of the exponential progresses of computing power, big data (storage capacity and availably of data) and deep learning algorithm.
  • This is very disruptive topic.  I do not agree with Kelly’s optimistic vision in his paper, nor with Ray Kurzweil. The disruption will start much earlier than the advent of the strong AI stage.  For instance, the tidal wave of ANI may cause such a havoc as to make AGI impossible for decades. This could be either for ethical reasons (laws slowing down the access to AGI resources because of the concerns with what “weak” AI will be already able to do in a decade) or for political reasons (the turmoil created by massive jobs destructions due to automatization).
  • Emotion and senses are part of the roadmap towards strong AI (AGI). Today’s focus is on cortex simulation as a model for future AI, but everything, from cognitive science to biology, suggests that it’s the complete nervous system from brain to body that will teach us how to grow efficient autonomous thinking. This is actually easier to state in a negative form: AI designed without emotions, through a narrow focus on growing cognitive and deductive thinking by emergent learning will most probably be less effective than a more balanced “society of minds” and almost certainly very hard to control.
  • Consciousness will emerge along the way towards strong AI. It will happen faster than we think, but it will be more progressive (dog-level, child-level, adult-level, god-knows-what-level, …). Strong AI will not grow “in a box”, it will grow from constant and open interactions with a vast environment.

Monday, March 23, 2015

Internet of Things and Smart Systems


Last month I had the pleasure to lead two roundtables about connected objects during an event jointly organized by Cité des Sciences and the National Academy of Technologies entitled “Connected Objects : a third digital revolution”. The first roundtable gave me the opportunity to discuss about the systemic vision that is necessary to understand and design connected objects, together with a very interesting list of companies:  Terraillon, Medissimo, Connected Cycle, Withings, Kolibree. This post proposes a summary of some of these ideas, with a focus on understanding what make connected objects smart and useful. This is not a summary of the complete discussion; it is filtered by my bias towards system analysis, as is expressed in a previous post from this blog. In a way, this is a follow-up to the talk that I gave at the 2013 Amsterdam conference on Smart Home, which I made available on Slideshare. It is also influenced by discussions at the IRT SystemX, the French technology research institute on “Systems of Systems”.

I will follow a simple outline:
  • First I will take a look at “connected objects” seen as objects with the additional benefits of being connected. My opening statement during the roundtable was that we should not care about this from a technology standpoint (what is added to the object) but focus on the customer experience. This may sound like a boring and self-evident statement, but my experience shows that (1) there is too much excitement about the technology (2) not enough effort with experience design, and that we are bound to see more disappointment with connected objects.
  • The second section will look at Internet of things (IoT) as a component of the Web Squared vision. With this second level of system analysis, connected objects interact with the complete Web in both directions, they bring additional senses (through sensors) to the Web and become contact / interaction points (bringing the Web “closer” to the user, what Joel de Rosnay calls the “clickable environment”). Obviously, the burning engine of this system is the value that we can bring though Big Data, a point that was abundantly made clear during our roundtable discussion and which is the topic of many books. However, a similar observation may be made that data follows usage and that focusing on customer experience first is the best way to avoid technology mirages.
  • The last section of this post will focus on “smart objects”, as defined by Olivier Ezratty – who was also a participant to the roundtable – as “objects that talk to one another”. There is a consensus that “we are not there yet” in the sense that experiences that are built on many connected objects interacting with each  other are rare, tend to involve objects from the same manufacturer, and to follow a closed set of possible interactions. “Smart objects” correspond to a more advanced “connected object ecosystem”, which is necessary to unlock more value to the end user.

1.   Connected Objects Seen as Objects

At a first glance, or at a first systemic level of analysis, a connected object gets two additional benefits from its connection:
  • Access to a deported interface, which is most often the smartphone. The tremendous capabilities of the smartphone (quality of the screen, touch interface, motion sensors, etc.), together with its ubiquity, make it a wonderful user interface for most scenarios.
  • Access to large storage and computing capabilities. This is true for computing, in the sense that one may do more on a smartphone or in the cloud that on the embedded chipset from the connected object. This is even more important for storage, since external storage gives both better capacity and durability. Connected objects have “memory”, as exemplified by the Withing connected scale.
We will look at the deeper benefits later on, but I would maintain that the large majority of connected devices that are available today are mostly selling the benefit of being controlled by the smartphone with some form of extended memory. That is, the additional benefit of true Web, Big Data, Smart System integration are either not there, not developed enough to be useful, or they are readily available in other ways.

My experience, especially in the realm of smart Homes where I have been quite active with my previous job, is that remote interaction with the smartphone does not carry enough weight to last long. One needs to add systemic value to the connected experiences, based on scenarios, context, and personalization to keep the interest going. Otherwise, once the novelty wears off, you are left with the underlying object: if it is a great, useful object in itself, you keep using it, otherwise you simply discard it. This is precisely the content of what I said during the Amsterdam Smart Home talk, with a focus on “life moments”, which are the daily events that may trigger scenarios.

The business model of such connected objects is, most often, to sell the object with a premium that corresponds to the improved service. The challenge is twofolds: (a) to deliver enough recurring value (b) to reduce the ownership cost/burden of the “connectedness”. The keyword “recurring” is important: it is easy to sell a connected object with the combination of aesthetic design and technology excitement, but if there is not recurring value, the business model is not sustainable. The ownership costs grows over time, with issues such as battery replacement / charging, network setup and update (the first wifi/Bluetooth pairing is easier because of the excitement of the recent purchase, the later ones, once the box and user guide card is gone, is trickier). Genevieve Bell, the anthropologist from Intel, explains this very well: each connected object is begging for attention, from its notifications to its battery charge, creating a true “cost” of ownership.

The Nurun design agency has a nice way of formulating the value challenge:  the value of the connected object, defined as immediate (out of the box), aggregate (over time) and emerging (such as social benefits) must outweigh the replacement cost. This is the only way to generate a sustainable business. As of today, and if the systemic value that we will discuss in the next sections is not there, there are few objects that will pass this test, and their success comes more from their intrinsic value (objects that prove useful and pleasant to use irrespectively of their connection) than their “connected” status.
To win this difficult equation, the technology battle is the excellence with service delivery (especially the app on the smartphone). Companies need to develop state-of-the-art skills with respect to digital software. This makes perfect sense since this is a “service business model”: delighting the customer with superior services associated with the object. This is where the “Web Giants’ lessons”, together with Lean Startup principles are relevant.

Giving memory and control to objects require sending data to the service provider, which prompts the issue of respecting data privacy. Our second roundtable was dedicated, among other things, to the issue of data privacy. It would require a different blog post to deal with this issue, but I would like to point out that, as of now, the absence of real customer value from her/his collected data is a bigger issue than the respect of data privacy, which is a great way to introduce the next section. Waze is a great example to remind us that applications which generate true user value gets access (consent) to lots of personal data.

2.       Internet of Things and Web Squared

If we pursue our thinking, and look at a more global system analysis, the connected object may benefit from two additional features:

  • It may send information to the cloud, using it sensors. The object is a mobile / wearable / personalized data capture device, which feeds “global services”, hosted on the Web. This corresponds to the “IoT as the senses of the Web” metaphor.
  • It may receive information from the cloud, and act upon them. The animated object (from a simple display – where the connected device acts as a more ergonomic and more convenient device than, say, the smartphone – to a complete animated behavior change – for instance a small robot whose interaction embodies a web service).

These two sides are the two parts of the “web square” vision: the connected object becomes part of something bigger. The object is not enough; it’s the digital experience that matters. The value does not come solely from what is in the object (sensors & actuators included), it comes in a large part from what is in the cloud. If you are not familiar with the “Web Squared” concept, the move “Eagle Eye” is a great and entertaining illustration.

Because of its very large scope, it is difficult to categorize the different business models that such a system approach enables. Without completeness, we can list:
  • Personalization and contextualization of a digital experience: the connected object is used to enrich a positive experience that is sold as a service. This leverages the first direction: object to cloud. Ambient digital experiences rely on connected objects.
  • Reciprocately, connected objects may better the digital experience though improved usability. In his book “Enchanted objects: design, human desire and the internet of things”, David Rose writes about glaneability, gesturability and wearability, which are three compelling reasons to use connected objects as a way to interact with the Web.
  • The complete B2B scope of connected objects, which is huge and outside the scope of this blog post, fits nicely into this object-to-cloud system view. Although the B2B model has its own logic, there are clear dependencies and cross-opportunities. For instance, metering devices, which are introduced with their separate business logic, may become part of customer digital experiences.
  • IoT/Cloud integration is a way to improve efficiency of business processes (I will not qualify them as digital, most business processes nowadays are digital in some form). This efficiency improvement may translate into cost reduction that may be passed to the consumer. Improving efficiency may come from cost reduction, (cf. metering example), risk avoidance or better characterization (insurance),  or performance improvement (cf. automated logistics). The “pure case” (when the value created by the efficiency gain is larger than the connected device) is uninteresting (objects become connected and the cost of their “status upgrade” is not seen by the customer) but common (more and more objects will be built with “connected object capabilities” – think of your printer’s ink cartridges).  The hybrid case (where the created efficiency value is not enough, but part of the business case for the connected object) is the most interesting one.

If we get back to our topic of “consumer connected objects”, the common umbrella for most of the digital experience that may be enriched in a systemic way through connected object is the “assistance” experience, from immediate to long-term assistance and coaching, from Siri or Google Now to healthcare services. Assistance requires context (personal data), knowledge (big data), personalization (machine learning) and problem solving (automated reasoning). The consensus during our roundtable discussion was that there needs to be a fair amount of each of these capabilities to provide a cloud assistance or coaching service that can bring sustainable recurrent value to the user. Most wearable devices, intended to be a piece of a wellbeing / health care digital experience, are still very far from meeting this goal, as can be seen by the mediocre smartphone applications and the fast decline in usage rate once the initial excitement wears off. The buzz about “quantified self” gives a good illustration of this point. Three years ago, I conjectured that quantified self would fit perfectly a combination of narcissists, geeks and people who enjoyed “system thinking” (if seeing a curve with the evolution of your bio-data tickles you, you’re in). This is a fairly large intersection – I fit just in there J – but it is still a niche. For most people, the “quantified self” experience left to oneself – which is exactly what I enjoy about my Withing connected scale – is too abstract and must be replaced by a true coaching experience.

This leads to two major challenges:
  • To generate knowledge from connected objects, one needs smart algorithms and data, and a little bit of time. To get data, one needs both usage and consent from the user, which both requires to deliver actual value. There is a chicken-and-egg conundrum: one needs data to deliver value, and value to attract the data. In most cases, the solution is to follow a two-step process, where the value generated by Big Data analysis comes in a second step, with a different value model to bootstrap usage.
  • The combination of information and technology required to produce a viable assistance or coaching service is accessible to large & technology-focused companies such as Apple or Google, but difficult to reach for most companies. This means that a partnership is required between different players, one of which is the connected object manufacturer. The object is not the center of the business value, what matters is reaching the critical mass of content (what to say), contextualization (when to say it), personalization (how to say it) and knowledge (why to say it). It is possible to add human interaction into the system loop, as a way to add knowledge or reasoning, which brings the digital experience closer to traditional coaching, but this usually comes at a cost. Still, there is likely a sweet spot for hybrid assistance or coaching services that mix the benefits of smart technology and human care.

The technology challenges associated with this systemic level are different from the previous one from Section 1. The first key domain is API (Application Programming Interfaces) architecture, used both for data collection and ecosystem partnerships. The importance of API comes from the second observation that partnerships will be required. The second domain is obviously data mining, as in Big Data, which must be either a strong point of the aspiring connected object contender, or one of its preferred partners. As stated earlier, most of the connected devices that have been introduced so far, especially in the field of connected health, fall short on delivering value with their data mining abilities.

There is a related challenge to this system vision, which is the social acceptability of “smart” cloud-based coaching services. The question of the “social acceptability” of artificial intelligence is also worth a separate discussion which I will address in my next post. During a presentation at the NATF, Dominique Cardon explained to us that the word “algorithm”, which had a very positive image a few decades ago as a symbol of technological progress, is now seen with more suspicion by the average citizen. “Smart” systems are under scrutiny, whereas social systems, where recommendation comes from the community, have a clear trust advantage.

3. Smart Objects Ecosystems

The ultimate goal of smart objects is not to improve, enrich or extend existing experiences, it is to create brand new ones. This will happen once smart objects start to act as a smart system, which supposes that they interact with one another. To build truly exciting and new experiences supposes an adaptive and intelligent behavior, obtained from distributed control and autonomous smart communication between objects. Today, the “state of the art” for mashing-up connected objects and web services seems to be IFTTT, which is deterministic, rule-based and centralized.

This long-term goal of what would be described as a smart object ecosystem is close to the general topic of this blog, in the spirit of Kevin Kelly organic/grown distributed systems. I have already mentioned the ideal of “calm computing” in a previous post. The three key principles of calm ubiquitous computing are the invisibility of the “smart dimension” (computing) – which does not mean invisible objects – the ability to stop down the assistance and the implicit machine learning – the smart object ecosystem must learn from the user, not the other way around. I often quote Adhoco, a Swiss smart home solution provider, as an example of existing smart objects service that learns from its users and operates under calm computing principles.
Another direct application of biomimicry is how to design a fail-safe smart system. Not only the control should not be centralized but distributed across the system elements, following complex systems design principles, in addition smart functions and devices must operate as assistance to lower level functions, and not the opposite. Following a simple principle that says that the more sophisticated the device, the more likely it will experience failure, control must be designed so that lower level automation may function independently from the more advanced “smart” or “adaptive” component.

A traditional failure of advanced smart home systems of the previous decade was the introduction of complex computerized systems to control low-level but vital automatic functions (such as light control or blind opening). The inevitable computer problems would then translate into a dysfunctional house, which is anything but smart. The appropriate modern design is both to avoid single-points-of-failure, and to make sure that high-level control logics are introduced as fail-safe “adds-on” to lower level of system logic.

The business model of smart object ecosystem is to provide with a new experience, or with a new level of satisfaction linked to an existing digital experience. The conviction that I expressed during this 2013 Amsterdam Smart Home interview is that it requires an “operator”, that is a company that takes responsibility for installing and maintaining the smart system. The alternative approach is to believe in interoperable standards and let the customer be the integrator of his own system, from her home to her health digital environment. It follows from the low level of maturity of “smart object interaction”, which we all agreed on during the roundtable, that letting the user be the integrator is a risky business scenario. This DIY (Do It Yourself) approach has two drawbacks, which I have seen firsthand with smart home experiments: first the smart object setup is a barrier to most consumers, but a few passionate geeks; second, the simplicity required by DIY installation prevents from delivering a true “smart system” behavior, which keeps the experience within the range of what is achieved with Section 1 and Section 2 approaches. This means that the value associated with the “target ideal experience” (of a smart home, for instance) is simply not there, which prevents raising a sustainable fee for the new service. The business challenge is to setup a brand with a clear promise, a distribution and installation network and the service maintenance infrastructure. I see the move of US telcos, such as AT&T or Verizon, towards smart home operators, as a significant signal. 

The technology challenges associated with this “third level of system integration” for connected objects are precisely linked to system integration. The skills and the software challenges are also different from what we saw in Section 1 and 2. Open system design, fault-tolerant architecture and machine learning are three key components of what is necessary to build a smart object ecosystem. Although digital and Big Data skills are bound to play an important role (level 3 encompasses levels 1 & 2), this is foremost a “system of system” integration game. Usability, availability, reliability, maintainability and adaptability are the key design challenges for smart object ecosystems.


On the one hand, I am expecting a lot of disappointment with respect to connected devices in the years to come because I believe that many of the barriers identified in this post still have to be removed. A very crude summary would be to say that connected objects are not enough, and that we are still far from the promise of smart “assistance” through connected objects, because there is a critical mass of data, learning and service innovation that requires more time and energy that what has been spent so far. On the other hand, I am a big believer in connected object because I think that the best is yet to come. This is just the beginning, not only will technology improve in all directions (miniaturization, better performance, new interaction channels, much better speech, image and pattern recognition, etc.) but time will make the conundrum identified in Section 2 become a virtuous cycle. The more time passes, the more we accumulate data which enriches the value that may be delivered through connected objects.

I believe that the distinction made here between three business models for mass market connected objects is useful, because it emphasizes different challenges and different expectations from the user. These three dimensions are not exclusive, many connected object strategies are bound to be a combination, but it matters to focus on the true benefits brought to the consumers. Here is a last attempt to summarize these three models:
  • The “service” business model uses objects to deliver improved services, with a key contribution of digital software excellence to the connected object experience.
  • The “efficiency” model uses objects to deliver better efficiency through big data, with a technology focus on API architecture and data mining.
  • The “new business” model provides a new experience that is made possible by a system of connected objects, where the technical challenge is the combination of usability, resilience and machine learning, from a “system of systems” point of view.

Friday, October 24, 2014

Big Data hides more than one paradigm shift

This post originates from a report which I wrote this summer for the NATF, which proposes a summary of what the ICT commission learned from its two-years cycle of interviews about Big Data. The commission decided to investigate about the impact of Big Data on French economy in 2012. Big Data is such a popular and hyped topic that it was not clear, at first, if a report would be necessary. So many books and reports have been published in the past two years (see the extract from the bibliography in the next section) that it made little sense to add a new one. However, throughout our interviews, and thanks to the FoE conference that NATF co-organized last year with NAE in Chantilly – which included Big Data as one of its four topics – we came to think that there was more to say about the topic than the usual piece about new customer insights, data scientists, the internet of things and the opportunities for new services.

Today I will focus on two ideas that may be characterized as paradigm shifts. I will keep this post short so it may be seen as a “teaser” for the full report which should be available soon. The first paradigm shift is a new way to analyze data based on systemic cycles and the real-time analysis of correlation. The old adage “correlation is not causation” is made “obsolete” because data mining is embedded into an operational loop that is judged, not by the amount of knowledge that is extracted, but by the dollar amount of new business that is generated. The second paradigm shift is about programming: Big data entails a new way to produce code in a massively distributed environment. This disruption comes from two fronts: the massive volume of data requires to distribute both data and procedure, on the one hand, and algorithmic tuning needs to be automated, on the other hand. Algorithms are grown as much as they are designed, they are derived from data though machine learning.
The full report contains an overview of what Big Data is because a report from NATF needs to be self-contained, but this is not necessary for a blog post. I assume that the reader has some familiarity with the Big Data topic. Otherwise, the Wikipedia page is a good place to start, followed by the upcoming bibliography entries, first of which Viktor Mayer-Schönberger and Kenneth Cukier’s book.

1. Big Data – A Revolution That Will Transform How We Live

This is the title of the book from Viktor Mayer-Schönberger and Kenneth Cukier, which covers the most famous paradigm shift of Big Data, which is its ability to transform our lives, from hard science, such as medicine, to marketing. The paradigm shift comes from the combination of what technology makes possible today – the ability to analyze very large amount of heterogeneous data in a very short amount of time – and the availability of the relevant data, which are the traces of our lives that have become digital. Thanks to the web, to smartphones, to technology which is everywhere in our lives and objects, there is a continuous stream of information that describes the world and our actions. Big Data may be described as the Information Technology which is able to mine this “digital logs” and produce new insights, opportunities and services. The constant improvement of technology (from Moore’s Law about processing to Kryder’s Law about storage) is matched by the increase in digital details about our lives. New connected objects, sensors and the growth of IoT (Internet of Things) mean that we are only seeing the beginning of what Big Data will be able to do in the future. 
One of the reason for not discussing these themes further is that the book from Viktor Mayer-Schönberger and Kenneth Cukier covers them very well so that I encourage you to read it. The other reason is that there are many other sources that develop these theses. Here is a short extract from our report’s bibliography :
[1] Commission Anne Lauvergeon. Un principe et sept ambitions pour l’innovation. 2013.
[2] John Podesta & al. Big Data : Seizing Opportunities, preserving values. Executive Office of the President, May 2014.
[3] François Bourdoncle. Peut-on créer un écosystème français du Big Data ?, Le Journal de l’Ecole de Paris n°108, Juillet/Aout 2014.
[5] Viktor Mayer-Schönberger, Kenneth Cukier. Big Data – A Revolution That Will Transform How We Live, Work and Think. John Murray, 2013.
[8] Gilles Babinet. L’ère numérique, un nouvel âge de l’humanité : Cinq mutations qui vont bouleverser votre vie. Le Passeur, 2014.
[10] Phil Simon. The Age of The Platform – How Amazon, Apple, Facebook and Google have redefined business. Motion Publishing, 2011.
[12] IBM Global Business Services, « Analytics : Real-world use of big data in telecommunications – How innovative communication service providers are extracting value from uncertain data”. IBM Institute for Business Value, Avril 2013.
[13] Thomas Dapp. “Big Data – The untamed force”, Deutsche Bank Research, May 5, 2014.
[15] David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani. “The Parable of Google Flu: Traps in Big Data Analysis
[16] Tim Harford. “Big data: are we making a big mistake?”, Financial Times,  March 28th, 2014. 
[19] Octo Technology. Les géants du Web : Culture – Pratiques - Architecture. Octo 2012.
[21] Tony Hey, Stewart Tansley, Kristin Tolle (eds). The Fourth Paradigm – Data-Intensive Scientific Discovery. Microsoft Research, 2009.
[22] Max Lin. “Machine Learning on Big Data – Lessons Learned from Google Projects”.
[24] Michael Kopp. “Top Performance Problems discussed at the Hadoop and Cassandra Summits”, July 17, 2013.
[25] Eddy Satterly. « Big Data Architecture Patterns ».
[26] Paul Ohm. “Broken Promises of Privacy: Responding to the Surprising Failure of Anonymization”. UCLA Law Review, Vol. 57, p. 1701, 2010
[27] CIGREF, « Big Data : La vision des grandes entreprises », 2013

2. Big Data as a new way to extract value from data

As introduced earlier, the key idea is to forget about causation, or trying to extract knowledge from data mining. A large part of Viktor Mayer-Schönberger and Kenneth Cukier’s book (chapter 5) is dedicated to the difference between causation and correlation. But what we found when we heard about “real big data systems”, such as those from Google or Criteo, is that these systems are anything but static data mining systems that aimed at producing knowledge. They are dynamic systems that constantly evolve and “learn” from the data, in a controlled loop. Most big data statistical tools, such as logistic regressions, or machine learning algorithms are still looking for correlations, but these correlations are not meant to hold intrinsic value, they are input for action, whose effect is measured in real-time. Hence knowing the “why” of the correlation, knowing if there is a causation, a reverse causation, a complex dependency circle which is the signature of complex systems … does not really matter. Nor does it matter (as much as in the past) to be assured that the correlation is stable and will last in time. The correlation detection is embedded into a control loop that is evaluated though the overall process financial result.

This Big Data approach towards data mining is about practice and experiments. It is also broader in scope than statistics or data mining, since the performance comes from the whole “dynamic system” implementation. It is also easy to deduce from this, which we do in our report with more details, that building such a Big Data system is a team effort, with a strong emphasis on technology and distributed systems and algorithms.  From a marketing perspective, the goal is no longer to produce “customer knowledge” – which meant to understand what the customer wants – but to build an adaptive process which leads to better customer satisfaction – which is actually less ambitious. If we consider the Walmart example that is detailed in the previously mentioned book [5], analyzing checkout receipts produces correlations that need not to be thought as “customer insights”. There is no need to find out why there is a correlation between purchases of diapers and beer packs. It is enough to put these two items close by and see if sales improve (which they did). In the virtual world of the Web, testing these hypotheses becomes a child play (no physical displacement necessary).

There is a natural risk of error if one takes these correlations out of their dynamic-loop context and tries to see them as prediction. Our different experts, including the speakers invited for the 2013 Frontiers of Engineering conference – in particular Jeff Hammerbacher from Cloudera et Thomas Hofmann from Google – were adamant about the fact that “Big Data produces information that one does not really understand”, which creates the risk of poor utilization. This is similar in a sense to the phenomenon of « spurious correlations », which says that one analyses a large cloud of data points with a very large number of variables, one finds statistically a large number of correlations without any meaningful significance. A great example of avoiding such pitfall is given by the “Google Flu Trends” (GFT) story. When they analyzed search requests that used works linked to Flu, Google researchers found that they could forecast flu epidemics propagation with a good level of accuracy. This claim was instantly absorbed, amplified and orchestrated as a proof of Big Data greatness. Then more detailed analysis [15] [16] showed the limits and the shortcoming of GFT. The article that is publish on Harvard’s blog [15] is actually quite balanced. Although it is now clear that GFT is not a panacea and shows more errors than other simpler and more robust forecasting methods, the articles also states that : « The initial vision regarding GFT – that producing a more accurate picture of the current prevalence of contagious diseases might allow for life-saving interventions – is fundamentally correct, and all analyses suggest that there is indeed valuable signals to be extracted ».

3Data is the new code

This great catch phrase was delivered to us by Henri Verdier, one of the many experts that was interviewed by the ICT commission. When Google’s teams look at a new startup, they compute the valuation mostly from the volume and the quantity of data that has been collected, with much less regard for the code that has been developed. Data valuation comes both from the difficulty to collect the data and its estimated future usage potential. Code is seen as an “artefact” that is linked to data, which is both destined to change and easy to replace. In this new world of Big Data, code is conceptually less important than the data it applied to. It is less important because it changes constantly, because it is made of simple sub-linear algorithms (the only ones that can be run onto petabytes of data) and because it is the result of a learning loop (simple algorithms in their principles, but zillions of parameters that require to be fine-tuned through experiments). To caricature the reasoning, Google could tell to these young startups : “I will buy your data and I will re-grow the code base using our own methods”.

This new way of programming does not apply only to new problems and new opportunities! This approach may be used to re-engineer more classical “information systems” through the combined application of commodity computing, massively parallel programming and open source data distribution software tools. This combination helps win one or two orders of magnitude with respect to cost, as was shown to us with numerous examples. In the previously mentioned book [5], one may learn about the VISA example, where Big Data technology was used to re-build an IT process with spectacular gains in cost and throughput. This “new way of programming”, centered on data, may be characterized in three ways:
  • Massively parallel programming because of the distribution of very large amount of data. The data distribution architecture becomes the software architecture because, as the volume grows, it becomes important to avoid “moving data”.
  • Sub-linear algorithms (whose compute time grows slower than the amount of data that they process) play a key role. We heard many great examples about the importance of such algorithms, such as the use of Hyperloglog counters in the computation of Facebook social graph diameter.
  • Algorithms need to be adaptive and tuned incrementally from their data. Hence machine learning becomes a key skill when one works on a very large amount of data.

During the 2013 FoE conference, Thomas Hoffman told us that “Big data is getting at the core of computer science”. This means that all current problems that receives the attention of today computer scientists, such as artificial intelligence, robotics, natural language processing, behavioral learning, and so on, all require the combination of these three characteristics: need for massive hence distributed computing power, huge amounts of data and machine learning to grow better algorithms.

This does not mean that « data as the new code » is a universal approach towards information system design. Massive distribution of data has its own constraints and faces fundamental (theory-proven) data architecture difficulties. They are known, for instance, as the CAP Theorem or the problem of snapshots algorithms in distributed computing. Simply put, it is not possible to get at the same time data consistency, high availability and fault tolerance if part of the network becomes unavailable. Big Data solutions usually pick a weakened form of consistency or availability. The logical consequence is that there remains domains – mostly related to transactions and ACID requirements as well as very low latency requirements – where “more classical “ architecture are still better suited.

4. Conclusion

These two paradigm shifts are accompanied by changes in culture, methods and tools. To finish this post and to summarize, I would quote: agile methods, open-source software culture and DevOps (continuous build, integration and delivery). It is stunningly obvious that one cannot succeed in developing the kind of closed-loop data mining systems described in Section 2, nor the machine-learning-data-driven algorithms described in section 3 without the help of agile methods.  Agile methods advocate incremental and short batch development cycles, organized around multi-skills teams, where everyone works in a synchronous manner on the same objective.  The same argument applies to the proper use of open-source software, though it is less obvious and comes from experience. It is not about using great software available for free, it is more about using continuously evolving software that represents the bleeding edge of big data technology. It is even more about the open source culture that fits like a glove to the concept of software factories (the topic of another post !). To succeed in this new software world, you need to love code and respect developers (and yes, I am aware of the paradox that this may cause together with the “data is the new code” motto). Last, there is no other way to produce continuously evolving code (which is implied by both these paradigm shifts, but is also true in the digital world) than switching to continuous build, integration and delivery, as exemplified by DevOps. I am quoting DevOps but I could also make a reference to the software factory idea (the two are closely related).

Not surprisingly, the reader who is familiar with “Les Géants du Web” [19] from Octo will recognize the culture which is common to the “Web Giants”, that is, the companies who are the most successful in the digital world, such as Amazon, Google or Facebook. There is no surprise because these companies are also amongst the world leaders in leveraging the promises of Big Data. Agile (hence collaborative) development is critical to Big Data which requires to mix computer science, information technology, statistical and domain matter (business) skills. Because Big Data requires to work on the “real” (large) sets of data, it means a strong collaboration between IT operations and development. This is made even more critical by the paradigm shift described in Section 2, since algorithmic development and tuning is embedded into an operation cycle, which is an obvious call for DevOps.

I will conclude with a few of the recommendation from the NATF report:
  • Big Data is much more than new opportunities to do new things. Fueled by a technology shift that is caused by drastic price drops (storage and computing), Big Data paradigm causes a disruption about how to build information systems.
  • Massive parallelism and huge volumes of data are bringing a new way of programming that is urgent to learn, and to teach. This goes for companies as well as universities or engineering schools.
  • The old world of cautious “analyze/model/design/run” waterfall linear projects is in competition with a new world of systemic loops “experiment/learn/try/check”.  This is true for science [21] as well as for business. Hence, Big data’s new paradigms needs to be taught in business schools as well as in engineering schools.

Readers who are familiar with  Francois Bourdoncle’s theses on Big Data will recognize them in these recommendations, which is quite natural since he was one of the experts audited by the ICT commission from the NATF.

Technorati Profile