Saturday, November 26, 2011

Lean IT, Devops and Cloud Programming

I have become more and more interested with lean IT over the years. It started with the book "The Art of Lean Software Development" by Curt Hibbs. 
I enjoyed this book because its introduction of lean is 100% compatible of what I learned from Jeffrey Liker and other great authors about TPS. This simple book helped me draw the lines between good software development methods such as extreme programming or agile programming, and lean management. For those of you who are not familiar with lean or extreme programming, here is a very crude summary of some of the most salient points of lean software development :
  • No delays: avoid as much as possible work that is sitting between two steps of the development process (what is called WIP – work in process – in the lean jargon). This is true at the process level (the goal is to design a streamlines/ “single piece flow” development organization) and the developer level. A practical goal is to avoid switching tasks as much as possible: focus on one thing and do it right!
  • Quality in (right first time) :  this lean principle translates into testing as early as possible (a tenet from agile programming) but also to use all techniques that improve the quality of the code, even at the expense of source code productivity since we all know that it is cheaper not to produce a bug than to remove it later. Here comes, for instance, the practice of pair programming and code reviews, but also “good practices” such as programming guidelines and standards
  • Fast delivery: the lean principle is to reduce the “lead time” of the software development process, which requires working on all stages. Removing in-between delays (cf. earlier) is necessary but not enough. Continuous integration is a core technique to achieve this goal, such as fast deployment techniques.
  • Short deliveries: It is more efficient to produce small pieces of software at a high rate than bigger pieces at a lower rate. This is another key principle from lean (“small batches”), which is doubly true for software development: not only are l smaller batches easier to build (a well-known law of software engineering), but the continuous evolution of customers needs and environment makes it easier to adapt with a small batch approach.
  • Less code: this is the application of the KISS principle !  Lean software development tries to stay away from complexity (see later in this post).  Unnecessary code may pop up from many sources, lean applies a technique called VSM (Value-Stream Mapping) and a posture of “muda removal”.  Muda (waste) removal means to go through the process with the “eyes of the customer” and remove all that does not produce value from her perspective. VSM is a tool that tracks the value creation and assigns it to each step of the process. Lean software development aims at producing the right product, without unnecessary features. It is also an architecture principle (stay away from complexity) designed at simpler and faster maintenance over the years.
  • Customer participation: the best way to produce only what is necessary from the user’s perspective is to ask her frequently ! This is why end-user/customer participation is a tenet of agile programming. When the customer is not available, the principle mixes with “small batches” to become : deliver fast, fail faster to succeed sooner.
  • Regular work effort : the leveling of the effort is a key principle from extreme programming, the equivalent of “heijunka” from lean programming. A few years ago, when I was still a CIO, I started thinking about “extreme IT” (applying extreme programming at the information system level), and the sustainability of the effort is a crucial point. Regularity has a counterpart, which is discipline. Using methods and tools (such as configuration management, source code versioning, automated testing) is crucial. One should take a look at the wonderful talk from Mark Striebeck on “Creating a testing culture” at Google.
  • Minimize handovers: complex tasks, such as writing software, are better accomplished when the responsibility chain is not truncated into too many segments. Working as a team is the only alternative to deliver complex projects (this is the topic of my third book). This is another insight from the Agile Manifesto, this is why today’s best practice it to assemble multi-disciplinary teams including software developers, marketers and designers.
There are a number of interesting books on this topic. The most classical ones are those from Mary Poppendieck, such as “Implementing Lean Software Development: from Concept to Cash”.  I had the pleasure of presenting my last book at the Lean IT Summit a month ago and to meet with Mary Poppendieck (a number of great talks, including hers, Michael Ballé’s and Georges Striebeck’s, are now available online).
There are also books which are not about software development per se but very closely related. I am especially fond of two books which I have reviewed in my French blog:
Thanks to Guillaume Fortaine, I have come to learn about Devops. Quoting from Wikipedia, “DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.  The more I read about Devops ( is a treasure trove of great articles), the more I find that Devops is the missing link between Agile/Extreme Programming and lean management (Toyota-style, hence Lean-Startup-style).  Although Devops claims to “help finishing what Agile Development started”, there are different flavors in what one may read on the web: bridging the development /admin & operations gap, making lean IT happen, making collaboration happen (and also),  bringing agility to operations,  which in turns makes it easier to leverage the benefits of new computing resources such as cloud programming. My first obvious interest with Devops has been the practical deployment of lean IT, as a followup of what I just explained earlier. For instance, Devops promote a number of interesting tools related to deployment automation, such  as GLU. It turns out that the “Cloud and Devops intersection” is also quite promising. Indeed, leveraging the strengths of cloud programming requires a shift in development/architecture culture that requires a “Devops-like” approach.

It happens that I gave a talk last week at the Model-Driven Day about « Complexity, Modularity and Abstraction” (the talk is available in the “My Shared Document” box on the left). The talk is about, among other things:
  • Complexity (why it is important, how to measure it and to tame complexity – since avoiding complexity altogether is not an option at the IS level)
  •  Sustainability: how to transform enterprise architecture into a regular practice. This is related to the concept of “extreme IT” : avoid the “heroic struggle” to move towards the “continuous transformation of the information system”. This is a major reason why I have been advocating for SOA as a company-wide enterprise architecture practice for many years.
  • Architecture-oriented services” : I made this pun to emphasize the difficulty to produce the “right” services through SOA. “Architecture-oriented” means services that have the right level of abstraction, that are modular and “easy-to-compose”. To my knowledge, there is no easy recipe for this, but the wisdom and folklore of 40 years of software architecture design apply.
  •  Cloud computing: I have added a small addition about “cloud-ready architecture” in the 4th edition of my first book. I strongly believe that Information Systems will change in a spectacular manner when we learn how to exploit massively parallel architecture (using tools/approaches such as Hadoop/MapReduce). Using Cloud Computing to provide with a few Saas-based front-office services is nice and relevant, but the big change comes when cloud-computing is applied to back-office services (provisioning, billing, data mining). This requires an architecture change, but mostly requires a culture change.
The thought that prompted me to write this post is as follows. Devops is the missing link between the themes of my MDD talk: managing complexity, delivering agility/modularity and moving to the new century of massively parallel computing (including, but not restricted to, cloud computing). I speak of "Cloud Programming" in the title because I agree with a comment made by Georges Reese: the Cloud is a computing resources defined by its API, i.e., a resource that is managed through programming. The exact quote is:
Cloud is, for the purposes of this discussion, is sort of the pinnacle of SOA in that it makes everything controllable through an API. If it has no API it is not Cloud. If you buy into this then you agree that everything that is Cloud is ultimately programmable.
 Not only the cloud is more agile (up/down scalability), it may be controled by the piece of software which is using it. From a systemic perspective, it's a "whole new game" :)


Matthias Marschall said...

Hi Yves,

thanks a lot for putting this post together. It is a really great overview on lean, agile, and DevOps.

I like your positioning of DevOps as "missing link between Agile/Extreme Programming and lean management". That really puts it nicely.

I hope, more and more folks understand how agile, lean, and DevOps work together to create more flow in software development organisations.

Yves Caseau said...

A great follow-up is this article on the OCTO blog:

Yves Caseau said...
This comment has been removed by the author.
Giom said...

Je remonte un vieux post parce qu'il y a un petit truc qui me turlupine à chaque fois que je le vois.
Tu dis souvent qu'il faut maintenant faire du "right first time". Autant je comprend très bien ce que tu dis sur le fait qu'il soit moins cher de faire bien tout de suite plutôt que de corriger plus tard, autant dans une vision plus macro je trouve que parfois la volonté de sortir des produits avec zéro défauts est une utopie qui fait perdre plus de temps qu'autre chose. Au niveau du développement de nouveaux services numériques j'avais plus l'impression qu'au contraire on migrait plutôt vers un fonctionnement "à la google" où l'on propose des versions beta aux clients le plus vite possible afin d'avoir des retours le plus tôt possible. Et maintenant que les gens ont compris qu'ils pouvaient mettre leurs appareils à jour on voit aussi sortir sur le marché des appareils non exempts de défauts mais qu'on corrigera avec une mise à jour ou dans une version 2...
Bref tout ça pour dire que le "right first time" d'accord au niveau des développements, mais pas au niveau de l'entreprise qui se refuserait à proposer à ses clients un produit non parfait.

Yves Caseau said...

Très bon point, qui mériterait un billet à lui tout seul. Il y a deux tensions dans le leanIT: (a) faire vite et faire zero défauts, (b) faire en juste à temps et anticiper/préparer/ cultiver. Pour faire une réponse rapide sur le premier point: il ne s'agit pas d'écrire du code avec toutes les exigences du premier coup, il faut en relacher certaines pour implémenter les petits lots (à la Google, à la Ries, à la prétoyping, etc.), en revanche, même pour un lot "beta" ou un MVP (minimum viable product), il faut viser l'écriture d'un code de qualité ! C'est vraiment un point clé: pour aller plus vite, on écrit moins de code, on remet des choses à plus tard, mais ce qu'on écrit est (1) propre et élégant (2) testé (3) architecturé/organisé. J'ai passé une grande partie de ma vie à écrire du code en mode ultra-rapide (ie, faire un POC en un week-end pour démontrer une idée), et je partage profondément l'idée lean que l'agilité n'est pas une excuse pour du code sale. Tout code sale (et souvent trop long) coûte incroyablement cher ... tot ou tard. Donc le slogan "quality first" s'interprete par "écrire toujours du code de qualité, et en écrire moins pour aller plus vite".

website designing development company chennai india said...

I have read this post. collection of post is a nice one ..that am doing website designing company chennai india and website development company chennai india. That I will inform about your post to my friends and all the best for your future posts..

Bailey Thompson said...

Because your business is similar to others does not make it the same, and these often subtle differences in the way different businesses operate and are managed mean that the systems and software that suit one business will not meet the requirements of another company.We offer a wide range of Web Development Leeds services for mobile handheld PDA devices.

Ruzina Bagum said...

This site is good because they give us a new thing and new ideas and new topic how good all of they are we should appreciate them because of these good thing.
seo company bangladesh

Albert Andrada said...

Really Nice Information,Thank You Very Much For Sharing.
Wordpress Development Company
Web Development Company

seema said...

very nice article.Thanks for sharing the Post...!

scrum method

Technorati Profile