Hype Driven Development

Software development teams often make decisions about software architecture or technological stack based on inaccurate opinions, social media, and in general on what is considered to be “hot”, rather than solid research and any serious consideration of expected impact on their projects. I call this trend Hype Driven Development, perceive it harmful and advocate for a more professional approach I call “Solid Software Engineering”. Learn more about how it works and find out what you can do instead.

New technology — new hope

Have you seen it? A team picking newest, hottest technology to apply in the project. Someone reads a blog post, it’s trending on Twitter and we just came back from a conference where there was a great talk about it. Soon after, the team starts using this new shiny technology (or software architecture design paradigm), but instead of going faster (as promised) and building a better product they get into trouble. They slow down, get demotivated, have problems delivering next working version to production. Some teams even keep fixing bugs instead of delivering new features. They need ‘just a few more days’ to sort it all out.

Hype Driven Development

Hype Driven Development (HDD) has many flavors and touches your project in many different ways:

• Reddit driven development — when a team or individual decide on technology/architecture/design based on what popular blogger wrote or what is hot on reddit, hackernews, blogs twitter, facebook, GitHub or other social media.
• Conference driven development — watch carefully what happens after people are back from conference. People get inspired. And that’s a two-edged sword. Starting to use newest hottest lib/framework/architecture paradigm without enough research might be a highway to hell.
• Loudest guy driven decisions — is when one guy is talking all the time about this new framework/lib/tech’s that he has no experience with, but talks about it all the time and finally the team decides to use it.
• Gem/lib/plugin driven development — a specially strong in Ruby On Rails community, where occasionally I can see a Gemfile so long that the only thing longer is the time it takes to load the app. It comes from the idea that every problem in rails should be solved with a gem. Sometimes it would take a couple of lines to build solution ourselves. But we’re just solving problems by adding libs, plugins, gems or frameworks.
• I would also mention here behavior popular among hype driven developers — Stack Overflow driven development — when developers copy-paste solutions from Stackoverflow (or in general from the internet) without really understanding them.

 

HDD is how teams bring doom on themselves

The problem with hype is that it easily leads to bad decisions. Both bad architectural decisions and technological stack decisions often hunt a team months or even years later. In worst case they may lead to another very problematic situation in software engineering: The Big Rewrite. Which almost never works out.

The root of all evil seems to be social media — where new ideas spread much faster than they get tested. Much faster than people are able to understand their pros and cons.

 

The Anatomy of Hype

Most hypes have similar structure. Here it goes:

  Step 1: Real problem and solution

They start in some company with a problem. A team within some company decides that solution to the problem is beyond the current technological stack, process or architecture. The company creates a new framework, library or paradigm and soon the problem is solved.

Step 2: Announcement, buzz and keywords

The team is excited to show their work to the rest of the world and soon they write blog posts and do talks on conferences. The problem oftentimes is non-trivial, so they are proud to present the impressive results of a non-trivial solution. People get excited about the new technology. The only problem is not everybody who gets excited is able to fully understand what the exact problem was and all the details of the solution. It was a non-trivial problem with a non-trivial solution after all. Takes more than a tweet, chit-chat or even blog post to explain. With communication tools like social media, blog posts and conference lightning talks the message gets blurred along the way.

Step 3: Mania starts

All shades of hype driven developers read blog posts and attend conferences. Soon the teams all over the world start using the new technology. Due to the blurred message — some of them make hasty decision to use framework even though it does not solve any of their actual problems. Yet the team does have expectation that this new technology will help.

  Step 4: Disappointment

As the sprints go by, the technology does not improve the team’s life as much as people hoped, but brings a lot of extra work. There’s a lot of rewriting the code and extra learning for the team. Teams slow down, management gets pissed off. People feel cheated.

Step 5: Realisation!

Finally the team does retrospection and realizes what are the tradeoffs of the new technology and for what purpose it would be more relevant. They get wiser… till the next hype shows up.

 

Examples of Hype:

Let’s examine some examples of hypes and see how those went through.

Example 1: React.js

Step 1: Facebook has a problem — advanced one page apps like Facebook itselves have, so many state changing events that it is hard to keep track what’s going on and keep the application state consistent.

Step 2: Facebook promotes new paradigm with buzzwords: functional, virtual DOM, components.

Step 3: Mania: Facebook has created the front-end framework of the future! Let’s write everything in react from now on!

Step 4: Wait there is a lot of work, but no quick return on investment!

Step 5: React is great for advanced one page app with lots of real-time notifications, but does not necessarily pay off for simpler applications.

Example 2: TDD is dead by DHH

Step 1: David Heinemeier Hansson (DHH, creator of Ruby on Rails framework) realises that it is hard to do TDD with Rails as this framework doesn’t have architecture supporting good OOP. Makes a pragmatic choice — not to write tests upfront.

Step 2: Hype starts with DHH blog post and conference talk. Hype keywords: TDD is DEAD.

Step 3: Let’s skip tests! Our Guru says so. We didn’t write them anyway. Now we’re at least not pretending. We’re finally honest.

Step 4: Wait! Even fewer things work now than before. We’ve built a buggy code.

Step 5: “TDD is not dead or alive. TDD is subject to tradeoffs, including risk of API changes, skill of practitioner and existing design” — Kent Beck.

Example 3: Microservices

Step 1: Big monolith application scales hard. There is a point when we can break them down into services. It will be easier to scale in terms of req/sec and easier to scale across multiple teams.

Step 2: Hype keywords: scalability, loose coupling, monolith.

Step 3: Let’s rewrite all to services! We have a ‘spaghetti code’ because we have a monolith architecture! We need to rewrite everything to microservices!

Step 4: Shit! It is now way slower to develop the app, difficult to deploy and we spend a lot of time tracking bugs across multiple systems.

Step 5: Microservices require a lot of devops skills in the team and with right investment might pay off as a way to scale the system and team. Before you reach serious scale issues it’s an overinvestment. Microservices are extracted not written. You must be this tall to use microservices.

  Example 4: NoSQL

Step 1: SQL databases have problems with high loads and unstructured data. Teams around the world start developing new generation of databases.

Step 2: Hype keywords: Scalability, BigData, High Performance.

Step 3: Our database is too slow and not big enough! We need NoSql!

Step 4: We need to join tables? That is a no go. Simple SQL operations are becoming increasingly challenging. Development is slow and our core problems are not solved.

Step 5: NoSql are tools to solve very specific problems (either extremely high volumes of data, unstructured data or very high load). SQL is actually a great tool and handles high load and huge data volumes well if used skillfully. The case for NoSql is still pretty rare in 2016.

Example 5: Elixir and Phoenix (or put your favorite lang/framework pair here)

Step 1: Web frameworks like Ruby On Rails don’t deal well with high performance applications, distributed applications and websockets.

Step 2: Hype keywords: Scalability, High Performance, Distributed, Fault-tolerant.

Step 3: Oh my good, our application is slow and our chat is not scalable!

Step 4: Wow, learning functional programming and distributed approach is not that easy. We are now really slow.

Step 5: Elixir and Phoenix is great framework, but takes significant effort to learn. It will pay back in a long run if you need specifically high performance app.

The list goes on and on:

In this crowded space of computer engineering we have a lot of areas where hypes are common. In JavaScript world new frameworks are born everyday. Node.js (keywords: event programming), reactive programming, Meteor.js (keywords: shared state), front-end MVC, React.js. You name it. In software engineering new architectures are born: Domain Driven Development, Hexagon, DCI. What is your favorite hype?

Good practices

So if we can’t rely on what the internet says and opinions of other people, then how to make clever decisions? Here come some good practises:

Test and research before you decide:

• Spikes — learn about your technology not from blogs, but from experience. Take 1 or 2 days to build a prototype of a new functionality in the new technology, before you make the decision. Let the team analyse pros and cons. You might take a couple of competitive technologies and let different parts of the team builds prototypes with different technology.
• Hackathon is a great way to build awareness of a team on tradeoffs of different technologies. Take 1 or 2 days for the whole team to hack all the technologies that are on the rise and look tempting to use. That will allow the team to make clever decisions on it’s own and take decisions based on their own experience.

When to go?

• In principle when return on investment is huge. Most technologies are created to solve a particular problem. Do you have that problem? Is it a big problem? Will it save a lot of time? Will the use of the technology pay back the cost of the entrance curve and rewriting? What if we initially slow down development by the factor of two? Or four? Is it still worth it?
• Great teams are allowed more — some teams are just faster than others at delivering the value. They get bored with what they do more easily. These teams can introduce new technologies more often. That’s no excuse not to use spikes and do hackathons. On the other hand, if the team has troubles delivering — proceed with caution.

  Hire the right people:

• Strong technical background is your friend — people who know different paradigms, understand theory of programming (e.g. algorithms, concurrency) and have good engineering culture tend to hype less.
• Experience — hype is stronger with young developers. With years people who have seen many technologies and been in troubles many times tend to get more balanced views of picking technology.