TIOBE index is flawed

Kotlin's late arrival into TIOBE's Top 20 prompted us to take a closer look at the TIOBE index itself. Spoiler: it has a major flaw.

For the first time, Kotlin has made it to the top 20 of the TIOBE index. One can boldly ask - why so late? After all, Kotlin is one of the most popular programming languages. The problem is not Kotlin or its users, the problem is the TIOBE index itself.

And in this article we are going to explain why you should stop using TIOBE index to gauge programming language popularity

Is it Kotlin or is TIOBE?

Kotlin has been gaining popularity for several decent years now. Its biggest leap happened when Google chose it as the official language for native Android apps. It also gains more and more recognition on the backend, simplifying the development process, thanks to a more expressive syntax.

Regarding the data in our IT Community Survey 2024, concerning knowledge of programming languages, Kotlin took 10th place - 7.1% of commercially working developers declared they use it at work regularly. In the StackOverflow Survey 2023, 9.7% of developers declared knowledge of Kotlin. This gave it the 15th position.

In the TIOBE ranking, Kotlin has made it to the top 20 for the first time. Okay, maybe it would be nothing unusual if it weren't for the languages that rank higher than Kotlin.

This elite group of languages, that are placed higher than Kotlin, includes:

Visual Basic (position 7)
Fortran (position 11)
Matlab
Scratch
Delphi/Object Pascal

Something’s fishy here.

None of these languages even appear near Kotlin, neither in our survey focused on the situation in Poland nor in the StackOverflow survey.

It's strange because TIOBE itself claims:

The TIOBE Programming Community Index is an indicator of the popularity of programming languages. The index is updated once a month. The ratings are based on the number of qualified engineers worldwide, courses, and third-party vendors.

If so, it's weird that JavaScript is only in 6th place (right before Visual Basic), C and C++ occupy 2nd and 3rd place, and Java and C# are behind them. Ever heard of TypeScript? That's good, because TIOBE did not.

The Problem with TIOBE

What appears in the index results directly from its methodology, which is presented on the TIOBE website:

Basically the calculation comes down to counting hits for the search query

+"<language> programming"

It seems to us that the number of results for such a query may not translate into the actual popularity of programming languages.

This index definition will reward languages taught at schools or universities - hence the presence of Matlab or Scratch in the index, or even Python's first place in the index.

Languages that do not have a very developed ecosystem will also be promoted. E.g. React results won't be counted as JavaScript-related. Besides, who the heck writes "JavaScript programming" anyway?

In TIOBE's methodology many retailers are counted as "search engines". This basically counts books available on the topic.

Now you know why we are not fans of the TIOBE index. And let's be clear - it's not about hating the TIOBE company - although when they write that their "ratings are based on the number of qualified engineers worldwide", it doesn't look good for them.

However, that's not the main point. We are on the internet, and not everything that glitters is gold. Sometimes it's worth checking where the results of one index or another originate, especially since the results of various popularity indexes are widely discussed in many online publications.

What to use instead of TIOBE?

When we started Bulldogjob a few years ago, the availability of data on tech in general, and especially on programming, was not great. Times have changed, and we now have much more data about our industry.

There are a few noteworthy surveys; here are some examples:

Also there are surveys specific for ecosystems, like Go Developer survey. You'll often found similar sources for other languages.

If you are looking for results based on what's going on in the interwebs, then consider RedMonk rankings, that analyze language popularity through GitHub and Stack Overflow statistics.

Each of the mentioned sources has its own strengths and weaknesses, so do not treat them as an objective source of truth. The key is to at least try to get to know them a little bit.