I’ve been seeing these surveys and statistics reference around The Chaos Report for nearly two decades. Often, as in this article here, they are used as citations supporting agile ways of working. And since I’m an aglist, you’d think I would be in full support of them. But I have three core problems with the reports themselves and the incessant quoting of them to support some position.
Trust?
First, I’m not sure I trust the data. Where does it originally come from AND are the collections accurate?
For example, I used to fill in project timesheets at the end of each week. I filled them in with the best recollection I had and with my perception of time spent. I realized over time that I was probably only reporting at 50% accuracy. And that was as an individual contributor. Aggregate that data over 100 developers over 1-+ projects every week. Would you trust what that data was telling you?
So, rolled-up statistics collected from a wide variety of companies doesn’t always make me that trustful and confident in the data.