Haystack vs. Pluralsight Flow

No Snake Oil

At Haystack Analytics, we only provide software engineering metrics that are proven to help teams - either through our own validation or empirical scientific evaluation.

For example; our North Star metrics include the Four Key Metrics. Rigorous appraisal of these Four Key Metrics (in research led by Dr Nicole Forsgren) has shown that higher performers are 2x more likely to meet their commercial goals (productivity, profitability, market share, number of customers) and their non-commercial goals (quantity of products or services, operating efficiency, customer satisfaction, quality of products or services and achieving organisational or mission goals). Indeed, companies which do well under these DevOps metrics have a 50% higher market cap growth over 3 years. The mechanics of how metrics drive organisational performance is also well understood and has been rationalised by industry leaders like Martin Fowler.

Unfortunately, many of our competitors don't apply the same level of rigour to validating their metrics. Indeed, in many cases unvalidated metrics may prove harmful and actively damage your organisation.

Both Pluralsight Flow (previously known as GitPrime) and WayDev contain a code quality metric called "churn", this is defined as "code which is deleted or rewritten shortly after being written (less than 3 weeks old)". This metric is then used to compute an "efficiency" metric which is assumed to be "the percentage of all contributed code which is productive work" for a given engineer. Not only have we seen no evidence that such a metric will actually help your organisation, it appears to fly in the face of the evidence we have on the importance of lowering Cycle Time.

Companies with lower Cycle Time (time from development to code in production) are able to rapidly test ideas to gain quick customer feedback, resulting in product that is more likely to satisfy a customer. If you test something and your customers don't like it - you should be able to remove it without being punished for increasing some meaningless "churn" metric.

More Harm Than Good?

Abusing metrics can often do more harm than good. At Haystack we firmly stand against the notion that individual level metrics can provide actionable insights and believe that they cause more damage to work culture as they are invasive, stress-inducing and misleading. As such, we only track delivery at a team level delivery while avoiding misleading output metrics like “lines of code”. We also provide additional insights to aid managers preventing harmful things like burnout. 

Pluralsight Flow focuses on metrics closely tied to lines of code such as "impact scores" and "tt100" which essentially measure the speed and difficulty of line changes. Their dashboard comes fully featured with a vast array of metrics, whilst Haystack prefers to present the North Star metrics and allow people to drill-in as needed.

Pluralsight Flow allows engineering leaders to track individual developers and developer output with metrics like lines of code to help them gain insights. With a core focus on individual reporting and features for team ranking. Evidence suggests this is a harmful practice which doesn't aid team performance.

In Project Oxygen, Google evaluated manager performance ratings and manager feedback from Google’s annual employee survey - they found that great managers are excellent coaches, but they do not micromanage.

Google also studied team performance in Project Aristotle evaluating a total of 180 teams (including 115 project teams in engineering) using hundreds of double-blind interviews and quantitate data sources. Google's researchers found that individual performance of team members is not significantly connected with team effectiveness, but psychological safety is essential.

Risk Alerting

Haystack integrates with Slack and email to provide real-time comprehensive alerts around sprint risks, blockers and burnout. Common issues like team burnout, pull requests stuck in discussion, code merged without review and high concurrent work can be flagged to the team for fast resolution of potential issues.

Pluralsight Flow spotlights problematic individuals, by allowing managers to compare individuals, with no alerting to remedy blockers.

More Cost Effective

Haystack not only offers superior value-for-money, but the product is also available at a lower cost.

Haystack offers both a "growth" plan that is available for $15 per developer a month with annual payment alongside a "startup" plan for $25 per developer a month which is available on a rolling monthly basis with no annual commitments.

Pluralsight Flow requires that contracts are locked in for an annual rate of $449 per developer a year ($37.42 per month) for their entry level pricing.

Support

Haystack Analytics provides dedicated support team of experienced engineering leaders available on-demand from our Slack Community. Every Haystack user has a dedicated private slack channel with our team of experts to help you deep dive into your data and teach you how to improve delivery. 

Pluralsight Flow provides customer support to help with things like troubleshooting your integrations.

Security

Haystack is the only tool that doesn’t read, store, or persist source code. We use the GitHub API to pull metadata such as timestamps and commit messages to produce all our insights. We also maintain industry best practices to ensure that all user information is safe.