We spent two years building an internal developer portal—only to discover it wasn’t solving our developers’ biggest pain points. Here’s how a simple developer survey changed everything. As a technical lead who focused on developer productivity at a high-growth startup, I often receive inbounds from founders working on developer tooling. During these chats, I find myself reiterating this same story over and over, so I thought it might be best to write it down for others as well.
No matter how many times you hear about the importance of solving real problems and talking to your customer, you don’t realize you’re doing it wrong until you’re doing it right. We spent over two years developing our own internal developer portal using Backstage, only to eventually realize it wasn’t going to solve our developers’ largest pain points at the time. By using a proper developer survey with real engagement, we were able to better understand developers’ highest priority issues and focus on the right things. Our developer productivity teams have been focusing on the right things and delivered more impact in a year than probably the previous 2 to 3 years.
The Rise of the Internal Developer Portal
In 2020, while working as a product engineer, I was approached by our platform team for an internal interview. It became clear they had a specific solution in mind—an internal developer portal. Shortly after, I joined a new team focused on building internal tools, beginning my journey in developer productivity.
This team’s mandate as mentioned was to build internal developer platforms to unify our diverse set of tooling into a single interface for our rapidly growing engineering organization. Although there were a few such platforms over the years, one of our core focuses from day one was building an internal software catalog and developer portal. Over the next two years from 2020-2022, we would build an internal developer portal utilizing Spotify’s Backstage. You can read more about that journey on the Brex Tech Blog. We were pretty early adopters of this technology, adopting Backstage when it was still in alpha, and a little ahead of the current hype train for using internal developer portals. While we had many learnings along the way, the main one I always reiterate is that the software catalog itself was not a game changer for us. Our developers didn’t really need the read only functionality and software knowledge base it provided, Instead, they found the most value in our internal developer portal for specific maintenance or “write” actions like creating new pieces of software or managing their softwares deployments(releases). From 2020-2022, we spent a lot of time and resources trying to increase engagement, growing our team from two to six engineers, but we were never able to get more than 20-30% of our developers to visit it on a weekly basis.
One piece of feedback that the Backstage team at Spotify often mentioned was make sure you’re solving for a real problem with your initial implementation. Internal developer portals have many different use cases and feature capabilities you can build, such as a software catalog, software creation, knowledge bases, custom functionality, user management, etc. It’s easy to try and build out a lot of this functionality right away, especially when there’s plugins to build on top of, but this increases your level of investment, delays value to developers, and adds unnecessary complexity. I myself would often echo this feedback and sentiment to those who reached out and were interested in building with Backstage, but it didn’t really click until later on.
Realizing the Gap: Implementing Internal Developer Surveys
In October 2022, after a major restructuring, our platform teams faced new resource constraints. We needed to provide more value with fewer resources—and to explicitly show that value. Right before this restructuring, we tried a developer survey tool called DX. It promised high engagement and clear insights—something we had struggled with in our previous attempts. Not only did we get unprecedented levels of engagement (90-100% of software engineers responded), but the data itself was incredibly useful. DX comes with a predefined question set based on industry leading research into developer sentiment and productivity. It made it incredibly easy to see our top priorities according to engineers, get additional comments on the priorities, and benchmark ourselves against other companies. After our restructuring, we knew this tool would help us be laser focused on the largest pain points and showcase our value.
When we reviewed the results, the message was clear: our developers’ biggest pain point wasn’t solvable via our developer portal. It was local development speed. At this point, it became clear to us that an internal developer portal wasn’t the right tool for the job, and we pivoted our focus to building local first tools for iterating on changesets.
Shifting Priorities: Putting the Developer Portal on Hold
Although we had spent the last two years heavily investing in an internal developer portal, and I myself had spent a lot of time learning about them, contributing to them, and advocating for them, it was necessary for us to divest from our internal developer portal and focus on the higher priority issues.
In about 4-6 weeks we built an MVP of a new tool that would allow developers to quickly and easily run their services locally while connected to our cloud development environment to test their code alongside other services. In a single command and a few seconds, developers could validate their code changes worked alongside our hundreds of microservices. The immediate feedback was positive and we had heavy power users from day one with a rapidly growing user base. We knew we were on the right track, although there were some kinks to work out along the way – though I’ll save that for another post.
Over the next few quarters we would invest more heavily into this tooling, simplifying the process and our overall local development user story based on feedback collected directly from user research as well as through the DX survey. By the end of the year we had doubled local development sentiment from 30 to 60 out of 100, achieving our reach goal, and dropped local development from the top priority to outside of the top 5.
So what now?
Hopefully this anecdote makes it clear why knowing and focusing on the right problems is so important. I often talk to startup founders who after a chat and a product demo ask if this is something Brex would be interested in. A couple of years ago, I wasn’t sure of the answer and would say maybe or let me talk to some other folks. However, after implementing an internal developer survey, the answer became simpler. If it wasn’t in the top 3 developer pain points, it was most likely not something we’d be interested in at this time.
If you’re a platform or engineering organization leader and you can’t instantly say the top 3 priorities for your engineering organization, you should focus on getting answers to those questions by running a quarterly developer survey with high engagement.
If you’re a founder for a developer tooling product, you should be asking companies what their developers’ top 3 priorities are and looking to solve those priorities with your tools or changing your pitch to show how you can help solve one of those priorities.
While it is often important to be a few steps ahead of your developers and thinking about the future, it’s often hard at a fast moving startup to focus on anything that isn’t currently a priority. Most likely, you ain’t going to need it (YAGNI). In the end, it wasn’t about the tool we built—it was about understanding what our developers truly needed. And we wouldn’t have learned that without asking the right questions.