Elixir was a deliberate choice when Brex was started, as outlined by our co-founder Pedro in this blog post. There are a few main reasons we picked Elixir. One is the fact it was built on the Erlang VM, which offers reliability and fault tolerance out of the box. Another is its functional nature, which forces immutability and makes complex code easier to reason about. Others include its great concurrency primitives and macros for extending the language. These are important features for a highly reliable financial infrastructure. As of 2021, a majority of Brex’s business logic has been developed in Elixir, which means all product engineers have had to onboard and learn Elixir. Brex has hired and onboarded over 150 engineers to write Elixir in a production system and has over 70 elixir services running in production, which gives us some unique insights into how to rapidly hire and onboard Elixir engineers. Over the last 3-4 years, we have encountered some issues and learned lessons that might be helpful to other organizations adopting Elixir.
This post was sponsored by digital product consultancy DockYard to support the Elixir community and to encourage its members to share their stories.
Hiring
Brex has more than doubled engineering every year since the beginning of 2019. In January of 2019 there were around 20-30 engineers. By January of 2020, we were at around 70 engineers, and as of January of 2021 we were around 200 engineers. Early on at Brex, we realized it was going to be hard to hire for Elixir experience specifically. Elixir is still a fairly young language, although more and more companies are adopting it. As we needed to hire rapidly, there wasn’t a plethora of Elixir engineers looking for new jobs to fill our open headcount. Therefore, at Brex we’ve adopted a language-agnostic hiring process, focused on hiring engineers who meet our overall technical expectations and align with our company’s values. Elixir knowledge is not a prerequisite to interview at Brex; in fact, very few engineers who end up joining Brex have any Elixir experience. This means almost all engineers who have joined Brex have had to learn Elixir on the job, which requires us to focus on onboarding and documentation around Elixir. Generally speaking, engineers are able to learn Elixir in under a month and be fully onboarded within two months. However, this doesn’t mean all engineers who join Brex have exactly the same experience. Engineers who come from dynamically typed languages (especially Ruby) or functional languages generally have an easier time onboarding and learning Elixir than engineers who come from statically typed languages. The lack of static types makes it hard for engineers with this background to grasp input and outputs of functions in Elixir. Typespecs can be helpful but they aren’t always correct, and without enforcement or a culture around them, a majority of our functions do not have them. Typespecs and dialyzer could help solve this issue, but if they are not adopted early it’s hard to retroactively roll them out across a large codebase without a concerted effort.
Macros
Over the last few years, we have written a lot of Elixir code (around a million lines of business logic), and in that time we’ve also made some mistakes with Elixir. One of these mistakes was around our use of macros. Macros are useful and have their place; however, if you follow the Elixir library guidelines it is recommended to avoid macros. Macros can hurt developer productivity due to their difficulty to reason about and debug. Although we have a lot of great macros at Brex that make our lives easier, there’s also a number of them we realize aren’t as useful. Now, some macros are rarely used or even discouraged because they constantly gave engineers issues, they were hard for product engineers to debug, or they didn’t provide enough value compared to another way of accomplishing the same thing. New engineers unfamiliar with Elixir found macros especially hard to reason about and read, including both DSLs and __using__ macros. Unlike the rest of Elixir, which is generally very legible, macros can be quite nuanced and obtuse if you are unfamiliar with them.
However, as I mentioned, there are a lot of good use cases for macros. An engineer at Brex, Lizzie Paquette, actually gave a great talk at Code Beam SF 20 about some of our internal macros which you can checkout on YouTube. We have many __using__ macros from internal libraries that allow engineers to easily and quickly implement complex abstractions. One such example is a library that allows engineers to easily define publishers and consumers of events which interact with our asynchronous events infrastructure (utilizing Kafka under the hood). Macros also allow us to create DSLs and extend the Elixir language so Engineers can focus on business problems. Examples of helpful DSL macros our engineers use internally include a simplified DSL for defining Elixir structs with type annotations similar to Ecto schemas, a DSL for defining Ecto.Schema changeset validations, and a DSL for defining enums. While I definitely agree macros should be used sparingly, these are some of the successful use cases we’ve seen internally.
Reducing Boilerplate and Enforcing Standards
When we initially started using Elixir, it was a fairly young language with a small but strong community. However, this also meant we didn’t find a lot of external libraries that fit our needs. Therefore, we decided to start writing internal libraries for certain use cases. The main two reasons for writing internal libraries were to reduce boilerplate and enforce standards. Some examples of reducing boilerplate were the macros mentioned in the previous section and the __using__ macros that are common in our libraries. __using__ macros allow us to abstract away a large amount of functionality so developers only need to write their own business logic. The DSLs we provide also allow engineers to simplify common use cases like Elixir structs and enums. Furthermore, we wanted to enforce standards through our libraries. An example of this is our wrapper around Ecto.Schema, which makes field values required and immutable by default but also allows for escape hatches where needed via schema field annotations. At this time, we have over 40 internal libraries that allow engineers to move quicker.
Nevertheless, there are drawbacks to having so many internal libraries. The largest issue we found with them is they were hard for new engineers to learn. It was common for new engineers to be unsure of where a Brex namespaced module came from or what it did. However, a benefit of using Elixir is its great tooling, including documentation. We took advantage of Elixir’s Hexdocs to auto generate and internally host our libraries’ documentation. But these docs were not discoverable on their own. So, in order to better teach new engineers about our internal libraries and give them the tools to learn themselves, we created an onboarding session around our internal libraries and some of the major ones they will use on a day to day basis. We also supplemented our Hexdocs documentation with some resources that live alongside the rest of our engineering documentation. Our internal libraries are incredibly helpful, but come with the need to properly document and educate new engineers on their use.
Conclusion
Elixir is a great language and the work the Elixir Core team has put into the tooling and core libraries is amazing. However, since Brex was an early adopter of Elixir and a hyper-growth startup, we encountered a number of speed bumps very quickly that we had to solve to continue scaling. Finding great Elixir developers quickly is still a fairly hard challenge, but there are plenty of great developers that can quickly learn the language, especially if they come from dynamic and functional language backgrounds. Using macros is a nuanced art, but when used well they can accelerate development for your organization. Hopefully these lessons can help other organizations both adopt and rapidly scale their Elixir usage without hitting some of our speed bumps.
Thank you to Lars Wikman and Susan Watkins for reading and editing drafts of this blog post