APIs, sometimes they feel like magic. I mean, it’s beautiful how you can take two independent systems, on two different servers, written in two different languages and make them act like they’re siblings. Well, that’s the intent anyway. It doesn’t always go according to plan. What follows is an account of our adventure in which our interaction with an API was more like distant cousins, very distant cousins. If you’re the nervous type, I’ll go ahead and tell you that it all works out in the end.
Our app makes use of a 3rd party vendor to send push messages to our users. We have a company account with the vendor that holds our user registrations. This ensures that messages with our name on them can only come from us and secures our users’ privacy in the process. With the upcoming changes in data protection laws, we wanted to consolidate our accounts to provide a more clear picture of our data security procedures. That meant re-registering our users with the proper account. All 5 million of them. One at a time.
The usual approach
We can’t be the first to cross this bridge, right? A quick internet search showed that were we not alone and many others had detailed their process for re-registration. A quick Java app and we were ready to test it out so we fed it 1000 dummy users to get an idea of timing. We patiently waited 181 seconds for the “Process Complete” notification. Ok, not bad, let’s see, 5 million records at .181 seconds a piece, that’s uhh, oh… That’s 10.5 days. So this won’t be done before lunch? We dug into the code but there wasn’t anything really wrong with it. It just wasn’t engineered to handle 5 million records. There had to be a better way.
And now from left field
In pooling our experiences from past jobs, hobby projects, and general knowledge we realized that it wasn’t the code that was the issue, it was the language. Now, there’s certainly nothing wrong with Java, it has been the gold standard for coding for decades, but that doesn’t mean it is always the best solution for every challenge. We decided to try out Elixir as a solution. Elixir is a functional, concurrent, general-purpose programming language that runs on the Erlang VM (BEAM). What that means is, it’s fast. Really fast. With just over 40 lines of Elixir we were able to process all 5 million of our user registrations in about 75 minutes. Roughly .5% of the time of the Java implementation. Looks like lunch won’t be late after all.
Never, never, never give up