Inngest: A better way to handle background jobs in serverless

As someone who works with Next.js on a daily basis, I am the first to admit that Vercel has done a great job at making the developer experience as smooth as possible, few other tools come close to the developer experience that comes with using Vercel as a team.. But working side by side with people who don't use serverless tools, I've come to realize that some of the most trivial problems in almost any other developer workflow are surprisingly annoying to deal with in a serverless environment.

Using serverless often means being limited to short-lived functions, no persistent state, and no native support for background jobs. Even simple workflows like sending delayed notifications, processing large datasets, or running scheduled tasks can feel like uphill battles. And while Vercel's serverless platform is powerful and intuitive in many respects, there's a notable gap when it comes to handling these stateful or long-running tasks.

Let's say your clients have a requirement to schedule an email 30 minutes after a user signs up. In Ruby on Rails, this could be easily solved with the following code: First we define the job, then we schedule it to run 30 minutes from now.

Now say you want to do the exact same thing using just Next.js and Vercel. Well you are out of luck, you cannot execute a function 30 minutes from now. Using a setTimeout and praying for the best? Sounds like a terrible idea that might work but function execution cannot be longer than 60 seconds.

Another common scenario is needing to run a long running background task, be it processing a large dataset or running a computationally intensive task. Or waiting for an extremely slow and unreliable third party API to return a response.

I have no idea what the folks at Streak are doing but I've had the great pleasure of having to integrate some of their APIs into a project and when I tell you that it was a nightmare, you have no idea. For some reason their APIs might randomly take up to 1 minute to return a response out of nowhere, and let me tell you that is not something fun to deal with when you are bound by 30 seconds execution time. Anyways, back from hating on Streak. Wait no they suck, you want to know more of it, check out their API reference. Why do I have to create an empty box and then update it to have all the fields that are supposed to be required? Oh wait no, there is a non-documented field you can pass to the API to make it work, and don't get me started on their Webhooks. Are you telling me there is no way of verifying the origin of your requests? How am I supposed to know if the request is coming from Streak or some random hacker? I had a real conversation with a real person at Streak about how to verify the origin of the request with no luck.

Sorry I got off track, now actually back to the point, you want to run a long running task, we all know what those are. You are in a serverless environment, what do you do?

Well, you have a few options, none of them great:

Even if you manage to get one of these approaches working, you still have to handle error cases, retries, and state management yourself. What happens if your function fails halfway through? How do you resume from where you left off? These are problems that traditional servers solve easily but become major headaches in a serverless environment.

import { inngest } from "@/inngest/client";
export default inngest.createFunction(
// Function config
  {
    event: "signup/user.created",
    retries: 3,
    onFailure: async ({ event, error }) => {
      await addToDeadLetterQueue(event, error);
    },
  },
  // Function handler
  async ({ event, step }) => {
    await step.sleep("wait-for-30-minutes", "30 minutes");
    await step.run("send-welcome-email", async () => {
      await sendEmail(event.data.user.email, "Welcome to the app!");
    });
  };
);

import { inngest } from "@/inngest/client";

await inngest.send({
  // The event name (this is typed in the inngest definition)
  name: "signup/user.created",
  // The event's data, can be typed with zod
  data: {
    user: {
      id: 123,
      email: "test@example.com",
    },
  },
});

npx inngest-cli@latest dev