How to use Netherite Storage Provider in Durable Functions

Netherite Storage Provider is a backend implementation of Durable Functions that delivers high performance.

It's still a preview version, but if you are using Functions Premium (EP1-3), it's worth evaluating it now.

GitHub logo microsoft / durabletask-netherite

A new engine for Durable Functions. https://microsoft.github.io/durabletask-netherite

Netherite

Netherite is a distributed workflow execution engine for Durable Functions (DF) and the Durable Task Framework (DTFx).

It is of potential interest to anyone developing applications on those platforms who has an appetite for performance, scalability, and reliability.

As Netherite is intended to be a drop-in backend replacement, it does not modify the application API. Existing DF and DTFx applications can switch to this backend with little effort.

Getting Started

To get started, you can either try out the sample, or take an existing DF app and switch it to the Netherite backend. You can also read our documentation.

The hello sample. For a quick start, take a look at hello sample. We included scripts that make it easy to build, run, and deploy this application. Also, this sample is a great starting point for creating your own projects.

Configure an existing DF app to use Netherite.…

View on GitHub

The use of Netherite requires some additional work, so we will briefly introduce how to use it.

Install Netherite

First, install the Microsoft.Azure.DurableTask.Netherite.AzureFunctions package in your Azure Functions project.

Install-Package Microsoft.Azure.DurableTask.Netherite.AzureFunctions -Version 0.4.0-alpha
dotnet add package Microsoft.Azure.DurableTask.Netherite.AzureFunctions --version 0.4.0-alpha

After installing the package, declare that you want to use Netherite in host.json. There are many configurations in the storageProvider, please refer to the official documentation.

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": false,
        "excludedTypes": "Request"
      }
    },
  },
  "extensions": {
    "durableTask": {
      "hubName": "<YourHubName>",
      "useGracefulShutdown": true,
      "storageProvider": {
        "type": "Netherite",
        "StorageConnectionName": "AzureWebJobsStorage",
        "EventHubsConnectionName": "EventHubsConnection"
      }
    }
  }
}

Set the connection string names for Storage and Event Hubs to the key names in the Azure Functions App Settings or local.settings.json.

Preserve System.Threading.Channels.dll

The current version 0.4.0-alpha does not work with an error when deployed to Azure Functions, so you need to add FunctionsPreservedDependencies to the csproj file to avoid this problem.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.DurableTask.Netherite.AzureFunctions" Version="0.4.0-alpha" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.5.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.13" />
  </ItemGroup>
  <ItemGroup>
    <FunctionsPreservedDependencies Include="System.Threading.Channels.dll" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Add a setting to prevent System.Threading.Channels.dll from being removed on deployment. This fix will allow it to work on Azure Functions.

This problem is expected to be fixed in a future version.

Create Function App and Event Hubs

Create Function App and Event Hubs Namespace by using Azure Portal or ARM Template.

  • Functions Premium (EP1-3)
    • Consumption Plan is not supported.
  • Event Hubs Namespace (Standard, 1TU)
    • Event Hubs are automatically created by Netherite

Since Netherite creates the Event Hub automatically, it is easier to use RootManageSharedAccessKey for the SAS key.

Use 64-bit worker process

FASTER needs to run in a 64-bit process, so change the Platform setting of Azure Functions to 64-bit.

If you run the application with the 32-bit Platform setting, the following exception will be thrown at runtime and it will not work properly.

Part** !!! Encountered exception while working on store in StoreWorker.Process: FASTER.core.FasterException: Overflow in AddressInfo - consider running the program in x64 mode for larger address space support

Note that when you deploy Azure Functions (Windows) from Azure Portal, 32-bit Platform will be selected by default.

Enable Runtime Scale Monitoring

Durable Functions with Netherite installed running on Functions Premium (EP1-3) need to have Runtime Scale Monitoring enabled in order to scale out.

Since Runtime Scale Monitoring requires warmed-up instances, set preWarmedInstanceCount to 1 when creating resources with ARM Template or Terraform.

After these steps, Azure Functions using Netherite is now working perfectly. Now you can verify the performance and have fun.

27

Tags:

AzureFunctionsSeverlessWorkflow