Streaming Salesforce Platform Events using gRPC and NodeJS
Salesforce has announced a pilot program, where they introduced a new way to communicate with their new Event Bus using the new “Pub/Sub API”.
As you may notice, there’s a lot of “new” in the sentence above, which means we will need to wait patiently, until this service is generally available to enjoy its full potential…
In the mean time, I’ve managed to gather the below details, that are publicly available today and have added a demo app with some thoughts based on my understanding.
I thought it would be a good time to play around with the underlying technology for this service (gRPC) and get a grasp of how this could potentially be leveraged. I’ve also built a small chat app using Lightning Web Components (LWC) to use as quick demo app (Instructions and links below).
TLDR;
I found out that gRPC can help to deliver a powerful real-time communication From and To a Salesforce org (EventBus).
Read more, if you are interested in real time communication between apps, event driven architecture with Salesforce and some cool server streaming capabilities.
The New Salesforce Event Bus
Let’s start first with the current publications around this new service.
As this article details, Salesforce decided to place and host their new Event Bus outside of the servers that are powering Sales Cloud and Service Cloud.
This change should bring us a few benefits:
- Allows to easily scale the service.
- It adopts micro-services architecture.
- It will be a Key part of how we’ll deliver new cross-cloud customer experiences.
The Pub/Sub API, will sit on top or basically acts as an interaction layer for the Event Bus (as a runtime engine).
- It’s based on gRPC, that allows us, to combine the power of a runtime engine with publish and subscribe in one simple interface.
- It will also provide a powerful client-driven flow control for event delivery.
What can it help us do better ?
The Pub/Sub API, as already alluded to above, will allow users to publish events, subscribe to events, request schema, request topic information and more…
- Basically, it eradicates the need for building out a custom CometD Streaming API client and also the need to use different API Interfaces or Protocols for various common tasks we need to do today.
The table below, shows the way we approach some tasks today using the range of API interfaces available. Where with the new Pub/Sub API, it’s all available within One simple and unified API Interface.
Some potential use cases:
- Subscribing to Event Monitoring events and publishing a Platform Event back into Salesforce to restrict a user’s profile when they log into Salesforce after working hours.
- Subscribing to Change Data Capture events and replicating order data in an external inventory system.
- Subscribing to a standard Platform Event like AppointmentSchedulingEvent and integrating with Google Calendar to update users’ calendars.
So what’s gRPC ?
Where did “gRPC” come from ?
- “RPC” — stands for “Remote processing calls”. The concept was introduced already back in the 90’s and has evolved since then.
- The “g” in “gRPC”, it’s kind of a secret…no one wants to shout it out, but it was originally pioneered by a team at Google (old project aka “stubby”).
- Today it’s an Open Source project with complete open specs and contributors. (Though development is still primarily executed by Google devs).
Overview of some of the API Patterns over time…
Over the years, we have seen a huge increase in volumes and a wide variety of client types (devices) that consume our API’s. This and obviously some ambition to tackle big problems from key players like Facebook and Google has led us to find all sorts of new and different approaches to handle the integration tasks.
Good old early adapters, love to watch a new player trying to come up with a new API approach and some are fast to declare, that the old way is dead and long live the new one.
The fact is really, that the API eco-system is still relatively small.
We need to be flexible and innovative in order to allow things to connect and talk easily.
The below link, shows some nice Pro’s and Con’s for each approach per use case.
In Short — We need to admit that it’s a journey, to provide a really fast and modern network communication which is fully optimised, scalable and easy to maintain.
Furthermore, there are also valid and interesting use cases where you would actually want to combine the techniques and use them altogether.
Main issues I could see that gRPC aims to solve:
- Strongly typed schema for request and response — our contract is well defined, kind of like SOAP/WSDL, but much easier to read, scale and maintain with advanced inteli-sense features in IDE.
- Language definition and Code generator to support different languages.
- HTTP/2 and gRPC allows a compressed and secured transaction of messages over the wire — its great for IOT and small devices with limited battery life.
- Bi-directional streaming in my opinion, would be one of the biggest features included in gRPC.
ProtoBuff
- “Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data — think XML, but smaller, faster, and simpler. ”
- It’s an Interface Definition Language (IDL) — You define how you want your data to be structured once. You can use special generated source code to easily write and read the structured data. Use it on a variety of data streams and compile it in a variety of languages.
- Can be used with more than 11 different coding languages.
The .proto file, provides us the “dictionary” by which the data is encoded and decoded to and from the Protocol Buffers binary format.
We will use it, as our schema definition for our request and response payloads from Salesforce.
The `proto` file will structure the data based on the schema definition, our gRPC server can then decode and encode our payload so data will be compact and easy to transmit.
On the side of our server file we have our gRPC client, which will read and push the data to our client apps, so it can render and visualise it as we wish.
Streaming a response to our client app
There are also several approaches for how to communicate notifications between our server to the client App.
While gRPC allows to stream the data on the server, in order to push those notifications to the client we will need to use one of the following:
Long Pole request/response
We could open a connection and pole for the results coming in using setTimeout() which will keep fetching data on a certain time interval.
Web-sockets
Web-Sockets offer bi-directional communication in real-time.
- Connections can both send and receive data from the client browser.
- Can transmit both binary data and UTF-8.
- When connections are terminated, Web-Sockets won’t automatically recover.
- Some enterprise firewalls with packet inspection have trouble dealing with Web-Sockets.
SSE/HTTP2
Through this interface, a client requests a particular URL to receive an event stream.
- Server-Sent Events (SSE) — Updates are pushed (rather than pulled, or requested) from a server to a browser.
- The connection is kept open (until it receives an instruction to be closed) by calling EventSource.close().
- Designed to use the JavaScript EventSource API to subscribe to a stream of data in any popular browser.
- Built-in support for re-connection and event-id and Transported over simple HTTP instead of a custom protocol.
Let’s Demo this in Action
This repository stores the full code.
The project holds 2 similar LWC apps, one in Salesforce org and the other is leveraging more or less the same code using LWC OSS running on localhost.
- Been trying to reuse the same code for both apps as much as possible here. gotta love re-usability.
- You’ll find that the main difference is only the one component that controls the Subscribe and Publish methods.
The Salesforce App, will use the `lightning/empApi` to Subscribe to the channel given by the user on the client and Apex to Publish events.
- The app is built like a messenger chat between 2 users (Salesforce app vs. localhost app). Each user will be able to send a message, that will Publish a Platform Event with some fields to describe the source of the message.
The OSS app will use Node JS and Express JS server setup, which we will add a gRPC server to run in parallel on a separate port.
The gRPC portion, at the moment (Till the Pub/Sub Api will be GA), will allow us to use JSForce connection to Subscribe and Publish Platform Events To and From Salesforce and display them in the form of a Chat.
- All operations will happen within One unified gRPC interface.
- Events will be pushed to the client app using Server Sent Events (SSE).
JSForce as of writing this article, is using cometD under the hood to subscribe and listen to the Streaming API.
Based on the details above, that’s exactly what should potentially change in the future and will allow a more streamlined way of consuming those events and probably more things to come…
Small Safe Harbor statement — The above article is only based on my understanding and interpretation of the technology. It’s not here to replace any official documentation and linked resources I found whilst researching on the web (Credit mentioned next to each link).
