Polling Publications
The Poll endpoint allows you to create a connection to the DealCloud Publications system. This connection allows you to consume events from the message queue of your specified Publication topic(s) through the topics parameter.
A Polling connection to DealCloud Publications effectively spins up a "consumer" that can be used to consume event messages. The connection also supports a specified TimeOut and Count which can be used to define the connection.
It is important to note that a connection stays open until either the Count or the TimeOut is reached, whichever is satisfied first. Therefore, using different combinations of the TimeOut and Count parameters in a polling connection request can help achieve different types of designs and connections for varying usecases and requirements.
Polling Parameters
| Parameter | Description |
|---|---|
| topics | Specify the Publication topic(s) you want to create a polling connection to |
| TimeOut | Defines the maximum connection time (in ms) of the polling request |
| Count | Defines the number of messages the connection should stay open to "listen" for |
Tip: Let's say you create a polling connection, with a timeout of 60000 and a count of 5. Your connection will be initiated and will last for either 60 seconds or until 5 events are returned, whatever happens first. So for example, if the queue receives the 5th event at 15 seconds, it will close out the connection and respond with the 5 events at around 15 seconds.
At that point, you can begin processing those messages while re-initiating the polling connection to continue 'listening' for new messages.
Request
POST {pubUrl}/{{poll}}
Authorization: {{auth}}
Content-Type: application/json
{
"Topics": [
"deals-events"
],
"TimeOut": 15000,
"Count": 0
}Response Formats
Data events have some special configurations. You can choose to publish data events as Cells or Rows. See more about the types of event types and responses
Response Errors
Topic doesn't exist or no messages in topic yet
HTTP/1.1 400 Bad Request
"One or more topics does not exist in Kafka. Probably no messages has been sent into
that topic yet."More than 1 parallel connection
HTTP/1.1 429 Maximum parallel API calls quota exceeded! maximum admitted parallel calls set to 1.Design Recommendations
What does TimeOut do?
Establishes the maximum connection time. When the timeout is reached, a response is returned unless the count (the number of messages requested), is satisfied first.
What should my TimeOut be?
We recommend setting a high timeout, so your connection can stay open and "listen" for new messages.
Do Publication connections support long polling?
Yes, long polling is supported and encouraged.
How do I use count?
Count is the maximum number of events to be return in one request. You should use a relatively low count number unless anticipating a large number of messages. A high count value should always be temporary. If the specified Count is higher than the number of event messages in the queue, the connection will remain open until the timeout is satisfied.
How do TimeOut and count work together?
Count sets the maximum number of events to return in one request. If there are fewer events in the queue than the count value, the connection will remain open until it times out. If there are more events in the queue than the count value, you will get a response before the timeout.
How do we build the most responsive system?
It's tempting to make many requests with high count and low timeout. However, this will not guarantee you the best performance.
To build the most responsive, low-latency system where you have events as soon as they're published, you should specify a low count and a high timeout. The best design alternates low count high timeout with high count high timeout. In this design, you default to low count ahd when you receive a response that contains data, you can increase the count until you receive an empty response.
Recommendation for number of requests.
Keep in mind that many polling connections may return in an empty results, indicating there weren't any new events. The best design is to make a new connection call right after your previous call returns. When there are many events your calls will return more frequently and you will make many requests and when there are few events you will make less calls.
How many topics can I poll against?
While you can poll against multiple topics at once, it is recommended that you only poll against one topic per poll in a single request.