Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
6c781cd
Update Apache Proton-J dependency (0.29.0 --> 0.31.0) (#407)
sjkwak Dec 13, 2018
71f8562
PartitionReceiver - add a method that provides an EventPosition which…
sjkwak Dec 13, 2018
0dd058d
Support IsPartitionEmpty property for PartitionRuntimeInformation (#399)
sjkwak Dec 14, 2018
0b7ca03
Move setPrefetchCount API to the ReceiverOptions class from the Parti…
sjkwak Dec 18, 2018
d1b68d1
Fixes several issues in the reactor related components (#411)
sjkwak Dec 21, 2018
9d04a98
Implement comparable on EventData (#395)
Fokko Dec 21, 2018
c37c848
Update receive/send link creation logic and improve tracing (#414)
sjkwak Jan 2, 2019
fec5a03
Prep for releasing client 2.0.0 and EPH 2.2.0 (#415)
sjkwak Jan 2, 2019
9305f06
Ensure that links are closed when transport error occurrs (#417)
sjkwak Jan 5, 2019
e6ef7f0
Prep for releasing client 2.1.0 and EPH 2.3.0 (#418)
sjkwak Jan 10, 2019
11c68b0
Update prefetch sendflow logic and increment version for new release …
sjkwak Jan 10, 2019
a1fa1a5
Fix args for proxy auth call to Authenticator (#421)
JamesBirdsall Jan 10, 2019
5fbdbe4
Prepare EPH 2.3.4 release (#423)
sjkwak Jan 15, 2019
3ddd268
Prepare EPH 2.4.0 release (#423) (#424)
sjkwak Jan 15, 2019
120231b
Handle proton:io errors with meaningful error msg (#427)
JamesBirdsall Jan 29, 2019
5826288
Minor changes to lease scanner (#428)
JamesBirdsall Feb 6, 2019
6edab2c
Make EventData.SystemProperties completely public (#435)
JamesBirdsall Mar 8, 2019
07b4072
Digest Support: init first connection with null headers (#431)
mssfang Mar 11, 2019
f4827ab
Fix lease scanner issues when Storage unreachable (#434)
JamesBirdsall Mar 12, 2019
50f99ca
message receiver - fix null pointer error and ensure that receive lin…
sjkwak Apr 4, 2019
325d471
Update version numbers for release (#440)
JamesBirdsall Apr 5, 2019
b2d7507
Update prefetch count for a receiver (#441)
sjkwak Apr 9, 2019
f5da494
Fix an issue of creating multiple sessions for $management & $cbs cha…
sjkwak Apr 15, 2019
830c255
Update version numbers for new release (#444)
sjkwak May 2, 2019
8de8dff
Merge commit '830c25509357e62a22d0d9bf889ea39169081038' into mergeLatest
conniey May 2, 2019
5ea92fa
Update spotbugs.xml report versions
conniey May 2, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions eng/spotbugs-aggregate-report/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@

<properties>
<azure-batch.version>5.0.1</azure-batch.version>
<azure-eventhubs.version>2.3.0</azure-eventhubs.version>
<azure-eventhubs-eph.version>2.5.0</azure-eventhubs-eph.version>
<azure-eventhubs.version>2.3.1</azure-eventhubs.version>
<azure-eventhubs-eph.version>2.5.1</azure-eventhubs-eph.version>
<azure-keyvault.version>1.2.0</azure-keyvault.version>
<azure-servicebus.version>2.0.0</azure-servicebus.version>
<azure-storage-blob.version>10.5.0</azure-storage-blob.version>
Expand Down
206 changes: 103 additions & 103 deletions eventhubs/data-plane/ConsumingEvents.md

Large diffs are not rendered by default.

130 changes: 65 additions & 65 deletions eventhubs/data-plane/PublishingEvents.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,33 @@
# Publishing Events with the Java client for Azure Event Hubs
# Publishing Events with the Java client for Azure Event Hubs

The vast majority of Event Hub applications using this and the other client libraries are and will be event publishers.
The vast majority of Event Hub applications using this and the other client libraries are and will be event publishers.
And for most of these publishers, publishing events is extremely simple and handled with just a few API gestures.

## Getting Started

This library is available for use in Maven projects from the Maven Central Repository, and can be referenced using the
following dependency declaration inside of your Maven project file:
following dependency declaration inside of your Maven project file:

```XML
<dependency>
<groupId>com.microsoft.azure</groupId>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs</artifactId>
<version>2.3.0</version>
<version>2.3.1</version>
</dependency>
```

For different types of build environments, the latest released JAR files can also be [explicitly obtained from the
Maven Central Repository](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22azure-eventhubs%22) or from [the Release distribution point on GitHub](https://github.com/Azure/azure-event-hubs/releases).

For different types of build environments, the latest released JAR files can also be [explicitly obtained from the
Maven Central Repository](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22azure-eventhubs%22) or from [the Release distribution point on GitHub](https://github.com/Azure/azure-event-hubs/releases).


For a simple event publisher, you'll need to import the *com.microsoft.azure.eventhubs* package for the Event Hub client classes.


For a simple event publisher, you'll need to import the *com.microsoft.azure.eventhubs* package for the Event Hub client classes.


```Java
import com.microsoft.azure.eventhubs.*;
```

Event Hubs client library uses qpid proton reactor framework which exposes AMQP connection and message delivery related
Event Hubs client library uses qpid proton reactor framework which exposes AMQP connection and message delivery related
state transitions as reactive events. In the process,
the library will need to run many asynchronous tasks while sending and receiving messages to Event Hubs.
So, `EventHubClient` requires an instance of `Executor`, where all these tasks are run.
Expand All @@ -37,63 +37,63 @@ So, `EventHubClient` requires an instance of `Executor`, where all these tasks a
ScheduledExecutorService executor = Executors.newScheduledThreadPool(8)
```

Using an Event Hub connection string, which holds all required connection information including an authorization key or token
(see [Connection Strings](#connection-strings)), you then create an *EventHubClient* instance.
Using an Event Hub connection string, which holds all required connection information including an authorization key or token
(see [Connection Strings](#connection-strings)), you then create an *EventHubClient* instance.

```Java
ConnectionStringBuilder connStr = new ConnectionStringBuilder()
.setNamespaceName("----ServiceBusNamespaceName-----")
.setEventHubName("----EventHubName-----")
.setSasKeyName("-----SharedAccessSignatureKeyName-----")
.setSasKey("---SharedAccessSignatureKey----");
.setSasKey("---SharedAccessSignatureKey----");

EventHubClient ehClient = EventHubClient.createSync(connStr.toString(), executor);
```

Once you have the client in hands, you can package any arbitrary payload as a plain array of bytes and send it. The samples
we use to illustrate the functionality send a UTF-8 encoded JSON data, but you can transfer any format you wish.
Once you have the client in hands, you can package any arbitrary payload as a plain array of bytes and send it. The samples
we use to illustrate the functionality send a UTF-8 encoded JSON data, but you can transfer any format you wish.

```Java
EventData sendEvent = EventData.create(payloadBytes);
ehClient.sendSync(sendEvent);
```
The entire client API is built for Java 8's concurrent task model, generally returning

The entire client API is built for Java 8's concurrent task model, generally returning
[*CompletableFuture<T>*](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html), so the library has these methods suffixed with *Sync* as their Synchronous counterparts/varaints.

## AMQP 1.0
Azure Event Hubs allows for publishing events using the HTTPS and AMQP 1.0 protocols. The Azure Event Hub endpoints
also support AMQP over the WebSocket protocol, allowing event traffic to leverage the same outbound TCP port as
HTTPS.
also support AMQP over the WebSocket protocol, allowing event traffic to leverage the same outbound TCP port as
HTTPS.

This client library is built on top of the [Apache Qpid Proton-J]() libraries and supports AMQP, which is significantly
more efficient at publishing event streams than HTTPS. AMQP 1.0 is an international standard published as ISO/IEC 19464:2014.
This client library is built on top of the [Apache Qpid Proton-J]() libraries and supports AMQP, which is significantly
more efficient at publishing event streams than HTTPS. AMQP 1.0 is an international standard published as ISO/IEC 19464:2014.

AMQP is session-oriented and sets up the required addressing information and authorization information just once for each
send link, while HTTPS requires doing so with each sent message. AMQP also has a compact binary format to express common
event properties, while HTTPS requires passing message metadata in a verbose text format. AMQP can also keep a significant
number of events "in flight" with asynchronous and robust acknowledgement flow, while HTTPS enforces a strict request-reply
AMQP is session-oriented and sets up the required addressing information and authorization information just once for each
send link, while HTTPS requires doing so with each sent message. AMQP also has a compact binary format to express common
event properties, while HTTPS requires passing message metadata in a verbose text format. AMQP can also keep a significant
number of events "in flight" with asynchronous and robust acknowledgement flow, while HTTPS enforces a strict request-reply
pattern.

AMQP 1.0 is a TCP based protocol. For Azure Event Hubs, all traffic *must* be protected using TLS (SSL) and is using
TCP port 5671.
AMQP 1.0 is a TCP based protocol. For Azure Event Hubs, all traffic *must* be protected using TLS (SSL) and is using
TCP port 5671.

This library will provide HTTPS support via WebSockets when Proton-J supports HTTPS.

## Connection Strings

Azure Event Hubs and Azure Service Bus share a common format for connection strings. A connection string holds all required
information to set up a connection with an Event Hub. The format is a simple property/value list of the form
{property}={value} with pairs separated by ampersands (&).
information to set up a connection with an Event Hub. The format is a simple property/value list of the form
{property}={value} with pairs separated by ampersands (&).

| Property | Description |
|-----------------------|------------------------------------------------------------|
|-----------------------|------------------------------------------------------------|
| Endpoint | URI for the Event Hubs namespace. Typically has the form *sb://{namespace}.servicebus.windows.net/* |
| EntityPath | Relative path of the Event Hub in the namespace. Commonly this is just the Event Hub name |
| EntityPath | Relative path of the Event Hub in the namespace. Commonly this is just the Event Hub name |
| SharedAccessKeyName | Name of a Shared Access Signature rule configured for the Event Hub or the Event Hub name. For publishers, the rule must include "Send" permissions. |
| SharedAccessKey | Base64-encoded value of the Shared Access Key for the rule |
| SharedAccessSignature | A previously issued Shared Access Signature token (not yet supported; will be soon) |

A connection string will therefore have the following form:

```
Expand All @@ -102,42 +102,42 @@ A connection string will therefore have the following form:

## Advanced Operations

The publisher example shown in the overview above sends an event into the Event Hub without further qualification. This is
the preferred and most flexible and reliable option. For specific needs, Event Hubs offers two extra options to
qualify send operations: Publisher policies and partion addressing.
The publisher example shown in the overview above sends an event into the Event Hub without further qualification. This is
the preferred and most flexible and reliable option. For specific needs, Event Hubs offers two extra options to
qualify send operations: Publisher policies and partion addressing.

### Partition Addressing

Any Event Hub's event store is split up into at least 4 partitions, each maintaining a separate event log. You can think
of partitions like lanes on a highway. The more events the Event Hub needs to handle, the more lanes (partitions) you have
to add. Each partition can handle at most the equivalent of 1 "throughput unit", equivalent to at most 1000 events per
Any Event Hub's event store is split up into at least 4 partitions, each maintaining a separate event log. You can think
of partitions like lanes on a highway. The more events the Event Hub needs to handle, the more lanes (partitions) you have
to add. Each partition can handle at most the equivalent of 1 "throughput unit", equivalent to at most 1000 events per
second and at most 1 Megabyte per second.

In some cases, publisher applications need to address partitions directly in order to pre-categorize events for consumption.
A partition is directly addressed either by using the partition's identifier or by using some string (partition key) that gets
A partition is directly addressed either by using the partition's identifier or by using some string (partition key) that gets
consistently hashed to a particular partition.

This capability, paired with a large number of partitions, may appear attractive for implementing a fine grained, per publisher
This capability, paired with a large number of partitions, may appear attractive for implementing a fine grained, per publisher
subscription scheme similar to what Topics offer in Service Bus Messaging - but it's not at all how the capability should be used
and it's likely not going to yield satisfying results.
Partition addressing is designed as a routing capability that consistently assigns events from the same sources to the same partition allowing
downstream consumer systems to be optimized, but under the assumption of very many of such sources (hundreds, thousands) share
the same partition. If you need fine-grained content-based routing, Service Bus Topics might be the better option.
and it's likely not going to yield satisfying results.

Partition addressing is designed as a routing capability that consistently assigns events from the same sources to the same partition allowing
downstream consumer systems to be optimized, but under the assumption of very many of such sources (hundreds, thousands) share
the same partition. If you need fine-grained content-based routing, Service Bus Topics might be the better option.

#### Using Partition Keys

Of the two addressing options, the preferable one is to let the hash algorithm map the event to the appropriate partition.
The gesture is a straightforward extra override to the send operation supplying the partition key:
The gesture is a straightforward extra override to the send operation supplying the partition key:

```Java
EventData sendEvent = EventData.create(payloadBytes);
> ehClient.sendSync(sendEvent, partitionKey);
```

#### Using Partition Ids

If you indeed need to target a specific partition, for instance because you must use a particular distribution strategy,
If you indeed need to target a specific partition, for instance because you must use a particular distribution strategy,
you can send directly to the partition, but doing so requires an extra gesture so that you don't accidentally choose this
option. To send to a partition you explicitly need to create a client object that is tied to the partition as shown below:

Expand All @@ -151,24 +151,24 @@ option. To send to a partition you explicitly need to create a client object tha
#### Publisher Policies

Event Hub Publisher Policies are not yet supported by this client and will be supported in a future release.

#### Special considerations for partitions and publisher policies

Using partitions or publisher policies (which are effectively a special kind of partition key) may impact throughput
and availability of your Event Hub solution.
Using partitions or publisher policies (which are effectively a special kind of partition key) may impact throughput
and availability of your Event Hub solution.

When you do a regular send operation that does not prescribe a particular partition, the Event Hub will choose a
partition at random, ensuring about equal distribution of events across partitions. Sticking with the above analogy,
all highway lanes get the same traffic.
When you do a regular send operation that does not prescribe a particular partition, the Event Hub will choose a
partition at random, ensuring about equal distribution of events across partitions. Sticking with the above analogy,
all highway lanes get the same traffic.

If you explicitly choose the partition key or partition-id, it's up to you to take care that traffic is evenly
distributed, otherwise you may end up with a traffic jam (in the form of throttling) on one partition while there's
little or no traffic on another partition.
If you explicitly choose the partition key or partition-id, it's up to you to take care that traffic is evenly
distributed, otherwise you may end up with a traffic jam (in the form of throttling) on one partition while there's
little or no traffic on another partition.

Also, like every other aspect of distributed systems, the log storage backing any partition may rarely and briefly slow
Also, like every other aspect of distributed systems, the log storage backing any partition may rarely and briefly slow
down or experience congestion. If you leave choosing the target partition for an event to Event Hubs, it can flexibly
react to such availability blips for publishers.
react to such availability blips for publishers.

Generally, you should *not* use partitioning as a traffic prioritization scheme, and you should *not* use it
for fine grained assignment of particular kinds of events to a particular partitions. *Partitions are a load
Generally, you should *not* use partitioning as a traffic prioritization scheme, and you should *not* use it
for fine grained assignment of particular kinds of events to a particular partitions. *Partitions are a load
distribution mechanism, not a filtering model*.
4 changes: 2 additions & 2 deletions eventhubs/data-plane/azure-eventhubs-eph/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@
<parent>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-clients</artifactId>
<version>2.3.0</version>
<version>2.3.1</version>
<relativePath>../pom.xml</relativePath>
</parent>

<modelVersion>4.0.0</modelVersion>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-eph</artifactId>
<version>2.5.0</version>
<version>2.5.1</version>

<name>Microsoft Azure SDK for Event Hubs Event Processor Host(EPH)</name>
<description>EPH is built on top of the Azure Event Hubs Client and provides a number of features not present in that lower layer</description>
Expand Down
2 changes: 1 addition & 1 deletion eventhubs/data-plane/azure-eventhubs-extensions/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<parent>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-clients</artifactId>
<version>2.3.0</version>
<version>2.3.1</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion eventhubs/data-plane/azure-eventhubs/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<parent>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-clients</artifactId>
<version>2.3.0</version>
<version>2.3.1</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ private void parseConnectionString(final String connectionString) {
this.transportType = TransportType.fromString(values[valueIndex]);
} catch (IllegalArgumentException exception) {
throw new IllegalConnectionStringFormatException(
String.format("Invalid value specified for property '%s' in the ConnectionString.", TRANSPORT_TYPE_CONFIG_NAME),
String.format(Locale.US, "Invalid value specified for property '%s' in the ConnectionString.", TRANSPORT_TYPE_CONFIG_NAME),
exception);
}
} else {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ public final class ClientConstants {
public static final String NO_RETRY = "NoRetry";
public static final String DEFAULT_RETRY = "Default";
public static final String PRODUCT_NAME = "MSJavaClient";
public static final String CURRENT_JAVACLIENT_VERSION = "2.3.0";
public static final String CURRENT_JAVACLIENT_VERSION = "2.3.1";
public static final String PLATFORM_INFO = getPlatformInfo();
public static final String FRAMEWORK_INFO = getFrameworkInfo();
public static final String CBS_ADDRESS = "$cbs";
Expand Down
2 changes: 1 addition & 1 deletion eventhubs/data-plane/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs-clients</artifactId>
<packaging>pom</packaging>
<version>2.3.0</version>
<version>2.3.1</version>

<name>Microsoft Azure Event Hubs SDK Parent</name>
<description>Java libraries for talking to Windows Azure Event Hubs</description>
Expand Down
Loading