Add an outbound connector

An outbound connector is a Spring Boot application. The starting point is a configuration class annotated with:

• @SpringBootApplication
• @OutboundConnector(name = "outbound-connector-name")

The core module scans the classpath for all classes residing in the energy.eddie package and annotated with the @OutboundConnector-annotation. For the outbound connector to be on the classpath, it needs to be added as a dependency to the core module. Outbound connectors are started in their own spring context and dispatcher servlet, which is registered in the parent context (core module). The outbound connector will behave like it is its own spring application with a few deviations. First, all beans defined in the core module are available in the child contexts. If the core defines a bean, it will be automatically available in the outbound connector. For more information, see section Beans of interest Second, each outbound connector runs in a dispatcher servlet, meaning it is possible to add Spring WebMVC Controllers. This is described in section Dispatcher-Servlet Third, outbound connectors have to be explicitly enabled, otherwise they will not be started by the core module. More on that in subsubsection the enabled property.

A short example for a minimal outbound connector:

import energy.eddie.api.agnostic.outbound.OutboundConnector;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
@OutboundConnector(name = "example")
public class ExampleOutboundConnector {
}

Interfaces

An outbound connector can support multiple interfaces. The interfaces are separated into two packages:

• energy.eddie.api.agnostic.outbound: The interfaces in this package are not bound by a version
• energy.eddie.api.v0_82.outbound: The current version of the CIM (Common Information Model)

@OutboundConnector

The @OutboundConnector-annotation denotes the starting point of an outbound connector. Must be used in combination with the @SpringBootApplication-annotation. The class annotated with the @OutboundConnector-annotation will be used to start the Spring context of the outbound connector. Furthermore, it is used to set the name of the outbound connector.

ConnectionStatusMessageOutboundConnector

The ConnectionStatusMessageOutboundConnector interface provides means to get a stream of connection status messages, which are emitted to the eligible party.

RawDataOutboundConnector

The RawDataOutboundConnector interface provides a stream of raw data messages. Raw data messages are messages that are received from region connectors and their region as is, without any changes, which can be useful for debugging purposes or operating on the data provided by metered data administrators itself.

PermissionMarketDocumentOutboundConnector

The PermissionMarketDocumentOutboundConnector interface provides a stream of permission market documents. Their purpose is similar to the connection status messages described in subsubsection ConnectionStatusMessageOutboundConnector. They provide information about status changes of a certain permission request in a CIM compliant format. For more information see section permission market documents.

ValidatedHistoricalDataMarketDocumentOutboundConnector

The ValidatedHistoricalDataMarketDocumentOutboundConnector interface provides a stream of validated historical data market documents. These are CIM compliant documents containing metered data, for more information see section validated historical data market documents.

AccountingPointEnvelopeOutboundConnector

The AccountingPointEnvelopeOutboundConnector interface provides a stream of accounting point market documents. The accounting point market documents are CIM compliant documents, for more information see section accounting point market documents.

TerminationConnector

The TerminationConnector interface provides the eligible party with means to change the status of a permission request. If a permission request has the status accepted, the eligible party can terminate a permission request by sending a termination document, which is a permission market document. See subsection termination documents and the permission process model documentation for more information. This interface does not produce any documents, but receives them.

RetransmissionOutboundConnector

WARNING

This interface is still work in progress.

The RetransmissionOutboundConnector interface provides the eligible party with means to request retransmission of data from a permission request. This interface receives retransmission requests and passes them on to the region connectors. The interface also informs about the result of the retransmission request in the form of a retransmission results. These results are only specific to permission requests and no order is guaranteed. Aka when sending multiple retransmission request for the same permission request there is no way to associate a retransmission result with a specific retransmission request, only with the permission request.

Currently, there are no cim documents defined for retransmission requests. So the internal RetransmissionRequest and RetransmissionResult can be used to send and receive retransmission requests.

General Topic Structure

To be compatible with different kings of messaging infrastructure (Kafka, NATS, RabbitMQ, Pulsar) the topic names (or queues, streams, whatever it's named in the respective messaging infrastructure) are made of a prefix and a suffix. Depending on the actual messaging solution, they are composed differently.

• Prefix: <direction>.eddie, e.g. ep.eddie
• Suffix: <datamodel/version>.<topic>, e.g. cim_0_82.permission-md

Applied to the different solutions:

• Kafka: Kafka-Topic: <Prefix>.<Suffix>, e.g. ep.eddie.cim_0_82.permission-md
• AMQP1.0 (out): Topic: <Prefix>.<Suffix>, e.g.: topic://ep.eddie.cim_0_82.permission-md
• AMQP1.0 (in): Queue: <Prefix>.<Suffix>, e.g.: queue://fw.eddie.cim_0_82.termination-md
• NATS: Subject: <Prefix>.<Suffix>, e.g. ep.eddie.cim_0_82.permission-md
• Pulsar: Tenant: configurable, using "eddie.app" as default; Namespace: <Prefix>; Topic <Suffix>, e.g. persistent://eddie.app/ep.eddie/cim_0_82.permission-md

Topic names are composed of the following fields:

Field	Values	Description
direction	ep / fw	ep: message from the EDDIE fw to the elibible party app, fw: message for the EDDIE fw
EDDIE-id	eddie	A constant prefix to group all eddie framework related topics
datamodel	cim_0_82 / cim_0_91_08	Datamodel combined with version
document type	see below	Type of document exchanged

CIM Topic Structure

CIM Document types:

Document type	Direction	Description
termination-md	fw	used to terminate a connection / subscription
permission-md	ep	informs about changes of the current permission status
accounting-point-md	ep	contains accounting point metadata from the MDA
validated-historical-data-md	ep	contains validated historical consumption data (VHCD) from the MDA
redistribution-transaction-rd	fw	contains VHCD updates to already received VHCD documents cim_1_0 only
near-real-time-data-md	ep	contains real time data from an AIIDA instance

• -md: market document
• -rd: request document

Shared Functionality

A lot of protocols have similar concepts, for example, AMQP's queue and Apache Kafka's topics. To generalize a few aspects, the outbound-shared module can be used. It provides classes for consistent naming or serialization and deserialization(SerDe).

Names

To get the names for endpoints, which can be HTTP endpoints, AMQP Queues, or Kafka topics, the TopicStructure class can be used. For names for headers, such as HTTP headers, Kafka headers, or AMQP properties, the Headers class provides constants for that. Of course, the endpoints and headers are not limited to those values, but it should provide standard names for endpoints/headers which provide the same data.

Serialization/Deserialization

For common formats, there are already serializers and deserializers in place, which can be reused.

• For JSON use the JsonMessageSerde class.
• For XML use the JsonMessageSerde class. Supports CIM documents as well as unknown types, which are serialized using the XmlMapper from jackson.

Custom SerDe

To implement a custom SerDe for other formats, such as CSV or protobuf, implement the MessageSerde interface and either extend the DefaultSerdeFactory or implement a custom SerdeFactory.

Security Configuration

Outbound Connectors that should be secured using spring security (like the Admin Console), can define a SecurityFilterChain using the OutboundConnectorSecurityConfig Annotation. The endpoints can then be secured as follows, but please note that the security config should only define rules for the paths of the certain outbound connector.

@OutboundConnectorSecurityConfig
public class OCSecurityConfig {
    @Bean
    public MvcRequestMatcher.Builder requestMatcher(HandlerMappingIntrospector introspector) {
        return new MvcRequestMatcher.Builder(introspector).servletPath("/" + ALL_OUTBOUND_CONNECTORS_BASE_URL_PATH + "/" + "oc-name");
    }

    @Bean
    public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        return http
                .csrf((csrf) -> csrf.requireCsrfProtectionMatcher(requestMatcher.pattern("*")))
                .authorizeHttpRequests((authorize) -> authorize
                        .requestMatchers(requestMatcher.pattern("**")).authenticated()
                        .anyRequest().permitAll()
                )
                // ... formLogin, oauth2Login, etc.
                .build();
    }
}

IMPORTANT

The filter chains are loaded by eddie core before the context of the outbound connector is built. Consequently, Beans that are part of the outbound connector are not available in the security config. Nevertheless, you can access e.g. @Values that are instantiated by eddie core.

Configuration

Since the outbound connector is a Spring Boot application, it is possible to load configurations from all sources as usual in Spring Boot. It is possible to use the @Value-annotation to get configuration values, as well as using @ConfigurationProperties to configure the outbound connector.

Naming conventions

While the names of the configuration properties are arbitrary the naming convention is to use outbound-connector.<outbound-connector-name>.<property-name>. This is useful because it helps differentiate between the configuration of different outbound connectors. In some cases this can be ignored, for example, when libraries are used, which read the properties themselves. In that case, it is often not possible to use the naming convention

Special configuration properties

There are a few predefined properties, which are either needed by the core module or used by convention.

The enabled property

Important: There is one special configuration property that must be set to true to enable the outbound connector. Otherwise, the outbound connector will not be started.

outbound-connector.<outbound-connector-name>.enabled=true

# Example
outbound-connector.kafka.enabled=true

The format property

The format property is a convention that allows users of the outbound connector to specify in which format the outbound connector should send its data. While some outbound connectors, like the admin console do not use that property, because they are showing the data in a human-readable format, others might support different formats in which data is produced and consumed. The Kafka outbound connector supports json and xml.

Outbound connectors that send data in a machine-readable format can benefit from defining and using this property, but the core module does not enforce the usage of the format property. There is no rule on which formats need to be supported, but most interfaces and their payload are intended to be serialized to JSON or XML.

outbound-connector.<outbound-connector-name>.format=format

# Example
outbound-connector.kafka.format=xml

Dispatcher Servlet (web interface)

Each outbound connector is started in its own spring context and runs in a separate dispatcher servlet. This means that each outbound connector can serve data via HTTP to configure it or even use HTTP to send and receive documents. The outbound connector is available under <hostname>:<eddie.management.server.port>/outbound-connectors/<outbound-connector-name>.

Beans of interest

The following beans are registered in the parent context and are available in the child contexts.

DataNeedsService

The DataNeedsService allows the outbound connector to query data needs, described in section data need configuration. This allows the outbound connector to implement complex routing logic based on the data need.