Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add Exporter interface
  • Loading branch information
Eric-Warehime committed Jun 28, 2022
commit aede594f47020cfe627208b8fd616b1d15ab9967
27 changes: 27 additions & 0 deletions exporters/exporter.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package exporters

import "github.com/algorand/go-algorand/rpcs"

// ExporterConfig is a generic type providing serialization/deserialization for exporter config files
type ExporterConfig interface{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be convinced otherwise, but I think this is a string.

Suggested change
type ExporterConfig interface{}
type ExporterConfig string

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we would have some serde module fetching configs from disk. The objects fetched from disk may be strings, but then we're going to want to marshal them into an equivalent of api.ExtraOptions, or idb.IndexerDbOptions.

Once we've constructed those objects, we'll need to merge in/validate/override properties that were set via the command line. Then whatever is left should get passed into the Exporter.

Let me know what your thoughts are on the above process for resolving cfg values...I haven't done any serious work on the config parsing yet so maybe it is a string, but it seemed like it wouldn't be to me given the above framework.


// Exporter defines the interface for plugins
type Exporter interface {
// Connect will be called during initialization, before block data starts going through the pipeline.
// Typically used for things like initializing network connections.
// The ExporterConfig passed to Connect will contain the Unmarhsalled config file specific to this plugin.
// Should return an error if it fails--this will result in the Indexer process terminating.
Connect(cfg ExporterConfig) error

// Shutdown will be called during termination of the Indexer process.
// There is no guarantee that plugin lifecycle hooks will be invoked in any specific order in relation to one another.
// Returns an error if it fails which will be surfaced in the logs, but the process is already terminating.
Shutdown() error

// Receive is called for each block to be processed by the exporter.
// Should return an error on failure--retries are configurable.
Receive(blockData *rpcs.EncodedBlockCert) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't quite right. Also, what do you think about wrapping it up in another object so that it's easier to add stuff without changing the function signature?

I'm also suggesting pass by value here, with the individual bits of data as pointers instead. We can make copies of the data object and allow plugins set each piece of data to nil if they want to (i.e. we are currently dropping the Certificate).

// ExportData is provided to the Exporter on each round.
type ExportData {
	// Block is the block data written to the blockchain.
	Block       *bookkeeping.Block

	// Delta contains a list of account changes resulting from the block. Processor plugins may have modify this data.
	Delta       *ledgercore.StateDelta

	// Certificate contains voting data that certifies the block. The certificate is non deterministic, a node stops collecting votes once the voting threshold is reached.
	Certificate *agreement.Certificate
}
Suggested change
Receive(blockData *rpcs.EncodedBlockCert) error
Receive(data ExportData) error


// Round returns the next round not yet processed by the Exporter. Atomically updated when Receive successfully completes.
Round() uint64
}