• Log Replication And Distribution

Log Replication and Distribution

The log "stream" is a conceptual entity that crosses nodes and processes, identified by a unique handle (or name). All processes that log to stream S log to the SAME stream. Other processes can receive all the logs sent to the stream.

There are 2 additional features needed in the log system.

1. Distribution: Other applications (anywhere on the cluster) can receive every log as it is written.

2. Replication: Logs are stored on both system controllers, regardless of their origin. Given 1, you can see how replication can be implemented at the application level. However this is such a common case SAFplus will implement it automatically.

These features require higher level SAFplus services which may not exist or be initialized. Therefore these features need to be optionally enabled and be carefully constructed to not cause undefined symbols when a subset of SAFplus is deployed.

Initialization

To implement these features we will need to use the name and group messaging library. However, we also want log to work WITHOUT group (for situations where group has not been initialized yet and for cases where a subset of SAFplus is being utilized).

Therefore, I want you to create a new directory under log7 and a new library to implement these features. I made a directory called "rep" under log7 and please call the library libclLogRep.so.

There needs to be a separate initialization function in safplusInitialize, but not a separate identifier bit. Since logRep is essentially "glue" between log and messaging, logRep should be automatically initialized if log, Group and Name are initialized.

You will need to add "hooks" from the log to logRep so that log can call logRep IF it is initialized, but in such a way that the compiler does not automatically include logRep if log exists. One way to do this is to define a structure with the needed functions in the log service (in c++ you'd use a class with pure virtual functions). Create a global pointer to that structure but set it to NULL. Of course, the functions are only called if the pointer is not null...

When/if logRep is initialized, it sets the global pointer to non-null -- pointing to its implementations.

Configuration

It should be noted that we really have 2 objects here; log streams (input) and log spoolers (outputs). For full generality, we could create separate configuration and then require additional configuration to join an input to an output. However, for simplicity we will put a single input/output pair in a single configuration. Applications can always create log streams and log receivers manually if full functionality is needed.

Configuration is defined in the SAFplusLog.yang file

Log streams have a configuration enum called "replication" with the following values:

• NONE: No replication
• SYSTEM_CONTROLLERS: replicate to the system controllers
• APPLICATIONS: replicate to interested applications.
• ANY: replicate to system controllers and interested applications.

Note this can actually be a bit-field. Also, if even if SYSTEM_CONTROLLERS is defined the log server does not need to refuse applications that want notifications. So the actual implementation semantics are that SYSTEM_CONTROLLERS and ANY behave exactly the same (this is easier to implement, see proposed implementation below).

        leaf replicate {
          type enumeration {
            enum NONE { description "No replication of this log stream"; value 0; }
            enum SYSTEM_CONTROLLERS { description "Replicate to the system controllers"; value 1;}
            enum APPLICATIONS { description "Replicate to interested applications"; value 2; }
            enum ANY { description "Replicate to the both system controllers and interested applications"; value 3; }
            }
        }

Log stream replication behavior is defined in the ServerConfig object:

• maximumRecordsInPacket: this defines the largest packet allowed.
• processingInterval: this defines the longest time logs can be delayed for aggregation before sending.

In replicated mode, configuration like fileName, fileLocation, fileUnitSize, etc. will be used by the Log Spooler Object (see below) to define how the log is output on each node that receives it.

Operation

Log Stream Creation

When a log stream is created, register the stream name with the Name service with the stream's handle. This lets applications look up the stream by name. This should be done (if Group and LogRep is initialized) regardless of the setting in the replication field since this look up will allow applications to issue logs by stream name not just by handle.

1. Stream object doesn't include "stream handle". How can we determine it?

1. The stream handle should be registered with the Name service only once, however assume there are several nodes (system controllers and payloads), when they starts, the their log servers start, then the stream handle will be registered multiple times. How do we avoid this?

It is meant to be registered multiple times as the different processes start up and race each other. This is OK if the name/handle pair is the same. So if a name AND handle is provided, look up the name in the Name service. If it does not match the handle you were given raise an exception (call the exception NameHandleMismatch or something similar). If just the handle is provided, then open the stream by handle (don't worry about whether a name is associated or not).

If 2 objects have the same handle, they are representations of the SAME conceptual entity. What that exactly means will depend on the entity. But for efficiency, we require that the "replication" enum to be not None to trigger this behavior (If "replication" is None, logs will only come from the local node and the stream is output to a local source and nowhere else). From now on, I'm going to imagine that the replication flag is set...

If the stream is open on multiple nodes, it is the SAME stream. Let's say the stream object on node A is set up to output to foo.txt, and the object on node B is set up to write the logs to a TCP socket, and on node C it has no output (all of these stream object have the same handle, they are the SAME conceptual stream). Now if log("%s",nodename) is called on each node (A,B,C), I'd expect foo.txt to look like:

 A
 B
 C

And the TCP socket output to be the same. However, its OK if the order across nodes is not guaranteed. That is, ALL logs written to the stream go to ALL outputs of that stream.

If the "replication" enum is not None:

Use a well-known sub-handle of the log stream to create a group. The members in this group will be all entities that are interested in receiving the logs from this stream.

It won't actually. Yes of course each node will have a separate Group C++ object, but all these objects will act as a single Group (provided they use the same handle). Its kind of like 2 processes opening the same file...

If the "replication" enum is SystemControllers or All, then add the well-known system controller's log spooler object handle (see below) to the group.

To output to system controllers, application authors could implement a log The purpose of the log

Note that any application can join this group by using the Name service to look up the log stream by name, and then adding itself using the Group service. Applications can even join the group BEFORE the log stream is created if the log stream's handle is "well-known".

If the "replication" field is changed at any time during operation, the log stream must be updated dynamically to reflect the changed state.

Issuing Replicated Logs

When a log is written, the appropriate LogRep virtual function should be called with the text of the log if it exists and if the "replication" field is set.

Inside this virtual function, multiple logs will be serialized into a single message for efficiency. This should not happen as a single text string. Use a format that allows the logs to be broken up on the receiver side, and identifies information like endian and message version. For example:

 Header:
  int16 IdAndEndian;  // This is a well-known number
  int8  version;
  int8  extra;
  Handle streamHandle;  // Because one receiver could be subscribed to multiple streams.
  int16 numLogs;

 Body: (repeated sets of length, value)
  int16 logLength;
  char  log[logLength];
  int16 logLength;
  char  log2[logLength];
  .
  .
  .

The number of logs serialized per message should vary, depending on the number of logs being issued. A high rate of logging should result in a large number of logs per message. A low rate of logging should result in a single log per message. You can use a leaky bucket algorithm to structure this.

The max # of logs per message and the maximum time to delay before issuing a log are defined in configuration.

What other configuration fields are needed?

Once the log is formatted, actually issuing the message is simple. Use the Group message send API:

send(data, length,GroupMessageSendMode::SEND_BROADCAST);

Receiving Logs

Please look at the Object Message (clObjectMessager.hxx) infrastructure to discover how to receive log messages.

Log Spooler Object

The log spooler object exists on the system controllers (and any other node if an application explicitly instantiates one) inside the log server.

It receives logs using the Group infrastructure just like any application would.

When it receives logs, it writes them to disk using the exact SAME code as in log. Do not copy the logic that writes logs to disk, call a function in the log service. But we do not want to write these logs to shared memory, etc, so you may need to create a special API for issuing logs directly and refactor the log service a little bit to isolate the logic that writes logs to disk.

Testing and Examples

An example application should be created that creates a replicated log stream and writes to it in a loop with varying rates. The logs written should be different but follow an algorithm so that the receiver can validate that they are correct. For example, a pseudo-random number generator with a known seed could be used to create a stream of data that is verifiable.

So the log could be "test log 8 [qieddiek]" where 8 is a "randomly" generated length from 1 to 1024 (say), and "qieddiek" is a "randomly" generated string of that length.

In another thread or process (use a command line parameter to change operational behavior), it should register to receive logs from that stream, print them to the screen, and validate them by comparison with the same pseudo-random number generator and seed.

Please write the receiver portion clearly and with descriptive comments so that it can be used as an example for other applications that want to receive logs.