Overview

The SAFplus Management system provides an integrated NETCONF, SNMP, CLI (local and remote), local API, and web access platform whose data model presents on these northbound interfaces as a single management data configuration and statistics tree. Yet data can be stored and requests serviced by many separate applications running on many nodes in a cluster. Incoming requests from multiple protocol handlers are automatically translated into a standard high performance message request format and routed to the application that is currently servicing that portion of the data tree. Data routing is dynamic. Applications can "bind" to portions of the tree at any time, allowing data servicing to be handed from one application to another and for crashing applications to be seamlessly restarted or failed over.

This guide describes how to add SAFplus Management into a cluster. It assumes that cluster applications are already management-enabled. To learn how to enable your applications for management access, please refer to the SAFplus Management Developer's Guide.

This document was prepared using the Ubuntu 14.04 Linux distribution. Your distribution may be different.

Installation

This section describes how to install SAFplus Management from packages provided by OpenClovis.

You can also "install" from source code by acquiring the source code via tarball or our repository and then building it. In that case, the binaries will be located in a different place ...[source directory]/target/[machine architecture]/, so all paths specified in this document must be converted to this location.

First add our key to your package manager

wget -q -O - http://download.openclovis.com/files/debian/OpenClovis.asc | sudo apt-key add -

Next, add our repository

sudo add-apt-repository "deb http://download.openclovis.com/files/debian $(lsb_release -s -c) contrib" 

Update the list of software packages in the repository

sudo apt-get update

Finally, install SAFplus Management

sudo apt-get install safplus-mgt

This installation has placed SAFplus Management binaries, libraries and helper files into the directory /opt/safplus/<version>/mgt.

Running

This section describes how to run SAFplus Management either on the development or on a target machine.

Installation

If you have installed safplus_mgt from a .deb or .rpm package, it will be installed to /opt/safplus/<version>/mgt. If you are building SAFplus Mgt from source, on the development machine binaries and other files are generated in the "target base directory" (.../target/[architecture]/) subdirectory. They can be run directly from this location. Or to run on another machine, first move the target base directory and all subdirectories to any directory in the target machine using any technique (scp, archive, etc). But for the purposes of this document, we will assume that it has been installed to the package location (/opt/safplus/<version>/mgt).

The first step is to enable NETCONF access through ssh. We have created a simple script to accomplish this in Ubuntu 14.04:

sudo /opt/safplus/<version>/mgt/mgt_install

If this script will not work for you, you simply need to change your /etc/ssh/sshd_config configuration file and add:

      Port 830
      Subsystem   netconf /opt/safplus/<version>/mgt/bin/netconf-subsystem

Then restart ssh via:

      service ssh restart

We also find that it is sometimes necessary to explicitly kill the sshd process to get it to access the changed configuration file.

To verify that this worked, let's see what process id is listening to port 830 and then make sure that process is "sshd":

mint17 mgt # netstat -tulpn | grep :830
tcp        0      0 0.0.0.0:830             0.0.0.0:*               LISTEN      10931/sshd      
tcp6       0      0 :::830                  :::*                    LISTEN      10931/sshd      
mint17 mgt # ps 10931
  PID TTY      STAT   TIME COMMAND
10931 ?        Ss     0:00 /usr/sbin/sshd -D

Configuration Files

Configuration for the NETCONF protocol occurs just like in the OpenYuma project (via the netconfd.conf file). Please see the OpenYuma documentation for more information and look at .../examples/eval/websvc/netconfd.conf for an example. By default, the netconfd.conf file should be located at /opt/safplus/<version>/mgt/share/netconf/netconfd.conf, although this location can be overridden via command line arguments.

Initial values for "native" (not using the SAFplus back-end) OpenYuma YANG modules are set via the startup-cfg.xml file. This is simply an XML file that specifies data formatted as defined by YANG modules. In particular, NETCONF access control happens here, since it is controlled by the "nacm" module (ietf-netconf-acm.yang). For more information, see the OpenYuma project and documentation relating to the particular YANG module that specifies the data structure. For use as a template, a simple startup-cfg.xml file that turns on access control is included here:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm">
    <enable-nacm>true</enable-nacm>
  </nacm>
</config>

But this example is only useful as a template because access control rules must be defined. An full example of setting up the nacm (access control) module is located in .../examples/eval/websvc/startup-cfg.xml.

This file should be located in /opt/safplus/<version>/mgt/share/netconf/, or in a location specified by the netconfd.conf file.

Environment Variables

An Up-to-date script to set up the management runtime environment can be found in the examples/eval/websvc/setup directory. This section provides an overview of the process.

First, it is important to set up your LD_LIBRARY_PATH and PYTHONPATH variables to point to the location of binaries and libraries:

THISDIR=`pwd`
export LD_LIBRARY_PATH=`pwd`/lib:`pwd`/plugin:$LD_LIBRARY_PATH
export PYTHONPATH=`pwd`/lib:`pwd`/bin:`pwd`/test:$PYTHONPATH

Next, set up the SAFplus logging and communications variables. Of course you will need to change your interface to one appropriate for your machine:

# Logging controls
export CL_LOG_SEVERITY=ERROR
export CL_LOG_CODE_LOCATION_ENABLE=1

# Pick the interface that you are using to communicate between nodes in your cluster.
export SAFPLUS_BACKPLANE_INTERFACE=eth0

If you are just communicating to local processes, you might use the loopback interface instead of eth0 as shown above. But to do so with the UDP transport you must also enable broadcasts over the loopback:

export SAFPLUS_BACKPLANE_INTERFACE=lo
# To use UDP broadcasts on the loopback interface I need to enable a broadcast address
ifconfig lo broadcast 127.255.255.255

If you are running this management application within the SAFplus HA Platform, you will need to configure SAFplus Platform appropriately (setting ASP_NODENAME to a node specified in your HA database). See the SAFplus Platform documentation for more details.

For network intensive applications, you might also want to configure Linux to provide more resources to networking:

sysctl -w net.core.wmem_max=20485760
sysctl -w net.core.rmem_max=20485760
sysctl -w net.core.rmem_default=20485760
sysctl -w net.core.wmem_default=20485760

Execution

SAFplus Management can be run within the SAFplus Availability/Scalability Platform or in a standalone configuration. To run within the SAFplus Platform, the "safplus_mgt" and "netsnmpd" binaries should be added to your cluster's model inside a 1+1 redundancy group. For more information about this see the SAFplus Platform documentation. Like any SAFplus application, note that it is also possible to run the application within the SAFplus cluster but started manually simply by running the applications on the command line or in the debugger when safplus_amf is already running.

To run SAFplus Management without high availability, simply run the binary when safplus_amf is not running. It will detect that it is in standalone mode and handle some of the platform services that safplus_amf normally provides. These commands should be run as root since SAFplus was installed in /opt, so begin with:

sudo -i

Now we need to describe how to communicate with the management enabled processes. For clustered access use your backplane ethernet interface. For local access use the loopback interface, and enable it for broadcast packets:

ifconfig lo broadcast 127.255.255.255
export SAFPLUS_BACKPLANE_INTERFACE=lo

Set up paths and environment variables and then start the safplus_mgt and netsnmpd binaries. In this case we'll use provided scripts:

cd /opt/safplus/7.0/mgt
source ./mgt_setup
./mgt_start

Testing

At this point, you should be able to access the system via NETCONF. You can use any NETCONF client that can handle NETCONF over SSH. For example, to test access via the SAFplus CLI, run:

safplus_cli

Then within the CLI run:

connect <any_name_you_choose> <user_name> <password> <machine's IP>

For example, if your system is configured to allow root SSH logins you could do:

connect localAdmin root <password> localhost

Next, "cd" to that machine and display the YANG data:

cd localAdmin
ls

If you are running the "websvc" example, you should see something like:

/networkElement/localAdmin:
  localAdmin (localhost)
    arp
    nacm
    safplusAmf
    serviceCfg
    serviceStats
    Andrew
    David

Otherwise, you will probably just see the arp, nacm and safplusAmf entities.

The SAFplus CLI is much more powerful than showin in this simple example. Please see the SAFplus Management CLI Guide for details.

Troubleshooting

• NETCONF clients connect and then disconnect instantly

You have either not enabled NETCONF in sshd_config, or the binding program does not exist. Add this to /etc/ssh/sshd_config and restart:

Port 830
Subsystem   netconf <replace with your path to SAFplus>/bin/netconf-subsystem

• Assertion `!"Cannot load dbal plugin"' failed.

You have not selected a database layer. copy libclSQLiteDB.so (or other layer) to /opt/safplus/7.0/mgt/plugin

• Assertion `!"Required env variable [SAFPLUS_BACKPLANE_INTERFACE] not defined"' failed.

You have not selected a network interface to communicate with management enabled applications. To access local apps only, try:

export SAFPLUS_BACKPLANE_INTERFACE=lo
ifconfig lo broadcast 127.255.255.255

• Assertion `!"Cannot load message transport plugin"' failed.

You have not selected a network transport to communicate with management enabled applications. Copy clMsgUdp.so to /opt/safplus/7.0/mgt/plugin

• My YANG modules are not visible in the CLI but 'standard' (e.g. nacm) modules are visible

Modules that use the SAFplus back end are handled specially. They must be copied to the directory specified by the SAFPLUS_MODULES_PATH environment variable. See Environment Variables