$Id: khneia_outline.txt,v 1.3 2004/01/07 05:11:47 nathan Exp $


khneia implementation
----------------------
(in glorious ASCII art)


The system consists of 2 main components. The master daemon and the client 
daemon. The master daemon will send heart beat requests and tasks to the 
clients. The client daemon will listen and respond to the master server's 
request. 

Client daemon
-------------

Once the client is launched, the first action performed is to register
with the master daemon. After registration, the client will listen for 
heart beat and tasks sent by the master daemon. 

This is the basic module outline for the khneia client.

     ----------
     | Client |
     ----------
	|
	-> sys_conf (read configuration files and set up options)
	|
	-> client_register (register the server onto the system)
	|
	-> client_listener (listen for heart beats and tasks)
	|
	-> khneia_sender (will properly send information to the server)
	|
	-> host_info (server information and stats)
	|
	-> disk_stats (disk usage information)
	|
	-> system_stats (memory and cpu information)
	|
	-> task (perform given tasks, not yet implemented)


sys_conf
--------

This module is responsible for reading the configuration file. The 
configuration file path must be set at compilation time. The default 
path is:

     /usr/local/etc/khneia_client.conf

The configuration file will contain the options for the client
daemon. The options are listed by <option> = <value>. When writing
options or values the <> brackets are not used. Comments are allowed
and everything after the octothorp (#) will be ignored until the
newline character \n. Also note for convenience the options are not
case sensitive, however the values are !

server     = <server name or ip> # server host name or IP address
serverport = <server port>       # server port default 8888
listenport = <listen port>	 # listen port default 8889
rsyncpath  = <path to rsync>     # path to the rsync binary


The sys_conf module will be written in flex++ for future
extendibility. Options and other configuration items will be very
easily added by adding simple lex rules. The module will have accessor
methods for each value and can be used at any point in the programs
runtime.

If the client daemon is running and the options are changed. The client 
daemon can be signaled to re-read the configuration file with the HUP 
signal.


register
--------

The register module will send the initial information to the server. When 
this initial information is sent. The sequence number will be set to 0. 
The register information will include the following packets:
	
	* Host information packet
	* System status packet
	* Disk packet

The register module will need to gather the server options from the sys_conf 
module. The register module will also have to use the sender module for 
sending all information.


Listener
------

The listener module will listen for task requests or heart beats from the 
master daemon. Once the listener is created it will listen on the port 
specified by the sys_conf module. The default listen port is 8889, and
can be specified in the conf file (using the listenport option)

Once a request has been accepted, the listener will fork a new process to
service the request.


Sender
------

The sender module will send all messages to the server. The message header 
will be added by this component. 


Host Info
---------

This module gathers information about the host system. The current 
information gathered is:
	
  a) hostname
  b) OS information (similar to uname -a) 
  c) Default route for this host
  d) IP address and mac address for every interface (except localhost)
  e) Version of khneia its running 


Disk Stats
----------

This module will report disk usage to the master. It will report the 
following information:

  a) Mount point
  b) Total size, free, used
  c) Filesystem type


System Stats
-------------

This module will gather system cpu and memory statistics. The cpu load
average will be reported to the 3 decimal place (its never really needed
beyond that).  The information reported will be the following:

  a) Memory total, used, free
  b) Swap total, used free
  c) CPU load 0, 1, 3 

Task
----

(not yet implemented)

The task module will handle the execution of the tasks. Currently there is 
only 1 supported task (rsync). Upon completion the task module will notify the
master daemon with a task complete packet.


Master daemon
-------------

The master daemon will send requests to the clients and then service their 
responses. These requests are sent on a timed interval specified in the
configuration file. Once a response is received, the master daemon will 
analyze the information. If the information is beyond limits then the master 
will send an alert. These alerts are just plain text email messages with 
proper alert information. After the analyzation the master daemon will 
store the information into the database.

This is the basic outline of the master daemon modules. 
     
     ----------
     | master |
     ----------
	|
	-> sys_conf (read configuration files and set up options)
        |
        -> master_listener (listen for heart beats and tasks)
        |
	-> master_poller (send requests on a timed interval)
	|
        -> sender (will properly send information to the client)
        |
	-> analyzer (analyze client information)
	|
	-> db_agents (store the information into the data base)
	|
        -> alerter (send alerts)

sys_conf
-------

This module is responsible for reading the configuration file. The 
configuration file path must be set at compilation time. The default 
path is:

     /usr/local/etc/khneia_master.conf

The configuration file will contain the options for the master
daemon. The options are listed by <option> = <value>. When writing
options or values the <> brackets are not used. Comments are allowed
and everything after the octothorp (#) will be ignored until the
newline character \n. Also note for convenience the options are not
case sensitive, however the values are !

dbserver   = <dbserver name/ip>   # database server hostname or IP
dbuser	   = <database user name> # database user name 
dbpass     = <database password>  # password for database (I know plain text.. bad)
dbport     = <dbserver port>      # database server port
listenport = <listen port>        # listen port default 8889
hbtime	   = <heart beat time>	  # time interval for requests (sec)
maxfork	   = <max amount of forks # the maximum forks allowed  

The sys_conf module will be written in flex++ for future
extendibility. Options and other configuration items will be very
easily added by adding simple lex rules. The module will have accessor
methods for each value and can be used at any point in the programs
runtime.

If the master daemon is running and the options are changed. The master
daemon can be signaled to re-read the configuration file with the HUP 
signal.


Master Listener 
---------------

The listener will listen on a port provided by the sys_conf module. Upon 
receiving a response from a client the listener will then spawn a new process
to service the response. 

The listener will also reset the value of the status to 0 once a register
packet is received. 


Master Poller 
-------------

The poller will send the requests to all the clients listed in the db based on 
the specified interval and request rules. The time interval will be set in 
the configuration file (hbtime). These polls will be sequenced to determine 
client response times. The responses are stored into the poll_log table.

Request rules:
  
  Clients listed in the poll table will be sent requests provided their status
  value is less then their threshold value (listed in the threshold table)  

The poller will also honor the maxfork value. This by default is 32 and since
both the Sender and the Listener will honor this value. Its possible for both
processes to fork off maxfork processes. Also when setting this value it should
be set to about 1/2 of the databases max connections allowed (see PostgreSQL docs). 
That way none of the processes are denied connections by the database.

The poller will also increment the hosts packet entry in the database by 1. This 
value will increase until it reaches the threshold or until it is reset to 0.
The reset process is set by the listener.


Master Sender 
-------------

The sender module will send all messages to the client. The message header,
the sequence number and the status for that host will be set by this module. 

Master Analyzer
---------------
(not yet implemented)

The analyser will analyze the packets received from the client. This analysis
will include the following.

     * determine packet type (register, heart beat, task response, task initiate)
     * determine the clients response time
     * CPU load out of allowed parameters
     * memory usage is out of allowed parameters
     * disk usage out of allowed parameters
     * swap usage out of allowed parameters

The allowed parameters can be set inside the data base. After the data has 
been analyzed it will be stored into the database. 


Master Alerter
--------------
(not yet implemented)

The alerter module will send email's to the people listed in the database. This list
will have 2 levels. The first level will be for warnings, the second level will 
recipient alarms.  These levels correspond to the soft and hard limits respectively.


khneia network protocol
-----------------------

To facilitate easy communication between the client and the master daemon. 
Khneia will have predefined packets. These packets will be sent using either 
TCP/IP.  Each packet will have a sequence number that will allow the master 
daemon to track response times for requests. Each packet will have a header 
and then the payload. The following packets are currently defined.


 [Header packet]

The header will be the same for all packets. The unique sequence number will 
be a 4 byte unsigned integer (this is a sequence in the database ).


  Field type       |  Field size (bytes)
 -----------------------------------------
  Sequence number  |   4
  Packet type      |   1
  Payload length   |   4
  Hostname         |   64


 [Host Information packet]

The system information packet will have the standard header along with the 
following information. To accommodate multi homed servers, these packets 
will not be a fixed length. The fields that can have multiple entries are 
denoted with a '*',

  Field type       |  Field size (bytes)
 -----------------------------------------
  OS information   |    128
  khneia version   |    32
  Default route    |    16
 *Device name      |    16
 *IP address       |    16
 *Netmask          |    16
 *Mac address      |    32


 [System Stats packet]

The system stats packets will contain the basic memory and cpu information. 
Since the heart beat interval can be set. The cpu information will 
include the 1, 5, 15 minute information. Also even though the free or 
used can be calculated, this information is included to ensure that 
the calculations are correct.

  Field type       |  Field size (bytes)
 -----------------------------------------
  CPU Load 1       |     6
  CPU Load 5       |     6
  CPU Load 15      |     6
  Memory total     |     4
  Memory used      |     4
  Memory free      |     4
  Swap total       |     4
  Swap used        |     4
  Swap free        |     4

 [Disk packet]

The disk packet will include information about the file system. Since a 
server can have multiple mounts. This packet will not be a fixed size. 
Also since Unix filenames can be very long, I have fixed the mount 
point size to 128 bytes. The packet will be sent as a large buffer.

  Field type        |  Field size (bytes)
 -----------------------------------------
   Mount point      |     128
   File system type |      32
   Total size       |       4
   Total used       |       4
   Total free       |       4

 [Server request]

The server request will be only contain the header packet. Since the server 
will not need to supply any further information.

 [Task initiate packet]

This packet is sent by the database and will have the standard header along
with a task sequence. This task sequence will be a unique unsigned 4 byte int.

  Field type       |  Field size (bytes)
 -----------------------------------------
   task sequence   |     4



 [Server rsync packet]

The server rsync packet will need to supply the client with the proper rsync 
server module as well as the local path.


  Field type       |  Field size (bytes)
 -----------------------------------------
   Rsync module    |    128
   Local path      |    512


Database tables
----------------

     Name      |   Type     
---------------+----------
 clients       | table   
 disk_info     | table    
 host_info     | table    
 network_info  | table    
 new_client_id | sequence 
 packets       | table    
 pkt_sequence  | sequence 
 poll          | table    
 poll_log      | table    
 system_stats  | table    
 threshold     | table    


 
clients table
--------------

this will store all the information for each client. Note that the clients
are unique by their hostname. This can be a FQDN so be careful !


host_info
---------

this will store all the host related information


disk_info table
----------------

This will store all the information about the systems disk. All the information
will be indexed by client_id


network_info table
-------------------

This table will store all the information. The inet address is used as a 


system_stats table
------------------

This table will keep all system stats info. Such as mem, cpu and swap usage.


poll table
-----------

This will hold entries for which systems we need to poll and which packets we 
need to request. Note that we will keep a status level, this level will be an
int starting at 0 -> N where N is the max level and N is > 1. This level 
is set to 0 initially and set to 1 when the client is polled. Once the client
responds then it will be set back to 0 (for packets with the sequence set) 
However this number will increase under these circumstances:
 
   I) when the client fails to respond prior to the next polling
 
  II) the client was unreachable or a network error occurred
 
This status value increases until  the max N value is reached (listed in the
threshold table as err_level). At that point then the client will no longer 
be polled, until its value is below its max N, or the client responds to a request. 


threshold
---------

This table will hold the clients thresholds. These are set as limits for the
particular max value.


poll_log table
---------------

This table will store the replies from the clients so that
a systems history can be traced.


pkt_sequence 
------------

This will keep track of the packets, note that this sequence will start
at 1, and then when the MAX sequence is reached it will cycle back to 0.
Since we need to support 4-byte ints we set MAX to 999999999

packets table
--------------

This table will have the names / int values for our packets that khneia will
use.