The Big Brother Systems and Network Monitor ------------------------------------------------------------------------ Installation and Configuration Manual Version 1.9i - Fri, 30 Dec 2005 20:19:58 PST ------------------------------------------------------------------------ 1.0 Quick and Dirty Install for Big Brother 1.1 Big Brother server 1.1.1 Big Brother server installation 1.1.2 Big Brother server upgrade 1.2 Big Brother client 1.2.1 Big Brother client installation 1.2.2 Big Brother client upgrade 1.3 Compiler directives 2.0 Configuring the BB hosts file 2.1 The etc/bb-hosts file 2.2 Basic format of etc/bb-hosts 2.3 etc/bb-hosts special directives 2.3.1 Display results in another HTML page 2.3.2 Display "page" results into an HTML subpage 2.3.3 Grouping hosts on the display (HTML tables) 2.3.4 Sending a summary to another BBDISPLAY 2.3.5 Testing Modem Banks 2.3.6 Specifying DHCP hosts (no fixed IP address) 2.4 Validating your etc/bb-hosts file 2.5 Pointers for writing entries in bb-hosts 3.0 Customizing your BB installation 3.1 The runbb.sh script 3.2 Configuring the etc/bbdef*.sh files 3.2.1 Configuring the etc/bbdef.sh file 3.2.2 Configuring the etc/bbdef-server.sh file 3.2.3 Configuring the etc/bbdef-client.sh file 3.3 Localizing programs paths 3.4 Creating BB clients 3.5 Configuring the Web Stuff 3.6 History 3.7 Adding your own tests 3.8 Starting BB at system boot 3.9 Summary of BB protocol 4.0 Configuring the Notification feature 4.1 A Little History 4.2 Setting up notification 4.3 Creating notification rules 4.4 Acknowledging notification messages 4.5 Manual notification (Web based) 4.6 SMS notification 4.7 Using kermit 4.8 Adding a custom notification 4.9 Notification/acknowledgement logs 4.10 Disabling notifications temporarely 4.11 Acknowledging a notification by e-mail 4.12 Displaying acknowledgements 5.0 Customizing your Big Brother display 5.1 HTMLized status logs 5.2 headers / footers (for reports too) 5.3 Creating skins 5.4 Inserting notes about your hosts 5.5 Extending the contents of the main HTML pages 5.6 Extending the contents of the history HTML pages 5.7 Working with historical logs 5.8 Hiding the bb.html/bb2.html page header 5.9 Pre-defined anchors in bb.html 5.10 Displaying multiple pages links on the same line 6.0 Miscellaneous Big Brother configuration options & tools 6.1 Specifying filesystem specific values in the disk test 6.2 Enabling security 6.3 Using bbnet to test TCP services 6.4 Sending custom one line status with bb 6.5 Changing the 'purple' interval 6.6 Specifying processes to look for 6.7 Sending arbritary data to a BBDISPLAY(s) 6.8 Disabling the messages file(s) zero-length test 6.9 Setting a Time To Live to a status message 6.10 Disabling the connectivity test 6.11 Reducing connectivity errors 6.12 Speeding up the network tests 6.13 Enabling WML ouput (WAP browsing) 6.14 Having BB notify you to reboot a server after is has been up XXX days 6.15 Using an alias name for the current host in etc/bb-hosts 6.16 Specifying CPU threshold values 6.17 Configuring messages file testing 6.18 Using the combo message 6.19 Tools to manage Big Brother data 6.20 Sending a BB message in cron 6.21 Defining characters that BB accepts in a status message 6.22 Integrating a LARRD graph with a status log 7.0 Other documentation sources ------------------------------------------------------------------------ This Installation and Configuration Guide is © Copyright Quest Software, Inc. 1997-2003 All rights reserved. ------------------------------------------------------------------------ ------------------------------------------------------------------------ Section 1: Quick and Dirty Install for Big Brother WARNING: For security reasons, it is best to install and run BB as its own user and not as root. (but use the root account during the install process, at the end you'll change ownership to the chosen BB user ID) *** READ THE README.SECURITY FILE BEFORE PROCEEDING *** 1.1 Big Brother Server 1.1.1 Big Brother Server installation For Big Brother Client installation please refer to its section further down below. You've extracted the BB archive and this is the directory structure that was created: bb1.9i-btf/ referred as $BBHOME 1. cd bb1.9i-btf This directory is often refered as BBHOME or $BBHOME cd ./install ./bbconfig where OS-NAME is bsdi sco3 sco freebsd solaris hpux9 hpux linux sunos netbsd osf ultrix irix unixware redhat redhatES aix dynix debian dgux caldera mandrake Note that is optional, BB will try to figure it out. If you are running Linux you may have to provide the proper distribution name: ./bbconfig redhat ./bbconfig redhatES ./bbconfig debian ./bbconfig caldera ./bbconfig mandrake ./bbconfig linux ./bbconfig will ask you questions about your setup be ready with: If the old-style structure be kept If you intend to use FQDN(Fully qualified domain names) Which host(s) is(are) the BBDISPLAY(s) Which host(s) is(are) the BBPAGER(s) If the current host is a BBDISPLAY/BBPAGER Default e-mail recipient for notification URL you intend to view BB with URL of the BB CGI scripts The user id of your web server 2. cd ../src type "make" then type "make install" If you have trouble compiling, refer to the README file located at src/README for those who are upgrading then go to "Upgrading a BB server" section 3. cd ../etc edit bb-hosts, put your hosts names in there. Refer to install/README. This is the core of Big Brother. You must read the docs here. edit bbdef*.sh files, and set alarm levels and things. If you want to use fully qualified domain name hosts then make sure you set FQDN=TRUE in etc/bbdef.sh. If this is the first host you install and intend to use it as the display/notification server as well as the server that test the network services then your bb-hosts should contain this line: xxx.xxx.xxx.xxx this.host.name # BBDISPLAY BBPAGER BBNET 4. ./bbchkcfg.sh Checks the bbdef*.sh/bbinc*.sh/bbsys.sh source scripts for invalid entries ./bbchkhosts.sh Checks the bb-hosts file for errors 5. cd ../.. ln -s bb where is the new version directory (e.g. bb1.9i-btf) This is useful as you don't have to change the directory in your startup script. (see section 11) cd bb chown -R bbuser . where bbuser is the user you defined in the install process. This makes sure that the bbuser can write/read into the BB directory sttructure as you will probably install while in the root account cd .. chown -R bbuser bbvar 6. ln -s /full_path_to_bb/www /WWW/bb (where /WWW is the Document Root dir). Make sure the permissions are correct. Configure your web server for this directory if need be. Also make sure that your web server follows symbolic links Note: use the location of the bb link from step 5 7. cd where is the new version directory (e.g. bb1.9i-btf) or cd where is the directory link you created in section 5 ./runbb.sh start examine the BBOUT file for any errors. By default the BBOUT log file is located in $BBHOME. The location of BBOUT is defined by the value of the BBOUT variable in bbdef.sh NB The HTML summary pages (bb.html/bb2.html) should be available 2 minutes after the startup. Don't panic the'll be there if you are patient. In your browser, you should be able to see the results at http://yourwebhost/bb/ or http://yourwebhost/bb/bb.html (assuming you used /bb as BB URL root) 8. Debug, and look at all the docs. 9. Check the online documentation! It lives under: http://bb4.com/bb/help/bb-help.html 10. This will have enabled you to have a display/pager server set up. At this point no clients are running. When you have all of your hosts defined in etc/bb-hosts then use install/bbclient to create a tarball for BB clients of the same OS/HW type. If you have different OS/HW platforms then reinstall BB on each one (then use the bbclient to create a tarball for each identical OS/HW client) and don't forget to copy your master bb-hosts file to it. Run through the install procedure to make sure that the clients are also installed properly(bbchkcfg.sh/bbchkhosts.cfg) 11. If you wish to start BB automatically at startup, we suggest the following command in your startup script: su - -c "cd ;./runbb.sh start" is the user you chose at install is the target directory in which BB was extracted or where you've created the bb directory link as per section 5. e.g. bb1.9i-btf For the Web Stuff ================= All Web-based things live in the $BBHOME/www directory. In order for BB to work correctly, this directory must be linked into your Web site somewhere. I suggest something like the following (this should have been done in step 6): ln -s /home/sean/bb/www /WWW/bb (where /WWW is the Document Root dir). You should then be able to access BB with the URL http://yourwebhost/bb/bb.html For the Pager Stuff =================== You need kermit and a modem(s). It should be working... 1. Edit etc/bbsys.sh and define the variables KERMIT (where kermit lives) and MAIL (with flag for subject header) if they are not defined in bbsys.local generated during the configuration. Note that the qpage and sendpage packages should also be considered for paging. If you're using SMS paging (mainly Europe) read Jacob's readme: install/README.SMS 2. For the notification configuration (who/when/how), please customize the etc/bbwarnsetup.cfg file and the etc/bbwarnrules.cfg file. Also consult the online help. 3. For Web-based paging/acknowlegement, web/bb-ack.sh should have been installed by ./install/bbconfig in your web server's /cgi-bin directory THIS STEP IS ONLY IF SOMEBODY CLICKS ON THE "PAGE/ACK" BUTTON OF bb.html/bb2.html . THE REAL PAGING/EMAIL/SMS NOTIFICATION IS TAKEN CARE BY STEP 2. 1.1.2 Big Brother Server Upgrade If you are upgrading from a previous version then do these steps after the preliminary install of BB: If your previous version of BB was pre-1.5 then follow these next steps otherwise jump to "Upgrading a BB server (Part 2)" We assume that the BBVAR/BBLOGS/BBHIST/BBHISTLOGS/BBDATA/BBDISABLE modified in $BBHOME/etc/bbinc.sh. cd BBTOP/bbvar (BBTOP is where you unpacked BB) rm -f logs acks hist histlogs notes data disabled mv OLDBBHOME/www/logs . mv OLDBBHOME/www/acks . mv OLDBBHOME/www/hist . mv OLDBBHOME/www/histlogs . mv OLDBBHOME/www/data . mv OLDBBHOME/disabled . Upgrading a BB server (Part 2) ============================== cd where is the directory of the new install cd www rm -r notes # Replace OLDBBHOME by the full path of BBHOME defined in your previous install mv OLDBBHOME/www/notes . cd .. cd ext If you have external scripts cp OLDBBHOME/ext/* . Only copy the ones you created !!! Don't overwrite the scripts distributed cp OLDBBHOME/ext/hist/* hist/ cp OLDBBHOME/ext/rep/* rep/ cp OLDBBHOME/ext/mkbb/* mkbb/ Only copy the ones you created !!! Don't overwrite the scripts distributed cp OLDBBHOME/ext/pg/* pg/ Only copy the ones you created !!! Don't overwrite the scripts distributed (This directory is new in BB 1.6) copy anything else in there (lardd,the state,...) to here cd .. cd etc cp OLDBBHOME/etc/bb-hosts . Depending of your current setup cp OLDBBHOME/etc/bbwarnrules.cfg . copy and edit appropriate config files cp OLDBBHOME/etc/bbwarnsetup.cfg . Run diff between old file and new cp OLDBBHOME/etc/security . file as format may have been changed vi bbdef.sh Set environment variables for your setup vi bbinc.sh vi bbdef-server.sh vi bbinc-server.sh cd ../.. # if you already have a link called bb used to access your BB install rm bb ln -s bb Setup bb as a link to the new bbdir directory such that it's easier to refer rc scripts to bb/ without having to change them every time you upgrade. If you have special changes that you have incorporated into BB then it is the time to carry them over now. 1.2.1 Big Brother Client installation You've extracted the BB archive and this is the directory structure that was created: bbc1.9i-btf/ referred as $BBHOME 1. cd bbc1.9i-btf This directory is often refered as BBHOME or $BBHOME cd ./install ./bbconfig where OS-NAME is bsdi sco3 sco freebsd solaris hpux9 hpux linux sunos netbsd osf ultrix irix unixware redhat redhatES aix dynix debian dgux caldera mandrake Note that is optional, BB will try to figure it out. If you are running Linux you may have to provide the proper distribution name: ./bbconfig redhat ./bbconfig redhatES ./bbconfig debian ./bbconfig caldera ./bbconfig mandrake ./bbconfig linux ./bbconfig will ask you questions about your setup be ready with: If you intend to use FQDN(Fully qualified domain names) 2. cd ../src type "make" then type "make install" If you have trouble compiling, refer to the README file located at src/README for those who are upgrading then go to "Upgrading a BB client" section 3. cd ../etc but first add an entry for this client in the etc/bb-hosts file on your BBDISPLAY/BBPAGER/BBNET host. Then copy that bb-hosts file into this directory: xxx.xxx.xxx.xxx this.host.name # Following that, edit bbdef*.sh, set alarm levels and things. If you want to use fully qualified domain name hosts then make sure is set FQDN=TRUE in etc/bbdef.sh. 4. ./bbchkcfg.sh Checks the bbdef*.sh/bbinc*.sh/bbsys.sh source scripts for invalid entries ./bbchkhosts.sh Checks the bb-hosts file for errors 5. cd ../.. ln -s bb where is the new version directory (e.g. bbc1.9i-btf) This is useful as you don't have to change the directory if you use a startup script. (see section 11) cd bb or cd chown -R . where bbuser is the user you defined in the install process. This makes sure that the bbuser can write/read into the BB directory sttructure as you will probably install while in the root account 6. ./runbb.sh start examine the BBOUT file for any errors. By default the BBOUT log file is located in $BBHOME. The location of BBOUT is defined by the value of the BBOUT variable in bbdef.sh 7. If there's any errors in BBOUT, debug, and look at all the docs. 8. Check the online documentation! It lives under: http://bb4.com/bb/help/bb-help.html 9. If you wish to start BB automatically at startup, we suggest the following command in your startup script: su - -c "cd ;./runbb.sh start" is the user you chose at install is the target directory in which BB was extracted or where you've created the bb directory link as per section 5. e.g. bbc1.9i-btf 1.2.2 Big Brother Client update An upgrade is almost identical to an installation except after you have executed install/bbconfig: cd etc/ cp OLDBBHOME/etc/bb-hosts . Depending of your current setup cp OLDBBHOME/etc/bb-dftab . file as format may have been changed cp OLDBBHOME/etc/bb-proctab . cp OLDBBHOME/etc/bb-cputab . cp OLDBBHOME/etc/bb-msgstab . cp OLDBBHOME/etc/bb-bbexttab . vi bbdef.sh Set environment variables for your setup vi bbinc.sh Check the differences with the previous installation vi bbdef-client.sh vi bbinc-client.sh cd ../ext If you have external scripts cp OLDBBHOME/ext/* . Only copy the ones you created !!! Don't overwrite the scripts distributed copy anything else in there (lardd,the state,...) to here cd .. # if you already have a link called bb used to access your BB install rm bb ln -s bb Setup bb as a link to the new bbdir directory such that it's easier to refer rc scripts to bb/ without having to change them every time you upgrade. 1.3 Compiler directives You change the behavior of BB by compiling it with different compiler options. All the options are defined in the CFLAGS variable on the Makefile -DBZERO If your OS doesn't support the bzero() function -DZOMBIE If zombie processes are generated while running BB This will get rid of them -DGETTIMEOFDAY To get duration statistics during TCP service checks -DREGEXEC If your OS supports regexec(),regcomp() -DREGEX If your OS support regex(),regcmp() -DRE_EXEC If your OS supports re_exec(),re_comp() -DTIMEH If your OS has time.h instead of sys/time.h -DSIGSETJMP If your OS supports sigsetjmp() and does not handle signals properly with setjmp() ------------------------------------------------------------------------ Section 2: Configuring the Big Brother hosts file 2.1 The etc/bb-hosts file The etc/bb-hosts file controls where Big Brother looks for things and the actions that are taken. The format is identical to the standard /etc/hosts file, except with additional directives for Big Brother. The etc/bb-hosts file should be kept identical on all BB client hosts as to reduce the risk of a misconfigured bb-hosts file. Even though clients only require entry lines for BBDISPLAY, BBPAGER and the local client, it is recommended that you distribute a master bb-hosts file to all clients. The only time you should not have an identical bb-hosts file is if you decide to have more than one BBNET host, in this case each BBNET host should only have tested entries in its bb-hosts file and those entries should not appear in any other BBNET host. 2.2 Basic format of etc/bb-hosts Lines are of the format: IP-ADDR HOSTNAME # DIRECTIVES IP-ADDR: XXX.XXX.XXX.XXX HOSTNAME: host.domain.com if FQDN="TRUE" host if FQDN="" FQDN (display with Fully Qualified Domain Name) is a variable that you set in the etc/bbdef.sh file. Directives the Big Brother knows about are: BBDISPLAY This host displays the HTML results This host receives incoming status reports, generates the HTMLized status reports, the bb.html and bb2.html web pages. There may be more than one BBDISPLAY if you want redundancy. BBDISPLAY host execute the bbd daemon to collect reports, and run the web/mkbb.sh & web/mkbb.sh scripts to generate the bb.html and bb2.html pages. You'll need a Web server on this host to display the results. BBPAGER This host acts as the notification server This host receives requests to notify admins and processes those requests. There may be more than one BBPAGER as well, for redundancy. Of course, you'll be notified more than once per problem. BBPAGER hosts likeweise run the bbd daemon so they can receive notification requests. To requests more than one BBDISPLAY/BBPAGER host, put the BBDISPLAY/BBPAGER directive on the config line for each host that acts as BBDISPLAY/BBPAGER. BBNET Indicates that this host checks the ip network services Many hosts can act as a BBNET. If you do have more than one BBNET, make sure that the bb-hosts file only contains the hosts that you will test against. You must be very careful because if a host is tested by many BBNETs then the status display for the services on that host will only show that last test status. You may get errors for that host even though it shows green, it's just that another BBNET send a status message afterwards and that message returned OK. On the other hand you may want to have tests from different location :) BBRELAY:<hostname or IP address> The host defined is a relay host This host will be relayed all incoming messages from any BBDISPLAY/BBPAGER hosts that has this directive defined on the same line as the BBDISPLAY/BBPAGER. So if a BBDISPLAY/BBPAGER entry in the bb-hosts file also contains a BBRELAY:<hostname|IP addr> directive then that host will be relayed all incoming messages from the current BBDISPLAY/BBPAGER. This is useful if you are moving a BBDISPLAY or a BBPAGER and you don't want to change all the BB clients bb-hosts files at once. 1.2.3.4 host1.domain99.com # BBDISPLAY BBPAGER BBRELAY:4.3.2.1 Host 4.3.2.1 will be sent all incoming messages that 1.2.3.4 receives. http://www_path The host is tested for http connections using the www_path https://www_path The host is tested for https connections using the www_path (requires lynx: http://lynx.browser.org/ or curl: http://curl.haxx.se/ ) Note that you can specify multiple URLs by joining the URLs with '|': http://www_path|http://www_path1 or by specifying them individually: http://www_path http://www_path1 ftp Test for ftp service smtp Test for smtp server pop3 Test host for pop3 server telnet Check telnet service ssh Test ssh server nntp Test nntp news server Check /etc/services for proper service name especially pop3 (sometimes referred as pop-3). These directives names MUST appear in /etc/services of the BBNET host that is doing the testing. Any text based service can be checked by putting the /etc/services name as the directive and adding in the list of services in the BBNETSVCS variable of etc/bbdef-server.sh. (See below for TCP ports that are note in /etc/services) By default, the test sends the "QUIT" string the to the tested service. The '!' can be prefixed to a service such it will be checked that the service is NOT running (!ftp). The '?' can also prefix a service. This prefix indicates that the service is considered a dialup service: it will generate a clear button if the service is down. NOTE: the ! and the ? prefixes can only be used with the basic TCP services that are handled by the bb-network.sh script. http* and dns/dig tests cannot use these prefixes. The :s , :q and :Q tags can be added to the directive by appending them to the directive: ftp:s or ftp:s:q This will enable special options to the bbnet program that does the TCP tests. :s - Just connect to the port, do not engage in a converstation with the service tested (silent) :q - Do not return timing statistics if you've enabled them (quiet) :Q - Do not return error messages usually returned by the bbnet program. You can also add a custom TCP port to test. You have to define the network service name in the BBNETSVCS variable of etc/bbdef-server.sh. In the bb-hosts you declare that new TCP test by putting in entry in this format: svcname:portnumber i.e. mytest:1234 - Test port 1234 and return it with the mytest status name dns Checks for name resolution server dig Same check as dns but using the dig command if the command is available. noping Don't do ping test for this host and display a clear dot. noconn Don't do ping test for this host and don't generate any colored dot noping Don't display this host on the page noping[:test1:test2:...] Don't display the defined tests for this host dialup If host is down then display a clear dot otherwise display a green status testip Use the IP address defined to execute network test. Don't use the hostname given. bbd Test a remote BBDISPLAY/BBPAGER You can add your own directives that you access through an external test. Known directives from user contributed tests: oracle/fping/trap/sybase/... These contributions can be found on the user contributions archive. (http://www.deadcat.net) This is a sample bb-hosts file: # # BIG BROTHER HOSTS FILE # 192.168.110.102 bobo # BBDISPLAY BBPAGER ftp smtp pop3 192.168.110.95 admin # http://admin/ 192.168.110.108 mynet # BBNET !nntp bobo is the display and notification server. mynet is the host that runs ip network services checks it will check the ftp/smtp/pop3 services on bobo, then the http service on admin and finally it makes sure that nntp is NOT running on mynet You can also group hosts within a seperate HTML table. There are three group tags defined: group, group-compress and group-only. Results can also be saved to a specific page using the 'page' directive. Changes to the etc/bb-hosts file are automatic. You do not have to restart BB to initialize the changes except if you make changes to BBDISPLAY/BBPAGER/BBNET locations. 2.3 etc/bb-hosts special directives 2.3.1 Display results in another HTML page page nyrouters New York Routers BB always creates two HTML pages: bb.html/bb2.html. When the "page" directive is encountered, BB will then create a new HTML page until another "page" directive is encountered and will save the output from that point on to that new HTML page. It will keep a pointer of the new HTML page in bb.html. The first argument is the directive, the second one is the name of the page (i.e. nyrouters.html as the .html tage is automatically added) and the remaining arguments is the caption that will appear in bb.html when it creates the link to that page. Always place your page directives after all hosts that you want to appear in the bb.html page. When you use a page directive then all subsequent output will be in an HTML page (the current one or a subsequent page directive) Each HTML page will be created in it's own directory to enable password protection (using your web server's features) based on the directory. N.B. summaries and dialup directives will always appear on bb.html/b2.html and not in any HTML subpages 2.3.2 Display "page" results into an HTML subpage page nyrouters New York Routers subpage manhattan Manhattan Routers ... subpage queens Queens Routers ... BB allows up to 3 levels of HTML display pages. The top pages bb.html and bb2.html. Another level using the "page" directive and finally the last level defined by the "subpage" directive. This will create a link in the current HTML page (created with "page") to a subpage. The subpage is created in it's own directory (as in "page"). It allows you to create a directory structure where there can be up to 3 different views. A topmost with bb.html/bb2.html. The next level with the "page" directive and the lowest level created with the "subpage" directory. This should be ample to secure the different views with user/password protection. 2.3.3 Grouping hosts on the display (HTML tables) The Web display may be broken into tables to create a more aesthetic and sensible display. It's also much faster to load small tables, than one giant table. So consider grouping your etc/bb-host entries logically and separating them using the "group" , "group-compress" or "group-only" directives. group Intranet Servers The group directive defines a block of hosts to be grouped in the same HTML table. All hosts lines following the group directive, until a new group/group-compress is defined, belong to that group. group-compress California Servers The group-compress is identical to the "group" directive except it will only display services (columns) containing data for that group. group-only conn|cpu|disk Restricted Services The group-only will create a table with only the colums defined in the directive. The columns are '|' delimited. Note that the text that follows the prevous directives is the title given to the HTML table. Note that you can embed HTML code in the title: Italic/H3/etc..., but use with caution: group <I>Intranet Servers</I> group-compress <I><H3>California Servers</H3></I> group-only conn|cpu|disk <B>Restricted Services</B> 2.3.4 Sending a summary to another BBDISPLAY summary quebec.bb 255.255.255.255 http://cafe.domain.com/bb/bb.html summary quebec.bb2 255.255.255.255 http://cafe.domain.com/bb/bb2.html This indicates to the BBDISPLAY machine that summary information of the bb.html, bb2.html, an HTML page or an HTML subpage be forwarded to the IP address noted by the 3rd item on summary line line. The summary can be sent to more than one BBDISPLAY. N.B. summary directive from other BBDISPLAYs (or the current one) will always appear at the bottom of the bb.html/b2.html and not in any HTML pages/subpages. 2.3.5 Testing Modem Banks dialup modem-bank 204.19.116.20 4 The dialup directive (not to be confused with the other dialup tag which displays a clear button if the host is down) is used to specify connectivity for a bank of modems. The 2nd parameter is the name to be displayed on the display page. The 3rd argument is the starting IP address of the modem bank. The last argument is the number of modems on that bank. A green dot will be displayed for IP addresses that are currently used and a clear dot will be displayed if the IP address is not used. Note that this test can take a long time if your modem banks have many unused modems as the test will time out to determine the connection status. N.B. dialup directive output will always appear at the bottom of bb.html/b2.html and not in any HTML page/subpages. 2.3.6 Specifying DHCP hosts (no fixed IP address) 0.0.0.0 dhcphost.domain # The 0.0.0.0 indicates that this host is a DHCP host and that the BBNET host will not try to run the connection test on that host using the IP address but only the hostname. 2.4 Validating your etc/bb-hosts file You can validate the etc/bb-hosts file by executing the etc/bbchkhosts.sh command. Note that it is a simple check and sometimes may complain about invalid qualifiers especially if they are associated with user contributed scripts. Also in some cases the script will complain but also mention that the error might not apply. Read the output carefully. 2.5 Pointers for writing entries in bb-hosts If you are going to use the same hostname in multiple entries then use the "noconn" directive for all entries for that hostname except the first entry. 1.2.3.4 some1.host # ftp smtp 2.3.4.5 some2.host # ftp smtp ... group-only ftp All ftp servers 1.2.3.4 some1.host # noconn 2.3.4.5 some2.host # noconn The first entry indicates all the tests to run while the second entry is used only as a display definition. ------------------------------------------------------------------------ Section 3: Customizing your BB installation 3.1 The runbb.sh script The runbb.sh script is the master BB script. It sets up the BB environment with the values from the configuration files. You MUST set the BBHOME variable to the directory path where you have installed BB. Without it, BB will not run properly. cd $BBHOME (to your BB install directory) To start BB: ./runbb.sh or ./runbb.sh start To stop BB: ./runbb.sh stop To restart BB: ./runbb.sh restart When BB starts/restarts it checks that there is no processes already running. If this is the case, BB will complain but will continue to start as it cannot determine if the process was really started by BB. BB checks for any processes that contain the BBHOME in its definition. If you ever have to add a script that runs as a deamon use the following template to add it in runbb.sh: echo "Starting snmptrapd" echo "Starting snmptrapd" >> $BBOUT MIBFILE=$BBHOME/cmu/etc/mib.txt export MIBFILE $BBHOME/cmu/bin/snmptrapd -B | $BBHOME/bin/bb $BBDISP - & set `$PS | $GREP snmptrapd | $GREP -v "$GREP"` > /dev/null echo $1 >> $BBPID This is from an SNMP trapd extension available at the FTP archive site (http://www.deadcat.net/). You'll notice that the PID is added to $BBPID, This way, you'll be able to stop the extension at the same time as BB when executing "runbb.sh stop". 3.2 Configuring the etc/bbdef*.sh files etc/bbdef*.sh files is where you configure how Big Brother reacts to a variety of situations that may arise. You can establish default levels for warnings(yellow), panics(red) values. Please note that any modifications to these files do not come into effect until the next restart of BB. It is best to stop/start BB immediately after these files has been modified. 3.2.1 Configuring the etc/bbdef.sh file bbdef.sh contains settings that affect both a client installation of BB as well as a server installation of BB. BBUSER is the user name that BB should run as. If you try to start BB using a different used than the one specified in BBUSER, BB won't start. STOPROOT prevents you from starting BB as root. It is recommended to you set STOPROOT=TRUE. Use FQDN (fully qualified domain naming) FQDN="" - Host names rendered on the BB status pages will only include the host name, not including the domain name. FQDN="TRUE" - Host names rendered on the BB status pages will include the host name and the domain name. BBTMP indicates the location of the BB temporary directory. You may want to locate it outside of the BB structure. BBOUT is the location of the BB error log file. BBPIC is the location of the file that contains the process ids of BB processes. The PAGELEVELS variable indicates to the 'bb' process at which level(s) it should send a notification message. The 'bb' program is used to send status messages and if the status level of the message is found in the PAGELEVELS variable then a notification message is also sent to the BBPAGER host. This is the default value. It can be overridden by the 'pagelevels:' token in the etc/bbwarnsetup.cfg file. The PAGELEVELSMAIL contains the color level(s) on which only e-mail is to be sent. The color level must also be found in PAGELEVELS. BBEXT contains the list of external scripts found in BBHOME/ext that are to be executed. *** NOTE *** etc/bb-bbexttab should be configured instead of using the BBEXT variable. The BBEXT variable has been kept for backward compatibility. BBDEFSLEEP is the lifecycle in seconds of an external script while STARTSLEEP is then initial interval between each external script at startup. MAXLINE is the maximum message size. Also defined in src/bb.h. DOCOMBO, MAXCOMBOMSGS and MAXCOMBOTTL are variables that define how multiple status logs can be sent in a single message. 3.2.2 Configuring the etc/bbdef-server.sh file Settings for a server installation are done in this file. Which program to use to check the HTTP connectivity: LYNX="/usr/local/bin/lynx -dump -head" - Use lynx if you need to check password protected pages. Please use the proper directory path as per your host installation. cUrl (another program) can also be used to check for protected pages. NONETPAGE defines which IP services not to notify on if an error is encountered. Include the services as they are named in the etc/bb-hosts file. This is only valid on a host defined as a BBNET host. BBREPWARN specifies the value at which the reporting feature returns red or yellow. Reporting will return green if 100% availability is determined, will be yellow until it reaches the value of BBREPWARN and then will return red below that value BBWEB="/bb" This variable defines the path to the root page of BB. Make sures it never ends with "/". All locations in BB are referenced from that location. BBWEBHOST is the full path to your BBDISPLAY server except the directory location: http://www.bb4.com BBWEBHTMLLOGS="${BBWEB}/html" This variable defines the path to the HTMLized status logs. This path is concatenated with the name of the HTMLized status file and is appended to the e-mail notification message for a quickpath to the error information. CGIBINURL is the directory location of the CGI scripts. BBREPURL is the location of the generated reports. The WEBHISTORY defines if you want to use the web-based history display script. This script gives you a 24hr HTML chart and the last 50 status changes. BBHIST_IGNOREBLUE tells the reporting to ignore maintenance time in statistics compiling if it's set to TRUE. The SAVESTATUSLOG variable defines if the historical log of the host.service is kept or not. The logs are saved in the $BBHISTLOGS directory. The BBLOGSTATUS variable determines if each log received is rendered into a static HTML page, dynamically by CGI or not at all (it appears as plain text). If it is set to DYNAMIC then a CGI will create the HTMLized status log only when requested. That's when you click on a colored dot on the BB display pages. The dynamic CGI is called bb-hostsvc.sh. If is set to DYNAMIC then a static HTML status log will be generated the www/html directory. If it is set to TEXT then no HTMLized status logs are available. The status will appear as plain text. You can define how many pings are executed before a connectivity test fails using the PINGTESTS variable. You can also set the sleep factor between tests using PINGSLEEPFCT. Each sleep within a single test is incremented by that factor. PINGWARN defines how many failed pings are required before the connectivity test yellow. If all pings fail then it returns red. NOPINGCLEAR lets you decide if you want clear dots for host that have the noping directive. If you set to TRUE then clear dots will be displayed for hosts with the noping as well as hosts with IP address 0.0.0.0 (DHCP hosts or hosts accessed only with its full name). You can use the noconn directive in conjunction with NOPINGCLEAR to select which hosts you don't want a clear dot. PURPLEDELAY sets the default expiry time stamp on an incoming status log. It is expressed in minutes. The IPTEST_2_CLEAR_ON_FAILED_CONN variable, if set to TRUE, defines that if a network test fails and that the connectivity (ping) test also failed then display the test results as clear. You can turn off this setting individually by test by using the ~ modifier. in etc/bbdef.sh: IPTEST_2_CLEAR_ON_FAILED_CONN=TRUE in etc/bb-hosts 123.123.123.123.host1 # ftp If the connectivity test to 123.123.123.123 failed and the ftp test also failed (very likely) then display the test with a clear dot. If you had set the directive to ~ftp then it would have been displayed as red. Note that this will work only if the basic connectivity is not disabled (CONNTEST=TRUE - enabled). SUMMARY_SET_BKG value determines if the summary dots are to be taken in account when determining the background color. Set it to anything but TRUE if you don't want summaries to set background. BBHISTEXT contains scripts names located in BBHOME/ext/hist that are to be executed when bb-hist.sh is run. This is to add site specific display information. BBMKBBEXT and BBMKBB2EXT defines scripts to execute at the end of the mkbb.sh and mkbb2.sh script respectively. Same basic idea as BBHISTEXT, you may extend their functionality that way. The scripts should be located in the ext/mkbb directory. The CONNTEST variable determines if the default connectivity testing is executed. Set to FALSE if you don't want the standard connectivity test. This is usually done when the fping.sh or fping.pl (from http://www.deadcat.net) external scripts are used. BBHOSTHISTLOG & BBALLHISTLOG are used when specifying if host based and system based historical information should be kept. This may be disabled when you are not interested in historical data. The BBNETTHREADS variables defines how many conccurent bb-network.sh tests should run to reduce the length of time of the network testing BBNETSVCS contains the valid TCP network tests that BB can perform. If you want to test a non-supported port then add the port name here. Read this section for more details. If you want WAP enabled display then set the WML_OUTPUT variable to TRUE. Depending on your device, you can set the size of the bb.wml/bb2.wml pages using WMLMAXCHARS. BBWAP defines where generated WAP files are stored. You can change the timeout values used by bbnet. Change the BBNETTIMER1/BBNETTIMER2 and BBNETTIMER3 to the appropriate values. The RUNOPTS variable defines runtime options for the bbd process. ENABLE_DISABLE - turn on enabling/disabling of notifications DATAMSG - accept "data" messages NOTESMSG - accept "notes" messages EMBEDHTML - If defined, Keep HTML code in status CONVHTMLTAGS - If defined, Change HTML <> tags to < and > LARRD - Inline code for LARRD when generating HTML status log CLEANCHARS contains the characters that should be removed from incoming status logs. 3.2.3 Configuring the etc/bbdef-client.sh file Settings for a client installation are done in this file. The DFWARN and DFPANIC variables contain the % levels at which the disk test will fail. By default DFWARN is 90% and DFPANIC is 95%. These values can be overriden by values in the etc/bb-dftab file described in section 5.2. CPUWARN and CPUPANIC are default levels based on the load average. CPUWARN is set at 150 and CPUPANIC at 300. These values are the load average (from uptime) multiplied by 100 (the dot is stripped from the value returned by uptime). You can also set these values in the etc/bb-cputab file. If you want the load average to be displayed in its original format (not stripped of its decimal point) then set the DISPREALLOADAVG to TRUE. You can define the processes for which you want to monitor if they are up and running. You do this by setting the PROCS and PAGEPROC variables. The processes included in PROCS will lead to a yellow condition if one or more the processes is down. Processes in PAGEPROC will generate a red condition if a process is down. But the preferred configuration method is the etc/bb-proctab file. MSGS and PAGEMSG are used to monitor error messages in the system logs. MSGS contain keywords to look for in the system logs. If a keyword is found then this will create a yellow condition and if the keyword is also in the PAGEMSG variable then it will be upgraded to a red condition. Note that the values of these variables can be overridden using the etc/bb-msgstab file You can also discard certain messages by defining portions of that message in the IGNMSGS variable. You can define multiple messages by delimiting each message by a ';'. The value can also be set in etc/bb-msgstab. MSGEXPIRE defines the amount of time the yellow/red conditions will last. Yellow conditions will last by the number of minutes defined by the first number will red conditions are defined by the 2nd number. i.e. 30:60, yellow=30m and red=60m. The CHKMSGLEN determines if the message file zero-length test is run. Set to TRUE or FALSE Define how many MSGS and PAGEMSG entries that are returned in a messages file(s) check. YELLOWMSGSLINES determines the number of yellow entries while REDMSGSLINES indicates how many red entries are returned. The OLDSTYLEBBMSGS variable indicates if the message file(s) check should not expire message just like pre-1.6d versions. You had to expire messages by clearing them by hand in the message file(s) by modifying the token string so it wouldn't match anymore. BBTOOMANYDAYSUP is used to notify if a host has been running for more than so many days. Some people use this to remind them to reboot some servers after a certain amount of time. WARNMINSONREBOOT and WARNCOLORONREBOOT determines which color the cpu column will bear the color defined in WARNCOLORONREBOOT and for how long after a server reboots. Note that only red/yellow are accepted and that the duration has to be less than 60 minutes. You can disable local test(s) by setting one or all of DOCPU/DOMSGS/DOPROCS/DODISK to FALSE. If you disable a test you can decide if you still want to send a message to the BBDISPLAY by setting the color in NOCPUCOLOR/NOMSGSCOLOR/NODISKCOLOR/NOPROCSCOLOR to the color of your choice or omit the color if you don't want any message sent to the BBDISPLAY. 3.2 Localizing programs paths Sometimes the OS specific programs paths found in the installation might be wrong for a particular host(s) OS. Change the paths in the etc/bbsys.local file. The variables contained in this file all have default values from the bbsys.OS file. The original bbsys.local is copied from the etc/bbsys.OS file during installation (bbconfig); so keep track of your changes as they will automatically be overwritten if you re-run bbconfig. 3.4 Creating BB clients Creating BB clients is very simple: cd $BBHOME cd install ./bbclient <client-hostname> <client-hostname> must be found in the etc/bb-hosts file. This will create a tarball for the client that has the same OS/HW platform type as the host on which the bblient command is executed. The tarball is found in the directory above $BBHOME. Don't forget to configure the etc/bb-proctab file the client after you've untarred the archive such that the correct processes are monitored. If you have different OS/HW platforms then install BB, as a client, on each OS/HW platform (then use the bbclient to create a tarball for each identical OS/HW client) and don't forget to copy your master bb-hosts file to it. Run through the install procedure to make sure that the clients are also installed properly (bbchkcfg.sh/bbchkhosts.cfg). 3.5 Configuring the Web Stuff All Web-based things live in the $BBHOME/www directory. In order for the bb stuff to work correctly, this directory must be linked into your Web site somewhere. I suggest something like the following: ln -s /home/sean/bb/www /WWW/bb (where /WWW is the Document Root dir of your web server). You should then be able to access BB with the URL http://.../bb/bb.html where ... is your webserver address. You also have to configure the script which generates the web-based history page if you set the WEBHISTORY variable to TRUE in etc/bbdef-server.sh. This page is accessible from the "History" button when you click on a colored dot in one of the display pages (bb.html/bb2.html). You must copy the web/bb-hist.sh script to your /cgi-bin directory and set the BBHOME. Make sure the script has the proper permissions to execute: system and web server permissions. If you want only a text based history display then set the WEBHISTORY variable to "FALSE" in etc/bbdef-server.sh. Set the BBHIST_IGNOREBLUE variable to TRUE in etc/bbdef-server.sh if you don't want the blue state to be used in the calculations of daily % use. By default the CGIBINURL (in etc/bbdef-server.sh) points to /cgi-bin but if your web server cofiguration has a different CGI directory then change it here or your history feature won't work. If you intend to enable manual notification/acknowledgement ("PAGE/ACK"button in bb.html/bb2.html) then copy the web/bb-ack.sh script to you /cgi-bin directory and set the BBHOME variable in the script. Make sure the script has the proper permissions to execute: system and web server permissions. Acknowledgments generate a log entry in $BBACKS/acklog file. 3.6 History Each time a status log is received it is checked against its previous status to see if it has changed. If it has, then the duration of the previous status is recorded and a new line is appended to the history file with the timestamp of the new status. There's a history file for each host.service (status log) received. The history files are located in the $BBHIST directory. Here's a sample: Sun Jan 24 18:30:45 1999 green 917220645 232068 Wed Jan 27 10:58:33 1999 red 917452713 1560 Wed Jan 27 11:24:33 1999 green 917454273 Notice that last line doesn't have a duration because it is the current status. The duration will be appended only when the status changes. If you do not wish to have historical data then rename $BBHIST to something else or remove it. 3.7 Adding your own tests You can easily add your own tests. Start with the template available at ext/ext-proto, add your code. Look at bb-local.sh, bb-network.sh for an example on how to send data to BB. Specify the name of your script in the etc/bb-bbexttab file. This file defines all scripts to run for each hosts. Start with the etc/bb-bbexttab.DIST file, copy to etc/bb-bbexttab and configure. Typical entries in etc/bb-bbexttab: <host> : <options> : scripts... www.bobo.com : : script1 script2;3600 script3;900 script4 www.baba.com : : script5 script6 Note that <options> is not yet implemented. make sure each defined script exists in the ext directory of the client install. See below for an explanation of the script2;3600 entry. Restart BB and your test should be running. But before you use it within BB, I suggest you test it for errors by using the method: cd /home/bb (or wherever your BB is located) BBHOME=/home/bb export BBHOME . ./etc/bbdef.sh cd ext ./yourexternaltest Look for errors, fix them, rerun your test until you're satisfied, then update bbdef.sh. Note that all temporary files should be created in $BBTMP and make sure you remove them after use. Also, remember that you don't have to deal with sending notification messages, the 'bb' process send a 'page' type message to the BBPAGER host when the status color is found in the PAGELEVELS variable defined in bbdef.sh. You can also set the frequency of the test by specifying the interval it should run at. In ext/bb-bbexttab, set an interval qualifier with the script name by appending the qualifier to the script name. Here's an example entry in etc/bb-bbexttab: www.bobo.com : : script1 script2;3600 script3;900 script4 script1 and script4 will run every 5mins (the default) while script2 and and script3 will run every 60 and 15 minutes respectively. You specify the interval in seconds. Note that the delimiter is ';'. It used to be ':' in BBEXT but it had to change due to the line entry format. When it's ready don't forget to update the svcerrlist token in the bbwarnsetup.cfg file on your BBPAGER host. You must assign a numeric code to your column name. You can also set the lifetime of the status sent by your script by following the instructions described in section 6.9: "Setting a Time To Live to a status message". 3.8 Starting BB at system boot Depending on U*X operation system version (Linux/BSD/Solaris/...) your startup procedures will vary from OS to OS. In short, you have to create a startup script that has a start/stop capabilities. Under a lot of OSes, you'll want to create your script in the init.d directory and create an S89bb link in rc3.d that links to init.d/S89bb (you may also want to create a K11bb link in rc3.d that'll be used when the system shuts down). Use an existing startup script has an example and substitute with these commands: To start BB, use this command: su - <bbuser> -c "cd <BBHOME>;./runbb.sh start" or su - <bbuser> -c "cd <BBHOME>;./runbb.sh restart" To stop BB, use this command: su - <bbuser> -c "cd <BBHOME>;./runbb.sh stop" <bbuser> is the user that BB will execute has make sure that bbuser has all permissions under BBHOME <BBHOME> is the location of your BB install e.g. su - bb -c "cd /home/bb;./runbb.sh start" 3.9 Summary of BB protocol Big Brother is based on a client/server paradigm. BB Client software sends a message to a BB daemon (either a BBDISPLAY, BBPAGER or a BBRELAY) over a TCP connection using port 1984. The client connects to the BB port socket writes its data and closes the port. The server listens on a that port socket then reads the incoming messages and closes the socket. Here's a quick summary of valid messages using an example: Status message: status www,bb4,com.disk red Wed Mar 7 21:16:33 EST 2001 Disk partitions on www,bb4,com OK /dev/wd0s1a 496053 25081 431288 5% / /dev/wd0s4e 1998122 1094955 743318 60% /home /dev/wd0s2e 1489830 1094928 275716 80% /usr *** Note that the status message can be also be sent within a combo message. You should only send "green" status thru a combo message. Each status message should be delimited by at least 2 newlines combo status ... status ... status ... Error message: page www,bb4,com.cpu red Thu Jan 25 17:36:30 EST 2001 up: 16 mins, 3 users, 105 procs, load=363 For both messages, host can be fully qualified or not www,bb4,com.disk www.disk Note that ',' replaces the '.' in the hostname for historical reason (I know, I know...) color can be red,green,yellow,purple,clear,blue text can be anything but usually on the first line it's just a summary while the rest of the text is a more detailed status. Usually the date is inserted as the first entry in the text. Summary message: summary quebec.mtl 255.255.255.255 red http://cafe.domain.com/bb/ quebec.mtl is the row/col definition quebec is the row mtl is the column 255.255.255.255 is the BBDISPLAY to send the summary to http://cafe.domain.com/bb/ is the URL that will be displayed when the red dot will be clicked on Acknowledgement messages: ack add_event 1234567 120 Rob acked this This is to acknowledge an event from a script 1234567 is the acknowledgement ID 120 is the amount in minutes of the TTL of the ack "Rob acked this" is an optional message ack rm_event www.bb4.com.disk 345 This is to signify that event has recovered. This should only be sent by a BBDISPLAY server daemon www.bb4.com.disk is the host.service combination that has recovered from an event. Note that commas are not needed in the hostname (confusing isn't it) ack addtag_event www.b4.com.disk [email protected] 1234567 120 999999999 Rob acked this This message is used to create a tag file on BBDISPLAYs to enable a checkmark to be displayed in the web pages. This should only be sent by a BBPAGER server daemon www.bb4.com.disk is the host.service combination to be acked. [email protected] is the recipient that is acking the event 1234567 is the ack number 120 length in minutes of the ack 99999999 is the current time in integer format "Rob acked this" is an optional message ack rmtag_event www.bb4.com.disk This message is used to remove a tag file on BBDISPLAYs that was created by an addtag_event message. www.bb4.com.disk is the host.service combination to remove Maintenance mode messages: (these message types are disabled by default) disable www.bb4.com* 120 Disk Removal This message will notify BB to disable all notifications for the host.service combinations that match the regular expression specified. Only a BBDISPLAY server should receive this message. www.bb4.com* is a regular expression that will match all services for www.bb4.com 120 is how long notifications will be disabled in minutes "Disk Removal" is an optional message enable www.bb4.com* This message will notify BB to reenable all notifications for the host.service combination specified. Only a BBDISPLAY server should receive this message. www.bb4.com* is a regular expression that will match all services for www.bb4.com. offline www.bb4.com.disk This message instructs BB to create the tagfile in the bbvar/disable directory. A BBDISPLAY server automatically sends this message to all BBPAGER servers. It shouldn't be used by scripts. www.bb4.com.disk is the host.service combination online www.bb4.com.disk This message instructs BB to remove the tagfile in the bbvar/disable directory. A BBDISPLAY server automatically sends this message to all BBPAGER servers. It shouldn't be used by scripts. www.bb4.com.disk is the host.service combination Data transfer messages: (these message types are disabled by default) data www.bb4.com.data .... .... .... The data message appends to the bbvar/data/www.bb4.com.data file the text specified by .... Note that the data text is appended. notes www.bb4.com.html .... .... .... The notes message write the text defined by .... to the www/notes/www.bb4.com.html file. Note the the data message overwrites the current file. This file can then be display by clicking on the hostname in the web display (bb.html/bb2.html). ------------------------------------------------------------------------ Section 4: Configuring the Notification feature (BBWARN) 4.1 A little History In the Beginning :), BB could only e-mail and notify by pager. The configuration required that all BB hosts (servers and clients) have the PAGER variable set with the recipients of the notifications. There were no possibilities of customizing the notification feature. This led to the: The Pager Protection Act of 1997 ================================ Use of BB has been linked to the untimely death of at least one pager in Texas. In order to protect pagers from their users, a notification scheme has been created to allow to specify a delay between the notifications. But this wasn't enough. BB netizens got restless and finally a solution was delivered by Robert-Andre Croteau, and described in an article in SysAdmin magazine. It was called BBWARN and it was good. It allowed an admin to specify rules which enable him/her to get notified based on the source of the error (host) and the service involved and also based on the day/time. Different rules could be set up and different recipients could be specified depending on the characteristics of the problem (host/service/day/time). Numerous changes have been made and various configuration options are available to the BB admin(s). Notification methods You can currently get notified by email, numeric pager and SMS communication device. Users have reported custom notification by alpha-numeric pagers and other PCS devices (some are available on the FTP archive or a search of the mailing list archives will lead you to them). To use the email feature, your BBPAGER host (in bb-hosts) must have an email program. This is not a problem on UNIX: the mail (mailx) program is available. Sending a notification message to a numeric pager requires kermit and a modem. The path of kermit should be defined in the KERMIT variable of bbsys.sh/bbsys.local. 4.2 Setting up notification All of the BB notification configuration is specified in the etc/bbwarnsetup.cfg file. Here are the various options that can be set. Note that instructions are included in the configuration file as comments. All tokens in the etc/bbwarnsetup.cfg file must be defined on the BBPAGER host. ***** The 'pagelevels:' token must be defined on all BB hosts (servers & clients) because the 'bb' executable sends the notification and that is done locally on each client host. ***** bbwarn: Set to "TRUE" is you want to enable the notification feature svcerrlist: This contains a list of service/code pairs. The service name is the column found in the HTML display and the code is the value displayed on a numeric message. If you add a custom check then the column name must have a corresponding code. ignforall: Is a regular expression that is used to temporarely disable notification a host.service combinations e.g. .*.cpu|.*.msgs|host11.* Don't notify for any cpu/msgs errors and any errors concerning host11. N.B. This feature can be duplicated but the ! rule (below). pagehelpcode: Numeric code to use when a user sends a manual notification. See section 4.5. ttyline: List of modems devices. You can specify more than one. prefix: Prefix to use when dialing out suffix: Suffix to use when dialing out (like a hangup) pagelevels: At which color level to send a notification. This token MUST be set on every host (clients & servers). You must restart BB if you modify it. pagelevelsmail: if a color defined here is also defined in pagelevels then only e-mail recipient will receive notifications when an error occurs at the pagelevelsmail defined color. Note that only recipient in the form [email protected] will be notified, if a recipient is prefixed with ep-, ext-XXX- or any other valid prefix, it will be ignored as they are not considered e-mail recipients. pagerecovered: Set to TRUE if you want to be notified when a problem has been fixed. This feature is only available when 'pagetype:' is set to EVENT. pagetype: Defines how the pager delay is handled. There are 4 choices (RCPT|EVENT|HOST|GROUP). RCPT: the recipient is not notified until pager delay expires. EVENT: the recipient is not notified for a particular host.service combination until the pager delay expires. HOST: the recipient is not notified for a particular host until the pager delay expires. GROUP: the recipient is not notified for a particular host.service within the same etc/bb-hosts group combination until the pager delay expires. pagemaster: Recpient(s) to receive an e-mail notification if a page notification could not be sent. pageaddhtmlpath: Set to TRUE if you want the HTML path of the status log to be appended to an e-mail notification. If this is set to TRUE, make sure the BBWEBHTMLLOGS variable is set correctly in etc/bbdef-server.sh. cfgdelim: entry delimiter in the etc/bbwarnrules.cfga briefrcpt: define the recipients that should receive a brief notifications message. You can you use regular expressions: ie. ep-* (all e-page recipients) This expression will be expanded to this regular expresssion: ep-.* The message will have this format: hostname.service - XXXYYYYYYYYYYYY <link to host.svc page in bb> Note: XXXYYYYYYYYYYY - Service code and IP address hg-xxxxxxx: You can have multiple hg-xxxxxxx tokens that define a group of hosts/devices. So you can e.g. create hostgroup 'hg-routers' as an alias or shorthand of a list your routers. These tokens can then be specified in the bbwarnrules.cfg file in the host-fields columns (1st/2nd column) instead of having to type all of them individually in all rules lines. pg-yyyyyyy: You can have multiple pg-yyyyyyy tokens that define a group of recipients. These tokens are expanded in the rules lines inthe recipients definition. On the client make sure that PAGELEVELS is set (etc/bbdef.sh) and on the server, if appropriate, that PAGELEVELSMAIL(etc/bbdef.sh) is set. Refer to section 3.2 for more details. 4.3 Creating notification rules You define the notification rules in the etc/bbwarnrules.cfg file. Rules are written in the following format: hosts;exhosts;services;exservices;day;time;recipients hosts: match on these hosts (* is a wildcard for all hosts) exhosts: exclude these hosts services: match on these services (* is wildcard for all hosts) exservices: exclude these services day: 0-6 (sunday-saturday) time: 0000-2359 recipients: email address, numeric pager, sms number, qpage recipients are defined with qp-<recipient> sendpage recipients are defined with sp-<recipient> Hylafax sendpage recipients are defined with hsp-<recipient> (note that Hylafax sendpage arguments will be added with the -p construct, regular sendpage don't) e-pager recipients are defined with ep-<recipient> SNMP traps are sent with trap-<destination IP address> (note that SNMP trap support is preliminary and may be subject to change) N.B. Please specify all recipients that do not allow long messages in the briefrcpt token in etc/bbwarnsetup.cfg. There's also a special format of the rule line: !hosts;exhosts;services;exservices;day;time;recipients If a rule line starts with ! the event thats matches the rule line will disable notification to any recipient defined on that rule line. If the recipients field is '*' then no notification will occur for that event. Here's an example !*;;*;;*;*;* This will in effect disable all notifications and render useless any other rule that you have defined :) !*;;*;;*;*;robert@localhost 9999999 This will remove robert@localhost and 9999999 from the list of recipients for the current event if they were defined on another rule that matched the same event. The value set by 'cfgdelim' (in bbwarnsetup.cfg) is your field delimiter. In this case the value is ';' Even though egrep regular expressions are allowed, do not use the .* construct, just use '*'. It will be replaced with .* in the regexp. It's just that * is more readable than .* Also the default 'pagedelay' value (see bbwarnsetup.cfg) which indicates how long before the next notification occurs can be overridden on the rule line for a specific recipient by appending the time value to the recipient: recipient:XX where XX is the value in minutes. For examples and complete information please refer to the etc/bbwarnrules.cfg and the etc/bbwarnsetup.cfg files. Escalation To escalate a notification, you use the following format for the recipient: recipient:^XX[-YY] XX is the initial wait before sending the notification YY is the delay for each subsequent call. If it is not specified then the 'pagedelay' value is used. An escalation can only be acknowledged by the recipient. Initial delay An initial delay can be specified when configuring a recipient. The format is: recipient:~XX[-YY] XX is the initial delay before sending the notification YY is the delay for each subsequent call. If it is not specified then the 'pagedelay' value is used. An initial delay can be reset if a recipient acknowledges a notification for all recipients of that notification (see below). There's a simple checking program that validates the contents of your bbwnarnrules.cfg file. It is located in $BBHOME/etc. cd $BBHOME/etc ./bbchkwarnrules.cfg Note that it only checks if your config file is not missing a column. 4.4 Acknowledging a notification When an admin is notified, the admin is always sent an acknowlegement tag with the message. This tag number is seven digits (XXXXXYY): the first five digits are a unique number and the last two digits are the recipient's ID for that notification (the recipient's ID can change, it is valid only for the current notification). Note that you can acknowledge using XXXXX99 to signify to acknowledge for all recipients of the XXXXX ack ID message except for escalated recipients(^rcpt) which can only ack themselves. In an e-mail notification, the ack number is in the subject and the body of the message: in the subject it is found at the beginning as in "!BB - XXXXXXX!", and in the body of the message it is at the beginning of the body delimited by [XXXXXXX]. In a numeric page, the acknowledgement is always after the BB numeric code. It is the last 7 digits of the numeric message. NOTE, if your pager doesn't support more than 21 digits, then you're out of luck, you'll be missing some digits. You could always try to shorten the ack number but that is left as an exercise for the reader ;-). If you get the first 5 digits then you can always use recipient 99 to acknowledge for all recipients but that may not fly at your site especially if you have multiple recipients. 4.5 Manual notification (Web based "PAGE/ACK" button) You can enable the manual notification feature and let the user reach you by pager/e-mail if they can't get a hold of you. If this is enabled, then you'll receive a numeric message starting with 911 (pagecode in etc/bbwarnsetup.cfg) followed by the phone number to call. Copy the web/bb-ack.sh script (if it was not copied during the installation) to your cgi-bin directory and make sure all permissions are correct. You can setup who gets called when, by adding rules to the etc/bbwarnrules.cfg file. The pseudo host to use is notify-admin*. There's an example in the config file. 4.6 SMS notification This a user custom notification scheme for SMS devices. What follows is the README file for his custom hack. Last updated: 1997-01-11 by [email protected] This section describes the usage of a kermit script that sends SMS messages. It is used as a mail -> SMS gateway, and also for the "Big Brother" network monitor(obviously). The kermit script for SMS can be found in etc/sms.scr My operator uses NOKIA server software, hopefully you can use this also on Motorola servers, without too much hassle. Terms SMS Short Message Service SMSC Short Message Service Center CIMD Computer Interface to Message Distribution Syntax kermit sms.scr device file number [number [number [...]]] device is modem device like /dev/cua0 The file should contain the message, only first line of file will be sent, and it MUST be terminated with LF, otherwise the kermit script won't be able to extract the message. SMS messages can only be 160 chars, and since nobody would want a lengthly message on a 8 character display, I haven't bothered to handle message splitting. If message is too long, it is truncated. If one or more SMS phonenumbers can be specified, the message will be sent to each one of them. Configuration I have done this the easy way, that is I start CIMD with faked checksums, if anybody manages to generate correct checksums, please mail me. Log in to SMSC You have to set two variables in sms.scr: \%b phonenumber to you operators SMSC \%c login to access CIMD For Europolitan (Swedish operator) you can use: \%b 46708222902 \%c cimd3 # (use faked checksums) This starts the CIMD server, and from now we must ACK all responses. Don't forget to include your modem prefix to \%b!! I don't use the "prefix" token from BB, since I also use this script as a mail->SMS gateway. If you are going to use this, you must anyhow configure this script, so I hope you don't mind. Start a CIMD session The first step in a CIMD session is to identify yourself You have to set two variables in sms.scr: \%d CIMD account \%e CIMD password Unfourtunately, I cant give you theese values ;) Now you have configured your script and can start to use it! Send to all recipients BB calls the script one time for each recipient, that's a bit of a waste, but it would just be to much work to get around. If you call it manually or from sendmail, you can use multiple recipients. More info The (english) spec of NOKIAS CIMD implementation can be found on: http://www.europolitan.se/europolitan/fick/tjanst/cimdspec.htm If you happen to know swedish, you can read more on Europolitans implementation on: http://www.europolitan.se/europolitan/fick/tjanst/faqcimd.htm Feedback Please let me know if you use this with a non NOKIA server, or if you fix real checksums! Jacob Lundqvist <[email protected]> (SMS:+46-708-555 456) Thanks Jacob !!! 4.7 Using kermit By default, BB expects kermit 5 to be installed on your BBPAGER host. On certain installations it is V6 and up that is installed. If that's your case then you must (might) modify which kermit script is used: copy numeric-k6.scr to numeric.scr in the script. (don't forget to make a backup of numeric.scr) But try with numeric.scr at first and if it doesn't work then replace it with numeric-k6.scr. Don't forget to set KERMIT in bbsys.sh. 4.8 Adding/modifying for a custom notification procedure You can create your own notification by editing your own notification script. For examples see: ext/pg/ex1 or ext/pg/ex2 . Your script must be saved in the $BBHOME/ext/pg directory. You would use a special prefix in the etc/bbwarnrules.cfg file. For example, you could use the recipient [email protected] to send a notification to the recipient called "[email protected]" using the "ex1" script. You create your own look for the notification message by using the following environment variables in your script in order to customize it. BBALPHAMSG - is the text body ACKCODE - is the security associated with the notification RCPT - is the recipient name (stripped of the ext-name- prefix) BBHOSTNAME - is the hostname in host.name.com format BBHOSTSVC - is the host.name.com.service tag BBHOSTSVCCOMMAS - is the host,name,com.service tag (notice that the '.' were replaced by ',') BBNUMERIC - is a numeric value made up of 22+ digits XXXYYYYYYYYYYYYZZZZZZZ XXX - corresponding code of service defined in bbwarnsetup.cfg YYYYYYYYYYYY - IP address ZZZZZZZ - security code - first 5 digits: event ID code last 2 digits: user ID code MACHIP - is the IP address of host in normalized 12 digits AAABBBCCCDDD BBSVCNAME - Name of the service of the event BBSVCNUM - numeric equivalent of the service name (from svcerrlist in bbwarnsetup.cfg) BBCOLORLEVEL - Color level causing the notification RECOVERED - 1 in recovery mode, anything else non-recovery mode DOWNSECSMSG - Default recovery message DOWNSECS - # of seconds to recovery After you've created your own script, make sure the permissions allow only the authorized users to modify it. 4.9 Notification/acknowledgement logs You'll find logs of who got notified for what reasons, logs about recoveries, who got a recovery message and logs about who acknowledged what notification. They are all located in the $BBACKS directory: acklog notifications.log recoveries.log recoverymsgs.log 4.10 Disabling notifications temporarely This feature is not automatically available. It must be defined in the RUNOPTS variable in etc/bbdef-server.sh. Add the ENABLE_DISABLE tag to RUNOPTS. You must restart BB after the change. Please be warned that if you enable this feature, a hacker could disable notifications while cracking into your systems. You can temporarely disable notifications without having to modify the etc/bbwarnrules.cfg file. All you have to do is send a 'disable' message to the BBDISPLAY(s) using the following format: MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO USE 'bb' directly ./bb $BBDISP "disable 'host regular expression' 'duration' [reason]" You can match multiple hosts/services by specifying a regular expression instead of a real host name. The duration is in minutes. You can also add an optional reason that will be displayed in the status. AGAIN, MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO USE 'bb' directly e.g. cd $BBHOME/bin ./bb $BBDISP "disable www.bb4.com.disk 240" Disable notifications for the disk event of www.bb4.com for 240 minutes. cd $BBHOME/bin ./bb $BBDISP "disable www.bb4.com* 240" Disable notifications for all events for host www.bb4.com for 240 minutes cd $BBHOME/bin ./bb $BBDISP "disable www.bb4.com* 240 Taking www.bb4.com offline for a new disk" Disable notifications for all events for host www.bb4.com for 240 minutes and specify the reason NOTE: duration can be expressed in seconds/minutes/hours/days 30s: 30 seconds 15m: 15 minutes 1h: 1 hour 1d: 1 day by default, minutes are used if a qualifier is not specified To reenable a disabled host(s), send the "enable" message ./bb $BBDISP "enable 'host regular expression'" As an example: cd $BBHOME/bin ./bb $BBDISP "enable www.bb4.com*" Enable notifications on all events for host www.bb4.com Note that when you use the enable message, the colored dot will stay blue until a new status is received by the BBDISPLAY(s) 4.11 Acknowledging a notification by e-mail You can acknowledge a notification by e-mail by setting up the bin/bb-mailack.sh in the .forward file of the used id that sent the notification message. On the BBPAGER host, set up a .forward file in the user account that's running the BB instance. The contents of the .forward file should be: |/bin/bb-mailack.sh Replace by the real BBHOME path, i.e. |/home/bb/bb/bin/bb-mailack.sh Then in the bb-mailack.sh script, set up these variables BBUSER - user id of the BB instance BBHOME - self describing ;) MAILFILE - Location of the user id mailbox in case the message is not an acknowledgement REPLYPFX - Prefix used to identify a reply. If you intend to use multiple mail programs to ack and if they don't provide the same prefix, I suggest you use your own prefix and enter it on the subject line instead of the one given by the mail program. DELAY - Default duration of the delay When you receive an e-mail notification, you can simply reply to the mail message to ack the notification and to delay the next notification for DELAY minutes if the problem persists. As long as the subject of the reply mail message is in the following format(It assumes the REPLYPFX is "Re:"): Re: !BB - XXXXXXX! You can specify the DELAY in the reply message also Re: !BB - XXXXXXX! DELAY=120 This will set the delay to 120 mins You can also specify a message that is saved in bbvar/acks/acklog Re: !BB - XXXXXXX! MSG=This is a message Anything after the MSG= is considered a message Finally, you can also tell the bb-mailack.sh to return a confirmation of the replied message (but it doesn't guarantee that the actual acknowledgement worked). Re: !BB - XXXXXXX! ACK=Y You can use all 3 directives on a subject line Re: !BB - XXXXXXX! ACK=Y DELAY=120 MSG=This is a message If the mail message does not start with "Re: !BB - XXXXXXX", then the message is saved at the location pointed by MAILFILE. 4.12 Displaying acknowledgements When an acknowledgement is sent to BB, it displays that fact by changing the colored dot of the acknowledged event to a checkmark of the same color. It also adds a status line at the bottom of the individual status page. You can also have the last 25 acknowledgements listed in the bb2.html page. This feature is enabled by default. To disable it, you must remove the acklog.sh script name in the BBMKBB2EXT variable in etc/bbdef-server.sh. You need to restart BB to have the change take effect. ------------------------------------------------------------------------ Section 5: Customizing your Big Brother display 5.1 Customizing the HTMLized status logs If you don't like the blinking gifs, you can change them to non-blinking by copying over the files found in www/gifs. The non-blinking gifs start with the nb- prefix. You can change Sean's mean face in the bb.html/bb2.html and HTMLized status logs by replacing the bb.gif file. This only applies to pre-1.3 versions. You can change the header/footer for the HTMLized status pages. The files are located in the /web directory. Note that your customized version can contain special tags which can be replaced by values: &BBDATE: current date/time &BBBACKGROUND: with the current background color (red/green/yellow/clear/purple) &BBRELDATE: Release date of the current version &BBREL: Current version number &BBCOLOR: Color of this status file &BBHOST: Name of host of current status file &BBSVC: Service name of current status file. &BBIPNAME: IP address of host, if 0.0.0.0 then use hostname &BBIP: IP address of host, always return IP address unless not found You can add tags by modifying the bbd.c file. Every time a status log is received by bbd, an HTML version of the status log is created by prefixing the status with the header file and suffixing with the footer file. You can put tags also in the status logs: &red &green &yellow &clear &purple These tags are replaced with a corresponding image source tag for the HTMLized version. Just like the bb-local.sh script does for the disk and procs tests. So, if you create your own custom check, enter these tags if you want to see a colored dot in the HTMLized status. NOTE: If you do not wish to have HTMLized status files, then rename the www/html directory and the main BB display pages will use the plain text status logs which are located in the $BBLOGS directory. This will only work if you had the install program create the old-style directory structure. 5.2 Customizing headers / footers (for reports too) bb.html/bb2.html and various HTML page generation can be customized by modifying their respectable headers/footers. These are located in the $BBHOME/web directory. The bb.html uses bb_header/bb_footer and bb2.html uses bb2_header/bb2_footer. You can also customize the output of the HTML file defined by the page directive in bb-hosts by creating <page name>_header/<page name>_footer files and you can also customize the HTML file defined by the subpage directive in bb-hosts with the files <page name>_<subpage name>_header and <page name>_<subpage name>_footer. So an example is division1_department3_header and division1_department3_footer. The reporting feature (bb_rep.html) also uses the same format. Except that bb.html uses bbrep_header/bbrep_footer. (bb2.html has no equivalent in reporting). As for the page directive, use <page name>rep_header/<page name>rep_footer. And for the subpage directive, use <page name>_<subpage name>rep_header and <page name>_<subpage name>rep_footer. Do not use the < and > characters in the filename, they were used for readability. NOTE: The link to bb4.com MUST be kept in the footer page as per your agreement of the license agreement. 5.3 Creating skins You can create your own look and feel by customizing various HTML aspects of Big Brother. You can use your own gifs by creating a new directory under $BBHOME/www. As an example, a directory 'psy' has been created with more psychedelic colored dots It lives in $BBHOME/www/psy. To use those images then set the BBSKIN variable in etc/bbinc.sh to "psy". To further modify the appearance of your display, modify the header/footer files found in $BBHOME/web. You also have some control over the bb.html/bb2.html and the status logs HTML pages by modifying the following variables in etc/bbinc.sh: MKBBLOCAL Title of the table for pages MKBBSUBLOCAL Title of the table for subpages MKBBREMOTE Title of the table of summaries MKBBTITLE HTML attributes for titles MKBBROWFONT HTML attributes for hostnames MKBBCOLFONT HTML attibutes for column names DAYS Text for days HOURS Text for hours MINS Text for minutes STATUNCHNMSG Text for unchanged status at bottom of status log RECVFROMMSG Text for IP location message at bottom of status log DOTHEIGHT Height of colored dots DOTWIDTH Widtn of colored dots BBSKIN Directory of images MKBBLOCALCOLS Number of columns of the table for pages/subpages 5.4 Inserting notes about your hosts You can setup HTML links on the host name in the bb.html/bb2.html pages to point to an information page for that particular host. If a file exists in the www/notes directory that matches the system name as displayed on the bb pages, then they are linked into both the bb.html page and the bb2.html summary page. These files can end in .htm, .html, .shtml, .php3 or nothing :) You can create these files manually or you can use the "notes" messages type: "notes <name of file> <data to write to the file>" Here's an example: cd $BBHOME/bin ./bb $BBDSIP "notes www.bb4.com.html <HTML><BODY>It's a hit!</BODY></HTML>" This feature is not automatically available. It must be defined in the RUNOPTS variable in etc/bbdef-server.sh. Add the NOTESMSG tag to RUNOPTS. You must restart BB after the change. Please be warned that if you enable this feature, a hacker could fill the disk with dummy notes file. Also note that the file size is limited to the incoming buffer of BB and by the size of the environment buffer. Also note that if you source the BB environment prior to sending the "notes" message and you have multiple BBDISPLAYs then the "notes" message will be sent to all BBDISPLAYs. If you want to send it to only one BBDISPLAY, then use that IP address as the command line argument and set BBDISPLAYS to "" just before sending your message. 5.5 Extending the contents of the main HTML pages You can tack on some extra HTML code at the bottom of the bb.html /bb2.html pages before the insertion of the footer. You save the scripts that generate the HTML code in the $BBHOME/ext/mkbb directory. And in etc/bbdef-server.sh, you specify the name of those scripts in the BBMKBBEXT variable for output on the bb.html page and in the BBMKBB2EXT variable for output in bb2.html. There's an example of a script in $BBHOME/ext/mkbb called eventlog.sh that displays the last global events in a table of the bb2.html page. Make sure that the scripts are only writable by the owner to prevent someone to execute malicious code. 5.6 Extending the contents of the history HTML pages You can tack on some extra HTML code at the bottom of a history display page before the insertion of the footer. You save the scripts that generate the HTML code in the $BBHOME/ext/hist directory. And in etc/bbdef-server.sh, you specify the name of those scripts in the BBHISTEXT variable. Make sure that the scripts are only writable by the owner to prevent someone to execute malicious code. 5.7 Working with historical logs There are many historical logs available with BB. These logs contain all events that occured based on the host.service combination, on the host itself or for all hosts. These logs are found in the $BBHIST directory. You can use these logs to view that last events that occured. By default, if you view a status log, you can click the history button to view the last 50 events for that host.service combination. You can also use the ext/mkbb/eventlog.sh to view the last 20 events that happened for all hosts (you can view more by changing the NUMEVENTS variable in eventlog.sh to the size you want). You can disable the logging of events by renaming the $BBHIST directory. You can set BBHOSTHISTLOG=FALSE or BBALLHISTLOG=FALSE to prevent the logging of host or all hosts events. You set those values in etc/bbdef-server.sh There's a lot of interesting information available in those logs. Check the FTP site for scripts that take advantage of that information. There's also a directory that keeps a copy of the status logs when they change state such that they can be viewed later on. The status logs are saved in the $BBHISTLOGS directory. A directory is created with the name of the hosts and then each service has its own directory where the status logs are saved based on the time of the occurance. 5.8 Hiding the bb.html/bb2.html page header You can position the bb.html/bb2.html pages at the beginning of the first block by accessing the page as follows: http://.../bb.html#begindata You can bookmark this position to hide the header. 5.9 Pre-defined anchors in bb.html http://.../bb.html#hosts-blk Go to the first host (same anchor as begindata) http://.../bb.html#pages-blk Go to the subpages table http://.../bb.html#summaries-blk Go to the summaries table http://.../bb.html#dialups-blk Go to the dialups table http://.../bb.html#hostname Go to the host line by using the hostname as the anchor: i.e. http://www.bb4.com/bb/bb.html#www.bb4.com 5.10 Displaying multiple pages links on the same line By default, each page link are displayed in an HTML table, one entry per line. You can now have more than one entry per line by modifying the value of MKBBLOCALCOLS in etc/bbinc-server.sh ------------------------------------------------------------------------ Section 6: Miscellaneous Big Brother configuration options 6.1 Specifying filesystem specific values in the disk test You can define warning and panic levels for specific filesystems. They are defined in the etc/bb-dftab file. This file contains definitions for all hosts all in a single file. You just need to redistribute across your hosts. The format is as follows: [hostname]:/filesystem:warning level:panic level example: /:85:95 www.maclawran.ca:/oradata:98:99 / uses 85% for warning level and 95% for panic, this is valid for all hosts (as no host name was define in the definition) Only on www.maclawran.ca /oradata uses 98% for warning and 99% for panic All other filesystems use the default values defined in the etc/bbdef-client.sh file. Use the etc/bb-dftab.DIST file as a starting point. 6.2 Enabling security When BB is installed, it accepts connections from any hosts. You can specify from which hosts only to accept status logs using the etc/security file. You can define individual hosts or networks in the etc/security file. Using this format: 192.168.1.0 192.168.2.125 10.0.0.0 Accept from network 192.168.1.0, for 192.168.2.125 host and the 10.0.0.0 network. 6.3 Using bbnet to test TCP services You can test any text based TCP services using bbnet. To do so add the test in bb-network.sh. If you need just a basic test, then just add the service to the list of other services. If you need to send a special command to the service daemon, then add a case statement entry like the imap test. Set change the textmsg variable to the text to be sent to the daemom. You can also use special options for bbnet. The -q/-Q/-s options are available. The -q option does not return the elasped time of test if bbnet was compiled with the -DGETTIMEOFDAY switch (which is only valid if the gettimeofday() function is available). The -Q option prevents the display of error messages. Finally the -s option only connects to the port to verify if the port can be opened. See the section on the etc/bb-hosts configuration file on how to tell the network test to use these options. 6.4 Sending custom one line status with bb The bb executable used to send statuses/notifications to the BBDISPLAY/BBPAGER can also be used by a custom check using standard input: custom_check | $BB $BBDISP - The custom_check program just needs to send a single line status to stdout (use the exact format of status files found in $BBLOGS). This is very useful if the custom check program is a daemon. 6.5 Changing the 'purple' interval By default, BB will send a 'purple' status to indicate that a status has not been received in the last 30 minutes (default). You can change this value to a more suitable value by modifying PURPLEDELAY in etc/bbdef-server.sh 6.6 Specifying processes to look for The PROCS & PAGEPROC variables in etc/bbdef-client.sh can be set with the names of the processes to monitor. A more sensible way ... is to use the etc/bb-proctab file. This file contains definitions for all hosts all in a single file. You just need to redistribute this file across your hosts. Host entries can be defined on multiple line entries for clarity in case you have many processes to check. The format is as follows: [hostname1]:process list for yellow : process list for red [hostname1]:process list for yellow : process list for red [hostname2]:process list for yellow : process list for red [hostname3]:process list for yellow : process list for red [hostname3]:process list for yellow : process list for red example: localhost:smtp:httpd www.maclawran.ca:!oracle "sleep 30":smtp !process - check that host does not exists "sleep 30" - use '"' when you want to check on a process that spawns multiple words You can also add logic to define min/max instances of a running process: httpd;>15 - more than 15 httpd must be running httpd;<=10 - less or equal to 10 hhtpd must be running httpd;5 - exactly 5 instances must be running if you don't specify the # of instances then it checks that at least one instance is running. For processes defined with "" as in "sleep 30", you must add the logic within the "": i.e. "sleep 30;>6" Use the etc/bb-proctab.DIST file as a starting point. If bb-proctab exists, then it overrides the PROCS/PAGEPROC values in bbdef-client.sh 6.7 Sending data to the BBDISPLAY to be handled by an external script You can send data to your BBDISPLAY host to be processed by an external script, a cron job or a command line program/script. The data is appended to the specified file name into the $BBDATA directory. To use this feature you have to send a "data" message to the BBDISPLAY in this format: "data <file name> <file data>" Here's an example: cd $BBHOME ./bb $BBDISP "data www.bb4.com.hits 952281437 10m 5000" The "952281437 10m 5000" would be appended to the $BBDATA/www.bb4.com.hits file. This feature is not automatically available. It must be defined in the RUNOPTS variable in etc/bbdef-server.sh. Add the DATAMSG tag to RUNOPTS. You must restart BB after the change. Please be warned that if you enable this feature, a hacker could fill the disk with dummy data files. Also note that if you source the BB environment prior to sending the "data" message and you have multiple BBDISPLAYs then the "data" message will be sent to all BBDISPLAYs. If you want to send it to only one BBDISPLAY, then use that IP address as the command line argument and set BBDISPLAYS to "" just before sending your message. 6.8 Disabling the messages file(s) zero-length test By default Big Brother verifies that none of the messages file specified in the MSGFILE variable is of zero-length. This is a simple test to make sure that a hacker hasn't linked your messages file(s) to /dev/null. Unfortunately, on some systems, when the logs are rotated they are left empty by the procedure that rotates the logs. If this is your case then you have 2 choices: Immediately have the log rotation, add this command: echo "`date`" >> <message file(s)> or use a syslogd feature to add a line in the messages file(s) or simply disable the zero-length check by setting the CHKMSGLEN variable to FALSE in bbdef-client.sh. Note, IMHO, the zero-length should always be there. Simply echo the data to the messages file(s) or use syslogd features to add an entry in the message file(s). Security is too important to ignore. Your best bet is to install a log scanner to analyze your messages file(s). Which one ? It is our of BB's scope to tell which one(s) but ask around for opinions of people who use log scanners. 6.9 Setting a Time To Live to a status message When you create external scripts, you can now tell how long the status is valid for. Just specify the length in minutes of the status. This is to prevent purple status on certain service that are not run in the regular interval of 5 minutes. To use this feature just follow this example $BB $BBDISP "status+1560 www,bb4,com.backup green `date` Backup Successful" A purple indicator will be triggered only if no status is received within 1560 minutes (26hours). 6.10 Disabling the connectivity test In some cases you may want to disable the connectivity test altogether because you don't need it or because you want to use the fping.sh external script available at ftp://ftp.deadcat.net . To disable it, just set the CONNTEST variable in etc/bbdef-server.sh, on the BBNET host, to FALSE. Also take in consideration that by disabling this test you will lose the ability to generate clear conditions for the other network tests because you will not have access to then connectivity status for that host. 6.11 Reducing connectivity errors If you are prone to a lot of red status for your connectivty tests even though the host responds when you ping by hand. To possibly reduce these false alarms, you may want to add/change the PINGPAR1 or PINGPAR2 parameters in etc/bbsys.local on the BBNET host. Adjust according to the BBNET's ping parameters. PINGPAR1 are options that goes before the hostname/ IP address in the ping command while PINGPAR2 are options that goes after the hostname/IP address. Adjust the number of packets sent for the ping test. Start with sending 2 packets and analyze the results. If it did not make a difference, try with 3 packets. Note that when you specify more than one packet to be sent, you inherently add a 1 second sleep on each new packet. This will delay, in seconds, your whole network test by (n-1) * hosts. If you test a large number of hosts, you will see a noticable difference in how often your hosts gets tested. 6.12 Speeding up the network tests If you have a faily large number of servers to monitor there's a very good possibility that the network tests takes a while to complete. You can speed up the tests by parallelizing the network tests from a single bb-network.sh script to running multiple copies of bb-network.sh at once. All you have to do is to increase the value of BBNETTHREADS in etc/bbdef-server.sh. Indicate how many bb-network.sh you'd like to have working concurrently. You can also speed up the network tests by using the fping program to do the connectivity test. You'll have to download fping and also get the fping.sh external script at the FTP archive site: http://www.deadcat.net. You can disable the regular ping test by setting CONNTEST to FALSE in etc/bbdef-server.sh. 6.13 Enabling WML ouput (WAP browsing) By default, BB does not generate WML files. YOu can enable the generation of the WML files by setting WML_OUTPUT to TRUE in etc/bbdef-server.sh. The files will be saved in the www/wml directory. You'll be able to view the output on a WAP browser by viewing the <BBWEB>/wml/bb.wml URL. Make sure your web server can render wml files. In your web server's mime.types file, make sure the following line is defined: text/vnd.wap.wml wml Add it if need be and stop/start your web server. Also note that some WAP device accepts can't accept large page size and you may have to reduce the value of WMLMAXCHARS to control the size of the bb.wml/bb2.wml pages. On the other hand, if your WAP device accepts large pages, then you can increase the value such that more info can appear on a single page. 6.14 Having BB tell you to reboot a server after is has been up XXX days You can have BB display the "cpu" column as yellow dot when a server has been up more than a pre-determined value. You set the BBTOOMANYDAYSUP value in etc/bbdef-client.sh to the number of days of uptime after you wish to schedule a reboot. You can also set this value in etc/bb-cputab. 6.15 Using an alias name for the current host in etc/bb-hosts Sometimes you may want to use an alias for the hostname instead of the hostname returned by the 'uname -a' command. To do so, on the BB client you wish to use an alias, create an etc/bbaliasname file that you want with the alias hostname as its contents and BB will use that hostname. Then use that alias hostname in the all bb-hosts files (BBDISPLAY, BBPAGER, BBNET). Also, you must declare the "testip" directive in the bb-hosts file of the BBNET host because the alias is unlikely resolvable by DNS otherwise the network tests will likely fail. 6.16 Specifying CPU threshold values BB uses the 5min load average value (from uptime) multiplied by 100 (the dot is stripped from the value returned by uptime). The CPUWARN & CPUPANIC variables in etc/bbdef-client.sh can be set with the load average thresholds to monitor. This was explained in section 3.2. A more sensible way ... is to use the etc/bb-cputab file. This file contains definitions for all hosts all in a single file. You just need to redistribute this file across your hosts. The format is as follows: hostname: misc settings : threshold value for yellow : threshold value for red localhost: misc settings : threshold value for yellow : threshold value for red : misc settings :threshold value for yellow : threshold value for red examples: localhost: : 150 : 300 www.maclawran.ca: : 250 : 350 www.bb4.com: daysup=999 : 75 : 150 Note that the order for matching is the following: 1) try to match for the hostname 2) use the localhost values is defined 3) use the ": XX : YY" line values if defined 4) use the CPUWARN/CPUPANIC values in etc/bbdef-client.sh if no match is found Use the etc/bb-cputab.DIST file as a starting point. If bb-cputab exists and if a match is found then the values override the CPUWARN/CPUPANIC values in bbdef-client.sh The "misc settings" field is reserved for other CPU test checks. Currently on "daysup" is valid and the number associated with it overrides the BBTOOMANYDAYSUP variable. 6.17 Configuring messages file testing The PAGEMSG, MSGS & IGNMSGS variables in etc/bbdef-client.sh can be used to define keywords to search for in messages file(s). Well, that's the old way. You can now define entries in the etc/bb-msgstab file to set those values. This file contains definitions for all hosts all in a single file. You just need to redistribute this file across your hosts. The format is as follows: hostname: filename(s) : misc settings : yellow string(s) : red string(s) : matched strings to ignore localhost: filename(s) : misc settings : yellow string(s) : red string(s) : matched strings to ignore : filename(s) : misc settings : yellow string(s) : red string(s) : matched strings to ignore example: www.bb4.com: /var/log/messages : : WARNING : NOTICE : not this message ; and this one either www1.bb4.com: /var/log/maillog : : refused : ERROR ; error ; BAD SU : from www.bb4.com To define more than one string in each field, the strings must be seperated by a ';'. The <misc settings> is currently not implemented. You can define multiple entries for the host in the etc/bb-msgstab file. If you want different settings for different files on the same host, use different lines. If a matched entry for the yellow/red string(s) contains the "matched strings to ignore" then the current match is ignored. Use the etc/bb-msgstab.DIST file as a starting point. If bb-msgstab exists and if a host matches an entry then the values override the PAGEMSG/MSGS/IGNMSGS values in bbdef-client.sh. If there's no matches for the current host then the PAGEMSG/MSGS/IGNMSGS values are used. You can reset the colored dot from red/yellow to green by removing the corresponding tmp/MSG.red.* and tmp/MSG.yellow.* files. Those files contain previous error messages and by default they are removed when they expire. 6.18 Using the combo message The combo message enables BB to send more than one status message at a time. This is useful when you create an external script that may have to send at lot of status messages (i.e. doing some network tests against all hosts). There are 3 steps in using the combo message: 1) Initialize combo message processing Before you can start using the combo message in your script, you must initialize its processing, add these lines at the beginning of your script: BBCOMBOID="$$" export BBCOMBOID $BBHOME/bin/bb-combo.sh start The BBCOMBOID variable acts as a unique tag for a particular run of your script. (Many scripts might be executing at the same time). 2) Building a combo message Instead of calling $BBHOME/bin/bb to send each status message to a BBDISPLAY, you replace that call with a bb-combo.sh call: Basic method: $BB $BBDISP "status $MACHINE.cpu $COLOR `date` DATA..." combo method: $BBHOME/bin/bb-combo.sh add "status $MACHINE.cpu $COLOR `date` DATA..." This will build a combo message. When the message reaches the MAXLINE size then the message is sent to the BBDISPLAY(s) and starts a new message. Note that only "green" status are saved in a combo message. All non-green status are sent immediately. 3) Ending combo message processing A partial combo message might be required to be sent to a BBDISPLAY(s) when your script is done. At the end of your script, add this line: $BBHOME/bin/bb-combo.sh end This will also remove temporary files that were used. If you need examples, take a look at the bb-local.sh and bb-network.sh scripts, they use the combo message. 6.19 Tools to manage Big Brother data bin/bbrm Removes data associated with a hostname or a service that is no longer monitored. bin/bbmv Renames data associated with a hostname or a service. bin/bbprune Archives historical data up to and including the month given as arguments. This tool creates archive files in the bbvar/archive directory. 6.20 Sending a BB message in cron If you want to use bin/bb send a single message to a BBDISPLAY/BBPAGER in cron, remember that bin/bb requires the BB environment to be loaded before it is executed. This loads environment variables required by bin/bb to execute properly. Use a crontab line similar to the following entry: 0 8-20 * * * BBHOME=/home/bb/bb;export BBHOME;. $BBHOME/etc/bbdef.sh;$BBHOME/bin/bb $BBDISP "status test.test red `date` This is a test" BBHOME=/home/bb/bb;export BBHOME;. $BBHOME/etc/bbdef.sh This sets the BB environement $BBHOME/bin/bb $BBDISP "status test.test red `date` This is a test" This sends the actual command. $BBDISP is the BBDISPLAY location. 6.21 Defining characters that BB accepts in a status message By default BB removes dubious characters from an incoming status log. The `,$,;,|,&,\ characters are removed. They are defined in the CLEANCHARS variable in bbdef-server.sh. Note that this is considered a security concern. If you use LARRD, you'll require to remove the '&' character from CLEANCHARS. 6.22 Integrating a LARRD graph with a status log LARRD is an extension that generates trending graphs based on the BB status data. You can find it at http://larrd.packetpushers.com/ . If you have installed LARRD along with your BB installation, you can have a status log associated LARRD graph displayed when you view the HTML generated page of a status log. This feature is not automatically available. It must be defined in the RUNOPTS variable in etc/bbdef-server.sh. Add the NOTESMSG tag to RUNOPTS. You must restart BB after the change. Also remove the '&' character from CLEANCHARS as LARRD requires it. ------------------------------------------------------------------------ Section 7: Other documentation sources $BBHOME/README $BBHOME/LICENSE $BBHOME/README.INSTALL $BBHOME/README.CHANGES $BBHOME/install/README $BBHOME/src/README
