Autopoll User Guide for UNIX BLAST features a sample script, Autopoll, which allows your UNIX system to call a series of remote computers and exchange information unattended. Autopoll performs the following tasks: - reads a list of sites to be polled, - connects to each site, - executes a BLAST transfer command file to transfer files, - disconnects, - scans the log file to determine which transfers were successful, - builds retry files as required, - and adds the results to a status file. Autopoll checks carefully for errors while polling. If an error is detected, the problem site is scheduled to be retried. Only the file transfer commands that failed are attempted again. ////////////////////// Installing Autopoll / ////////////////////// Autopoll consists of eight scripts that were copied into your BLAST directory (usually /usr/blast) when the BLAST program was installed in your system. The scripts are: autopoll.scr -- master script autoinit.scr -- initializes variables and files autoierr.scr -- reports initialization errors autodisp.scr -- draws screen displays autoline.scr -- reads site information autosw.scr -- strips switches off of @filename (10.8.x only) autopsnd.scr -- checks log for status of SENDs audoprcv.scr -- checks log for status of GETs autoparx.scr -- updates status files The scripts may be moved to any convenient directory in your system. For instance, you could segregate Autopoll from other BLAST files by creating a poll directory: cd /usr/blast mkdir poll mv auto*scr poll In addition to these script files, you must have a BLAST setup called autopoll. This setup resides in the BLAST setup directory (usually /usr/blast), and it must include a valid communications port or hunt file and other connection information, such as modem type and baud rate. Optionally, you may also specify the script autopoll.scr in the Script File field of the setup, which will simplify the command line to start Autopoll. //////////////////// Starting Autopoll / //////////////////// Autopoll must be started from the directory in which the Autopoll scripts and support files (site and transfer command files) are found. If blast.exe is not in this directory, you need to add the full path for blast.exe to your PATH or give the full path in the command line. If autopoll.scr has been entered in the Script File field of the autopoll setup, the format for invoking Autopoll from the command line is: blast autopoll max_cycles site_file [start_time] [TRACE] However, if autopoll.scr has not been entered in the Script File field of the setup, the command line must include the script explicitly: blast autopoll -sautopoll.scr max_cycles site_file ... Other command line switches may be required under certain conditions. For example, if your aim is to run Autopoll from cron, you must disable terminal output by including -b or -n on the command line. The command line parameters have the following meaning: autopoll -- the autopoll setup -sautopoll.scr -- the autopoll script (.scr extension is not needed) max_cycles -- the maximum number of attempts to complete all specified transfers site_file -- the filename "stub" of the site description file start_time -- [optional] in 24 hour format, the time that Autopoll will begin polling. If this parameter is omitted, Autopoll begins polling immediately. TRACE -- [optional] command to file a capture file of the entire polling session--used primarily for troubleshooting Here are some example command lines: blast autopoll 3 retail 10:45 blast autopoll 1 northwest -n & blast autopoll 2 nightly TRACE In the first example, a maximum of three attempts will be made to poll the sites listed in the file retail.dat, starting at 10:45 am. Notice that the command line specifies just the stub ("retail") of the site filename, retail.dat. (Autopoll appends a variety of extensions to the filename stub to determine the names of special files.) In the second example, one attempt will be made immediately to poll the sites in northwest.dat; the job will be placed in the background (&) with terminal output suppressed (-n). The third example illustrates the use of the trace option. A start time can be optionally specified with TRACE. //////////////// The Site File / //////////////// The site file is the "master list" of information about the sites to be polled. Site files may use any valid filename, but the extension must be ".dat". Each line in the site file holds the parameters needed to connect with and transfer files to and from one site. Each line, or site record, consists of five fields separated by exclamation marks ("!", also called "bangs"), in the form: setup_name!site_name!phone_number!baud_rate!TCF_name where setup_name specifies a setup to be used for polling. If omitted, the field defaults to autopoll. site_name contains a descriptive label for the site. If omitted, the field defaults to the Description field of setup_name. phone_number specifies the phone number to be used for the site. If omitted, Autopoll uses the Phone Number field of setup_name. baud_rate specifies the baud rate to be used for this site. If omitted, Autopoll uses the Baud Rate field of setup_name. TCF_name specifies the Transfer Command File to be used for this site. If omitted, this field defaults to autopoll.tcf. Each line must contain four bangs. Any fields that are to be skipped must be indicated by consecutive bangs ("!!"). Blank lines and lines beginning with a pound sign ("#"), space, or tab will be skipped, so you may freely comment your site file. Lines may not exceed 100 characters in length. Examples: ....|....1....|....2....|....3....|....4....|....5.... !Blaster!1(919)542-0939!! store06!!!!nightly.tcf NewYork!Albany!782-8311!19.2!ny.tcf In the first example, no setup is specified so autopoll.su will be loaded. The site name will be "Blaster", overriding the Description field of the setup; the phone number will be 1(919)542-0939; the baud rate will be taken from the setup since that field is blank; and the transfer command file will default to autopoll.tcf since the field is blank. In the second site record, the setup store06.su will be loaded. The site name, phone number, and baud rate will default to the values given in store06.su. The command file will be nightly.tcf. In the last site record, the file NewYork.su will be loaded. The site name will be "Albany", the phone number will be 782-8311, the baud rate will be set to 19.2 kbps, and the command file will be ny.tcf. ////////////////////////////// Transfer Command File (TCF) / ////////////////////////////// Autopoll uses a standard BLAST Transfer Command File (TCF) to specify the files to be sent and received. You may use a unique TCF for each site or the same TCF for several sites. A complete description of the Transfer Command File format can be found in the User Manual. Autopoll treats wildcards and remote commands (such as remote print and remote rename) as "try once" specifications. These transfers and commands are attempted during the first cycle only. Even if errors occur, Autopoll does not attempt the transfers or commands again. For this reason, wildcards and remote commands must be used with caution. ////////////// Setup Files / ////////////// Autopoll uses standard BLAST Setup files (.su files) to specify site-dependent configuration information. You may run Autopoll with either one or multiple setup files. Please note, whether you use one or multiple setup files, you must always create an autopoll.su setup file. You may use one setup file (autopoll.su) if all of the following conditions exist: The system type for each remote site is the same. The login for each remote site is the same. The password for each remote site is the same. You specify the phone number for each remote site in the site file. You must use multiple setup files if any of the following conditions exist: The remote sites have different system types. The remote sites have different logins. The remote sites have different passwords. When using multiple setup files, you may specify the phone number and baud rate for each remote site in either the site file or in the setup file for the site. A complete description and instructions for creating setup files can be found in the User Manual. /////////////////////// Where to Store Files / /////////////////////// Autopoll script files, transfer command files, and site files must be stored in the same directory, which must be your current working directory. //////////////////////// Configuration Example / //////////////////////// Suppose you have been asked to set up a polling network for a client who has a central UNIX system and two remote UNIX sites. The sites are configured as follows: Site name Phone Login, password 1. Sam's Discount Mart 542-0307 buz, apollo11 2. Metro Army Surplus 542-5694 neil, saturn5 Because the logins are different, different BLAST setup files are created for each site ("sam" and "metro"). The site file, retail.dat, for the polling network then becomes: ....|....1....|....2....|....3....|....4....|....5.... Retail Site List for My Polling Network sam!Sam's Discount!542-0307!!sams.tcf metro!Metro Army Surplus!542-5694!!metro.tcf The first line of the file is treated as a comment since it begins with a space. The last two lines are the site records. Actually, the site records may be duplicating information already in the setups since BLAST setups contain fields for the Phone Number and Description. If so, the site records could be simplified: ....|....1....|....2....|....3....|....4....|....5.... Retail Site List for My Polling Network (Phone number and Description taken from setups) sam!!!!sams.tcf metro!!!!metro.tcf The site file now has an additional comment line; otherwise it is equivalent to the previous site list. According to the site list, a command file called sams.tcf will be executed when Autopoll connects to Sam's Discount Mart. The command file metro.tcf will be executed when Autopoll connects to Metro Army Surplus (The rules for writing command files are described in the User Manual). Suppose you needed to get two files from Sam and send him one. The TCF sams.tcf might look like this: ....|....1....|....2....|....3....|....4....|....5.... +/usr/buz/acq12.txt /usr/client/sam1 +/usr/buz/wk_82 /usr/client/sam2 /usr/tmp/read_me/OVW The "+" sign in column 1 of the first two lines signifies that BLAST will per- form a GET. Thus, BLAST will get /usr/buz/acq12.txt and /usr/buz/wk_82 from Sam. The files will be called /usr/client/sam1 and /usr/client/sam2 respectively on the local system. The last line of the TCF signifies a SEND because it does not begin with "+". Thus, BLAST will send /usr/tmp/read_me to the remote system. Since the remote filename is not specified, BLAST will use the same name on the remote side. The /OVW switch signifies that BLAST may overwrite an existing file of the same name on the remote system. Luckily, metro.tcf is similar to sams.tcf: ....|....1....|....2....|....3....|....4....|....5.... +/usr/neil/acq12.txt /usr/client/metro1 +/usr/neil/wk_82 /usr/client/metro2 /usr/tmp/read_me/OVW These files--retail.dat, sams.tcf, and metro.tcf--are created using a standard text editor (e.g., vi) and saved in the same directory as the Autopoll scripts. With the required files ready, the BLAST command line to start Autopoll could then be: blast autopoll 3 retail which specifies a maximum of three attempts to complete the polling session with retail.dat. ////////////////////////////////////////////// Other Files That Use the Site Filename Stub / ////////////////////////////////////////////// Autopoll distinguishes several special files by appending different extensions to the site filename stub. The extensions are shown below. Extension Created by Meaning Example* --------- --------------- --------------------- ---------- .dat user (required) Site file retail.dat .stp user (optional) Stop file retail.stp .hdr user (optional) Banner file retail.hdr .log Autopoll Short summary file retail.log .prn Autopoll Long summary file retail.prn .cap Autopoll Capture file retail.cap ---------------------------------------------------------- *where the site filename stub is "retail". Site File. The site file (e.g., retail.dat) is the master list of information ========== about the sites to be polled. The format of the site file was discussed earlier. Stop File. The stop file (e.g., retail.stp) is used to exit prematurely but ========== gracefully from a polling session. Autopoll checks for the existence of the stop file in the Autopoll directory before connecting to every site. If the file is found, the polling session is terminated. The contents of the file are irrelevent, so touch is a convenient way to create a stop file: touch retail.stp Autopoll deletes the stop file before exiting. Banner File. The banner file (e.g., retail.hdr) is an optional file that ============ Autopoll will print before the summary file is printed at the end of polling. Printing is performed by the BLAST LPRINT statement. Summary Files. The short summary file (e.g., retail.log) is printed at the end ============== of polling. The long summary file (e.g., retail.prn) is prepared by Autopoll but not printed. The file is saved in the Autopoll directory. More information about the summary files is given below. Capture File. The capture file is opened when TRACE is specified on the command ============= line. The capture file records the incoming data stream of the entire polling session, e.g., modem initialization commands and login dialogs. Caution! With large site lists the capture file can become very large. This feature is used primarily for troubleshooting. /////////////////////////// Long and Short Summaries / /////////////////////////// The Autopoll status files summarize the activities of a polling session. Two summary files are generated. Printed at the end of polling, the short summary is most helpful when polling goes well since a quick glance is all it takes to see the outcome of the polling session. The long summary is helpful for troubleshooting. A typical short summary file looks like this: ******************** 02/09/96 11:15:29 ******************** Cycle 1 1. FAILED: Sam's Discount < Error transferring 3 file(s). > 2. success: Metro Army Surplus Cycle 2 1. success: Sam's Discount ***************************************************************** A long summary file looks like this: 02/09/96 ************************************************************** 11:15:33 * Cycle: 1 Site: 1 * * Name: Sam's Discount * Phone: 542-0307 * TCF: sams.tcf * Log: C1S001.log * *-------------------- SESSION INFORMATION -------------------- * Filetransfer error -8: DCD lost during transfer * Error transferring 3 file(s). Log file follows: * * **** BLAST Professional UNIX 10.7.3 on remote system [uov] * LOSS OF CARRIER, ending Filetransfer * File transfer interrupted, 12% of file acq12.txt received ************************************************************** 02/09/96 ************************************************************** 11:16:30 * Cycle: 1 Site: 2 * * Name: Metro Army Surplus * Phone: 542-5694 * TCF: metro.tcf * Log: C1S002.log * *-------------------- SESSION INFORMATION -------------------- * No errors encountered. * Log file has been deleted. ************************************************************** 02/09/96 ************************************************************** 11:18:49 * Cycle: 2 Site: 1 * * Name: Sam's Discount * Phone: 542-0307 * TCF: C1S001.tcf * Log: C2S001.log * *-------------------- SESSION INFORMATION -------------------- * No errors encountered. * Log file has been deleted. ************************************************************** 02/09/96 ************************************************************** 11:20:41 * Polling complete: all sites polled successfully. ************************************************************** ////////////////////// Autopoll under cron / ////////////////////// cron is a UNIX-supplied scheduler. To run Autopoll under cron, you will need to create a crontab file that launches BLAST. Usually, the crontab specifies a shell procedure that starts BLAST after setting some environment variables like BLASTDIR and SETUPDIR. A typical crontab file could consist of a single entry: 10 2 * * * /usr/doug/gopoll This line instructs cron to start the program /usr/doug/gopoll at 2:10 every day. The gopoll program may be a shell procedure, such as: PATH=$PATH:/usr/blast BLASTDIR=/usr/blast SETUPDIR=/usr/doug BPRINTER="lp -d laser %s" export PATH BLASTDIR SETUPDIR BPRINTER cd /usr/blast/poll blast autopoll 6 nightly -b This shell procedure sets environment variables needed by BLAST and ensures that the correct directory has been entered before starting BLAST. The appropriate values of these environment variables are likely to be different, of course, on your system. The -b switch disables terminal output. Consult the UNIX man page on cron and crontab for more information about cron. ////////////////// Tips and Tricks / ////////////////// Keep it Simple. Polling sessions can quickly become complicated if there are =============== many remote sites and several file transfers that must be performed with each one. Create simple but sensible directory structures to support the polling network. As a rule of thumb, command files should contain lines no longer than 80 characters so that they can be easily viewed and edited on standard terminals. Problems Don't "Just Go Away". In a large polling network, it is not uncommon ============================== to have a few remote sites that consistently stumble. Especially frustrating are intermittent difficulties. Take some time to study problem sites carefully. - Are the phone lines reliable? Can a phone line be dedicated to polling (at least during the polling session)? - Are the modems compatible with each other? - Is BLAST or BHOST being initiated correctly on the remote? - Are the expected files consistently present? Are directory and file permissions set appropriately? Tune BLAST Protocol Parameters. Certain BLAST protocol parameters can be tuned =============================== for better performance with Autopoll: Logon Timeout: 20 Inactivity Timeout: 20 DCD Loss Response: ABORT These settings permit Autopoll to react more quickly to lost connections than the default settings allow. You may also wish to experiment with compression levels and packet size to find settings for best throughput. Exploiting BPRINTER. The summary file is printed by the BLAST script LPRINT ==================== command, which is tied to the BPRINTER environment variable. You can alter the behavior of LPRINT by changing the definition of BPRINTER in the shell environment. For example, Autopoll can append summaries to an archive instead of printing them if BPRINTER performs a catenation: BPRINTER="cat %s >> /usr/blast/summaries" export BPRINTER BLAST substitutes %s for the name of the short summary when the command is executed. For more information about BPRINTER, see the BLAST User Manual. ///////////////////// Modifying Autopoll / ///////////////////// Because Autopoll is written in BLAST's scripting language, it is very easy to modify. The Autopoll scripts are thoroughly commented, and the BLAST User Manual contains a complete description of BLAST's scripting language with many examples. Feel free to customize Autopoll to your exact needs. User-Supplied Scripts. The behavior of Autopoll can also be changed by writing ====================== one or more user-supplied scripts. Autopoll checks for the existence of these scripts at various points during execution. If it finds a user-supplied script, the script is CALLed. Autopoll tests the value of @STATUS when the script returns; polling continues normally if @STATUS equals 0; otherwise the site is marked as failed. User-supplied scripts reside in the same directory as the Autopoll scripts. They are CALLed at the following places during execution: autousr0.scr - before the first site is polled autousr1.scr - before every attempt to CONNECT autousr2.scr - before every attempt to start FILETRANSFER autousr3.scr - before every attempt to DISCONNECT autousr4.scr - before Autopoll terminates Because BLASTscript variables are global, a user-supplied script must be careful not to disturb the contents of any variables needed by Autopoll. The following variables may be changed freely by any user-supplied script: @STATUS @EFERROR @input @temp @xferOK @msg @start You can also create new variables if you wish. To prevent confusion, it is suggested that new variables begin with "u", e.g., @uvar2. File I/O with User-Supplied Scripts. Autopoll opens file handles 1-7 at various ==================================== points during execution. The handles have the following functions: 1 - read-only - current site (or retry) file 2 - - utility I/O 3 - - utility I/O 4 - - utility I/O 5 - write-only - complete polling results 6 - write-only - retry file for next cycle 7 - write-only - brief polling results (printed out) Any of the handles reserved for utility I/O may be opened by user-supplied scripts as long as the handles are freed before the scripts return to Autopoll (i.e., each user script must close its own files). User scripts may also write to the status files, handle 5 and handle 7, if they wish. An example of this technique is shown below. Autopoll closes the log file before calling user-supplied scripts. If a user script opens a log of its own, the log must be closed before execution returns to Autopoll. NOTE: If TRACE is specified, a user-supplied script cannot use the TCAPTURE command because TRACE uses the capture buffer for its own use. Sample User-Supplied Script. The following user-supplied script creates a ============================ command file on a remote UNIX system consisting of all the files in a given target directory. The file is retrieved for use by Autopoll in the normal way. This script provides a way to GET an unspecified number of files from the remote system without sacrificing Autopoll's ability to recover from file transfer errors. # autousr2.scr # # Autopoll user-supplied script # # This script assumes that Autopoll is logged into a UNIX-based system! # # The script creates a command file on the remote UNIX machine # consisting of all files in the given directory. The # command file is then retrieved so Autopoll can use it # in the normal way. # # To create the command file, awk processes the # output from ls. Since we will be getting each file in # the listing, a '+' is inserted at the front of each # filename. The /FWD switch is appended to the filename # so the file will be deleted from the remote system if the # transfer is successful. On the local (Autopoll) side, /OVW is # appended to the filename so the file will overwrite an existing # file. Assuming the target directory is /u/out and the command # file is x.tcf, the necessary command is: # # ls /u/out | awk '{printf("+/u/out/%s/FWD %s/OVW\n", $1, $1)}' > x.tcf # # The script returns 0 if there is no error. The following errors # are returned: # # 1 - unable to detect remote shell prompt ($, %, or #) # 2 - target directory is empty # 3 - can't start BLAST on remote system # set @upath = "/u/out" # adjust path to suit! set @temp = "ls " strcat @temp, @upath strcat @temp, " | " # e.g, "ls /u/out | " strcat @temp, "awk '{printf(\"+" # attach awk command strcat @temp, @upath strcat @temp, "/%s/FWD %s/OVW\\n\", $1, $1)}' > " strcat @temp, @worktcf # @worktcf is Autopoll variable tsend cr # get the attention of the remote ttrap 2, "$", "#", "%" if @STATUS = "0" return 1 # return error wait 1 idle tsend "ls ", @upath, " | wc -l", cr # how many files? ttrap 2, " 0" # no files in directory? if @STATUS = "1" fwrite 5, @b, " Remote directory ", @upath, " is empty." write "remote directory ", @upath, " is empty!" return 2 # dir is empty, nothing to do! end write "preparing command file remotely" # inform user wait 1 idle tsend @temp, cr # run the awk script ttrap 2, "$", "#", "%" if @STATUS = "0" return 1 filetransfer # now retrieve the command file if @STATUS not = "0" return 3 # can't start BLAST on remote get @worktcf @worktcf to esc return @EFERROR # returns 0 if no error # # End of script. /////////////////////////// Configuration Worksheets / /////////////////////////// The following "worksheets" may help you organize the large amount of information needed to set up a polling network successfully. A. List the machines in your polling network. Site Name Phone Modem Port BLAST ver. Login,password ------- ------ ------- -------- ------ ---------- -------------- Central X X 1. 2. 3. ... --------------------------------------------------------------------------- B. Decide whether different setup files will be needed for each site. If so, create the setups and list their names. Remember, autopoll loads the setup AUTOPOLL by default. Different setups? Yes / No Site Name Setup ---------- ------------- ------------- 1. 2. 3. ... ------------------------------------------ C. Set up the remote sites and test each connection manually. Make sure the following sequence of keyboard commands work flawlessly: Connect -- dials the modem and logs in if necessary Filetransfer -- enters BLAST filetransfer ESC -- exits BLAST filetransfer Disconnect -- logs off and hangs up the phone D. Name the site file. Build the entries in the site file with any standard text editor (e.g., vi), selecting appropriate name(s) for the TCF files. site filename: __________.dat Setup Name Phone Baud TCF -------- ------------ ------- ------ ----------- -------------------------------------------------------- E. Write the site file and save it in the autopoll directory. F. List the files to be transferred with each site, and the direction of transfer (S=send, G=get): Site S/G Remote name Local name Options ------- --- ---------------------- ---------------------- -------- 1. 2. 3. ... -------------------------------------------------------------------------- G. Write the various TCF files and put them in the autopoll directory. H. Decide how many cycles to allow for polling and when to start polling: Cycles: Start: I. Build the command line to start autopoll: blast autopoll Max_cycles Site_file [Start_time] J. cd to the autopoll directory, type the command line, and watch the script fly! --oOo-- # ident "autodocu.txt 1.3"