irq plugin: Added plugin to collect interrupt counters.
[collectd.git] / src / collectd.pod
1 =head1 NAME
2
3 collectd - System statistics collection daemon
4
5 =head1 SYNOPSIS
6
7 collectd I<[options]>
8
9 =head1 DESCRIPTION
10
11 collectd is a daemon that collects various system statistics periodically and
12 stores them into RRD-files. Which data is collected depends on compile-time
13 settings. The following features may be available:
14
15 =over 4
16
17 =item
18
19 Apache server stats (I<apache>)
20
21 =item
22
23 Apple hardware sensors (I<apple_sensors>, Darwin only)
24
25 =item
26
27 Battery status (I<battery>)
28
29 =item
30
31 CPU utilization (I<cpu>)
32
33 =item
34
35 Mountpoint usage (I<df>)
36
37 =item
38
39 Disk and partition usage/throughput (I<disk>)
40
41 =item
42
43 DNS traffic (I<dns>)
44
45 =item
46
47 Email usage (I<email>)
48
49 =item
50
51 Harddisk temperatures (I<hddtemp>)
52
53 =item
54
55 Irq (I<irq>)
56
57 =item
58
59 System load averages (I<load>)
60
61 =item
62
63 Motherboard monitor (I<mbmon>)
64
65 =item
66
67 Memory usage (I<memory>)
68
69 =item
70
71 MySQL statistics (I<mysql>)
72
73 =item
74
75 NFS utilization (I<nfs>, Linux only)
76
77 =item
78
79 Network latency (I<ping>)
80
81 =item
82
83 Number of processes (I<processes>, Linux only)
84
85 =item
86
87 lm_sensors information (I<sensors>, Linux only)
88
89 =item
90
91 Serial port traffic (I<serial>, Linux only)
92
93 =item
94
95 Swap usage (I<swap>)
96
97 =item
98
99 Tape drive usage (I<tape>, Solaris only)
100
101 =item
102
103 Network traffic (I<traffic>)
104
105 =item
106
107 Number of users logged into the system (I<users>)
108
109 =item
110
111 System ressources used by VServers (I<vserver>)
112
113 =item
114
115 Wireless network stats (I<wireless>)
116
117 =back
118
119 =head1 OPTIONS
120
121 =over 4
122
123 =item B<-C> I<E<lt>config-fileE<gt>>
124
125 Specify an alternative config file. This is the place to go when you wish to
126 change B<collectd>'s behavior. The path may be relative to the current working
127 directory.
128
129 =item B<-P> I<E<lt>pid-fileE<gt>>
130
131 Specify an alternative pid file. This overwrites any settings in the config 
132 file. This is thought for init-scripts that require the PID-file in a certain
133 directory to work correctly. For everyday-usage use the B<PIDFile>
134 config-option.
135
136 =item B<-f>
137
138 Don't fork to the background. I<collectd> will also B<not> close standard file
139 descriptors, detach from the session nor write a pid file. This is mainly
140 thought for 'supervisioning' init replacements such as I<runit>.
141
142 =item B<-h>
143
144 Output usage information and exit.
145
146 =back
147
148 =head1 MODES
149
150 collectd can operate in four different operating modes. The modes are described
151 below.
152
153 The simplest mode is the so called B<local mode>. Data is collected locally and
154 written in RRD files that reside in I<DataDir>. This is the default mode when
155 collectd is linked against C<librrd>.
156
157 The modes B<client mode> and B<server mode> are used to send data over a
158 network and receive it again.
159
160 In B<client mode> the daemon collects the data locally and sends its results
161 to one or more network addresses. No RRD files are written locally in this
162 case. If collectd is not linked against C<librrd> this is the default mode.
163
164 If started in B<server mode> the daemon will listen on one or more interfaces
165 and write the data it receives to RRD files. No data is collected locally.
166
167 In the last mode, B<log mode>, data is collected locally and written in
168 text files that reside in I<DataDir>.
169
170 Please refer to L<collectd.conf(5)> for the configuration options and default
171 values.
172
173 =head1 SPECIAL PLUGINS
174
175 =head2 apache
176
177 This module connects to an Apache webserver and expects the output produced by
178 B<mod_status.c>. If requires B<libcurl> to set up the HTTP connection and issue
179 the request(s). The following is a sample config for the Apache webserver. The
180 use of C<ExtendedStatus on> is mandatory.
181
182   ExtendedStatus on
183   <IfModule mod_status.c>
184     <Location /mod_status>
185       SetHandler server-status
186     </Location>
187   </IfModule>
188
189 This plugin requires further configuration. Please read L<collectd.conf(5)>.
190
191 =head2 cpufreq
192
193 This module reads F</sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq> (for
194 the first CPU installed) to get the current CPU frequency. If this file does
195 not exist make sure B<cpufreqd> (L<http://cpufreqd.sourceforge.net/>) or a
196 similar tool is installed and an "cpu governor" (that's kernel module) is
197 loaded.
198
199 =head2 email
200
201 This plugin collects data indirectly by providing a UNIX socket that external
202 programs can connect to. A simple line based protocol is used to communicate
203 with the plugin:
204
205 E-Mail type (e.g. "ham", "spam", "virus", ...) and size (bytes):
206
207   e:<type>:<size>
208
209 If C<size> is less than or equal to zero, C<size> is ignored.
210
211 Spam score:
212
213   s:<value>
214
215 Successful spam checks (e.g. "BAYES_99", "SUBJECT_DRUG_GAP_C", ...):
216
217   c:<type1>[,<type2>,...]
218
219 Each line is limited to 256 characters (including the newline character). 
220 Longer lines will be ignored.
221
222 =head2 mysql
223
224 Requires B<mysqlclient> to be installed. It connects to the database when
225 started and keeps the connection up as long as possible. When the connection is
226 interrupted for whatever reason it will try to re-connect. The syslog will
227 contain loud complaints in case anything goes wrong.
228
229 This plugin issues C<SHOW STATUS> and evaluates C<Bytes_{received,sent}>,
230 C<Com_*> and C<Handler_*> which correspond to F<traffic-mysql.rrd>,
231 F<mysql_commands-*.rrd> and F<mysql_handler-*.rrd>. Also, the values of
232 C<Qcache_*> are put in F<mysql_qcache.rrd> and values of C<Threads_*> are put
233 in F<mysql_threads.rrd>. Please refer to the B<MySQL reference manual>,
234 I<5.2.4. Server Status Variables> for an explanation of these values.
235
236 =head2 sensors
237
238 The B<sensors> module uses lm_sensors to retrieve sensor-values. This means
239 that all the needed modules have to be loaded and lm_sensors has to be
240 configured (most likely by editing F</etc/sensors.conf>. Read
241 L<sensors.conf(5)> for details.
242
243 The B<lm_sensors> homepage can be found at
244 L<http://secure.netroedge.com/~lm78/>.
245
246 =head2 mbmon
247
248 The B<mbmon> module uses mbmon to retrieve temperature, voltage, etc.
249
250 collectd connects to B<localhost> (127.0.0.1), port B<411/tcp>.
251 The B<Host> and B<Port> options can be used to change these
252 default values. See L<collectd.conf(5)> for details. C<mbmon> has to be
253 running to work correctly. If C<mbmon> is not running timeouts may appear
254 which may interfere with other statistics..
255
256 C<mbmon> must be run with the -r option ("print TAG and Value format");
257 Debian's /etc/init.d/mbmon script already does this, other people
258 will need to ensure that this is the case.
259
260 =head2 hddtemp
261
262 To get values from B<hddtemp> collectd connects to B<localhost> (127.0.0.1),
263 port B<7634/tcp>. The B<Host> and B<Port> options can be used to change these
264 default values. See L<collectd.conf(5)> for details. C<hddtemp> has to be
265 running to work correctly. If C<hddtemp> is not running timeouts may appear
266 which may interfere with other statistics..
267
268 The B<hddtemp> homepage can be found at
269 L<http://www.guzu.net/linux/hddtemp.php>.
270
271 =head2 irq
272
273 The B<irq> modules uses F</proc/interrupts> to retrieve interrupts per second.
274 If there is more than one CPU all counters are added for each interrupt.
275
276 =head2 vserver
277
278 B<VServer> support is only available for Linux. It cannot yet be found in a 
279 vanilla kernel, though. To make use of this plugin you need a kernel that has 
280 B<VServer> support built in, i.e. you need to apply the patches and compile 
281 your own kernel, which will then provide the /proc/virtual filesystem that is
282 required by this plugin.
283
284 The B<VServer> homepage can be found at L<http://linux-vserver.org/>.
285
286 =head1 RRD FILES
287
288 The RRD files are created automatically. The size of the RRAs depend on the
289 compile time settings of I<step> and I<width>. With the default values (I<step>
290 = B<10>, I<width> = B<1200>) the following RRAs are created:
291
292   RRA:AVERAGE:0.1:1:8640
293   RRA:AVERAGE:0.1:50:1210
294   RRA:AVERAGE:0.1:223:1202
295   RRA:AVERAGE:0.1:2635:1201
296   RRA:MIN:0.1:1:8640
297   RRA:MIN:0.1:50:1210
298   RRA:MIN:0.1:223:1202
299   RRA:MIN:0.1:2635:1201
300   RRA:MAX:0.1:1:8640
301   RRA:MAX:0.1:50:1210
302   RRA:MAX:0.1:223:1202
303   RRA:MAX:0.1:2635:1201
304
305 By default collectd uses a 10 second I<step>. Thus the RRAs contain the
306 following timespans. If you've changed the I<step> at compile time you will
307 have calculate resolution and timespan yourself.
308
309   PDP per CDP |  Resolution  | Data points | Timespan
310   ------------+--------------+-------------+---------
311             1 | 10.0 seconds !        8640 ! 1 day
312            50 |  8.3 minutes |        1210 | 1 week
313           223 | 37.2 minutes |        1202 | 1 month
314          2635 |  7.3 hours   |        1201 | 1 year
315
316 The DS'es depend on the module creating the RRD files:
317
318 =over 4
319
320 =item Apache traffic (F<apache/apache_bytes.rrd>)
321
322   DS:count:COUNTER:HEARTBEAT:0:134217728
323
324 =item Apache requests (F<apache/apache_requests.rrd>)
325
326   DS:count:COUNTER:HEARTBEAT:0:1048576
327
328 =item Apache scoreboard (F<apache/apache_scoreboard-I<E<lt>nameE<gt>>.rrd>)
329
330   DS:count:GAUGE:HEARTBEAT:0:U
331
332 =item Apple temperature sensor (F<apple_sensors/temperature-I<E<lt>nameE<gt>>.rrd>)
333
334   DS:value:GAUGE:HEARTBEAT:U:U
335
336 =item Apple fanspeed sensor (F<apple_sensors/fanspeed-I<E<lt>nameE<gt>>.rrd>)
337
338   DS:value:GAUGE:HEARTBEAT:U:U
339
340 =item Battery charge (F<battery-I<E<lt>nameE<gt>>/charge.rrd>)
341
342   DS:charge:GAUGE:HEARTBEAT:0:U
343
344 =item Battery current (F<battery-I<E<lt>nameE<gt>>/current.rrd>)
345
346   DS:current:GAUGE:HEARTBEAT:U:U
347
348 =item Battery voltage (F<battery-I<E<lt>nameE<gt>>/voltage.rrd>)
349
350   DS:voltage:GAUGE:HEARTBEAT:U:U
351
352 =item CPU (F<cpu-I<E<lt>numE<gt>>.rrd>)
353
354   DS:user:COUNTER:HEARTBEAT:0:100
355   DS:nice:COUNTER:HEARTBEAT:0:100
356   DS:syst:COUNTER:HEARTBEAT:0:100
357   DS:idle:COUNTER:HEARTBEAT:0:100
358   DS:wait:COUNTER:HEARTBEAT:0:100
359
360 =item CPU frequency (F<cpufreq-I<E<lt>numE<gt>>.rrd>)
361
362   DS:value:GAUGE:HEARTBEAT:0:U
363
364 =item Mountpoints (F<df-I<E<lt>pathE<gt>>.rrd>)
365
366   DS:used:GAUGE:HEARTBEAT:0:U
367   DS:free:GAUGE:HEARTBEAT:0:U
368
369 =item Diskstats (F<disk-I<E<lt>majorE<gt>>-I<E<lt>minorE<gt>>.rrd>)
370
371   DS:rcount:COUNTER:HEARTBEAT:0:U
372   DS:rmerged:COUNTER:HEARTBEAT:0:U
373   DS:rbytes:COUNTER:HEARTBEAT:0:U
374   DS:rtime:COUNTER:HEARTBEAT:0:U
375   DS:wcount:COUNTER:HEARTBEAT:0:U
376   DS:wmerged:COUNTER:HEARTBEAT:0:U
377   DS:wbytes:COUNTER:HEARTBEAT:0:U
378   DS:wtime:COUNTER:HEARTBEAT:0:U
379
380 =item Diskstats (F<partition-I<E<lt>majorE<gt>>-I<E<lt>minorE<gt>>.rrd>)
381
382   DS:rcount:COUNTER:HEARTBEAT:0:U
383   DS:rbytes:COUNTER:HEARTBEAT:0:U
384   DS:wcount:COUNTER:HEARTBEAT:0:U
385   DS:wbytes:COUNTER:HEARTBEAT:0:U
386
387 =item E-Mail count (F<email/email-I<E<lt>typeE<gt>>.rrd>)
388
389   DS:count:GAUGE:HEARTBEAT:0:U
390
391 =item E-Mail size (F<email/email_size-I<E<lt>typeE<gt>>.rrd>)
392
393   DS:size:GAUGE:HEARTBEAT:0:U
394
395 =item HDD Temperature (F<hddtemp-I<E<lt>majorE<gt>>-I<E<lt>minorE<gt>>.rrd>)
396
397   DS:value:GAUGE:HEARTBEAT:U:U
398
399 =item Irq (F<irq-I<E<lt>irqnumberE<gt>>-I<E<lt>descriptionE<gt>>.rrd>)
400
401   DS:value:COUNTER:HEARTBEAT:U:U
402
403 =item System load (F<load.rrd>)
404
405   DS:shortterm:GAUGE:HEARTBEAT:0:100
406   DS:midterm:GAUGE:HEARTBEAT:0:100
407   DS:longterm:GAUGE:HEARTBEAT:0:100
408
409 =item Memory usage (F<memory.rrd>)
410
411   DS:used:GAUGE:HEARTBEAT:0:9223372036854775807
412   DS:free:GAUGE:HEARTBEAT:0:9223372036854775807
413   DS:buffers:GAUGE:HEARTBEAT:0:9223372036854775807
414   DS:cached:GAUGE:HEARTBEAT:0:9223372036854775807
415
416 =item MySQL commands and handlers (F<mysql_commands-I<E<lt>commandE<gt>>.rrd> and F<mysql_handler-I<E<lt>handlerE<gt>>.rrd>)
417
418   DS:value:COUNTER:HEARTBEAT:0:U
419
420 =item MySQL query cache (F<mysql_qcache.rrd>)
421
422   DS:hits:COUNTER:HEARTBEAT:0:U
423   DS:inserts:COUNTER:HEARTBEAT:0:U
424   DS:not_cached:COUNTER:HEARTBEAT:0:U
425   DS:lowmem_prunes:COUNTER:HEARTBEAT:0:U
426   DS:queries_in_cache:GAUGE:HEARTBEAT:0:U
427
428 =item MySQL threads (F<mysql_threads.rrd>)
429
430   DS:running:GAUGE:HEARTBEAT:0:U
431   DS:connected:GAUGE:HEARTBEAT:0:U
432   DS:cached:GAUGE:HEARTBEAT:0:U
433   DS:created:COUNTER:HEARTBEAT:0:U
434
435 =item NFSv2 Procedures (F<nfs2_procedures-I<(client|server)>.rrd>)
436
437   DS:null:COUNTER:HEARTBEAT:0:U
438   DS:getattr:COUNTER:HEARTBEAT:0:U
439   DS:setattr:COUNTER:HEARTBEAT:0:U
440   DS:root:COUNTER:HEARTBEAT:0:U
441   DS:lookup:COUNTER:HEARTBEAT:0:U
442   DS:readlink:COUNTER:HEARTBEAT:0:U
443   DS:read:COUNTER:HEARTBEAT:0:U
444   DS:wrcache:COUNTER:HEARTBEAT:0:U
445   DS:write:COUNTER:HEARTBEAT:0:U
446   DS:create:COUNTER:HEARTBEAT:0:U
447   DS:remove:COUNTER:HEARTBEAT:0:U
448   DS:rename:COUNTER:HEARTBEAT:0:U
449   DS:link:COUNTER:HEARTBEAT:0:U
450   DS:symlink:COUNTER:HEARTBEAT:0:U
451   DS:mkdir:COUNTER:HEARTBEAT:0:U
452   DS:rmdir:COUNTER:HEARTBEAT:0:U
453   DS:readdir:COUNTER:HEARTBEAT:0:U
454   DS:fsstat:COUNTER:HEARTBEAT:0:U
455
456 =item NFSv3 Procedures (F<nfs3_procedures-I<(client|server)>.rrd>)
457
458   DS:null:COUNTER:HEARTBEAT:0:U
459   DS:getattr:COUNTER:HEARTBEAT:0:U
460   DS:setattr:COUNTER:HEARTBEAT:0:U
461   DS:lookup:COUNTER:HEARTBEAT:0:U
462   DS:access:COUNTER:HEARTBEAT:0:U
463   DS:readlink:COUNTER:HEARTBEAT:0:U
464   DS:read:COUNTER:HEARTBEAT:0:U
465   DS:write:COUNTER:HEARTBEAT:0:U
466   DS:create:COUNTER:HEARTBEAT:0:U
467   DS:mkdir:COUNTER:HEARTBEAT:0:U
468   DS:symlink:COUNTER:HEARTBEAT:0:U
469   DS:mknod:COUNTER:HEARTBEAT:0:U
470   DS:remove:COUNTER:HEARTBEAT:0:U
471   DS:rmdir:COUNTER:HEARTBEAT:0:U
472   DS:rename:COUNTER:HEARTBEAT:0:U
473   DS:link:COUNTER:HEARTBEAT:0:U
474   DS:readdir:COUNTER:HEARTBEAT:0:U
475   DS:readdirplus:COUNTER:HEARTBEAT:0:U
476   DS:fsstat:COUNTER:HEARTBEAT:0:U
477   DS:fsinfo:COUNTER:HEARTBEAT:0:U
478   DS:pathconf:COUNTER:HEARTBEAT:0:U
479   DS:commit:COUNTER:HEARTBEAT:0:U
480
481 =item Network latency / Ping (F<ping-I<E<lt>hostnameE<gt>>.rrd>)
482
483   DS:ping:GAUGE:HEARTBEAT:0:65535
484
485 =item Processes (F<processes.rrd>)
486
487   DS:running:GAUGE:HEARTBEAT:0:65535
488   DS:sleeping:GAUGE:HEARTBEAT:0:65535
489   DS:zombies:GAUGE:HEARTBEAT:0:65535
490   DS:stopped:GAUGE:HEARTBEAT:0:65535
491   DS:paging:GAUGE:HEARTBEAT:0:65535
492   DS:blocked:GAUGE:HEARTBEAT:0:65535
493
494 =item lm_sensors fanspeed sensor (F<sensors-I<E<lt>instE<gt>>.rrd> or F<lm_sensors-I<E<lt>chipE<gt>>/fanspeed-I<E<lt>instE<gt>>.rrd>)
495
496   DS:value:GAUGE:HEARTBEAT:U:U
497
498 =item lm_sensors temperature sensor (F<sensors-I<E<lt>instE<gt>>.rrd> or F<lm_sensors-I<E<lt>chipE<gt>>/temperature-I<E<lt>instE<gt>>.rrd>)
499
500   DS:value:GAUGE:HEARTBEAT:U:U
501
502 =item lm_sensors voltage sensor (F<sensors-I<E<lt>instE<gt>>.rrd> or F<lm_sensors-I<E<lt>chipE<gt>>/voltage-I<E<lt>instE<gt>>.rrd>)
503
504   DS:voltage:GAUGE:HEARTBEAT:U:U
505
506 =item Serial port traffic (F<serial-I<E<lt>numE<gt>>.rrd>)
507
508   DS:incoming:COUNTER:HEARTBEAT:0:U
509   DS:outgoing:COUNTER:HEARTBEAT:0:U
510
511 =item Spam score (F<email/spam_score.rrd>)
512
513   DS:score:GAUGE:HEARTBEAT:0:U
514
515 =item Spam checks (F<email/spam_check-I<E<lt>typeE<gt>>.rrd>)
516
517   DS:hits:GAUGE:HEARTBEAT:0:U
518
519 =item Swap usage (F<swap.rrd>)
520
521   DS:used:GAUGE:HEARTBEAT:0:1099511627776
522   DS:free:GAUGE:HEARTBEAT:0:1099511627776
523   DS:cached:GAUGE:HEARTBEAT:0:1099511627776
524   DS:resv:GAUGE:HEARTBEAT:0:1099511627776
525
526 =item Tape drive usage (F<tape-I<E<lt>nameE<gt>>.rrd>)
527
528   DS:rcount:COUNTER:HEARTBEAT:0:
529   DS:rmerged:COUNTER:HEARTBEAT:0:U
530   DS:rbytes:COUNTER:HEARTBEAT:0:U
531   DS:rtime:COUNTER:HEARTBEAT:0:U
532   DS:wcount:COUNTER:HEARTBEAT:0:U
533   DS:wmerged:COUNTER:HEARTBEAT:0:U
534   DS:wbytes:COUNTER:HEARTBEAT:0:U
535   DS:wtime:COUNTER:HEARTBEAT:0:U
536
537 =item Network traffic (F<traffic-I<E<lt>interfaceE<gt>>.rrd>)
538
539   DS:incoming:COUNTER:HEARTBEAT:0:U
540   DS:outgoing:COUNTER:HEARTBEAT:0:U
541
542 =item Interface packets (F<if_packets-I<E<lt>interfaceE<gt>>.rrd>)
543
544   DS:rx:COUNTER:HEARTBEAT:0:U
545   DS:tx:COUNTER:HEARTBEAT:0:U
546
547 =item Interface errors (F<if_errors-I<E<lt>interfaceE<gt>>.rrd>)
548
549   DS:rx:COUNTER:HEARTBEAT:0:U
550   DS:tx:COUNTER:HEARTBEAT:0:U
551
552 =item Users (F<users.rrd>)
553
554   DS:users:GAUGE:HEARTBEAT:0:65535
555
556 =item VServer load (F<vserver-I<E<lt>xidE<gt>>/load.rrd>)
557
558   DS:shortterm:GAUGE:HEARTBEAT:0:100
559   DS:midterm:GAUGE:HEARTBEAT:0:100
560   DS:longterm:GAUGE:HEARTBEAT:0:100
561
562 =item VServer threads (F<vserver-I<E<lt>xidE<gt>>/threads.rrd>)
563
564   DS:total:GAUGE:HEARTBEAT:0:65535
565   DS:running:GAUGE:HEARTBEAT:0:65535
566   DS:uninterruptible:GAUGE:HEARTBEAT:0:65535
567   DS:onhold:GAUGE:HEARTBEAT:0:65535
568
569 =item VServer network traffic (F<vserver-I<E<lt>xidE<gt>>/traffic-I<E<lt>nameE<gt>>.rrd>)
570
571   DS:incoming:COUNTER:HEARTBEAT:0:9223372036854775807
572   DS:outgoing:COUNTER:HEARTBEAT:0:9223372036854775807
573   DS:failed:COUNTER:HEARTBEAT:0:9223372036854775807
574
575 =item VServer processes (F<vserver-I<E<lt>xidE<gt>>/vs_processes.rrd>)
576
577   DS:total:GAUGE:HEARTBEAT:0:65535
578
579 =item VServer memory usage (F<vserver-I<E<lt>xidE<gt>>/vs_memory.rrd>)
580
581   DS:vm:GAUGE:HEARTBEAT:0:9223372036854775807
582   DS:vml:GAUGE:HEARTBEAT:0:9223372036854775807
583   DS:rss:GAUGE:HEARTBEAT:0:9223372036854775807
584   DS:anon:GAUGE:HEARTBEAT:0:9223372036854775807
585
586 =item Wireless link quality (F<wireless-I<E<lt>interfaceE<gt>>.rrd>)
587
588   DS:quality:GAUGE:HEARTBEAT:0:U
589   DS:power:GAUGE:HEARTBEAT:U:0
590   DS:noise:GAUGE:HEARTBEAT:U:0
591
592 =back
593
594 =head1 SEE ALSO
595
596 L<collectd.conf(5)>, L<rrdtool(1)>, L<sensors(1)>, L<hddtemp(8)>,
597 L<mbmon(1)>,
598 L<kstat(3KSTAT)>
599
600 =head1 AUTHOR
601
602 Florian Forster E<lt>octo@verplant.orgE<gt>
603
604 =cut