RE: Why are the services not started automatically on reboot?

[Sonderegger, Markus]

Hello Lutz
Thanks for your comments.

> You are using RHEL I think?
Yes

> Are you using Systemd to start your applications or handle Monit the 
> application startup only?
Monit

> Any dependency defined for the services?
Yes

> There is no reason to do a "monit start all" after your system is up and 
> running. As long as you use the default setting ("onreboot start") Monit 
> start all defined applications.
Good news, that is also my understanding.

If changed 2 service checks from PROGRAM to PROCESS and now the debug output in 
monit.log looks much more understandable.
I do not know why service check PROGRAM generated strange behavior like calling 
several time the START method on the same service  but it could be related to 
having the check program taking time to check if the applications run as they 
are heavy and take its time to start up. The check status is only collected 
after the next poll cycle.
I will investigate in more detail when I find time for it.

What still puzzles me that monit is not calling the START method for services 
which have no dependency and for the service which is at the base of the 
dependency tree (all other services depend from it) but calls the RESTART 
method.
Here the definition of the base service others depend on it
CHECK PROCESS mqm_TBCOM MATCHING "^/opt/mqm/bin/amqzxma0 -m TBCOM -u mqm$"
    START   = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/strmqm 
TBCOM'" UID mqm GID mqm
    STOP    = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/endmqm 
-i TBCOM'" UID mqm GID mqm
    RESTART = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; 
/opt/mqm/bin/endmqm -p TBCOM; /opt/mqm/bin/strmqm TBCOM'" UID mqm GID mqm
    IF NOT EXIST THEN RESTART
    IF NOT EXIST THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
    IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR
    IF 2 RESTARTS WITHIN 5 CYCLES THEN EXEC "/u01/app/bcom/monit/logger.sh 
FATAL"

The log shows that the RESTART method gets called.
[CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' exec: 
'/u01/app/bcom/monit/logger.sh CRITICAL'
[CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' trying to restart
[CEST Aug 14 22:07:14] debug    : 'appia_APPIA' process test failed [pid=22195] 
-- No such process
[CEST Aug 14 22:07:14] debug    : Cannot open proc file '/proc/2377/stat' -- No 
such file or directory
[CEST Aug 14 22:07:14] debug    : system statistic error -- cannot read 
/proc/2377/stat
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' restart: '/bin/bash -c . 
/u01/app/bcom/settings/bash_profile; /opt/mqm/bin/endmqm -p TBCOM; 
/opt/mqm/bin/strmqm TBCOM'
[CEST Aug 14 22:07:22] debug    : AMQ8146E: IBM MQ queue manager not available.
...

For all other services which are dependent of the base service monit correctly 
calls the START method.
Any ideas why this happens?
Could it be that the services do have an EXEC method defined. I use it to log 
RESTART of services into syslog calling the logger command.
It would be very nice if monit ALERT would be configurable in a sense that not 
only mailing is possible but another transport could be called e.g. logger 
command using syslog.

I also noticed that
CHECK DIRECTORY mqsi_mqsibackout_CS WITH PATH 
/u01/app/mqsi/filenodes/CcpcsfileToReconSwift940/CCPCSFILE.TO.RECON.SWIFT.940/mqsibackout
    IF CHANGED TIMESTAMP THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
generates the following log entry I do not understand as the directory has not 
changed
[CEST Aug 14 22:07:27] debug    : 'mqsi_mqsibackout_CS' directory exists
[CEST Aug 14 22:07:27] debug    : 'mqsi_mqsibackout_CS' is directory
[CEST Aug 14 22:07:27] error    : 'mqsi_mqsibackout_CS' modify/change time for 
/u01/app/mqsi/filenodes/CcpcsfileToReconSwift940/CCPCSFILE.TO.RECON.SWIFT.940/mqsibackout
 changed from N/A to Fri, 21 Jun 2019 16:54:48
[CEST Aug 14 22:07:27] info     : 'mqsi_mqsibackout_CS' exec: 
'/u01/app/bcom/monit/logger.sh CRITICAL'

Many thanks for any help.
Regards
Markus

-----Original Message-----
From: monit-general <monit-general-bounces+markus.sonderegger=address@hidden> 
On Behalf Of Lutz Mader
Sent: Sonntag, 11. August 2019 18:19
To: This is the general mailing list for monit <address@hidden>
Subject: Re: Why are the services not started automatically on reboot?

Hello Markus,
let us know the used Monit version (the output from monit -V) and the Linux 
version you used.

With regards, Lutz

p.s.
You are using RHEL I think?

Are you using Systemd to start your applications or handle Monit the 
application startup only?

Any dependency defined for the services?

p.s.
There is no reason to do a "monit start all" after your system is up and 
running. As long as you use the default setting ("onreboot start") Monit start 
all defined applications.

To prevent the default behaviour you can use "onreboot laststate", but Monit 
does not start stopped applications after a system restart then.
But you can stop applications and do some maintenance after a system restart 
and start the aplications manually.
You can use "onreboot" with any service definition also.

I use something like this, for example (with Monit 5.25.2).
check process Postgres with pidfile /var/run/pgsql/data.pid
  onreboot laststate
  start program "/bin/launchctl load -F
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  stop program "/bin/launchctl unload
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  if failed unixsocket /var/run/pgsql/.s.PGSQL.5432 then restart #  if failed 
host macmini.local port 5432 then restart
  if 3 restarts within 30 cycles then unmonitor


*****JuliusBaer Disclaimer***** This e-mail is for the intended recipient only 
and may contain confidential or privileged information. If you have received 
this e-mail by mistake, please contact us immediately and completely delete it 
(and any attachments) and do not forward it or inform any other person of its 
contents. If you send us messages by e-mail, we take this as your authorisation 
to correspond with you by e-mail, however, we reserve the right not to execute 
orders and instructions transmitted by e-mail at any time and without further 
explanation. If you do not wish to receive any further e-mail correspondence 
please let us know. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, amended, corrupted, lost, 
destroyed, arrive late or incomplete, or contain viruses. Neither the Julius 
Baer Group nor the sender accept liability for any errors or omissions in the 
content of this message which arise as a result of its e-mail transmission. 
Please note that all e-mail communications to and from the Julius Baer Group 
may be monitored. This communication is for informational purposes only. It is 
not intended as an offer or solicitation for the purchase or sale of any 
financial instrument or as an official confirmation of any transaction.

smime.p7s
Description: S/MIME Cryptographic Signature