How to configure NTP client to sync with NTP server during system startup (boot) in (RHEL 7 / CentOS 7) Linux

While working on NTP server on my setup I realised that in Red Hat Enterprise Linux when we a service restart for NTPD or chronyd, the local time does not syncs with the NTP server immediately and it takes some time before syncing the local clock

So we have to manually sync the clock with NTP server using ntpdate

But what if a machine went for a reboot with incorrect date and time?

I also wanted to validate this scenario and it turns out NTPD in Red Hat 7 is very poor in handling this compared to SuSE Enterprise Linux 11.

To validate this

I changed the date of my setup

server2:~ # date --set "01 Jan 2018"
Mon Jan 1 00:00:00 IST 2018

Logged a message to understand the time of reboot

server2:~ # logger rebooting

My NTP server is configured properly and is enabled

server2:~ # systemctl is-enabled ntpd
enabled

Next reboot the blade to see if it takes the new date and time

server2:~ # reboot
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.

From the syslog we see during the shutdown stage the logs were stored with wrong time

Jan 1 00:00:05 server2 deepak: rebooting
Jan 1 00:00:21 server2 systemd: Stopped Dump dmesg to /var/log/dmesg.
Jan 1 00:00:21 server2 systemd: Stopping Dump dmesg to /var/log/dmesg...

Fortunately our hwclock had the correct date and time hence once the system started up it had proper time

Jan 1 00:00:21 server2 systemd: Stopping custom security scripts at start-up...
Jan 1 00:00:21 server2 systemd: Stopping OpenSSH server daemon...
Jan 1 00:00:21 server2 systemd: Stopping (null)...
Feb 20 15:34:50 server2 kernel: microcode: microcode updated early to revision 0x3a, date = 2017-01-30
Feb 20 15:34:50 server2 kernel: Initializing cgroup subsys cpuset
Feb 20 15:34:50 server2 kernel: Initializing cgroup subsys cpu
Feb 20 15:34:50 server2 kernel: Initializing cgroup subsys cpuacct

So let us change the date and time of our hwclock

server2:~ # hwclock --set --date "01 jan 2018"

server2:~ # hwclock ;date
Mon 01 Jan 2018 12:00:14 AM IST -0.547321 seconds
Tue Feb 20 15:41:16 IST 2018

server2:~ # logger rebooting again

server2:~ # reboot

From our syslog as we see the node went for shutdown with proper time since our date had correct time (only hwclock was changed)

Feb 20 15:41:07 server2 deepak: rebooting again
Feb 20 15:41:17 server2 systemd: Stopping custom firewall configuration using iptables...
Feb 20 15:41:17 server2 systemd: Unmounting RPC Pipe File System...
Feb 20 15:41:17 server2 systemd: Stopping system-systemdx2dfsck.slice.
Feb 20 15:41:17 server2 systemd: Stopped Stop Read-Ahead Data Collection 10s After Complete

Next once again as we see the system attempts to startup, hwclock time is used for logging purpose which for our case was incorrectly set before reboot

Feb 20 15:41:17 server2 systemd: Stopping Pro-active monitoring utility for unix systems...
Feb 20 15:41:17 server2 systemd: Stopping LSB: LinuxICCM Scanner...
Feb 20 15:41:17 server2 systemd: Stopping Self Monitoring and Reporting Technology (SMART) Daemon...
Jan 1 00:02:57 server2 journal: Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 62.8G available â current limit 4.0G).
Jan 1 00:02:57 server2 kernel: microcode: microcode updated early to revision 0x3a, date = 2017-01-30
Jan 1 00:02:57 server2 kernel: Initializing cgroup subsys cpuset
Jan 1 00:02:57 server2 kernel: Initializing cgroup subsys cpu
Jan 1 00:02:57 server2 kernel: Initializing cgroup subsys cpuacct

But NTPD never syncs the local clock date and time with NTP server until our node is completely UP

Once the node is up after few minutes I observed the local time syncing with NTP

Jan 1 00:04:04 server2 rsyslogd: action 'action 0' resumed (module 'builtin:ompipe') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jan 1 00:04:04 server2 rsyslogd: action 'action 0' resumed (module 'builtin:ompipe') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jan 1 00:04:04 server2 systemd: Started Stop Read-Ahead Data Collection.
Feb 20 15:47:32 server2 systemd: Time has been changed
Feb 20 15:47:32 server2 rsyslogd: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
Feb 20 15:48:19 server2 su: (to root) deepak on pts/0

So you see NTPD is really very slow at this and not very good.
But actually we have few options with NTPD which can be used to tweak this behaviour

We can use ntpdate to forcefully sync the localclock using NTP server before NTPD starts

In Red Hat 7 we have below unit file for ntpdate.service

# systemctl cat ntpdate
# /usr/lib/systemd/system/ntpdate.service
[Unit]
Description=Set time via NTP
After=syslog.target network.target nss-lookup.target
Before=time-sync.target
Wants=time-sync.target

[Service]
Type=oneshot
ExecStart=/usr/libexec/ntpdate-wrapper
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

By default this service is disabled, so enable this service

# systemctl enable ntpdate.service
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpdate.service to /usr/lib/systemd/system/ntpdate.service.

Next modify the NTP server which ntpdate would sync in /etc/ntp/step-tickers

server2:~ # cat /etc/ntp/step-tickers
# List of NTP servers used by the ntpdate service.

192.168.1.100

Next lets again change the hwclock date and time

server2:~ # hwclock --set --date "01 jan 2018"

Add a logger message to monitor the logs

server2:~ # logger "rebooting to test ntpdate"

Lastly reboot the blade

server2:~ # reboot

Now lets monitor our syslog once my node is UP
Here the node went for reboot

Feb 20 15:54:03 server2 deepak: rebooting to test ntpdate
Feb 20 15:54:06 server2 systemd: Stopping custom firewall configuration using iptables...

Here my system started to boot

Feb 20 15:54:06 server2 systemd: Stopping Serial Getty on ttyS0...
Feb 20 15:54:06 server2 systemd: Stopping Getty on tty1...
Jan 1 00:02:56 server2 journal: Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 62.8G available â current limit 4.0G).
Jan 1 00:02:56 server2 kernel: microcode: microcode updated early to revision 0x3a, date = 2017-01-30
Jan 1 00:02:56 server2 kernel: Initializing cgroup subsys cpuset
Jan 1 00:02:56 server2 kernel: Initializing cgroup subsys cpu
Jan 1 00:02:56 server2 kernel: Initializing cgroup subsys cpuacct

Now since we enabled our ntpdate service I observed that during the system startup ntpdate does a force sync with NTP server

Jan 1 00:03:16 server2 hp-ams[2476]: amsHelper Started . .
Jan 1 00:03:16 server2 systemd: Started HP Agentless Management Service daemon.
Jan 1 00:03:16 server2 cntdb.security: - Setup B&R chain ...
Feb 20 15:57:11 server2 systemd: Time has been changed
Feb 20 15:57:11 server2 ntpdate[2612]: step time server 192.168.1.100 offset 4377234.731233 sec
Feb 20 15:57:11 server2 systemd: Started Set time via NTP.

But again I do not like such dependency and my expectation is that the very first message which gets logged from hwclock must be proper so we have to make sure that before system goes down the hwclock is in sync with local clock

IMPORTANT NOTE: With Red Hat Enterprise 7 when the system clock is synchronized by the Network Time Protocol (NTP) or Precision Time Protocol (PTP), the kernel automatically synchronizes the hardware clock to the system clock every 11 minutes.

But still considering worst case scenario the hwclock was not set properly with system clock and system went for a reboot with wrong hwclock time?

My solution for this was to create a systemd unit file which will be called before shutdown and during startup to make sure

First local time syncs with NTP server
then hwclock is synced with local clock

So with this during reboot our node will always come up with proper time

server1:~ # systemctl cat set_my_clock.service
# /usr/lib/systemd/system/set_my_clock.service
[Unit]
Description=Syncing system and hardware clock

[Service]
Type=oneshot
ExecStart=/etc/init.d/set_my_clock start
ExecStop=/etc/init.d/set_my_clock stop
RemainAfterExit=true

[Install]
WantedBy=basic.target

NOTE: You can also create this file under "/etc/systemd/system" as that is the recommended location from Red Hat

Next create the main script which will do the work
NOTE: I have enabled debugging to monitor the execution during reboot, you can remove the set -x

# cat /etc/init.d/set_my_clock
#!/bin/bash
set -x
case "$1" in

start|stop)
echo "Set Sys time according to Hardware Clock";
# This will force sync the local system clock with NTP server
/sbin/ntpdate -u 192.168.1.00 192.168.1.101;

# This will sync our hwclock with the system clock time
/sbin/hwclock --systohc;
;;

*)
echo "Usage: $0 {start|stop}"
exit 1
;;
esac

echo "done !"
exit 0

Enable this script

# systemctl enable set_my_clock.service

It is time to validate our fix
Let's change the hwclock date and time

server1 # hwclock --set --date "05 Feb 2018"

server1 # date --set "01 Jan 2018"
Mon Jan 1 00:00:00 IST 2018

Lets have a logger message to monitor the logs

server1:~ # logger rebooting

Reboot the node

server1:~ # reboot

Here the shutdown started

Jan 1 00:00:13 server1 deepak: rebooting
Jan 1 00:00:15 server1 systemd: Stopping Availability of block devices...
Jan 1 00:00:15 server1 systemd: Stopped Stop Read-Ahead Data Collection 10s After Completed Startup.

Our script was executed properly during system shutdown but then as you see it is unable to change the current time of the syslog

Jan 1 00:00:15 server1 set_my_clock: + case "$1" in
Jan 1 00:00:15 server1 set_my_clock: + echo 'Set Sys time according to Hardware Clock'
Jan 1 00:00:15 server1 set_my_clock: Set Sys time according to Hardware Clock
Jan 1 00:00:15 server1 set_my_clock: + /sbin/ntpdate -u 192.168.1.100 192.168.1.101

although the system is coming up with proper time so we know our hwclock was synced properly with system clock above

Feb 20 16:41:54 server1 kernel: Initializing cgroup subsys cpuset
Feb 20 16:41:54 server1 kernel: Initializing cgroup subsys cpu
Feb 20 16:41:54 server1 kernel: Initializing cgroup subsys cpuacct

Again during startup

Feb 20 16:42:02 server1 systemd: Starting Syncing system and hardware clock...
Feb 20 16:42:02 server1 set_my_clock: + case "$1" in
Feb 20 16:42:02 server1 set_my_clock: + echo 'Set Sys time according to Hardware Clock'
Feb 20 16:42:02 server1 set_my_clock: Set Sys time according to Hardware Clock
Feb 20 16:42:02 server1 set_my_clock: + /sbin/ntpdate -u 192.168.1.100 192.168.1.101
Feb 20 16:42:02 server1 set_my_clock: 20 Feb 16:42:02 ntpdate[1118]: no servers can be used, exiting
Feb 20 16:42:02 server1 set_my_clock: + /sbin/hwclock --systohc

So with this now our system will come up with proper date and time and we need not be dependent on NTP service daemon for system startup.

I hope the article was useful.