Dynamic MOTD is sometimes displayed partially

Since I started running my own servers, I’ve been using a dynamic MOTD (Message of the Day) to display useful information when logging into my server. The main benefit compared to a static MOTD is that it can be dynamically generated by custom shell scripts. That means you can show up-to-date information about the system, such as disk usage, memory usage, system load, running docker containers, and much more.

Because I primarily run Debian as my server OS, I’ve been using the update-motd feature that comes with the OS. It uses the PAM module pam_motd.so which in turn runs the scripts in /etc/update-motd.d/ to generate the dynamic MOTD. All my motd scripts are available on GitHub.

Those scripts had worked reliably for years. But every once in a while, I notice that the dynamic Message of the Day (MOTD) is not displayed correctly when I log in via SSH. Sometimes it doesn’t show up at all, and other times it only shows a partial message. Almost like the MOTD is being cut off at a seemingly random point.

But when running run-parts --lsbsysinit /etc/update-motd.d manually, there is no issue at all. It always runs perfectly fine and shows the complete output of all scripts. And it totally seems as if I was not the only one experiencing this issue. There’s this post on AskUbuntu titled Ubuntu 12.04: pam_motd sometimes displays only partial MOTD describing a similar problem. Especially the “sometimes” part is what makes this issue so strange.

After fighting with this strange behavior, today I was finally able to reliably reproduce the issue and managed to find the root cause. Well at least for my specific case - your case might be different.

Debugging the problem

First, I wanted to check if the run-parts command was being executed correctly. I ran watch -n 0.1 'ps aux | grep "run-parts"' in one existing ssh session and then started a second ssh session on my second screen. What this does is to show me all running processes that contain “run-parts” in their name, refreshing the output every 0.1 seconds.

When logging in, I could see two new processes starting up. At first, I wasn’t sure if it was really two separate processes or maybe just a glitch. So I continued looking for further clues.

I added this line to one of the scripts in /etc/update-motd.d/ whose output wasn’t shown correctly, to see if it was executed at all:

echo "Script X ran at $(date)" >> /tmp/motd-debug.log

That confirmed my suspicion. I found that the script was executed twice when I logged in via SSH:

Script X ran at Fri May  9 23:08:39 CEST 2025
Script X ran at Fri May  9 23:08:39 CEST 2025

This behavior is unexpected because the script should only run once per login session. That must mean that there are two login sessions created at the same time, creating some sort of race condition. So I started thinking about what could cause two ssh sessions to be created simultaneously.

On Windows, I use MobaXterm to connect to my servers and one of the features of MobaXterm is a file browser that shows in the sidebar. That file browser uses sftp to connect to the server and read the file system. This is done by creating a second SSH connection in the background, which can be used for browsing the file system or transferring files.

It is a great feature, but it seems to cause the MOTD scripts to run twice in quick succession, leading to the partial display of the message.

Checking the pam_motd.so source code, it becomes clear where the problem lies.

if (do_update && (stat("/etc/update-motd.d", &st) == 0)
    && S_ISDIR(st.st_mode))
{
    mode_t old_mask = umask(0022);
    if (!system("/usr/bin/env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin run-parts --lsbsysinit /etc/update-motd.d > /run/motd.dynamic.new"))
        rename("/run/motd.dynamic.new", "/run/motd.dynamic");
    umask(old_mask);
}

When two sessions are created at the same time, both will call run-parts to generate the MOTD. Both processes will then try to write the output into the /run/motd.dynamic.new file. And the last writer wins. Although, if the second process did not finish writing, the motd file will be incomplete. It’s a classic race condition.

Let’s look at the sequence of events in more detail: The first process is started a few milliseconds before the second one, and it will create the motd.dynamic.new file and start writing to it. The second process will then try to write to the same file before the first process finished writing the whole file. After the second process starts its write operation, the first process renames the file from motd.dynamic.new to motd.dynamic and opens the motd.dynamic file for reading. This does not affect the second write operation, as the file descriptor for the second process is still valid and points to the same inode.

At this stage, the file is being written to by the second process, while the first process has already renamed the file and is about to display the file’s content to the user. Depending on how far the second process has progressed with writing to the file, the first process will read and display only a partial MOTD.

This whole situation is where the race condition occurs, and the MOTD is only partially displayed, depending on the write progress of the second process. We can even observe this behavior by using inotifywait to monitor the /run/ directory for changes to the motd.dynamic file:

~$ inotifywait -m /run/ | grep motd
Setting up watches.
Watches established.
/run/ CREATE motd.dynamic.new
/run/ OPEN motd.dynamic.new
/run/ MODIFY motd.dynamic.new
/run/ MODIFY motd.dynamic.new (x17)
/run/ OPEN motd.dynamic.new
/run/ MODIFY motd.dynamic.new
/run/ MODIFY motd.dynamic.new
/run/ CLOSE_WRITE,CLOSE motd.dynamic.new
/run/ MODIFY motd.dynamic.new
/run/ MOVED_FROM motd.dynamic.new
/run/ MOVED_TO motd.dynamic
/run/ OPEN motd.dynamic
/run/ ACCESS motd.dynamic
/run/ CLOSE_NOWRITE,CLOSE motd.dynamic
/run/ MODIFY motd.dynamic
/run/ MODIFY motd.dynamic (x17)
/run/ CLOSE_WRITE,CLOSE motd.dynamic
/run/ OPEN motd.dynamic
/run/ ACCESS motd.dynamic
/run/ CLOSE_NOWRITE,CLOSE motd.dynamic

Solution

Actually, for me the solution is quite simple. I disabled the sftp file browser feature in MobaXterm by going to Sessions > Edit Session > Advanced SSH settings and changed SSH-browser type to None. After some playing around it also seems to work when changed to SCP. That’s because SCP does not create an interactive SSH session in the background and hence does not run the run-parts command.

After fixing the problem, inotifywait only shows a single write process for the MOTD file:

~$ inotifywait -m /run/ | grep motd
Setting up watches.
Watches established.
/run/ CREATE motd.dynamic.new
/run/ OPEN motd.dynamic.new
/run/ MODIFY motd.dynamic.new
/run/ MODIFY motd.dynamic.new (x15)
/run/ CLOSE_WRITE,CLOSE motd.dynamic.new
/run/ MOVED_FROM motd.dynamic.new
/run/ MOVED_TO motd.dynamic
/run/ OPEN motd.dynamic
/run/ ACCESS motd.dynamic
/run/ CLOSE_NOWRITE,CLOSE motd.dynamic

There might be other reasons for this issue as well. For example, if you have multiple ssh sessions starting at the same time for other reasons. On a busy linux server with multiple users, this might happen regularly. Another issue could be if you have configured pam_motd in /etc/pam.d/sshd as well as /etc/pam.d/login. Eventually, you want to check for simultaneous write operations and for multiple ssh sessions started at the same time, if you experience similar issues.

But a proper fix to this issue would be to use a file locking mechanism within the pam_motd.c source code to ensure that only one process can write to the MOTD file at a time.