Re: bin/59657: syslogd outputs BOM in the message

["pr%xn--rvztrtkrfrgp-bbb7j2b8f0b9d7a21oft.com@localhost via gnats" <gnats-admin%NetBSD.org@localhost>]

The following reply was made to PR bin/59657; it has been noted by GNATS.

From: pr%xn--rvztrtkrfrgp-bbb7j2b8f0b9d7a21oft.com@localhost
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: bin/59657: syslogd outputs BOM in the message
Date: Mon, 27 Oct 2025 23:23:28 +0000

 On Sun, Sep 21, 2025 at 09:25:01AM +0000, Christos Zoulas via gnats wrote:
 >  Now in practice according to ChatGPT:
 >  Almost all modern syslog implementations do not emit a BOM, even for =
 >  UTF-8 content.
 >  
 >  Many receivers are tolerant and just assume UTF-8 without requiring BOM.
 >  
 
 ChatGPT that fount of knowledge.
 
 I'd call the FreeBSD implementation the opposite of tolerant of UTF-8:
 
 957 	/*
 958 	 * Removes characters from log messages that are unsafe to display.
 959 	 * TODO: Permit UTF-8 strings that include a BOM per RFC 5424?
 960 	 */
 961 	static void
 962 	parsemsg_remove_unsafe_characters(const char *in, char *out, size_t outlen)
 
 >  
 >  And:
 >  RFC 5424 says the BOM is required if you send UTF-8 MSG.
 >  
 
 Yes, that was a terrible idea along with adding an optional BOM to
 UTF-8.
 
 apropro it's a can of worms:
 https://corp.unicode.org/pipermail/unicode/2020-June/008713.html
 
 >  
 >  Perhaps adding a flag to select the behavior? What should the default =
 >  be?
 
 The BOM is the flag, we don't need to show the flag as a prefix for 
 every message. We don't notice the optional UTF-8 BOM "" at the
 beginning of files because editors etc. don't show it.
 
 Although it has just occured to me I'm only thinking about this
 appearing locally and not being passed onto a remote syslogd. We
 shouldn't strip it in that case (the BOM is now the remote syslogd's
 problem).
 
 So it'll need to be preserved for F_FORW & F_TLS but not for local 
 likely to be read by a human destinations like F_FILE, F_WALL etc.
 
 This is actually much messier than I first thought.
 
 At the end of the day it is just something that can be filtered out by
 whomever is viewing it.
 
 emacs:
 2025-10-27T23:08:58.611141+00:00 funcube potato - - - dr<U+f8>mst<U+f8>rre
 2025-10-27T23:09:00.741754+00:00 funcube potato - - - _drømstørre
 vim:
 2025-10-27T23:08:58.611141+00:00 funcube potato - - - dr<U+f8>mst<U+f8>rre
 2025-10-27T23:09:00.741754+00:00 funcube potato - - - <feff>drømstørre
 
 Cheers,
 Ben