If you live within the USA however, it might be a different story insomuch as the NSA for example, has the modus operandi to monitor all traffic that exits (or enters) the USA and isn't supposed to use its spy apparatus within the USA.
Lamentably, there's also CALEA and "An Act to deter and punish terrorist acts in the United States and across the globe, to enhance law enforcement investigatory tools, and for other purposes." (aka the PATRIOT Act) which legislatively facilitate warrant-less wiretapping domestically within the USA as well last I checked.
Signal's use of phone
#s, more or less puts it right within the targeting apparatus of the latter, and you don't have to look too hard to see Signal logs entered into Court transcripts as evidence which is a fucking awful look for anything professing to be "secure". ;(
CC:
@mjg59@nondeterministic.computer