Harald Sack<p>Building on the 90s, statistical n-gram language models, trained on vast text collections, became the backbone of NLP research. They fueled advancements in nearly all NLP techniques of the era, laying the groundwork for today's AI. </p><p>F. Jelinek (1997), Statistical Methods for Speech Recognition, MIT Press, Cambridge, MA</p><p><a href="https://sigmoid.social/tags/NLP" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>NLP</span></a> <a href="https://sigmoid.social/tags/LanguageModels" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>LanguageModels</span></a> <a href="https://sigmoid.social/tags/HistoryOfAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>HistoryOfAI</span></a> <a href="https://sigmoid.social/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a> <a href="https://sigmoid.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://sigmoid.social/tags/historyofscience" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>historyofscience</span></a> <a href="https://sigmoid.social/tags/ISE2025" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ISE2025</span></a> <span class="h-card" translate="no"><a href="https://sigmoid.social/@fizise" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>fizise</span></a></span> <span class="h-card" translate="no"><a href="https://wisskomm.social/@fiz_karlsruhe" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>fiz_karlsruhe</span></a></span> <span class="h-card" translate="no"><a href="https://fedihum.org/@tabea" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>tabea</span></a></span> <span class="h-card" translate="no"><a href="https://sigmoid.social/@enorouzi" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>enorouzi</span></a></span> <span class="h-card" translate="no"><a href="https://fedihum.org/@sourisnumerique" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>sourisnumerique</span></a></span></p>
eupolicy.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
This Mastodon server is a friendly and respectful discussion space for people working in areas related to EU policy. When you request to create an account, please tell us something about you.
Administered by:
Server stats:
192active users
eupolicy.social: About · Profiles directory · Privacy policy
Mastodon: About · Get the app · Keyboard shortcuts · View source code · v4.4.1
#textprocessing
0 posts · 0 participants · 0 posts today
Veronica Olsen 🏳️🌈🇳🇴🌻<p>Back when I first wrote text processing code in the 90s on my Amiga 1200, I always used the ¤ symbol as a placeholder character for splitting and replacing to exclude things I wanted skipped without affecting character count. It was available on the Norwegian keyboard, and practically never used in text.</p><p>Recently I discovered that Unicode has two "Not a character" symbols perfect for the same usage: \uFFFE and \uFFFF.</p><p>They can be really useful!</p><p><a href="https://mastodon.online/tags/Code" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Code</span></a> <a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Unicode</span></a></p>
Veronica Olsen 🏳️🌈🇳🇴🌻<p>2. Immediately after the split, replace U+FFFF with newline, but keep both versions of the line, and pass the one with the U+FFFF to the text paragraph parser. Everything else (like headings) gets the cleaned one.</p><p>3. After paragraph lines with a single break between them (belonging to the same paragraph) have been processed, THEN I replace the U+FFFF characters there.</p><p>It seems to work, but it took me like 3-4 hours to crack. 😅 </p><p>4/4</p><p><a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Unicode</span></a></p>
Veronica Olsen 🏳️🌈🇳🇴🌻<p>I tried using the alternative line and paragraph separators from Unicode, but splitlines accepts them too. Then I discovered these Unicode characters:</p><p>U+FFFE <noncharacter-FFFE> not a character.<br>U+FFFF <noncharacter-FFFF> not a character.</p><p>The solution, then was:</p><p>1. Replace all occurrences of [br] with or without a trailing newline, using regex pattern "(?i)(?<!\\)(\[br\]\n?)", with a U+FFFF character.</p><p>3/4</p><p><a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Unicode</span></a></p>
Veronica Olsen 🏳️🌈🇳🇴🌻<p>This works fine in principle, but it is incredibly hard to figure out exactly when to make the replacement.</p><p>For instance, if I do it too early, the parser will split on the breaks as I use splitlines() early on. If I do it too late, I get double line breaks some places.</p><p>2/4</p><p><a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Unicode</span></a></p>
Veronica Olsen 🏳️🌈🇳🇴🌻<p>I've been struggling with solving an issue with my text editor project. The editor is plain text and uses a blank line to separate paragraphs.</p><p>The editor has an option to preserve or not preserve single line breaks inside paragraphs when generating the output.</p><p>However, some users want to not preserve them, but still want to be able to add hard breaks sometimes. So I've been trying out using [br] as a hard break shortcode.</p><p>1/4</p><p><a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Unicode</span></a></p>
Veronica Olsen 🏳️🌈🇳🇴🌻<p>I started working on a Python class to write MS Office Word documents from already tokenized formatted text. It took me 5 hours to get a working version that can handle most of the formatting I need.</p><p>I have already done this with the Open Document format. It took me significantly longer, but I do steal some code from that code for DocX.</p><p>That said, DocX is actually easier to generate the XML for it turns out.</p><p><a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.online/tags/Code" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Code</span></a> <a href="https://mastodon.online/tags/Documents" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Documents</span></a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TextProcessing</span></a></p>
barefootliam<p>Getting ready to run an online introductory XSLT course for people writing or maintaining stylesheets.</p><p><a href="https://floss.social/tags/XSLT" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>XSLT</span></a> <a href="https://floss.social/tags/XML" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>XML</span></a> <a href="https://floss.social/tags/Schematron" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Schematron</span></a> <a href="https://floss.social/tags/XSpec" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>XSpec</span></a> <a href="https://floss.social/tags/declarative" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>declarative</span></a> <a href="https://floss.social/tags/functionalProgramming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>functionalProgramming</span></a> <a href="https://floss.social/tags/textProcessing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>textProcessing</span></a> <a href="https://floss.social/tags/digitalHumanities" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>digitalHumanities</span></a> <a href="https://floss.social/tags/JATS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>JATS</span></a></p>
TrendingLive feeds
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.
Create accountLoginDrag & drop to upload