cynicalsecurity :cm_2:<p>Serious question regarding LLMs.</p><p>I have been trying to train a model specifically for one thing: helping me with <a href="https://bsd.network/tags/OpenBSD" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenBSD</span></a> PF¹ configurations.</p><p>Using a Jolla Mind2², which uses llama, I have uploaded the PDF of "The Book of PF (3rd Edition)" (by <span class="h-card"><a href="https://mastodon.social/@pitrh" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>pitrh</span></a></span>) and the PDFs of the various presentations given on PF.</p><p>Then I tried asking some questions and, well, the bit which I find incredibly puzzling is that it gets the answer right (for some basic configurations) but the notation is wrong! As some presentations / book pages use, for example, the -> character, then the LLM uses that for direction in a PF rule so you get</p><p>pass on egress from any -> egress:0 port 80</p><p>which is really puzzling.</p><p>Note that, in my little mind, having constrained the data set to what I imagine was the best data available, I was expecting pretty impressive results but.. no.</p><p>Anyone willing to spend a little time to explain why to me? I am really not ranting, I don't want to vibe PF, I just want something help me have better insights or improve my rules by making suggestions based on good data (i.e. not just searching for it).</p><p>__<br>¹ <a href="https://www.openbsd.org/faq/pf/" rel="nofollow noopener" target="_blank"><span class="invisible">https://www.</span><span class="">openbsd.org/faq/pf/</span><span class="invisible"></span></a><br>² <a href="https://www.jollamind2.com" rel="nofollow noopener" target="_blank"><span class="invisible">https://www.</span><span class="">jollamind2.com</span><span class="invisible"></span></a><br>³ <a href="https://nostarch.com/book-of-pf-4th-edition" rel="nofollow noopener" target="_blank"><span class="invisible">https://</span><span class="ellipsis">nostarch.com/book-of-pf-4th-ed</span><span class="invisible">ition</span></a></p>