#Development #Explorations
Pixel data from encoders to decoders · A closer look at the tiny building blocks of an image https://ilo.im/165v7s
_____
#Pixels #Images #Codecs #Encoders #Decoders #Compression #WebDev #Frontend #CSS #JavaScript
#Development #Explorations
Pixel data from encoders to decoders · A closer look at the tiny building blocks of an image https://ilo.im/165v7s
_____
#Pixels #Images #Codecs #Encoders #Decoders #Compression #WebDev #Frontend #CSS #JavaScript
#Design #Tools
Kompressor · A smart and powerful image converter/compressor https://ilo.im/165hq2
_____
#Image #Conversion #Compression #MacOS #ProductDesign #UiDesign #WebDesign #Development #WebDev #Frontend
Would adding Brotli Compression help shrink ePubs?
https://shkspr.mobi/blog/2025/07/would-adding-brotli-compression-help-shrink-epubs/
The ePub format is the cross-platform way to package an eBook. At its heart, an ePub is just a bundled webpage with extra metadata - that makes it extremely easy to build workflows to create them and apps to read them.
Once you've finished authoring your ePub, you've got a folder full of HTML0, CSS, metadata documents, and other resources. The result is then stored in a standard Zip file and is then renamed to .epub
. This is known as the Open Container Format (OCF).
There are actually a few different compression schemes for Zip files, but the specification says:
OCF ZIP containers MUST include only stored (uncompressed) and Deflate-compressed ZIP entries within the ZIP archive.
The Deflate algorithm is venerable1 and, while incredible for its time, has been superseded by more modern compression schemes. For example, Brotli.
What happens if we unzip an ePub and then recompress it with Brotli? Will that dramatically reduce the file size?
Steps
unzip book.epub -d book/
tar -cvf book.tar book/
zip -9 book.tar.zip book.tar
brotli -k -q 11 book.tar
Results
I took a random(ish) sample from Standard eBooks and a few from my personal stash2.
Book 1Book 2Book 3Book 4Contents768KB911KB389KB594KBDeflate250KB248KB103KB175KBBrotli190KB187KB82KB137KBThe good news is that ePubs compress pretty well already! That isn't much of a surprise - compression algorithms love the repetitious nature of HTML and human-readable text. Obviously Brotli is better but, on the file sizes we're talking about, not dramatically better. Saving 60KB is OK - but in a world of terabyte sized SD cards does it matter?
Brotli is also computationally harder to decompress, which makes it slightly less attractive for low-powered eReaders.
It's also possible to make a small saving by reducing the complexity and verbosity of the CSS and HTML.
However, that's not the real problem.
I lied to you
An ePub contains more than just text and text-based metadata. It can contain web fonts, images, even music. The above books had all their fonts and media stripped out. Let's run the experiment again but, this time, including everything in the original book.
Book 1Book 2Book 3Book 4Contents23MB3.8MB0.76MB0.93MBDeflate22MB1.7MB0.46MB0.51MBBrotli22MB1.5MB0.43MB0.47MBAll of a sudden, Brotli makes next to no difference. Yes, the textual compression is still there, but it is overshadowed by the huge cost of the media files.
Mixed Media
The ePub 3.3 specification lays out which multimedia formats are acceptable. As well as the older formats like gif, png, and jpeg - newer formats like WebP are acceptable. Similarly, TTF fonts are listed in the standard along with WOFF2.
Modern image and font formats have better compression than their ancestors. Indeed, WOFF2 uses Brotli as its compression scheme.
The biggest filesize saving in ePubs comes from properly compressing images and fonts.
Can You Picture That?
It is a matter of opinion as to what resolution is best suited to an ePub. Most modern eReaders have, at best, 300ppi resolution. They're also normally monochrome. But eBooks aren't always read on low-resolution, black and white eInk screens - so it probably makes sense to have high-resolution colour images in order to future-proof books.
But the compression of those images is not a matter of opinion. Lossless compression algorithms are well supported for legacy and modern image formats.
Let's take a specific example. Twenty Years at Hull House is the 22MB book above. Less than a MB of that is for text, the rest is images.
The largest illustration in the book is a 1937x1971, transparent PNG weighing in at 1MB. Increasing the lossless compression level takes it down to 840KB. Reducing the palette to something more suitable takes it to 640KB. If you were releasing this as an ePub 3.3 file, using WebP would take the image to a hair over 600KB.
Basically, a 20%-40% filesize reduction with no loss of fidelity.
Across all the PNG images in the ePub, I was able to easily get the filesize from 20MB to 16MB.
Converting to lossless WebP got it down to 13MB.
What The Font?
Fonts can be shrunk in a number of ways. The most obvious way is to compress to WOFF2 which, as described above, uses Brotli compression.
Based on my quick tests, a typical ePub's TTF will see about a 50% reduction in font size. For typical "English" language fonts, that's a reduction from 30KB to 15KB. So big relative compression, but small absolute compression.
Complex decorative fonts can go from 800KB to 80KB. But it is rare for a font to exceed a megabyte.
If it does, that usually means that it has more glyphs than strictly necessary. If your book is written entirely in the Latin alphabet, do you really need all those fancy accents, Chinese ideographs, and emoji? Probably not.
I've previously written about Subsetting Fonts and the perils of excessive trimming.
Back to Basics
Brotli is magic - but changing the compression algorithm for the ePub standard is probably a false economy. The text portion of modern eBooks is already fairly small and compresses with reasonable efficiency.
The best compression gains come from either using next-generation image and font formats or, if legacy compatibility is necessary, using the most aggressive compression settings for traditional images.
OK! It is actually XHTML, but let's not quibble. ↩︎
That's a fancy way of saying "old". ↩︎
I couldn't be bothered automating this. Go ahead a run it on every ePub if you want something more representative. ↩︎
blog! “Would adding Brotli Compression help shrink ePubs?”
The ePub format is the cross-platform way to package an eBook. At its heart, an ePub is just a bundled webpage with extra metadata - that makes it extremely easy to build workflows to create them and apps to read them.
Once you've finished authoring your ePub, you've got a folder full of HTML, CSS,…
Read more: https://shkspr.mobi/blog/2025/07/would-adding-brotli-compression-help-shrink-epubs/
⸻
#compression #epub
What to know about President Tump’s ‘chronic venous insufficiency’ https://www.byteseu.com/1210848/ #America #azizzadeh #BloodClot #ChronicVenousInsufficiency #compression #condition #cvi #Doctor #Heart #kwong #LowLeg #people #PresidentTrump #treatment #UnitedStates #UnitedStatesOfAmerica #US #USNews #USA #USANews #VascularSurgery #vein
#compression #7zip
Pour illustrer les paramètres "compression maximale" dont j'ai parlé là : https://sebsauvage.net/links/?s0zmfA
Illustration avec un jeu : Loophole.
Décompressé: 3,18 Go
7z "ultra" (-mx=9) : 1,33 Go
7z avec mes réglages : 0,53 Go
(Et zpaq -m4 fait un peu mieux : 0,49 Go)
Bien sûr c'est un exemple qui marche bien, le gain ne sera pas forcément aussi bon sur d'autres données.
7-Zip 25.00 is out!
Now supports 64+ CPU threads for faster compression ️
bzip2 speeds improved by up to 40%, plus important security fixes
Update now for better performance and safety!
#ArchiveDeJeux
#Compression #7zip
Comme il m'a été demandé, voici les paramètres que j'utilise désormais pour obtenir la meilleure compression avec 7-Zip.
Inconvénient :
- on passe en mono-thread (la compression prend *beaucoup* plus de temps ; la décompression sera un peu moins rapide).
- cela consomme plus de RAM (10 Go à la compression, 1 Go à la décompression).
1 Go pour la décompression ne me semble pas déraisonnable pour les machines de nos jours.
Introducing Iris-WebP, a better WebP encoder: https://halide.cx/iris/index.html
Iris offers up to >20% better compression than libwebp, and trades blows with libaom for AVIF.
My goal is to build an image-first encoder ecosystem for the modern Web, so that new web image codecs aren't generally superfluous.
So I have hundreds of videos of ~1 minute recorded from my phone ~10 years ago, and they generally don’t have that great compression, nor they are stored in a modern and advanced video format.
For archiving purposes, I want to take advantage of my workstation’s mighty GPU to process them so that the quality is approximately the same, but the file size would be strongly reduced.
Nevertheless, compressing videos is terribly hard, and way more complex than compressing pictures, so I wouldn’t really know how to do this, what format to use, what codec, what bitrate, what parameters to keep an eye on, etc.
I don’t care if the compression takes a lot of time, I just want smaller but good looking videos.
Any tips? (Links to guides and tutorials are ok too)
Also, unfortunately I am forced to use Windows for this (don’t ask me why ), but I know nothing about Windows because I hate it. Practical software suggestions are very much welcome, too!
@vildis +9001%
Why they don't go with tar
or pax
or plain bzip2
/ xz
is beyond me.
The `bzip2` crate switches from C to 100% Rust, https://trifectatech.org/blog/bzip2-crate-switches-from-c-to-rust/.
Use less memory. Faster to run. Simpler cross-compilation. That’s it.
The crate has been audited too, neat!
I thoroughly enjoyed Antonio Somaini’s lecture tonight on the Politics of Latent Spaces at the conference Art in the Age of Average. The new AI-thoritarians.
His reflections on compression as a cultural and epistemic process were truly inspiring — and the sources cited were excellent, too ;)
#AI #LatentSpaces #DigitalCulture #Compression #Vectorization #NeuralNetworks #ArtAndAI #MachineVision #epistemiccompression #AIAesthetics @databasecultures
Salut, je cherche une app mobile libre pour compresser des images jpg par lot. Une appli à me recommander ?
#help #android #mobile #photo #app# #fdroid #compression #manipulationImage
Revisited some #compression algorithms after enabling multi-threading for xz. Here's results for my rip of the original Metal Gear Solid for PS1.
Source: 1.4GB
7zip: 802MB, 31 seconds
BZip2: 923MB, 1:12
GZip: 909MB, 33 seconds
XZ: 807MB, 32 seconds
ZSTD: 875MB, 3 seconds
”LLM’s are storages of millions of vector programs, that are formed by passive exposure to human output.”
No. There are so many things wrong in this!
-Not programs because they don’t execute anything
-If you give an #LLM a mathematically recursive task, it inferences in exact same time regardless of complexity
-It’s more like #compression than programs, really
-And ”passive exposure”??!! Wtf are these people smoking?!!
Tells more about the psychologies of people building #ai
#compression
Alors là, je n'avais jamais vu une telle différence entre 7zip et zpaq :
Titanfall 2 : 63,7 Go.
En 7z : 39,4 Go
En zpaq (m4) : 24,8 Go.
14,6 Go de différence !
An interesting side-note:
Because #sunoai and other #genai music models have been trained illegally from compressed music obtained without permission from streaming & online services, it turns out there’s an almost sure-fire method of detecting #ai music output as well:
Psychoacoustic compression artefacts
Because the models learn everything, they’ve not learned the full spectra of instruments, but rather what the brain doesn’t discard when hearing them.
What happens when AI compresses the entire Internet into a single oracle, flattening messy outliers into streamlined approximations? Is that efficiency worth the cost? My article "Honey, AI Shrunk the Archive" is now up as a pre-print https://blog.still-water.net/ai-as-miniature-archive