webadmin/bpfire
1
0
Fork 0
mirror of https://github.com/vincentmli/bpfire.git synced 2026-04-28 03:33:25 +02:00
Code Issues Packages Projects Releases Wiki Activity

xz: Performance and compression tuning (V1)

Browse Source 
Hi,

as mentioned in the "list", we're trying to make the archives as small as possible
using 'xz'-compression.

In order to achieve this, this patch tests the size of the memory available on the
host system.

It sets the xz-memory-limit (--memory=[N]Mib) to 70% of the available working memory,
a maximum of four xz-'worker threads' (-T4) and a compression rate of '-8' (-8).

These options are written to the 'XZ_OPT=' environment.

For details see:
https://linux.die.net/man/1/xz

I have set the number of available xz-'worker threads' (-T) to four (-T4), because during
the final tests the '-T0' parameter led to error messages snd stopped: 'cannot allocate memory'.
It wouldn't even run with 90%.

Furthermore, testing with '-T0' led to countless messages filling  up '_build.packages.log'.

E.g.:
...
xz: Adjusted the number of threads from 8 to 2 to not exceed the memory usage
limit of 1557 MiB
...

Tests took place on a 32bit-Ubuntu 16.04.4-system with 8 GB RAM and an Intel I7-2600.
Build time was about 04:30 hrs. Perhaps a 64bit-system would perform better (higher
values), but my goal was to make this run on as many systems as possible, so I choosed
these averages.

If minimum requirements (1024 MB RAM) are not met, building stops.

Current results:

'next', untuned:
ipfire-2.19.2gb-ext4.i586-full-core121.img.gz	=> 332951687 Bytes
ipfire-2.19.i586-full-core121.iso		=> 228589568 Bytes

'next', XZ_OPT: -T4 -8, 70% RAM:
ipfire-2.19.2gb-ext4.i586-full-core121.img.gz	=> 329725723 Bytes
ipfire-2.19.i586-full-core121.iso		=> 217055232 Bytes

These two resulting archives are 14760300 Bytes smaller than before.

Best,
Matthias

Signed-off-by: Matthias Fischer <matthias.fischer@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
This commit is contained in:
Matthias Fischer
2018-05-20 09:30:23 +02:00
committed by Michael Tremer
parent 814a7d126b
commit 0d7a6f03d4
3 changed files with 19 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
lfs/Config
View File

@@ -209,7 +209,7 @@ define PAK
tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \
-p --numeric-owner
rm -f /install/packages/package/files.tmp
cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *
cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *
rm -r /install/packages/package/tmp
-cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp
mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES