This patch uses the new Zstandard algorithm to compress the file system
image on the ISO image. This comes with these advantages:
* Compression is about twice as fast than XZ with the parameters we have
selected here
* We use a lot less memory during compression and can therefore utilise
all processor cores of the build machines
* Decompression (when installing IPFire and when creating the
flash-image) is substantically faster
The downside is that the generated ISO image is slighty larger (~10MiB)
which I am okay with as a trade-off for the points mentioned above.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Arne Fitzenreiter <arne_f@ipfire.org>
we have no supported armv5tel board left so we can switch to the higher
arch. This now can use the vpu (still in softfp calling convention to
not break existing installations.)
this fix many compile problems, also boost is now working again.
Signed-off-by: Arne Fitzenreiter <arne_f@ipfire.org>
If you have 2GB RAM the build of dnsdist will fail because MAX_PARALLELISM was
set to zero by RAM/2048 because a bit of RAM is used by the system.
This patch ensure that the lowest PARALLELISM value is 1.
Signed-off-by: Arne Fitzenreiter <arne_f@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
System capabilities are stored in extended file system attributes
which are by default not stored in tar balls.
This patch ensures that they are packaged and extracted.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Those are now created in their own temporary directory, so that
no other files can be included by accident.
We also package with fewer temporary files.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Arne Fitzenreiter <arne_f@ipfire.org>
Go leaves temporary build files in the directory
which we do not need and we should clean up after
every build.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Arne Fitzenreiter <arne_f@ipfire.org>
Exporting XZ_OPT caused that every time xz was called, it automatically
enabled parallelism. The make systemm also launches multiple processes
at the same time to use more processor cores at the same time.
The combination of this causes memory exhaustion even on large systems
and has no performance gain. Therefore this is disabled by default
and only enabled where we need it which is already the case.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
This will now adjust MAKETUNING to not launch more processes
than MAX_PARALLELISM. Handy to limit builds that use a lot of memory.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
For details see:
https://www.gnu.org/software/tar/
Since new 'tar' has changes in commandline option parsing,
it was necessary to adjust 'lfs/Config, 'lfs/cdrom' and
'lfs/core-updates' (thanks to Marcel for diffs and Michael for
corrections).
I have tried to make only the most necessary changes.
As always, please check.
Best,
Matthias
Signed-off-by: Matthias Fischer <matthias.fischer@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Some packages ship an older version which does not recognise
aarch64. To overcome this, we just update these scripts when
necessary.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
This caused that the xz command was called without any
extra arguments. This will now create the tar archive
first and then pass the archive through xz with our
command line switches.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Changes since V1:
- Tuned some more 'tar'-command lines
- Included 'lfs/core-updates'
- Some fine-tuning
Hi,
Current results with V2 (clean builds):
'next':
'packages' => 255 MB
'ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 319 MB
'ipfire-2.19.i586-full-core121.iso' => 218 MB
Total => 792 MB
'xz-tuning':
'packages' => 228 MB
'ipfire-2.19.2gb-ext4.i586-full-core121.img.gz' => 318 MB
'ipfire-2.19.i586-full-core121.iso' => 207 MB
Total => 753 MB (-39 MB)
It would be nice if someone could review and test these. ;-)
If necessary, I can upload a complete patch.
Best,
Matthias
Signed-off-by: Matthias Fischer <matthias.fischer@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Hi,
as mentioned in the "list", we're trying to make the archives as small as possible
using 'xz'-compression.
In order to achieve this, this patch tests the size of the memory available on the
host system.
It sets the xz-memory-limit (--memory=[N]Mib) to 70% of the available working memory,
a maximum of four xz-'worker threads' (-T4) and a compression rate of '-8' (-8).
These options are written to the 'XZ_OPT=' environment.
For details see:
https://linux.die.net/man/1/xz
I have set the number of available xz-'worker threads' (-T) to four (-T4), because during
the final tests the '-T0' parameter led to error messages snd stopped: 'cannot allocate memory'.
It wouldn't even run with 90%.
Furthermore, testing with '-T0' led to countless messages filling up '_build.packages.log'.
E.g.:
...
xz: Adjusted the number of threads from 8 to 2 to not exceed the memory usage
limit of 1557 MiB
...
Tests took place on a 32bit-Ubuntu 16.04.4-system with 8 GB RAM and an Intel I7-2600.
Build time was about 04:30 hrs. Perhaps a 64bit-system would perform better (higher
values), but my goal was to make this run on as many systems as possible, so I choosed
these averages.
If minimum requirements (1024 MB RAM) are not met, building stops.
Current results:
'next', untuned:
ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 332951687 Bytes
ipfire-2.19.i586-full-core121.iso => 228589568 Bytes
'next', XZ_OPT: -T4 -8, 70% RAM:
ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 329725723 Bytes
ipfire-2.19.i586-full-core121.iso => 217055232 Bytes
These two resulting archives are 14760300 Bytes smaller than before.
Best,
Matthias
Signed-off-by: Matthias Fischer <matthias.fischer@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
This will allow us to run multiple builds on the same
system at the same time (or at least have them on disk).
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
This is a very weird way to distribute sources in 2017.
Let's save the environment and stop using CDs.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
The build environment is using a number of variables which
occasionally conflicted with some other build systems.
This patch cleans that up by renaming some variables and
later unexporting them in the lfs files.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
We also move the initscript for also to src/initscripts/packages
and use this new macro to install the initscript
Signed-off-by: Jonatan Schlag <jonatan.schlag@ipfire.org>
There is no longer a SSH service on source.ipfire.org.
The usual login on git.ipfire.org should be used instead.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
