Don't do one repository for all remotes. Do one *clone* per
remote.
This also means that users no longer download information twice,
in practice, because the backup repository will only be downloaded
if the main one didn't work.
Theoretically, this change is makes the process less efficient, but
in practise it's more reliable now.
We do now use --mirror on the git clone command for caches, but we
already did git pull --all before.
This just ensures that we absolutely have all local code.
NOTE:
The new code isn't used by default. To use it, you must do:
export XBMK_CACHE_MIRROR="y"
Otherwise, the old behaviour will continue to be used. This is
because the new code, while correct, puts more strain on upstream
servers (more code being downloaded), and can result in higher amounts
of disk space being used. The old behaviour wasn't broken, so we'll
also support that method.
TODO: perhaps also have a check in place to re-use both caches,
where available, regardless of XBMK_CACHE_MIRROR?
Signed-off-by: Leah Rowe <leah@libreboot.org>
also, the check is -e, not -d, because we
might be operating on a symlink.
it's a bit hacky but this should work.
the previous change (now reverted) broke
re-use of the main cache/ in release work
directories.
Signed-off-by: Leah Rowe <leah@libreboot.org>
xbmkdir checks if a directory exists, before running
mkdir, and then still uses -p
i was testing xbmk on arch linux today, and noticed
that it errored out when a directory already exists.
i'm mitigating against buggy or differently behaving
mkdir implementations this way, by wrapping around
it.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the --status flag seems to be a GNUism
as stated in the previous commit, i import sbase
suckless now, so as to have a consistent implementation
of sha512sum.
this ensures that its output is reliable, when i'm using
the output of this command within backticks.
Signed-off-by: Leah Rowe <leah@libreboot.org>
this allows me to remove several eval calls, and the
errors relating to configs can now show exactly which
function they occured in, allowing for easier debugging.
once again, eval should be used sparingly if at all.
Signed-off-by: Leah Rowe <leah@libreboot.org>
what this function does will differ wildly,
depending on whether it's a child instance
or a parent instance of xbmk.
break up this function accordingly.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the following checks on the path mitigates the
lack of error handling in the findpath command
that sets pypath.
this was all thought of when i initially wrote
this code. it's perfectly fine.
Signed-off-by: Leah Rowe <leah@libreboot.org>
on further inspection, the following check ensures that the
python version number is 3.
if anything went wrong, the possibility alluded to in the
comment wouldn't actually matter in practise.
Signed-off-by: Leah Rowe <leah@libreboot.org>
otherwise, it may get created as the root user, disabling
further use of lbmk until manual user intervention.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the new check is still retained, to the extent that
the lock file still contains the TMPDIR string, and
it's checked whether this changed during execution.
however, the current TMPDIR handling is over-engineered
and prevents the re-use of project source caches when
doing release builds; this means that the release builds
happen much more slowly, especially for slow internet
connections.
this change *fixes* that bug. now release builds once
again re-use the main cache/ directory.
Signed-off-by: Leah Rowe <leah@libreboot.org>
be a bit less pedantic about if else clauses. leave the
big ones still with then on separate lines, where else
is specified.
also unroll a few condensed code lines where i missed
a few.
sloccount 2303 in lbmk. that's still only slightly bigger
than libreboot 20260907 which was 2180, and still much
smaller than libreboot 20230625 which was 3322.
this is *without* the condensed codelines, so now the only
thing that's reduced is the overall amount of logic present
in the build system.
and i should clarify that lbmk is presently much more powerful
than both of those two versions (20160907/20230625).
the 2016 one is useful for comparison historically, since that
was the last major version of libreboot prior to the great
second coming of leah in 2021; and the 2023 june release was
basically the last one before the great audits of 2023 to
2025 began.
not to brag (not much anyway), but all of this means that lbmk
is an insanely efficient build system, considering all the
features it has and what it does.
i unrolled the condensed code style in lbmk, making the scripts
a lot easier to read, because i received complainst about the
condensed style previously used; nicholas chin and alper nebi
yasak both told me that it sucked, and riku viitanen had hinted
at that same fact several months prior.
so hopefully now, lbmk is a bit nicer. those and other people
often find it challenging to challenge me because for reason
they assume i'll get upset and fly off the handle, but it's the
opposite. i want constant criticism, so that i know to improve!
Signed-off-by: Leah Rowe <leah@libreboot.org>
otherwise, running ./mk dependencies as root will
create xbmkwd/ (temporary directories) as root,
which will then prevent non-root instances of lbmk
from being able to make temporary files.
Signed-off-by: Leah Rowe <leah@libreboot.org>
use the new functionality in err(), whereby a given
function name and arguments can be provided, for
debugging purposes.
something similar was already done in a few places,
and replaced with this unified functionality.
this patch will make xbmk much easier to debug, under
fault conditions.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the newer way handles escaped characters better, and it
can be nested more easily. it's also more readable.
personally, i prefer the old way, because it's more
minimalist, but it occurs to me that a lot of people
nowadays don't know about backticks, but they do know
of the modern way.
to make the code more readable, i have modernised it.
Signed-off-by: Leah Rowe <leah@libreboot.org>
you can still remove it with the -f flag on rm, but
xbmk only does that on exit from the main parent
instance, or after each build stage in release.sh
because of this, the user could still manually override
the lock file; this would cause running instances of lbmk
to restart wrongly as parent instances.
there's no way to fix any of this, but users don't normally
put -f in their rm commands.
however, this is also a preventative bug fix. if a bug
is ever caused in the future, where the lock file is
created erroneously, the write protection will prevent that,
so long as *it* is still done.
Signed-off-by: Leah Rowe <leah@libreboot.org>
e.g. gnupath, xbmkpath
these currently go in XBMK_CACHE/, which is bad
because they're meant to be temporary.
XBMK_CACHE is for permanent files.
Signed-off-by: Leah Rowe <leah@libreboot.org>
We currently use GNU-only options in the date command,
when initialising a Git repository.
This isn't a problem in practise, on non-GNU implementations
if not initialising a Git repository, because it's only
used in that situation.
In practise, only those systems with GNU coreutils and libc
are used to compile releases, so this is OK for me at least.
Future portability improvements will correct the issue, and
then this error check can be removed.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the release functions in release.sh rely on the
version string *not* being a path containing slashes.
just a single string e.g. "foo", not e.g. "foo/bar"
this is because several checks there make that
assumption. in practise, we always ensure that tags
and such do not contain these characters.
however, someone else working on their own version
of xbmk might not know of this design flaw, so let's
try to correct it in code.
we can add more filtering as designed, in the relevant
function (xbmk_sanitize_version).
Signed-off-by: Leah Rowe <leah@libreboot.org>
we no longer separate them. xbloc was the on-disk
tmp directory, whereas xbtmp used to be in /tmp
which we assumed to be tmpfs (it may not be, but
often is on many workstation setups - and our
documentation recommended doing this).
as mentioned in the previous commit, benchmarking
shows little speed difference using tmpfs /tmp
versus on-disk /tmp, for our purposes at least.
therefore, the handling of tmp files is being
greatly simplified.
Signed-off-by: Leah Rowe <leah@libreboot.org>
This way, all operations will be done inside the xbmk
work directory. This is being done, so that I can then
reliably sandybox certain commands in future commits,
for example the "rm" command.
This will also allow me to unify the location of all
temporary files, in future commits. I previously used
the /tmp directory because it's tmpfs-based on many
setups, and this is great for performance. However, in
practise, I never noticed any difference in performance
when benchmarking it (testing /tmp on-disk versus tmpfs).
Signed-off-by: Leah Rowe <leah@libreboot.org>
this is a general function that sets variables,
but there are many types of variables to be set.
rather than have all the logic inside this function,
handle it in subfunctions called by xbmk_set_env.
Signed-off-by: Leah Rowe <leah@libreboot.org>
we don't need these anymore, because we now know
whether or not the lock file exists in these cases.
this is because child/parent instance determination
is now done based on the presence of that file, rather
than how TMPDIR is set; and TMPDIR is now set accordingly,
via more robust logic as in previous patching.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it is extremely unlikely to occur, but this patch reduces
the likelihood even further. that unlikely occurance is:
when creating a TMPDIR, it's possible that it was already
created before. this is OK on child instances, where that
is the intended behaviour (unified TMPDIR), but not for
parent instances.
Signed-off-by: Leah Rowe <leah@libreboot.org>
we previously checked whether xbmk was running a child
instance, based on the initialisation of TMPDIR, but
this relied on unreliable string substitutions, which
could not be made inherently reliable. there were also
no checks on whether the given TMPDIR, even if correct,
was a directory or whether it was a symlink; there were
also no checks on whether it changed.
now with this change, child instances are detected by
the presence of the lock file. the parent instance
writes the generated TMPDIR location in that file, and
this is checked again in the child instance, to ensure
that the TMPDIR didn't change; it also errors out if
the TMPDIR doesn't exist or if it is a symlink.
Signed-off-by: Leah Rowe <leah@libreboot.org>
the current behaviour is a relic from the older lbmk
design, before recent auditing.
the current logic would cause xbmk to continue execution,
going into a child process with .git/ being a symlink.
The .git/ directory should never be a symlink, because
it is extremely error-prone.
Signed-off-by: Leah Rowe <leah@libreboot.org>
This reverts commit fb7aaa78bb.
it caused a few issues. will re-do later
the old code isn't really broken, just inefficient, because
several files are scanned twice, but in practise the overhead
isn't that great
The error occurs sometimes, when bruteforcing me.bin:
ERROR ./mk: Unhandled error for: mv /home/user/lbmk/tmp/me.bin /home/user/lbmk/cache/tmpdl/check
This revert should fix the issue, for now.
i'm adding characters to 7ztest, which isn't being passed
on through because everything runs in subshells; the next
pass would default back to the original string, so a given
file may be checked multiple times.
fix this by mitigation; use the random string from mktemp
as a suffix instead.
in practice, this has not affected performance much, but it
will nevertheless avoid unnecessary work by xbmk.
Signed-off-by: Leah Rowe <leah@libreboot.org>
I never really intended for this to be configurable,
but the cache directory is also used during release
builds.
There's too much that can go wrong, letting the user
decide where their cache is. Simplify it by hardcoding.
Signed-off-by: Leah Rowe <leah@libreboot.org>
it's just two lines, and we want much more granular
control of where the lock is enforced. it should be
JUST after confirming that the instance is a parent.
it is at this moment that we should bail if a lock
file exists, because this signals that another instance
of xbmk is running.
Signed-off-by: Leah Rowe <leah@libreboot.org>
