diff options
author | Patrick Williams <patrick@stwcx.xyz> | 2023-12-06 01:48:56 +0300 |
---|---|---|
committer | Patrick Williams <patrick@stwcx.xyz> | 2023-12-06 18:21:06 +0300 |
commit | da295319aa739dfa657fb4a1ba7e81804ba48658 (patch) | |
tree | d8bb670855f82ac4f2a50e8ca78b4534c0865485 /poky/scripts/lib | |
parent | 4e5690417ceb3d4e031bf3fcd306f724026f9d3d (diff) | |
download | openbmc-da295319aa739dfa657fb4a1ba7e81804ba48658.tar.xz |
subtree updates
meta-arm: 1dff3300fb..0b61cc659a:
Ross Burton (1):
meta-arm/selftest: add test that PAC/BTI instructions are used
meta-openembedded: 991e6852a5..5ad7203f68:
Alexander Kanavin (1):
fio: revert "fio: upgrade 3.32 -> 2022"
BELOUARGA Mohamed (1):
monocypher: add crypto library recipe
Dylan Turner (1):
apache2: v2.4.57 to v2.4.58 to fix CVE-2023-43622
Hongxu Jia (1):
freeradius: make sub packages to runtime depends on freeradius
Kai Kang (1):
libnma: remove conflict xml file
Khem Raj (12):
nlohmann-json: Fix SRCREV_FORMAT and do not package git metadata into ptests
ptest-packagelists-meta-oe.inc: Move poco to slow tests
sdbus-c++-libsystemd: Upgrade to 254
sdbus-c++-tools: Upgrade to 1.4.0
gstd: Fix systemd user unit packaging
basu: Update to latest master
sdbus-c++: Install ptests into PTEST_PATH
liblognorm:Add asprintf to autoconf function check macro
gnome-console,gnome-terminal: Depend on vte from core layer
Revert "gnome-terminal: Remove recommendation on vte-prompt"
vte9: Drop recipe
basu: Update the SRCREV to get lld fix
Luca Fancellu (1):
linuxptp: Update downstream patches
Markus Volk (9):
libcacard: fix version string in libcacard.pc
cups-filters: fix Makefile race condition
system-config-printer: Add packageconfig for polkit
pipewire: upgrade 0.3.85 > 1.0.0
libcacard: set meson version based on PV
spice: Set meson version based on PV
spice-gtk: Set meson version based on PV
libdecor: update 0.2.0 -> 0.2.1
xdg-desktop-portal-gnome: upgrade 45.0 -> 45.1
Naveen Saini (2):
tbb: upgrade 2021.9.0 -> 2021.11.0
tbb: enable NUMA/Hybrid CPU support
Patrick Wicki (6):
squid: update from v5.7 to v6.5
squid: add nm dispatcher reload hook
squid: add auth packageconfig
squid: move configs to sub package
squid: add url-rewrite-helpers packageconfig
squid: add systemd service
Patrick Williams (1):
glog: Disable 64bit atomics on armv{5,6}
Peter Kjellerstedt (1):
redis: Inherit pkgconfig
Ross Burton (1):
python3-validators: add new recipe
Wang Mingyu (26):
ctags: upgrade 6.0.20231119.0 -> 6.0.20231126.0
dnfdragora: upgrade 2.1.4 -> 2.1.5
gensio: upgrade 2.7.7 -> 2.8.0
frr: upgrade 9.0.1 -> 9.1
capnproto: upgrade 1.0.1 -> 1.0.1.1
libbpf: upgrade 1.2.2 -> 1.3.0
paho-mqtt-cpp: upgrade 1.2.0 -> 1.3.1
tomoyo-tools: upgrade 2.5.0 -> 2.6.1
python3-aiohttp: upgrade 3.9.0 -> 3.9.1
python3-bitstring: upgrade 4.1.2 -> 4.1.3
python3-dbus-fast: upgrade 2.14.0 -> 2.15.0
python3-humanize: upgrade 4.8.0 -> 4.9.0
python3-ipython: upgrade 8.17.2 -> 8.18.0
python3-mypy: upgrade 1.7.0 -> 1.7.1
python3-pdm: upgrade 2.10.3 -> 2.10.4
python3-pexpect: upgrade 4.8.0 -> 4.9.0
python3-pychromecast: upgrade 13.0.7 -> 13.0.8
python3-pydantic: upgrade 2.5.1 -> 2.5.2
python3-pymisp: upgrade 2.4.178 -> 2.4.179
python3-pytest-xdist: upgrade 3.4.0 -> 3.5.0
python3-sentry-sdk: upgrade 1.35.0 -> 1.37.1
python3-types-setuptools: upgrade 68.2.0.1 -> 68.2.0.2
python3-virtualenv: upgrade 20.24.6 -> 20.24.7
redis: upgrade 7.2.2 -> 7.2.3
ser2net: upgrade 4.5.1 -> 4.6.0
thingsboard-gateway: upgrade 3.4.2 -> 3.4.3.1
alperak (12):
squashfs-tools-ng: upgrade 1.1.4 -> 1.2.0
tmate: Fix finding msgpack 6+
msgpack-c: upgrade 4.0.0 -> 6.0.0
msgpack-cpp: upgrade 4.1.1 -> 6.1.0
brotli: upgrade 1.0.9 -> 1.1.0
icewm: upgrade 2.9.9 -> 3.4.4
iotop: upgrade 1.21 -> 1.25
liblognorm: upgrade 1.0.1 -> 2.0.6
libmodbus: upgrade 3.1.7 -> 3.1.10
libpwquality: upgrade 1.4.4 -> 1.4.5
libspiro: upgrade 20200505 -> 20221101
gtkwave: upgrade 3.3.111 -> 3.3.117
poky: 2696bf8cf3..028b6f6226:
Adrian Freihofer (1):
cmake-qemu.bbclass: support qemu for cmake
Alassane Yattara (9):
bitbake: toaster/tests: Update methods wait_until_~ to skip using time.sleep
bitbake: toaster/tests: Override table edit columns TestCase from image recipe page
bitbake: toaster/tests: Test software recipe page
bitbake: toaster/tests: Added Machine page TestCase
bitbake: toaster/tests: Added Layers page TestCase
bitbake: toaster/tests: Added distro page TestCase
bitbake: toaster/tests: Bug-fix on tests/functional/test_project_page
bitbake: toaster/tests: Test single layer page
bitbake: toaster/tests: Test single recipe page
Alex Kiernan (4):
rust: Delete python2 configparser code path
rust: Drop TARGET_VENDOR export
eudev: Upgrade 3.2.12 -> 3.2.14
rust: Drop targets and hosts override magic
Alexander Kanavin (15):
python3-pyproject-hooks: fix upstream version check
cmake: upgrade 3.27.5 -> 3.27.7
desktop-file-utils: upgrade 0.26 -> 0.27
erofs-utils: upgrade 1.6 -> 1.7.1
webkitgtk: update 2.40.5 -> 2.42.2
epiphany: upgrade 44.6 -> 45.1
virglrenderer: upgrade 0.10.4 -> 1.0.0
libxkbcommon: upgrade 1.5.0 -> 1.6.0
mpg123: upgrade 1.31.3 -> 1.32.3
icu: upgrade 73-2 -> 74-1
p11-kit: upgrade 0.25.0 -> 0.25.2
glib-2.0: install gio-querymodules into bindir as well as libexecdir for native
meson: update 1.2.2 -> 1.3.0
repo: update 2.37 -> 2.39
rt-tests: update 2.5 -> 2.6
Bruce Ashfield (1):
lttng-modules: fix build for v6.7+
Changhyeok Bae (1):
iptables: upgrade 1.8.9 -> 1.8.10
Charlie Johnston (2):
bitbake.conf: Add gsutil as hosttool for gcp fetcher.
bitbake: fetch2: Ensure GCP fetcher checks if file exists before download.
Jan Vermaete (1):
systemd: fixed typo
Joao Marcos Costa (1):
documentation.conf: fix do_menuconfig description
Joshua Watt (2):
bitbake: bitbake-hashclient: Add commands to get hashes
bitbake: hashserv: sqlite: Ensure sync propagates to database connections
Julien Stephan (6):
devtool: fix update-recipe dry-run mode
lib/oe/recipeutils.py: remove trailing white-spaces
devtool: finish/update-recipe: restrict mode srcrev to recipes fetched from SCM
devtool: tag all submodules
devtool: add support for git submodules
oeqa/selftest/devtool: add test for git submodules
Justin Bronder (1):
contributor-guide: add License-Update tag
Kareem Zarka (2):
wic: bootimg-efi: Make kernel image installation configurable
oeqa/selftest/wic: Add tests for kernel image installation
Khem Raj (8):
shared-mime-info: Fix build with clang-17+
libsoup-2.4: Fix build with clang-17 and libxml2-2.12
busybox: Enable utmp support on musl systems
virglrenderer: Fix build with clang
llvm: Upgrade to 17.0.6
rust-common.bbclass: Define rust arch for x32 platforms
vte: Upgrade to 0.74.1
vte: Separate out gtk4 pieces of vte into individual packages
Lee Chee Yang (3):
wic: add test for partition hidden attributes
migration-guides: add release notes for 4.3.1
openssl: upgrade to 3.2.0
Malte Schmidt (1):
wic: rawcopy: add support for zstd decompression
Marco Felsch (1):
json-c: fix icecc compilation
Markus Volk (3):
bluez5: fix connection for ps5/dualshock controllers
cups: Add root,sys,wheel to system groups
vte: upgrade 0.72.2 -> 0.74.0
Martin Hundeb?ll (1):
libpam: split /etc/environment into pam-plugin-env package
Matsunaga-Shinji (1):
cve-check: Modify judgment processing using "=" in version comparison
Michael Opdenacker (4):
systemd-compat-units.bb: fix postinstall script
dev-manual: layers: update link to YP Compatible form
contributor-guide: fix command option
migration-guides: release 3.5 is actually 4.0
Niko Mauno (1):
rust-llvm: Allow overriding LLVM target archs
Patrick Williams (1):
shared-mime-info-native: handle old GCC for AlmaLinux8
Peter Marko (2):
cve-update-nvd2-native: remove unused variable CVE_SOCKET_TIMEOUT
cve-update-nvd2-native: make number of fetch attemtps configurable
Richard Haar (1):
bitbake: bitbake: tests: Fix duplicate test_underscore_override test
Richard Purdie (2):
bitbake: ui/ncurses: Add missing function call to avoid traceback
bitbake: cooker: Avoid eventlog variable listing lockups
Robert Yang (2):
gnu-config: Update to latest revision
gettext: Upgrade 0.22 -> 0.22.3
Ross Burton (3):
core-image-minimal-initramfs: don't install a kernel into the initramfs
autoconf: upgrade to 2.72d
Revert "cve-check: Modify judgment processing using "=" in version comparison"
Sundeep KOKKONDA (3):
rust: Split rustdoc into a separate package
glibc: stable 2.38 branch updates
binutils: stable 2.41 branch updates
Tim Orling (8):
python3-sphinxcontrib-applehelp: 1.0.4 -> 1.0.7
python3-sphinxcontrib-devhelp: 1.0.2 -> 1.0.5
python3-sphinxcontrib-htmlhelp: 2.0.1 -> 2.0.4
python3-sphinxcontrib-qthelp: 1.0.3 -> 1.0.6
python3-sphinxcontrib-serializinghtml: 1.1.5 -> 1.1.9
vim: upgrade 9.0.2068 -> 9.0.2130
python3-cryptography-vectors: add RECIPE_NO_UPDATE_REASON
python3-cryptography{-vectors}: 41.0.5 -> 41.0.7
Trevor Gamblin (2):
python3-ptest: skip test_storlines
patchtest: shorten patch signed-off-by test output
Viswanath Kraleti (1):
systemd-boot: Fix build issues on armv7a-linux
Wang Mingyu (27):
bind: upgrade 9.18.19 -> 9.18.20
diffoscope: upgrade 251 -> 252
ell: upgrade 0.59 -> 0.60
git: upgrade 2.42.1 -> 2.43.0
gnutls: upgrade 3.8.1 -> 3.8.2
libdrm: upgrade 2.4.117 -> 2.4.118
libgcrypt: upgrade 1.10.2 -> 1.10.3
libksba: upgrade 1.6.4 -> 1.6.5
libxslt: upgrade 1.1.38 -> 1.1.39
log4cplus: upgrade 2.1.0 -> 2.1.1
python3-certifi: upgrade 2023.7.22 -> 2023.11.17
python3-setuptools: upgrade 68.2.2 -> 69.0.2
python3-wcwidth: upgrade 0.2.9 -> 0.2.11
python3-hypothesis: upgrade 6.89.0 -> 6.90.0
python3-pyasn1: upgrade 0.5.0 -> 0.5.1
python3-scons: upgrade 4.5.2 -> 4.6.0
python3-urllib3: upgrade 2.0.7 -> 2.1.0
ethtool: upgrade 6.5 -> 6.6
gi-docgen: upgrade 2023.1 -> 2023.3
init-system-helpers: upgrade 1.65.2 -> 1.66
libsolv: upgrade 0.7.26 -> 0.7.27
python3-idna: upgrade 3.4 -> 3.6
ofono: upgrade 2.1 -> 2.2
python3-sphinx-rtd-theme: upgrade 1.3.0 -> 2.0.0
python3-trove-classifiers: upgrade 2023.11.14 -> 2023.11.22
python3-wheel: upgrade 0.41.3 -> 0.42.0
resolvconf: upgrade 1.91 -> 1.92
Xiangyu Chen (2):
shadow: Fix for CVE-2023-4641
bash: changes to SIGINT handler while waiting for a child
Zahir Hussain (1):
cmake: Unset CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES
meta-raspberrypi: 8231f97534..fde68b24f0:
Lorenzo Arena (1):
docs: fix syntax for overriding fs type for initramfs image
Change-Id: Idc6f6b1e913442bae03dfec9f207924c56f31056
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Diffstat (limited to 'poky/scripts/lib')
-rw-r--r-- | poky/scripts/lib/devtool/__init__.py | 24 | ||||
-rw-r--r-- | poky/scripts/lib/devtool/standard.py | 284 | ||||
-rw-r--r-- | poky/scripts/lib/devtool/upgrade.py | 51 | ||||
-rw-r--r-- | poky/scripts/lib/recipetool/append.py | 4 | ||||
-rw-r--r-- | poky/scripts/lib/wic/plugins/source/bootimg-efi.py | 7 | ||||
-rw-r--r-- | poky/scripts/lib/wic/plugins/source/rawcopy.py | 3 |
6 files changed, 238 insertions, 135 deletions
diff --git a/poky/scripts/lib/devtool/__init__.py b/poky/scripts/lib/devtool/__init__.py index 702db669de..b4f998a5bc 100644 --- a/poky/scripts/lib/devtool/__init__.py +++ b/poky/scripts/lib/devtool/__init__.py @@ -233,6 +233,30 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None): bb.process.run('git checkout -b %s' % devbranch, cwd=repodir) bb.process.run('git tag -f %s' % basetag, cwd=repodir) + # if recipe unpacks another git repo inside S, we need to declare it as a regular git submodule now, + # so we will be able to tag branches on it and extract patches when doing finish/update on the recipe + stdout, _ = bb.process.run("git status --porcelain", cwd=repodir) + found = False + for line in stdout.splitlines(): + if line.endswith("/"): + new_dir = line.split()[1] + for root, dirs, files in os.walk(os.path.join(repodir, new_dir)): + if ".git" in dirs + files: + (stdout, _) = bb.process.run('git remote', cwd=root) + remote = stdout.splitlines()[0] + (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root) + remote_url = stdout.splitlines()[0] + logger.error(os.path.relpath(os.path.join(root, ".."), root)) + bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, os.path.join(root, ".."))), cwd=os.path.join(root, "..")) + found = True + if found: + useroptions = [] + oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d) + bb.process.run('git %s commit -m "Adding additionnal submodule from SRC_URI\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=os.path.join(root, "..")) + found = False + if os.path.exists(os.path.join(repodir, '.gitmodules')): + bb.process.run('git submodule foreach --recursive "git tag -f %s"' % basetag, cwd=repodir) + def recipe_to_append(recipefile, config, wildcard=False): """ Convert a recipe file to a bbappend file path within the workspace. diff --git a/poky/scripts/lib/devtool/standard.py b/poky/scripts/lib/devtool/standard.py index d53fb81007..ad6e346279 100644 --- a/poky/scripts/lib/devtool/standard.py +++ b/poky/scripts/lib/devtool/standard.py @@ -234,10 +234,14 @@ def add(args, config, basepath, workspace): if args.fetchuri and not args.no_git: setup_git_repo(srctree, args.version, 'devtool', d=tinfoil.config_data) - initial_rev = None + initial_rev = {} if os.path.exists(os.path.join(srctree, '.git')): (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree) - initial_rev = stdout.rstrip() + initial_rev["."] = stdout.rstrip() + (stdout, _) = bb.process.run('git submodule --quiet foreach --recursive \'echo `git rev-parse HEAD` $PWD\'', cwd=srctree) + for line in stdout.splitlines(): + (rev, submodule) = line.split() + initial_rev[os.path.relpath(submodule, srctree)] = rev if args.src_subdir: srctree = os.path.join(srctree, args.src_subdir) @@ -251,7 +255,8 @@ def add(args, config, basepath, workspace): if b_is_s: f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree) if initial_rev: - f.write('\n# initial_rev: %s\n' % initial_rev) + for key, value in initial_rev.items(): + f.write('\n# initial_rev %s: %s\n' % (key, value)) if args.binary: f.write('do_install:append() {\n') @@ -823,8 +828,8 @@ def modify(args, config, basepath, workspace): _check_compatible_recipe(pn, rd) - initial_rev = None - commits = [] + initial_revs = {} + commits = {} check_commits = False if bb.data.inherits_class('kernel-yocto', rd): @@ -880,15 +885,23 @@ def modify(args, config, basepath, workspace): args.no_extract = True if not args.no_extract: - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) - if not initial_rev: + initial_revs["."], _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + if not initial_revs["."]: return 1 logger.info('Source tree extracted to %s' % srctree) + if os.path.exists(os.path.join(srctree, '.git')): # Get list of commits since this revision - (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree) - commits = stdout.split() + (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_revs["."], cwd=srctree) + commits["."] = stdout.split() check_commits = True + (stdout, _) = bb.process.run('git submodule --quiet foreach --recursive \'echo `git rev-parse devtool-base` $PWD\'', cwd=srctree) + for line in stdout.splitlines(): + (rev, submodule_path) = line.split() + submodule = os.path.relpath(submodule_path, srctree) + initial_revs[submodule] = rev + (stdout, _) = bb.process.run('git rev-list --reverse devtool-base..HEAD', cwd=submodule_path) + commits[submodule] = stdout.split() else: if os.path.exists(os.path.join(srctree, '.git')): # Check if it's a tree previously extracted by us. This is done @@ -905,11 +918,11 @@ def modify(args, config, basepath, workspace): for line in stdout.splitlines(): if line.startswith('*'): (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree) - initial_rev = stdout.rstrip() - if not initial_rev: + initial_revs["."] = stdout.rstrip() + if not initial_revs["."]: # Otherwise, just grab the head revision (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree) - initial_rev = stdout.rstrip() + initial_revs["."] = stdout.rstrip() branch_patches = {} if check_commits: @@ -976,10 +989,11 @@ def modify(args, config, basepath, workspace): ' ln -sfT ${KCONFIG_CONFIG_ROOTDIR}/.config ${S}/.config.new\n' ' fi\n' '}\n') - if initial_rev: - f.write('\n# initial_rev: %s\n' % initial_rev) - for commit in commits: - f.write('# commit: %s\n' % commit) + if initial_revs: + for name, rev in initial_revs.items(): + f.write('\n# initial_rev %s: %s\n' % (name, rev)) + for commit in commits[name]: + f.write('# commit %s: %s\n' % (name, commit)) if branch_patches: for branch in branch_patches: if branch == args.branch: @@ -1202,44 +1216,56 @@ def _get_patchset_revs(srctree, recipe_path, initial_rev=None, force_patch_refre branchname = stdout.rstrip() # Parse initial rev from recipe if not specified - commits = [] + commits = {} patches = [] + initial_revs = {} with open(recipe_path, 'r') as f: for line in f: - if line.startswith('# initial_rev:'): - if not initial_rev: - initial_rev = line.split(':')[-1].strip() - elif line.startswith('# commit:') and not force_patch_refresh: - commits.append(line.split(':')[-1].strip()) - elif line.startswith('# patches_%s:' % branchname): - patches = line.split(':')[-1].strip().split(',') - - update_rev = initial_rev - changed_revs = None - if initial_rev: + pattern = r'^#\s.*\s(.*):\s([0-9a-fA-F]+)$' + match = re.search(pattern, line) + if match: + name = match.group(1) + rev = match.group(2) + if line.startswith('# initial_rev'): + if not (name == "." and initial_rev): + initial_revs[name] = rev + elif line.startswith('# commit') and not force_patch_refresh: + if name not in commits: + commits[name] = [rev] + else: + commits[name].append(rev) + elif line.startswith('# patches_%s:' % branchname): + patches = line.split(':')[-1].strip().split(',') + + update_revs = dict(initial_revs) + changed_revs = {} + for name, rev in initial_revs.items(): # Find first actually changed revision stdout, _ = bb.process.run('git rev-list --reverse %s..HEAD' % - initial_rev, cwd=srctree) + rev, cwd=os.path.join(srctree, name)) newcommits = stdout.split() - for i in range(min(len(commits), len(newcommits))): - if newcommits[i] == commits[i]: - update_rev = commits[i] + if name in commits: + for i in range(min(len(commits[name]), len(newcommits))): + if newcommits[i] == commits[name][i]: + update_revs[name] = commits[name][i] try: stdout, _ = bb.process.run('git cherry devtool-patched', - cwd=srctree) + cwd=os.path.join(srctree, name)) except bb.process.ExecutionError as err: stdout = None if stdout is not None and not force_patch_refresh: - changed_revs = [] for line in stdout.splitlines(): if line.startswith('+ '): rev = line.split()[1] if rev in newcommits: - changed_revs.append(rev) + if name not in changed_revs: + changed_revs[name] = [rev] + else: + changed_revs[name].append(rev) - return initial_rev, update_rev, changed_revs, patches + return initial_revs, update_revs, changed_revs, patches def _remove_file_entries(srcuri, filelist): """Remove file:// entries from SRC_URI""" @@ -1294,14 +1320,17 @@ def _remove_source_files(append, files, destpath, no_report_remove=False, dry_ru raise -def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None): +def _export_patches(srctree, rd, start_revs, destdir, changed_revs=None): """Export patches from srctree to given location. Returns three-tuple of dicts: 1. updated - patches that already exist in SRCURI 2. added - new patches that don't exist in SRCURI 3 removed - patches that exist in SRCURI but not in exported patches - In each dict the key is the 'basepath' of the URI and value is the - absolute path to the existing file in recipe space (if any). + In each dict the key is the 'basepath' of the URI and value is: + - for updated and added dicts, a dict with 2 optionnal keys: + - 'path': the absolute path to the existing file in recipe space (if any) + - 'patchdir': the directory in wich the patch should be applied (if any) + - for removed dict, the absolute path to the existing file in recipe space """ import oe.recipeutils from oe.patch import GitApplyTree @@ -1315,54 +1344,60 @@ def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None): # Generate patches from Git, exclude local files directory patch_pathspec = _git_exclude_path(srctree, 'oe-local-files') - GitApplyTree.extractPatches(srctree, start_rev, destdir, patch_pathspec) - - new_patches = sorted(os.listdir(destdir)) - for new_patch in new_patches: - # Strip numbering from patch names. If it's a git sequence named patch, - # the numbers might not match up since we are starting from a different - # revision This does assume that people are using unique shortlog - # values, but they ought to be anyway... - new_basename = seqpatch_re.match(new_patch).group(2) - match_name = None - for old_patch in existing_patches: - old_basename = seqpatch_re.match(old_patch).group(2) - old_basename_splitext = os.path.splitext(old_basename) - if old_basename.endswith(('.gz', '.bz2', '.Z')) and old_basename_splitext[0] == new_basename: - old_patch_noext = os.path.splitext(old_patch)[0] - match_name = old_patch_noext - break - elif new_basename == old_basename: - match_name = old_patch - break - if match_name: - # Rename patch files - if new_patch != match_name: - bb.utils.rename(os.path.join(destdir, new_patch), - os.path.join(destdir, match_name)) - # Need to pop it off the list now before checking changed_revs - oldpath = existing_patches.pop(old_patch) - if changed_revs is not None: - # Avoid updating patches that have not actually changed - with open(os.path.join(destdir, match_name), 'r') as f: - firstlineitems = f.readline().split() - # Looking for "From <hash>" line - if len(firstlineitems) > 1 and len(firstlineitems[1]) == 40: - if not firstlineitems[1] in changed_revs: - continue - # Recompress if necessary - if oldpath.endswith(('.gz', '.Z')): - bb.process.run(['gzip', match_name], cwd=destdir) - if oldpath.endswith('.gz'): - match_name += '.gz' - else: - match_name += '.Z' - elif oldpath.endswith('.bz2'): - bb.process.run(['bzip2', match_name], cwd=destdir) - match_name += '.bz2' - updated[match_name] = oldpath - else: - added[new_patch] = None + GitApplyTree.extractPatches(srctree, start_revs, destdir, patch_pathspec) + for dirpath, dirnames, filenames in os.walk(destdir): + new_patches = filenames + reldirpath = os.path.relpath(dirpath, destdir) + for new_patch in new_patches: + # Strip numbering from patch names. If it's a git sequence named patch, + # the numbers might not match up since we are starting from a different + # revision This does assume that people are using unique shortlog + # values, but they ought to be anyway... + new_basename = seqpatch_re.match(new_patch).group(2) + match_name = None + for old_patch in existing_patches: + old_basename = seqpatch_re.match(old_patch).group(2) + old_basename_splitext = os.path.splitext(old_basename) + if old_basename.endswith(('.gz', '.bz2', '.Z')) and old_basename_splitext[0] == new_basename: + old_patch_noext = os.path.splitext(old_patch)[0] + match_name = old_patch_noext + break + elif new_basename == old_basename: + match_name = old_patch + break + if match_name: + # Rename patch files + if new_patch != match_name: + bb.utils.rename(os.path.join(destdir, new_patch), + os.path.join(destdir, match_name)) + # Need to pop it off the list now before checking changed_revs + oldpath = existing_patches.pop(old_patch) + if changed_revs is not None and dirpath in changed_revs: + # Avoid updating patches that have not actually changed + with open(os.path.join(dirpath, match_name), 'r') as f: + firstlineitems = f.readline().split() + # Looking for "From <hash>" line + if len(firstlineitems) > 1 and len(firstlineitems[1]) == 40: + if not firstlineitems[1] in changed_revs[dirpath]: + continue + # Recompress if necessary + if oldpath.endswith(('.gz', '.Z')): + bb.process.run(['gzip', match_name], cwd=destdir) + if oldpath.endswith('.gz'): + match_name += '.gz' + else: + match_name += '.Z' + elif oldpath.endswith('.bz2'): + bb.process.run(['bzip2', match_name], cwd=destdir) + match_name += '.bz2' + updated[match_name] = {'path' : oldpath} + if reldirpath != ".": + updated[match_name]['patchdir'] = reldirpath + else: + added[new_patch] = {} + if reldirpath != ".": + added[new_patch]['patchdir'] = reldirpath + return (updated, added, existing_patches) @@ -1530,6 +1565,12 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi recipedir = os.path.basename(recipefile) logger.info('Updating SRCREV in recipe %s%s' % (recipedir, dry_run_suffix)) + # Get original SRCREV + old_srcrev = rd.getVar('SRCREV') or '' + if old_srcrev == "INVALID": + raise DevtoolError('Update mode srcrev is only valid for recipe fetched from an SCM repository') + old_srcrev = {'.': old_srcrev} + # Get HEAD revision try: stdout, _ = bb.process.run('git rev-parse HEAD', cwd=srctree) @@ -1556,13 +1597,12 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi if not no_remove: # Find list of existing patches in recipe file patches_dir = tempfile.mkdtemp(dir=tempdir) - old_srcrev = rd.getVar('SRCREV') or '' upd_p, new_p, del_p = _export_patches(srctree, rd, old_srcrev, patches_dir) logger.debug('Patches: update %s, new %s, delete %s' % (dict(upd_p), dict(new_p), dict(del_p))) # Remove deleted local files and "overlapping" patches - remove_files = list(del_f.values()) + list(upd_p.values()) + list(del_p.values()) + remove_files = list(del_f.values()) + [value["path"] for value in upd_p.values() if "path" in value] + [value["path"] for value in del_p.values() if "path" in value] if remove_files: removedentries = _remove_file_entries(srcuri, remove_files)[0] update_srcuri = True @@ -1576,11 +1616,10 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi patchfields['SRC_URI'] = '\\\n '.join(srcuri) if dry_run_outdir: logger.info('Creating bbappend (dry-run)') - else: - appendfile, destpath = oe.recipeutils.bbappend_recipe( - rd, appendlayerdir, files, wildcardver=wildcard_version, - extralines=patchfields, removevalues=removevalues, - redirect_output=dry_run_outdir) + appendfile, destpath = oe.recipeutils.bbappend_recipe( + rd, appendlayerdir, files, wildcardver=wildcard_version, + extralines=patchfields, removevalues=removevalues, + redirect_output=dry_run_outdir) else: files_dir = _determine_files_dir(rd) for basepath, path in upd_f.items(): @@ -1632,15 +1671,15 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil else: patchdir_params = {'patchdir': relpatchdir} - def srcuri_entry(basepath): + def srcuri_entry(basepath, patchdir_params): if patchdir_params: paramstr = ';' + ';'.join('%s=%s' % (k,v) for k,v in patchdir_params.items()) else: paramstr = '' return 'file://%s%s' % (basepath, paramstr) - initial_rev, update_rev, changed_revs, filter_patches = _get_patchset_revs(srctree, append, initial_rev, force_patch_refresh) - if not initial_rev: + initial_revs, update_revs, changed_revs, filter_patches = _get_patchset_revs(srctree, append, initial_rev, force_patch_refresh) + if not initial_revs: raise DevtoolError('Unable to find initial revision - please specify ' 'it with --initial-rev') @@ -1658,11 +1697,11 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil # Get updated patches from source tree patches_dir = tempfile.mkdtemp(dir=tempdir) - upd_p, new_p, _ = _export_patches(srctree, rd, update_rev, + upd_p, new_p, _ = _export_patches(srctree, rd, update_revs, patches_dir, changed_revs) # Get all patches from source tree and check if any should be removed all_patches_dir = tempfile.mkdtemp(dir=tempdir) - _, _, del_p = _export_patches(srctree, rd, initial_rev, + _, _, del_p = _export_patches(srctree, rd, initial_revs, all_patches_dir) logger.debug('Pre-filtering: update: %s, new: %s' % (dict(upd_p), dict(new_p))) if filter_patches: @@ -1677,18 +1716,31 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil updaterecipe = False destpath = None srcuri = (rd.getVar('SRC_URI', False) or '').split() + if appendlayerdir: files = OrderedDict((os.path.join(local_files_dir, key), val) for key, val in list(upd_f.items()) + list(new_f.items())) files.update(OrderedDict((os.path.join(patches_dir, key), val) for key, val in list(upd_p.items()) + list(new_p.items()))) + + params = [] + for file, param in files.items(): + patchdir_param = dict(patchdir_params) + patchdir = param.get('patchdir', ".") + if patchdir != "." : + if patchdir_param: + patchdir_param['patchdir'] += patchdir + else: + patchdir_param['patchdir'] = patchdir + params.append(patchdir_param) + if files or remove_files: removevalues = None if remove_files: removedentries, remaining = _remove_file_entries( srcuri, remove_files) if removedentries or remaining: - remaining = [srcuri_entry(os.path.basename(item)) for + remaining = [srcuri_entry(os.path.basename(item), patchdir_params) for item in remaining] removevalues = {'SRC_URI': removedentries + remaining} appendfile, destpath = oe.recipeutils.bbappend_recipe( @@ -1696,7 +1748,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil wildcardver=wildcard_version, removevalues=removevalues, redirect_output=dry_run_outdir, - params=[patchdir_params] * len(files)) + params=params) else: logger.info('No patches or local source files needed updating') else: @@ -1713,14 +1765,22 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil _move_file(os.path.join(local_files_dir, basepath), path, dry_run_outdir=dry_run_outdir, base_outdir=recipedir) updatefiles = True - for basepath, path in upd_p.items(): - patchfn = os.path.join(patches_dir, basepath) + for basepath, param in upd_p.items(): + path = param['path'] + patchdir = param.get('patchdir', ".") + if patchdir != "." : + patchdir_param = dict(patchdir_params) + if patchdir_param: + patchdir_param['patchdir'] += patchdir + else: + patchdir_param['patchdir'] = patchdir + patchfn = os.path.join(patches_dir, patchdir, basepath) if os.path.dirname(path) + '/' == dl_dir: # This is a a downloaded patch file - we now need to # replace the entry in SRC_URI with our local version logger.info('Replacing remote patch %s with updated local version' % basepath) path = os.path.join(files_dir, basepath) - _replace_srcuri_entry(srcuri, basepath, srcuri_entry(basepath)) + _replace_srcuri_entry(srcuri, basepath, srcuri_entry(basepath, patchdir_param)) updaterecipe = True else: logger.info('Updating patch %s%s' % (basepath, dry_run_suffix)) @@ -1734,15 +1794,23 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil os.path.join(files_dir, basepath), dry_run_outdir=dry_run_outdir, base_outdir=recipedir) - srcuri.append(srcuri_entry(basepath)) + srcuri.append(srcuri_entry(basepath, patchdir_params)) updaterecipe = True - for basepath, path in new_p.items(): + for basepath, param in new_p.items(): + patchdir = param.get('patchdir', ".") logger.info('Adding new patch %s%s' % (basepath, dry_run_suffix)) - _move_file(os.path.join(patches_dir, basepath), + _move_file(os.path.join(patches_dir, patchdir, basepath), os.path.join(files_dir, basepath), dry_run_outdir=dry_run_outdir, base_outdir=recipedir) - srcuri.append(srcuri_entry(basepath)) + params = dict(patchdir_params) + if patchdir != "." : + if params: + params['patchdir'] += patchdir + else: + params['patchdir'] = patchdir + + srcuri.append(srcuri_entry(basepath, params)) updaterecipe = True # Update recipe, if needed if _remove_file_entries(srcuri, remove_files)[0]: diff --git a/poky/scripts/lib/devtool/upgrade.py b/poky/scripts/lib/devtool/upgrade.py index 9cd50be3a2..10827a762b 100644 --- a/poky/scripts/lib/devtool/upgrade.py +++ b/poky/scripts/lib/devtool/upgrade.py @@ -90,7 +90,7 @@ def _rename_recipe_files(oldrecipe, bpn, oldpv, newpv, path): _rename_recipe_dirs(oldpv, newpv, path) return _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path) -def _write_append(rc, srctreebase, srctree, same_dir, no_same_dir, rev, copied, workspace, d): +def _write_append(rc, srctreebase, srctree, same_dir, no_same_dir, revs, copied, workspace, d): """Writes an append file""" if not os.path.exists(rc): raise DevtoolError("bbappend not created because %s does not exist" % rc) @@ -119,8 +119,9 @@ def _write_append(rc, srctreebase, srctree, same_dir, no_same_dir, rev, copied, if b_is_s: f.write('EXTERNALSRC_BUILD:pn-%s = "%s"\n' % (pn, srctree)) f.write('\n') - if rev: - f.write('# initial_rev: %s\n' % rev) + if revs: + for name, rev in revs.items(): + f.write('# initial_rev %s: %s\n' % (name, rev)) if copied: f.write('# original_path: %s\n' % os.path.dirname(d.getVar('FILE'))) f.write('# original_files: %s\n' % ' '.join(copied)) @@ -182,10 +183,15 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee uri, rev = _get_uri(crd) if srcrev: rev = srcrev + paths = [srctree] if uri.startswith('git://') or uri.startswith('gitsm://'): __run('git fetch') __run('git checkout %s' % rev) __run('git tag -f devtool-base-new') + __run('git submodule update --recursive') + __run('git submodule foreach \'git tag -f devtool-base-new\'') + (stdout, _) = __run('git submodule --quiet foreach \'echo $sm_path\'') + paths += [os.path.join(srctree, p) for p in stdout.splitlines()] md5 = None sha256 = None _, _, _, _, _, params = bb.fetch2.decodeurl(uri) @@ -256,29 +262,32 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee __run('git %s commit -q -m "Commit of upstream changes at version %s" --allow-empty' % (' '.join(useroptions), newpv)) __run('git tag -f devtool-base-%s' % newpv) - (stdout, _) = __run('git rev-parse HEAD') - rev = stdout.rstrip() + revs = {} + for path in paths: + (stdout, _) = _run('git rev-parse HEAD', cwd=path) + revs[os.path.relpath(path,srctree)] = stdout.rstrip() if no_patch: patches = oe.recipeutils.get_recipe_patches(crd) if patches: logger.warning('By user choice, the following patches will NOT be applied to the new source tree:\n %s' % '\n '.join([os.path.basename(patch) for patch in patches])) else: - __run('git checkout devtool-patched -b %s' % branch) - (stdout, _) = __run('git branch --list devtool-override-*') - branches_to_rebase = [branch] + stdout.split() - for b in branches_to_rebase: - logger.info("Rebasing {} onto {}".format(b, rev)) - __run('git checkout %s' % b) - try: - __run('git rebase %s' % rev) - except bb.process.ExecutionError as e: - if 'conflict' in e.stdout: - logger.warning('Command \'%s\' failed:\n%s\n\nYou will need to resolve conflicts in order to complete the upgrade.' % (e.command, e.stdout.rstrip())) - __run('git rebase --abort') - else: - logger.warning('Command \'%s\' failed:\n%s' % (e.command, e.stdout)) - __run('git checkout %s' % branch) + for path in paths: + _run('git checkout devtool-patched -b %s' % branch, cwd=path) + (stdout, _) = _run('git branch --list devtool-override-*', cwd=path) + branches_to_rebase = [branch] + stdout.split() + for b in branches_to_rebase: + logger.info("Rebasing {} onto {}".format(b, revs[os.path.relpath(path,srctree)])) + _run('git checkout %s' % b, cwd=path) + try: + _run('git rebase %s' % revs[os.path.relpath(path, srctree)], cwd=path) + except bb.process.ExecutionError as e: + if 'conflict' in e.stdout: + logger.warning('Command \'%s\' failed:\n%s\n\nYou will need to resolve conflicts in order to complete the upgrade.' % (e.command, e.stdout.rstrip())) + _run('git rebase --abort', cwd=path) + else: + logger.warning('Command \'%s\' failed:\n%s' % (e.command, e.stdout)) + _run('git checkout %s' % branch, cwd=path) if tmpsrctree: if keep_temp: @@ -288,7 +297,7 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee if tmpdir != tmpsrctree: shutil.rmtree(tmpdir) - return (rev, md5, sha256, srcbranch, srcsubdir_rel) + return (revs, md5, sha256, srcbranch, srcsubdir_rel) def _add_license_diff_to_recipe(path, diff): notice_text = """# FIXME: the LIC_FILES_CHKSUM values have been updated by 'devtool upgrade'. diff --git a/poky/scripts/lib/recipetool/append.py b/poky/scripts/lib/recipetool/append.py index 9dbb1cc4b5..4b6a7112c2 100644 --- a/poky/scripts/lib/recipetool/append.py +++ b/poky/scripts/lib/recipetool/append.py @@ -299,7 +299,7 @@ def appendfile(args): if st.st_mode & stat.S_IXUSR: perms = '0755' install = {args.newfile: (args.targetpath, perms)} - oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: sourcepath}, install, wildcardver=args.wildcard_version, machine=args.machine) + oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: {'path' : sourcepath}}, install, wildcardver=args.wildcard_version, machine=args.machine) tinfoil.modified_files() return 0 else: @@ -353,7 +353,7 @@ def appendsrc(args, files, rd, extralines=None): logger.warning('{0!r} is already in SRC_URI, not adding'.format(source_uri)) else: extralines.append('SRC_URI += {0}'.format(source_uri)) - copyfiles[newfile] = srcfile + copyfiles[newfile] = {'path' : srcfile} oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines) tinfoil.modified_files() diff --git a/poky/scripts/lib/wic/plugins/source/bootimg-efi.py b/poky/scripts/lib/wic/plugins/source/bootimg-efi.py index 4f30926f1a..13a9cddf4e 100644 --- a/poky/scripts/lib/wic/plugins/source/bootimg-efi.py +++ b/poky/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -400,9 +400,10 @@ class BootimgEFIPlugin(SourcePlugin): exec_native_cmd(objcopy_cmd, native_sysroot) else: - install_cmd = "install -m 0644 %s/%s %s/%s" % \ - (staging_kernel_dir, kernel, hdddir, kernel) - exec_cmd(install_cmd) + if source_params.get('install-kernel-into-boot-dir') != 'false': + install_cmd = "install -m 0644 %s/%s %s/%s" % \ + (staging_kernel_dir, kernel, hdddir, kernel) + exec_cmd(install_cmd) if get_bitbake_var("IMAGE_EFI_BOOT_FILES"): for src_path, dst_path in cls.install_task: diff --git a/poky/scripts/lib/wic/plugins/source/rawcopy.py b/poky/scripts/lib/wic/plugins/source/rawcopy.py index ccf332554e..21903c2f23 100644 --- a/poky/scripts/lib/wic/plugins/source/rawcopy.py +++ b/poky/scripts/lib/wic/plugins/source/rawcopy.py @@ -58,7 +58,8 @@ class RawCopyPlugin(SourcePlugin): decompressor = { ".bz2": "bzip2", ".gz": "gzip", - ".xz": "xz" + ".xz": "xz", + ".zst": "zstd -f", }.get(extension) if not decompressor: raise WicError("Not supported compressor filename extension: %s" % extension) |