pkgsrc-Changes archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
CVS commit: pkgsrc/pkgtools/python-versions-check
Module Name: pkgsrc
Committed By: wiz
Date: Sat Jul 1 09:26:59 UTC 2023
Added Files:
pkgsrc/pkgtools/python-versions-check: DESCR Makefile PLIST
pkgsrc/pkgtools/python-versions-check/files: python-versions-check
python-versions-check.1
Log Message:
pkgtools/python-versions-check: import python-versions-check-1.0
python-versions-check is a script for pkgsrc developers that compares
the Python versions supported by a package to the ones supported
by its dependencies and the packages depending on it.
This helps recognizing problems when a package wants to use another
package which does not support all the Python versions it supports.
To generate a diff of this commit:
cvs rdiff -u -r0 -r1.1 pkgsrc/pkgtools/python-versions-check/DESCR \
pkgsrc/pkgtools/python-versions-check/Makefile \
pkgsrc/pkgtools/python-versions-check/PLIST
cvs rdiff -u -r0 -r1.1 \
pkgsrc/pkgtools/python-versions-check/files/python-versions-check \
pkgsrc/pkgtools/python-versions-check/files/python-versions-check.1
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Added files:
Index: pkgsrc/pkgtools/python-versions-check/DESCR
diff -u /dev/null pkgsrc/pkgtools/python-versions-check/DESCR:1.1
--- /dev/null Sat Jul 1 09:26:59 2023
+++ pkgsrc/pkgtools/python-versions-check/DESCR Sat Jul 1 09:26:59 2023
@@ -0,0 +1,6 @@
+python-versions-check is a script for pkgsrc developers that compares
+the Python versions supported by a package to the ones supported
+by its dependencies and the packages depending on it.
+
+This helps recognizing problems when a package wants to use another
+package which does not support all the Python versions it supports.
Index: pkgsrc/pkgtools/python-versions-check/Makefile
diff -u /dev/null pkgsrc/pkgtools/python-versions-check/Makefile:1.1
--- /dev/null Sat Jul 1 09:26:59 2023
+++ pkgsrc/pkgtools/python-versions-check/Makefile Sat Jul 1 09:26:59 2023
@@ -0,0 +1,30 @@
+# $NetBSD: Makefile,v 1.1 2023/07/01 09:26:59 wiz Exp $
+
+PKGNAME= python-versions-check-1.0
+CATEGORIES= pkgtools
+
+MAINTAINER= wiz%NetBSD.org@localhost
+HOMEPAGE= https://www.pkgsrc.org/
+COMMENT= Tool for checking Python versions in dependencies
+LICENSE= 2-clause-bsd
+
+WRKSRC= ${WRKDIR}
+USE_LANGUAGES= # empty
+INSTALLATION_DIRS= bin ${PKGMANDIR}/man1
+REPLACE_PYTHON+= python-versions-check
+
+PYTHON_VERSIONS_INCOMPATIBLE= 27
+
+post-extract:
+ ${CP} ${FILESDIR}/python-versions-check ${WRKSRC}
+
+do-configure:
+
+do-build:
+
+do-install:
+ ${INSTALL_SCRIPT} ${WRKDIR}/python-versions-check ${DESTDIR}${PREFIX}/bin
+ ${INSTALL_MAN} ${FILESDIR}/python-versions-check.1 ${DESTDIR}${PREFIX}/${PKGMANDIR}/man1
+
+.include "../../lang/python/application.mk"
+.include "../../mk/bsd.pkg.mk"
Index: pkgsrc/pkgtools/python-versions-check/PLIST
diff -u /dev/null pkgsrc/pkgtools/python-versions-check/PLIST:1.1
--- /dev/null Sat Jul 1 09:26:59 2023
+++ pkgsrc/pkgtools/python-versions-check/PLIST Sat Jul 1 09:26:59 2023
@@ -0,0 +1,3 @@
+@comment $NetBSD: PLIST,v 1.1 2023/07/01 09:26:59 wiz Exp $
+bin/python-versions-check
+man/man1/python-versions-check.1
Index: pkgsrc/pkgtools/python-versions-check/files/python-versions-check
diff -u /dev/null pkgsrc/pkgtools/python-versions-check/files/python-versions-check:1.1
--- /dev/null Sat Jul 1 09:26:59 2023
+++ pkgsrc/pkgtools/python-versions-check/files/python-versions-check Sat Jul 1 09:26:59 2023
@@ -0,0 +1,231 @@
+#!/usr/bin/env python3
+#
+# $NetBSD: python-versions-check,v 1.1 2023/07/01 09:26:59 wiz Exp $
+#
+# Copyright (c) 2023 The NetBSD Foundation, Inc.
+# All rights reserved.
+#
+# This code is derived from software contributed to The NetBSD Foundation
+# by Thomas Klausner.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+#
+
+'''For a given Python package, find all packages that use it
+(by looking at DEPENDS, BUILD_DEPENDS, TOOL_DEPENDS)
+the compare acceptable and incompatible versions.'''
+
+import argparse
+from collections import defaultdict
+import glob
+import os
+import pathlib
+import re
+import sys
+
+# only accept includes with ../../ or in the current directory
+include_re = re.compile(r'\s*\.\s*include\s+"(\.\./\.\./[^/]*/[^/]*/|)([^/]*)"')
+depends_re = re.compile(r'[^#]*DEPENDS.*:(\.\./\.\./.*)')
+# '+=', '?='
+pva_re = re.compile(r'PYTHON_VERSIONS_ACCEPTED\s*\+?\??\s*=\s*([0-9 ]*)')
+pvi_re = re.compile(r'PYTHON_VERSIONS_INCOMPATIBLE\s*\+?\??\s*=\s*([0-9 ]*)')
+
+
+# all available Python versions
+existing = set([])
+# dictionary for pkg_path -> dependencies
+includes = defaultdict(set)
+# dictionary for pkg_path -> allowed Python versions
+python_versions = defaultdict(set)
+
+
+def supported_versions(pkg_path):
+ '''Return Python versions supported by a package.'''
+ if pkg_path in python_versions:
+ return python_versions[pkg_path]
+ return existing
+
+
+def extract_python_versions(path, apply_existing=True):
+ '''Find the supported Python versions for a package.'''
+ accepted = set([])
+ if apply_existing:
+ accepted = existing
+ with open(path, 'r', encoding='utf-8') as input_file:
+ for line in input_file.readlines():
+ if m := pva_re.match(line):
+ accepted = set(m.group(1).split())
+ elif m := pvi_re.match(line):
+ accepted = accepted - set(m.group(1).split())
+ return accepted
+
+
+def extract_includes(path, dict_key=None):
+ '''Read the interesting parts of a Makefile (fragment).'''
+ pkg_path = get_pkg_path(path)
+ directory = path[:path.rfind('/')+1]
+ if not dict_key:
+ dict_key = pkg_path
+ # elif dict_key in includes:
+ # # already handled
+ # return includes[dict_key]
+ any_python_include = False
+ with open(path, 'r', encoding='utf-8') as input_file:
+ for line in input_file.readlines():
+ if m := include_re.match(line):
+ file_path = m.group(1)
+ file_name = m.group(2)
+ # skip any unexpanded variables - we're not a full parser
+ # skip 'mk' includes
+ if '/mk/' in file_path or '${' in file_path or '${' in file_name:
+ continue
+ if 'lang/python/' in file_path:
+ any_python_include = True
+ # local includes and Makefile.common includes are parsed immediately
+ if len(file_path) == 0:
+ extract_includes(directory + m.group(2), dict_key)
+ elif pkg_path + '/' in file_path or file_name == 'Makefile.common':
+ full_path = absolute_path(file_path + file_name)
+ extract_includes(full_path, dict_key)
+ # non-local ones are dependencies
+ elif 'lang/lua/' in file_path \
+ or 'print/texlive' in file_path \
+ or 'lang/python' in file_path:
+ # has no Makefile
+ continue
+ else:
+ includes[dict_key].add(get_pkg_path(file_path))
+ elif m := depends_re.match(line):
+ file_path = m.group(1)
+ if '${' in file_path:
+ continue
+ includes[dict_key].add(get_pkg_path(m.group(1)))
+ elif m := pva_re.match(line):
+ python_versions[dict_key] = set(m.group(1).split())
+ elif m := pvi_re.match(line):
+ python_versions[dict_key] = supported_versions(dict_key) - set(m.group(1).split())
+ if not any_python_include:
+ includes[dict_key] = set([])
+ return includes[dict_key]
+
+
+def absolute_path(path):
+ '''Convert relative path to absolute one.'''
+ if path.startswith('../../'):
+ return args.pkgsrcdir + '/' + path[6:]
+ return path
+
+
+def get_pkg_path(full_path):
+ '''Strip pkgsrcdir from path.'''
+ cand = full_path
+ if cand.startswith(args.pkgsrcdir):
+ cand = cand[len(args.pkgsrcdir)+1:]
+ elif cand.startswith('../../'):
+ cand = cand[6:]
+
+ if cand.count('/') == 2:
+ cand = cand[:cand.rfind('/')]
+ return cand
+
+
+def report_problem(first, first_versions, second_versions):
+ '''Pretty-print a problem with mismatching Python versions.'''
+ difference = first_versions ^ second_versions
+ difference = sorted([int(x) for x in difference])
+ first_versions = sorted([int(x) for x in first_versions])
+ print(f'{first}: supports {first_versions}, missing: {difference}')
+
+
+parser = argparse.ArgumentParser(description='find all packages that' +
+ ' link against a given package')
+parser.add_argument('package', nargs='?',
+ help='package whose dependencies we want to check (default: current directory)')
+parser.add_argument('-p', dest='pkgsrcdir', default='/usr/pkgsrc',
+ help='path to the pkgsrc root directory', action='store')
+parser.add_argument('-w', dest='wip', default=False,
+ help='include wip in search', action='store_true')
+args = parser.parse_args()
+
+if not args.package:
+ args.package = get_pkg_path(os.getcwd())
+
+if not pathlib.Path(args.pkgsrcdir).exists() or \
+ not pathlib.Path(args.pkgsrcdir + '/doc').exists() or \
+ not pathlib.Path(args.pkgsrcdir + '/mk').exists():
+ print(f'invalid pkgsrc directory "{args.pkgsrcdir}"')
+ sys.exit(1)
+
+existing = extract_python_versions(args.pkgsrcdir + '/lang/python/pyversion.mk', False)
+supported = extract_python_versions(args.pkgsrcdir + '/' + args.package + '/Makefile')
+
+searchlist = set([args.package])
+result = set([])
+while searchlist:
+ entry = searchlist.pop()
+ # already handled?
+ if entry in result:
+ continue
+ result.add(entry)
+ searchlist |= extract_includes(args.pkgsrcdir + '/' + entry + '/Makefile')
+
+# print(f"dependencies for {args.package}: {sorted(result)}")
+print(f"Supported Python versions for {args.package}: {sorted([int(x) for x in supported_versions(args.package)])}")
+print(f"Checking packages used by {args.package}:")
+for entry in result:
+ entry_versions = supported_versions(entry)
+ # print(f"python version for {entry}: {entry_versions}")
+ if supported_versions(args.package) > entry_versions:
+ report_problem(entry, entry_versions, supported_versions(args.package))
+
+makefiles = glob.glob(args.pkgsrcdir + '/*/*/Makefile*')
+makefiles.extend(glob.glob(args.pkgsrcdir + '/*/*/*.mk'))
+makefiles = list(filter(lambda name: name.find('/mk/') == -1
+ and not name.endswith('buildlink3.mk')
+ and not name.endswith('cargo-depends.mk')
+ and not name.endswith('go-modules.mk'),
+ makefiles))
+if not args.wip:
+ makefiles = list(filter(lambda name: name.find('/wip/') == -1, makefiles))
+
+print(f"Checking packages using {args.package}:")
+checked_packages = set([])
+makefile_content = {}
+for makefile in makefiles:
+ extract_includes(makefile)
+
+searchlist = set([args.package])
+handled = set([])
+while searchlist:
+ entry = searchlist.pop()
+ if entry in handled:
+ continue
+ handled.add(entry)
+
+ entry_versions = supported_versions(entry)
+ for package, dependencies in includes.items():
+ if entry in dependencies:
+ package_versions = supported_versions(package)
+ if entry_versions < package_versions:
+ report_problem(package, package_versions, entry_versions)
+ python_versions[package] = entry_versions
+ searchlist.add(package)
Index: pkgsrc/pkgtools/python-versions-check/files/python-versions-check.1
diff -u /dev/null pkgsrc/pkgtools/python-versions-check/files/python-versions-check.1:1.1
--- /dev/null Sat Jul 1 09:26:59 2023
+++ pkgsrc/pkgtools/python-versions-check/files/python-versions-check.1 Sat Jul 1 09:26:59 2023
@@ -0,0 +1,85 @@
+.\" $NetBSD: python-versions-check.1,v 1.1 2023/07/01 09:26:59 wiz Exp $
+.\"
+.\" Copyright (c) 2023 The NetBSD Foundation, Inc.
+.\" All rights reserved.
+.\"
+.\" This code is derived from software contributed to The NetBSD Foundation
+.\" by Thomas Klausner.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+.\" POSSIBILITY OF SUCH DAMAGE.
+.\"
+.Dd June 10, 2023
+.Dt PYTHON-VERSIONS-CHECK.PY 1
+.Os
+.Sh NAME
+.Nm python-versions-check.py
+.Nd check Python versions for a package and its dependencies and users
+.Sh SYNOPSIS
+.Nm
+.Op Fl hw
+.Op Fl p Ar pkgsrcdir
+.Op Ar category/package
+.Sh DESCRIPTION
+.Nm
+is a script for pkgsrc developers that compares the Python versions
+supported by a package to the ones supported by its dependencies and
+the packages depending on it.
+.Pp
+.Nm
+checks
+.Ar category/package
+or the package in its current working directory, if no argument is
+provided.
+.Pp
+.Nm
+supports the following options:
+.Bl -tag -width 12n -offset indent
+.It Fl h
+Show the usage.
+.It Fl p Ar pkgsrcdir
+Use
+.Ar pkgsrcdir
+as root of the pkgsrc tree.
+Defaults to
+.Pa /usr/pkgsrc .
+.It Fl w
+Also bump packages in the
+.Pa wip/
+subtree.
+.El
+.Sh EXAMPLES
+Example for the output if there is a problem:
+.Bd -literal
+# python-versions-check.py math/py-numpy
+Supported Python versions for math/py-numpy: [39, 310, 311]
+Checking packages used by math/py-numpy:
+Checking packages using math/py-numpy:
+math/py-scipy: supports [38, 39, 310, 311], missing: [38]
+.Ed
+This means that while
+.Pa math/py-numpy
+only supports Python version 3.9, 3.10, and 3.11,
+.Pa math/py-scipy
+also supports Python version 3.8.
+This will break e.g. bulk builds because the required
+.Pa py38-numpy
+package will not exist.
Home |
Main Index |
Thread Index |
Old Index