tech-pkg archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Structuring configuration file versioning support in pkgsrc
Hi, I'm working on adding support for configuration files versioning to
pkgsrc.
I already posted some details and the proposal in the past, so I'll try
to be concise:
the idea is to store example configuration files in a version control
system to keep track of changes across upgrades, without changing
pkgsrc behavior in copying them to etc if the configuration file is
already there.
A tool yet to be written would then aid the user in installing the
configuration via an interactive 3-way merge, in backing up known to be
working configuration for the system or a subset of packages to a
specific vcs branch, and so on.
The solution should support different kinds of version control systems
(eg, cvs, git, mercurial, svn, fossil come to mind), and provide
support for using remote configuration repositories.
Furthermore, it should be able, if set so, to pull matching
configuration files adequate to the role of the system from the site
vcs repo and directly copy them to etc when installing or upgrading
a package.
The idea was to have sh scripts provided with the package do the job,
by creating a new script, eg, VCSWORK, implementing needed operations
wrapping around different tools and version control systems.
+FILES should mostly stay the way it is, with new functions to copy "c"
files to the local VCS working directory unconditionally and to install
configuration files unconditionally if required when working in "pull"
mode.
INSTALL should invoke appropriate VCSWORK functions in VCS-PRE (repo pull)
and VCS-POST (+FILES is called to copy new configuration examples into
the vcs working directory, then files are added/committed to the repo,
the repo is pushed if remote) phases.
VCS-POST should also handle configuration files selection, pulling and
call the new +FILES function for unconditional installation if pkgsrc is
set up to make use of this feature.
The new code could be included at compile and package creation time
only if CONF_FILES are shipped with the package.
As I said, a separate tool would be needed for interactive
configuration merging and backup/restore operations.
The INSTALL script would need additional parameters to select which VCS
to use among the supported ones, the working directory, the remote repo
URL and authentication details, if configuration is to be pulled from a
site-specific resource, the system role and so on.
The path I'm working on is to have new vcs parameters in
pkg_install.conf, have them parsed by pkg_add and passed to the INSTALL
script when running VCS-PRE and VCS-POST phases, if needed.
This to avoid hardcoding configuration parameters in the package at
build time, a solution which would not work well for public
repositories of prebuilt packages.
I have extended pkg_install to support new options and, before I go on
writing scripts and patches to mk/pkginstall, I'd like to hear from you
if this is an acceptable solution (another one would be to have all
vcs-related functionality handled by a new pkg_install tool, and have
this tool parse its configuration when called from INSTALL). I'm
obviously open to suggestions and corrections!
Keivan
Index: pkgtools/pkg_install/files/add/add.h
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/add/add.h,v
retrieving revision 1.19
diff -u -r1.19 add.h
--- pkgtools/pkg_install/files/add/add.h 14 Sep 2010 22:26:18 -0000 1.19
+++ pkgtools/pkg_install/files/add/add.h 17 May 2018 22:38:32 -0000
@@ -45,6 +45,7 @@
int make_hierarchy(char *);
void apply_perms(char *, char **, int);
-int pkg_perform(lpkg_head_t *);
+int pkg_perform(lpkg_head_t *);
+int pkg_perform_vcs(lpkg_head_t *, vcs_params_t *);
#endif /* _INST_ADD_H_INCLUDE */
Index: pkgtools/pkg_install/files/add/main.c
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/add/main.c,v
retrieving revision 1.32
diff -u -r1.32 main.c
--- pkgtools/pkg_install/files/add/main.c 27 Dec 2015 12:36:42 -0000 1.32
+++ pkgtools/pkg_install/files/add/main.c 17 May 2018 22:38:32 -0000
@@ -36,7 +36,7 @@
#include "lib.h"
#include "add.h"
-static char Options[] = "AC:DIK:P:RVfhm:np:t:Uuv";
+static char Options[] = "AC:DIK:P:RVfhm:np:t:Uuv:k";
char *Destdir = NULL;
char *OverrideMachine = NULL;
@@ -45,6 +45,8 @@
Boolean NoRecord = FALSE;
Boolean Automatic = FALSE;
Boolean ForceDepends = FALSE;
+Boolean noVCS = FALSE;
+
/*
* Normally, updating fails if the dependencies of a depending package
* are not satisfied by the package to be updated. ForceDepending
@@ -60,7 +62,7 @@
usage(void)
{
(void) fprintf(stderr, "%s\n%s\n%s\n",
- "usage: pkg_add [-AfhInRuVv] [-C config] [-P destdir] [-K pkg_dbdir]",
+ "usage: pkg_add [-AfhInRuVvk] [-C config] [-P destdir] [-K pkg_dbdir]",
" [-m machine] [-p prefix]",
" [[ftp|http]://[user[:password]@]host[:port]][/path/]pkg-name ...");
exit(1);
@@ -71,6 +73,8 @@
{
int ch, error=0;
lpkg_head_t pkgs;
+ vcs_params_t vcs_config;
+
setprogname(argv[0]);
while ((ch = getopt(argc, argv, Options)) != -1) {
@@ -138,6 +142,9 @@
case 'v':
Verbose = TRUE;
break;
+ case 'k':
+ noVCS = TRUE;
+ break;
case 'h':
case '?':
@@ -149,7 +156,15 @@
argc -= optind;
argv += optind;
+ if (noVCS || (strcasecmp(pkg_useconfvcs, "no") == 1)) {
+ vcs_config.pass_to_INSTALL = FALSE;
+ vcs_config.varnum = 0;
+ }
+ else {
+ vcs_config.pass_to_INSTALL = TRUE;
+ }
pkg_install_config();
+ pkg_install_vcs_config(&vcs_config);
if (Destdir != NULL) {
char *pkgdbdir;
@@ -198,7 +213,7 @@
TAILQ_INSERT_TAIL(&pkgs, lpp, lp_link);
}
- error += pkg_perform(&pkgs);
+ error += pkg_perform_vcs(&pkgs, &vcs_config);
if (error != 0) {
warnx("%d package addition%s failed", error, error == 1 ? "" : "s");
exit(1);
Index: pkgtools/pkg_install/files/add/perform.c
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/add/perform.c,v
retrieving revision 1.110
diff -u -r1.110 perform.c
--- pkgtools/pkg_install/files/add/perform.c 26 Feb 2018 23:45:01 -0000 1.110
+++ pkgtools/pkg_install/files/add/perform.c 17 May 2018 22:38:33 -0000
@@ -126,8 +126,10 @@
{ 0, NULL, 0, 0 },
};
+static int pkg_do_vcs(const char *, int, int, vcs_params_t *);
static int pkg_do(const char *, int, int);
+
static int
end_of_version(const char *opsys, const char *version_end)
{
@@ -963,11 +965,27 @@
/*
* Run the install script.
*/
-static int
-run_install_script(struct pkg_task *pkg, const char *argument)
-{
+
+static int run_install_script_vcs(struct pkg_task *pkg, const char *argument, vcs_params_t *vcs_config) {
int ret;
+ int len;
char *filename;
+ char **fused_params;
+ if (vcs_config->pass_to_INSTALL == TRUE) {
+ fused_params = malloc((2 + vcs_config->varnum)*sizeof(char *));
+ fused_params[0] = xstrdup(pkg->pkgname);
+ fused_params[1] = xstrdup(argument);
+
+ for (int i = 2; i < vcs_config->varnum + 2; i++) {
+ len = 2;
+ len+= strlen(vcs_config->vcs_params[i-2].name);
+ if (vcs_config->vcs_params[i-2].var != NULL && *(vcs_config->vcs_params[i-2].var) != NULL) {
+ len+= strlen(*(vcs_config->vcs_params[i-2].var));
+ }
+ fused_params[i] = malloc(sizeof(char)*len);
+ fused_params[i] = fuse_vcs_config(vcs_config->vcs_params + i-2);
+ }
+ }
if (pkg->meta_data.meta_install == NULL || NoInstall)
return 0;
@@ -987,19 +1005,44 @@
ret = 0;
errno = 0;
- if (fcexec(pkg->install_logdir, filename, pkg->pkgname, argument,
- (void *)NULL)) {
- if (errno != 0)
- warn("exec of install script failed");
- else
- warnx("install script returned error status");
- ret = -1;
+ if (vcs_config->pass_to_INSTALL == TRUE) {
+
+
+ if (pfcexec(pkg->install_logdir, filename, fused_params)) {
+ if (errno != 0)
+ warn("exec of install script failed");
+ else
+ warnx("install script returned error status");
+ ret = -1;
+ }
+ for (int i = 0; i < 2 + vcs_config->varnum; i++)
+ free(fused_params[i]);
+ free(fused_params);
+ }
+ else {
+ if (fcexec(pkg->install_logdir, filename, pkg->pkgname, argument,
+ (void *)NULL)) {
+ if (errno != 0)
+ warn("exec of install script failed");
+ else
+ warnx("install script returned error status");
+ ret = -1;
+ }
}
free(filename);
-
return ret;
}
+static int
+run_install_script(struct pkg_task *pkg, const char *argument)
+{
+ vcs_params_t dummy_vcs_config;
+ dummy_vcs_config.pass_to_INSTALL = FALSE;
+ dummy_vcs_config.version = 1;
+ dummy_vcs_config.varnum = 0;
+ return run_install_script_vcs(pkg, argument, &dummy_vcs_config);
+}
+
struct find_conflict_data {
const char *pkg;
const char *old_pkg;
@@ -1370,8 +1413,7 @@
/*
* Install a single package.
*/
-static int
-pkg_do(const char *pkgpath, int mark_automatic, int top_level)
+static int pkg_do_vcs(const char *pkgpath, int mark_automatic, int top_level, vcs_params_t *vcs_config)
{
char *archive_name;
int status, invalid_sig;
@@ -1460,7 +1502,7 @@
goto clean_memory;
if (check_other_installed(pkg))
- goto clean_memory;
+ goto clean_memory;
if (check_explicit_conflict(pkg))
goto clean_memory;
@@ -1494,13 +1536,17 @@
* Normal installation.
* Install/update dependencies first and
* write the current package to disk afterwards.
- */
+ */
if (check_dependencies(pkg))
goto clean_memory;
if (write_meta_data(pkg))
goto nuke_pkgdb;
}
+ if (vcs_config->pass_to_INSTALL == TRUE) {
+ run_install_script_vcs(pkg, "VCS-PRE", vcs_config);
+
+ }
if (run_install_script(pkg, "PRE-INSTALL"))
goto nuke_pkgdb;
@@ -1510,6 +1556,10 @@
if (run_install_script(pkg, "POST-INSTALL"))
goto nuke_pkgdb;
+ if (vcs_config->pass_to_INSTALL == TRUE) {
+ run_install_script_vcs(pkg, "VCS-POST", vcs_config);
+
+ }
/* XXX keep +INSTALL_INFO for updates? */
/* XXX keep +PRESERVE for updates? */
@@ -1569,8 +1619,39 @@
clean_find_archive:
free(pkg);
return status;
+
+
+}
+
+static int
+pkg_do(const char *pkgpath, int mark_automatic, int top_level)
+{
+ vcs_params_t dummy_vcs_params;
+ dummy_vcs_params.pass_to_INSTALL = FALSE;
+ dummy_vcs_params.varnum = 0;
+ dummy_vcs_params.version = 1;
+
+ return pkg_do_vcs(pkgpath, mark_automatic, top_level, &dummy_vcs_params);
+}
+
+
+int
+pkg_perform_vcs(lpkg_head_t *pkgs, vcs_params_t *vcs_config)
+{
+ int errors = 0;
+ lpkg_t *lpp;
+
+ while ((lpp = TAILQ_FIRST(pkgs)) != NULL) {
+ if (pkg_do_vcs(lpp->lp_name, Automatic, 1, vcs_config))
+ ++errors;
+ TAILQ_REMOVE(pkgs, lpp, lp_link);
+ free_lpkg(lpp);
+ }
+
+ return errors;
}
+
int
pkg_perform(lpkg_head_t *pkgs)
{
@@ -1586,3 +1667,4 @@
return errors;
}
+
Index: pkgtools/pkg_install/files/add/pkg_add.1
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/add/pkg_add.1,v
retrieving revision 1.48
diff -u -r1.48 pkg_add.1
--- pkgtools/pkg_install/files/add/pkg_add.1 21 Mar 2018 17:32:44 -0000 1.48
+++ pkgtools/pkg_install/files/add/pkg_add.1 17 May 2018 22:38:33 -0000
@@ -137,6 +137,8 @@
Display help and exit.
.It Fl I
If an installation script exists for a given package, do not execute it.
+.It Fl k
+Do not pass VCS options to the INSTALL script
.It Fl K Ar pkg_dbdir
Override the value of the
.Dv PKG_DBDIR
Index: pkgtools/pkg_install/files/lib/lib.h
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/lib/lib.h,v
retrieving revision 1.69
diff -u -r1.69 lib.h
--- pkgtools/pkg_install/files/lib/lib.h 26 Feb 2018 23:45:02 -0000 1.69
+++ pkgtools/pkg_install/files/lib/lib.h 17 May 2018 22:38:33 -0000
@@ -200,6 +200,18 @@
plist_t *tail; /* tail of list */
} package_t;
+typedef struct {
+ //enums would be better
+ const char *name;
+ const char **var;
+} vcs_config_variable_t;
+
+typedef struct {
+ int varnum;
+ int version;
+ Boolean pass_to_INSTALL;
+ vcs_config_variable_t *vcs_params;
+} vcs_params_t;
#define SYMLINK_HEADER "Symlink:"
#define CHECKSUM_HEADER "MD5:"
@@ -378,8 +390,10 @@
/* Parse configuration file */
void pkg_install_config(void);
+void pkg_install_vcs_config(vcs_params_t *);
/* Print configuration variable */
void pkg_install_show_variable(const char *);
+char * fuse_vcs_config(vcs_config_variable_t *);
/* Package signature creation and validation */
int pkg_verify_signature(const char *, struct archive **, struct archive_entry **, char **);
@@ -440,6 +454,7 @@
extern const char *gpg_keyring_sign;
extern const char *gpg_keyring_verify;
extern const char *gpg_sign_as;
+extern const char *pkg_useconfvcs;
extern char fetch_flags[];
extern const char *pkg_vulnerabilities_dir;
Index: pkgtools/pkg_install/files/lib/parse-config.c
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/lib/parse-config.c,v
retrieving revision 1.15
diff -u -r1.15 parse-config.c
--- pkgtools/pkg_install/files/lib/parse-config.c 16 Jun 2010 23:02:49 -0000 1.15
+++ pkgtools/pkg_install/files/lib/parse-config.c 17 May 2018 22:38:33 -0000
@@ -85,6 +85,23 @@
const char *ignore_advisories = NULL;
const char tnf_vulnerability_base[] = "http://ftp.NetBSD.org/pub/NetBSD/packages/vulns";;
const char *acceptable_licenses = NULL;
+//gsoc start
+const char *pkg_useconfvcs = "yes";
+const char *pkg_vcsworkdir;
+const char *vcs_type;
+const char *vcs_isremote = "no";
+const char *vcs_netprotocol;
+const char *vcs_uri;
+const char *vcs_authvar = "PKG_VCS_AUTHSTRING";
+const char *vcs_branch = "pkgauto";
+const char *vcs_installedbranch = "etclocal";
+const char *vcs_deployconf = "no";
+const char *vcs_deployrole = "any";
+const char *vcs_deployhostname;
+const char *vcs_allowinteraction = "no";
+
+
+
static struct config_variable {
const char *name;
@@ -116,6 +133,19 @@
{ "PKGVULNURL", &pkg_vulnerabilities_url },
{ "VERBOSE_NETIO", &verbose_netio },
{ "VERIFIED_INSTALLATION", &verified_installation },
+ { "PKG_USECONFVCS", &pkg_useconfvcs },
+ { "PKG_VCSWORKDIR", &pkg_vcsworkdir },
+ { "VCS_TYPE", &vcs_type },
+ { "VCS_ISREMOTE", &vcs_isremote },
+ { "VCS_NETPROTO", &vcs_netprotocol },
+ { "VCS_URI", &vcs_uri },
+ { "VCS_AUTHVAR", &vcs_authvar },
+ { "VCS_BRANCH", &vcs_branch },
+ { "VCS_INSTALLEDBRANCH", &vcs_installedbranch },
+ { "VCS_DEPLOYCONF", &vcs_deployconf },
+ { "VCS_DEPLOYROLE", &vcs_deployrole },
+ { "VCS_DEPLOYHOSTNAME", &vcs_deployhostname },
+ { "VCS_ALLOWINTERACTION", &vcs_allowinteraction },
{ NULL, NULL }, /* For use by pkg_install_show_variable */
{ NULL, NULL }
};
@@ -175,6 +205,7 @@
int do_cache_index;
char *value;
+
parse_pkg_install_conf();
if ((value = getenv("PKG_DBDIR")) != NULL)
@@ -183,6 +214,7 @@
pkgdb_set_dir(config_pkg_dbdir, 1);
config_pkg_dbdir = xstrdup(pkgdb_get_dir());
+
if ((value = getenv("PKG_REFCOUNT_DBDIR")) != NULL)
config_pkg_refcount_dbdir = value;
else if (config_pkg_refcount_dbdir == NULL)
@@ -248,6 +280,52 @@
(active_ftp && *active_ftp) ? "a" : "",
(ignore_proxy && *ignore_proxy) ? "d" : "");
}
+void
+pkg_install_vcs_config(vcs_params_t *vcs_config)
+{
+ char *value;
+ if (strcasecmp(pkg_useconfvcs, "yes"))
+ {
+ //set up working dir
+ if (((value = getenv("PKG_VCSWORKDIR")) == NULL) && (pkg_vcsworkdir == NULL)) {
+ pkg_vcsworkdir = xstrdup("FROM_PREFIX");
+ //form pkgetc under PREFIX inside the INSTALL script
+ }
+ }
+
+ if ((value = getenv(vcs_authvar))!= NULL)
+ vcs_uri = xasprintf("%s%s@%s", vcs_netprotocol, value, vcs_uri);
+ vcs_config->version = 1;
+ if (vcs_config->pass_to_INSTALL == TRUE) {
+ vcs_config->varnum = 11;
+ vcs_config->vcs_params = malloc(vcs_config->varnum * sizeof(vcs_config_variable_t));
+
+ vcs_config->vcs_params[0].name = "PKG_VCSWORKDIR";
+ vcs_config->vcs_params[0].var = &pkg_vcsworkdir;
+
+ vcs_config->vcs_params[1].name = "VCS_TYPE";
+ vcs_config->vcs_params[1].var = &vcs_type;
+ vcs_config->vcs_params[2].name = "VCS_ISREMOTE";
+ vcs_config->vcs_params[2].var = &vcs_isremote;
+ vcs_config->vcs_params[3].name = "VCS_NETPROTO";
+ vcs_config->vcs_params[3].var = &vcs_netprotocol;
+ vcs_config->vcs_params[4].name = "VCS_URI";
+ vcs_config->vcs_params[4].var = &vcs_uri;
+ vcs_config->vcs_params[5].name = "VCS_BRANCH";
+ vcs_config->vcs_params[5].var = &vcs_branch;
+ vcs_config->vcs_params[6].name = "VCS_INSTALLEDBRANCH";
+ vcs_config->vcs_params[6].var = &vcs_installedbranch;
+ vcs_config->vcs_params[7].name = "VCS_DEPLOYCONF";
+ vcs_config->vcs_params[7].var = &vcs_deployconf;
+ vcs_config->vcs_params[8].name = "VCS_DEPLOYROLE";
+ vcs_config->vcs_params[8].var = &vcs_deployrole;
+ vcs_config->vcs_params[9].name = "VCS_DEPLOYHOSTNAME";
+ vcs_config->vcs_params[9].var = &vcs_deployhostname;
+ vcs_config->vcs_params[10].name = "VCS_ALLOWINTERACTION";
+ vcs_config->vcs_params[10].var = &vcs_allowinteraction;
+ }
+
+}
void
pkg_install_show_variable(const char *var_name)
@@ -269,3 +347,9 @@
if (*var->var != NULL)
puts(*var->var);
}
+char * fuse_vcs_config(vcs_config_variable_t *param) {
+ //generate a string with the parameter name and value separated by the UnitSeparator ascii char, 0x1F
+ if (param->var != NULL)
+ return xasprintf("%s\x1F%s", param->name, *(param->var));
+ return xasprintf("%s\x1FUNSET",param->name);
+}
Index: pkgtools/pkg_install/files/lib/pkg_install.conf.5.in
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pkg_install/files/lib/pkg_install.conf.5.in,v
retrieving revision 1.21
diff -u -r1.21 pkg_install.conf.5.in
--- pkgtools/pkg_install/files/lib/pkg_install.conf.5.in 5 Nov 2014 14:40:01 -0000 1.21
+++ pkgtools/pkg_install/files/lib/pkg_install.conf.5.in 17 May 2018 22:38:33 -0000
@@ -205,6 +205,36 @@
.It Dv interactive
The user is always asked interactively when installing a package.
.El
+.It Dv PKG_USECONFVCS
+Defaults to yes, set to no in order to disable the handling of package provided configuration files in a version control system.
+.It Dv PKG_VCSWORKDIR
+The directory where to copy configuration files and examples before committing them to the versioning system. If the VCS is local, this is also the local repository.
+.It Dv VCS_TYPE
+The version control system to use, defaults to CVS. If a non default solution is set, take care of initializing the repository before installing packages!
+Supported values are: CVS, GIT, HG, SVN, FOSSIL...
+.It Dv VCS_ISREMOTE
+Should pkgsrc connect to a remote repository? Possible values are "yes" or "no", by default a local repository is used.
+.It Dv VCS_NETPROTO
+String specifying the network protocol to use if you are connecting to a remote repository, in the format required by the version control system in use.
+
+This is the prepended to authentication credentials, if specified.
+.It Dv VCS_AUTHVAR
+This variables specifies the name of the ENVIRONMENT VARIABLE storing the user:password combination used when connecting to a remote VCS repository.
+By default
+.Pa PKG_VCS_AUTHSTRING
+is read from the shell environment.
+.It Dv VCS_BRANCH
+Change the branch name PKGSRC should automatically commit provided example configuration files to.
+.It Dv VCS_INSTALLEDBRANCH
+Change the branch name used by a tool yet to be written to store configuration files installed in the system with local modifications.
+.It Dv VCS_DEPLOYCONF
+PKGSRC can, if a remote repository is set, work the opposite way and try to pull from the repository configuration files for the package being installed or upgraded, matching the system role.
+Can be set to either "yes" or "no", defaults to "no".
+.It Dv VCS_DEPLOYROLE
+String stating which is the role of this installation in the network, used to pull adequate, working configuration.
+.It Dv VCS_DEPLOYHOSTNAME
+After the role is selected, a best match on the system hostname is attempted, in case installation-specific configuration exists.
+Set this option to use a hostname different from the one set system wide.
.El
.Sh FILES
.Bl -tag -width ".Pa @SYSCONFDIR@/pkg_install.conf"
Home |
Main Index |
Thread Index |
Old Index