Discussion:
Using jail.conf array parameters in exec.* commands
(too old to reply)
Jeff Kletsky
2017-02-15 01:48:01 UTC
Permalink
Raw Message
TL;DR

Is there a patch available to allow substitution of "array" parameters
into the strings used for exec.prestart, exec.poststop, and the like?


Longer:

I'd like to be able to use array parameters in exec.* commands,
but trying to do so results in

jail: test-two: exec.prestart: array cannot be substituted inline

A quick check of the source didn't suggest it would work, but the
bash-isms ${host.interface[*]} and ${host.interface[@]} failed too.


The immediate application is in creating what will be the vnet.interface(s)
using netgraph on the appropriate host interfaces

vnet.interface = ng0, ng1;

Works well, once the interfaces are created, but I'd rather not have
to define that list twice in each jail.


If I could write something like:

vnet.interface = ng0, ng1;
host.interface = re0, re0.100;

and then iterate over them in the prestart and poststop scripts to:

* Create ng0 connected to re0
* Create ng1 connected to re0.100

it would be easier to maintain than having to write something like:

vnet.interface = ng0, ng1;
parent.interfaces = "re0 re0.100";
cloned.interfaces = "ng0 ng1";



I'm open to ideas here, including if I can somehow "write back" to
vnet.interface based on exec.prestart parsing the "custom" variables.


TIA,

Jeff
Ernie Luzar
2017-02-15 10:30:31 UTC
Permalink
Raw Message
Post by Jeff Kletsky
TL;DR
Is there a patch available to allow substitution of "array" parameters
into the strings used for exec.prestart, exec.poststop, and the like?
I'd like to be able to use array parameters in exec.* commands,
but trying to do so results in
jail: test-two: exec.prestart: array cannot be substituted inline
A quick check of the source didn't suggest it would work, but the
The immediate application is in creating what will be the vnet.interface(s)
using netgraph on the appropriate host interfaces
vnet.interface = ng0, ng1;
Works well, once the interfaces are created, but I'd rather not have
to define that list twice in each jail.
vnet.interface = ng0, ng1;
host.interface = re0, re0.100;
* Create ng0 connected to re0
* Create ng1 connected to re0.100
vnet.interface = ng0, ng1;
parent.interfaces = "re0 re0.100";
cloned.interfaces = "ng0 ng1";
I'm open to ideas here, including if I can somehow "write back" to
vnet.interface based on exec.prestart parsing the "custom" variables.
TIA,
Jeff
An alternate method to coding the jail.conf vnet.interface parameter is
to use the "ifconfig vnet" command to enable it and "ifconfig -vnet"
command to disable it in your netgraph script that starts and stops the
vnet jail. Doing so would eliminate your current desire for array
processing in the jail.conf definition all together.

I use the bridge/epair method my self because its so much easier to
understand. If you don't mind sharing, I sure would like to see your
netgraph script for vnet jail control once you get it working.
Jeff Kletsky
2017-02-17 17:58:48 UTC
Permalink
Raw Message
Thanks for the suggestion of trying to use 'ifconfig interface vnet jail'
in the scripts themselves.

I'll get my scripts up once I've got them running again confidently
and can get proper licensing on them.

TL;DR

* Is there a clean way to "catch" failures in jail(8) creation after
exec.prestart completes, such as vnet.interface failing?

* Is there a good way to execute commands in the host environment once
jail(8) brings up the jail, but before exec.start runs?



The rest:

I've been thinking about that for a while, especially as there isn't a
way to "catch" an execution error in jail(8) itself, such as the vnet
transition failing. (Yes, I'll open an issue on that once I'm convinced
I can't do it with the current jail functionality.)

To be able to call 'ifconfig interface vnet jail' the jail needs to
exist already:

# ifconfig ngeth3 vnet t2
ifconfig: jail "t2" not found

Further, the network needs to be up and running when services are
started. ntpd, anything that binds to a specific interface (rather
than *), anything that needs DNS (such as nginx providing proxy
services), ...


jail(8) tells me I have the following hooks available

exec.prestart -- jail isn't created yet
exec.start -- runs *in* the jail; typically starts execution
exec.poststart -- runs in the host, after exec.start completes

There isn't a "jail up, but not executing yet" hook in the host
environment that I am aware of.

There is a somewhat ugly approach along the lines of:

exec.prestart -- do the setup on the host side
exec.start -- '/bin/true' or 'return 0'-- don't do anything
exec.poststart -- 'ifconfig interface vnet jail'-like things
'jexec jail sh /etc/rc > ${exec.consolelog}'


Is there a better approach that someone out there knows of?


Thanks!

Jeff
Post by Ernie Luzar
Post by Jeff Kletsky
TL;DR
Is there a patch available to allow substitution of "array" parameters
into the strings used for exec.prestart, exec.poststop, and the like?
[...]
An alternate method to coding the jail.conf vnet.interface parameter
is to use the "ifconfig vnet" command to enable it and "ifconfig -vnet"
command to disable it in your netgraph script that starts and stops the
vnet jail. Doing so would eliminate your current desire for array
processing in the jail.conf definition all together.
Post by Ernie Luzar
I use the bridge/epair method my self because its so much easier to
understand. If you don't mind sharing, I sure would like to see your
netgraph script for vnet jail control once you get it working
Ernie Luzar
2017-02-17 23:01:24 UTC
Permalink
Raw Message
Post by Jeff Kletsky
Thanks for the suggestion of trying to use 'ifconfig interface vnet jail'
in the scripts themselves.
I'll get my scripts up once I've got them running again confidently
and can get proper licensing on them.
TL;DR
* Is there a clean way to "catch" failures in jail(8) creation after
exec.prestart completes, such as vnet.interface failing?
* Is there a good way to execute commands in the host environment once
jail(8) brings up the jail, but before exec.start runs?
I've been thinking about that for a while, especially as there isn't a
way to "catch" an execution error in jail(8) itself, such as the vnet
transition failing. (Yes, I'll open an issue on that once I'm convinced
I can't do it with the current jail functionality.)
To be able to call 'ifconfig interface vnet jail' the jail needs to
# ifconfig ngeth3 vnet t2
ifconfig: jail "t2" not found
Further, the network needs to be up and running when services are
started. ntpd, anything that binds to a specific interface (rather
than *), anything that needs DNS (such as nginx providing proxy
services), ...
jail(8) tells me I have the following hooks available
exec.prestart -- jail isn't created yet
exec.start -- runs *in* the jail; typically starts execution
exec.poststart -- runs in the host, after exec.start completes
There isn't a "jail up, but not executing yet" hook in the host
environment that I am aware of.
exec.prestart -- do the setup on the host side
exec.start -- '/bin/true' or 'return 0'-- don't do anything
exec.poststart -- 'ifconfig interface vnet jail'-like things
'jexec jail sh /etc/rc > ${exec.consolelog}'
Is there a better approach that someone out there knows of?
Thanks!
Jeff
Lets make this simple. Do not use the "service jail jailname start"
command to start / stop your jails. Your mixing legacy rc.conf jail
method with jail.conf method. All ways use the jail(8) command itself to
start/stop your jails.

If you do this in a script then you can check the jail resulting return
code to determine if the jail start/stop failed. But there is no
information to tell you why it failed. In all most all cases it's caused
by jail.conf parameters syntax coding error or invalid value content.
Really pretty simple to determine cause by looking at the jail.conf
content for the offending vnet jail.

Change your mind set from thinking you have to use the exec.* hooks to
configure the vnet jails netgraph network setup.

Just have individual jail.conf files for each vnet jail with no vnet
interface defined.

Now you can start the jail with just the standard exec.start line and
standard exec.stop line. Once your script has issued the jail(8) command
to start the jail then follow it with all the netgraph commands to
enable its network. The vnet jail it self has no knowledge of any
network connectivity at start up, you can wrap either bridge/epair or
netgraph around it and it don't care.

This was learned the hard way.
Jeff Kletsky
2017-02-17 23:57:10 UTC
Permalink
Raw Message
Thanks again for your thoughts on this.

I *am* using "pure" jail(8) and jail.conf(5) techniques and have been
for many years now. What I'd like to get to is a robust way to start
jails the way I have been

# jail -c some_jail

and just have it work reliably, especially when there is setup that
needs to be done on the host to enable the smooth running of the jail.

Preferably, *all* configuration of the jail and its connectivity is
done in jail.conf, be it /etc/jail.conf or a jail-specific one. I'm
trying to avoid going back to having to define another service that
wraps a call to jail(8) which would just start the jail with persist
set, then have to set up networking in the wrapper, jexec /etc/rc,
and then deal with shutdown of a jail created with persist set.

I agree that *usually* the reason a jail won't start is
misconfiguration. However, there are other conditions that can occur,
such as lack of resources. If you want to more gracefully respond to
this, the wrapper script would need either to parse the appropriate
jail.conf, or to have all the pertinent information available in
another form. Having individual jail.conf files for each jail at
least makes easier to parse, at the expense of not being able to
define global and regional jail properties that are inherited across
the appropriate jails.


The network has to be up and connected *before* /etc/rc runs,
especially where services in the jail need network interfaces present
to bind to specific addresses, to mount network file systems, or have
access to critical services, such as DNS (for example, nginx will fail
to start if it can't resolve proxy host names).

Since there is no jail vnet or jail ID (number) available, you can't
have jail(8) run needed operations in the jail.conf-declared
exec.prestart command, including, for example:

* ifconfig interface vnet jail
* ipfw add action proto from src to dst jail prisonID
* ipfw add lookup jail table_name

By the time jail(8) will run the jail.conf declared exec.poststart,
exec.start has already run to completion inside the jail.


How do you handle getting the network up *before* /etc/rc or the
specific service is started in the jail?


I unfortunately suspect you're right that I can't use the existing
jail(8) and jail.conf(5) approach without wrapping the whole thing in
a script. The hooks, even for networking, don't seem to be there.


Jeff
Post by Ernie Luzar
Lets make this simple. Do not use the "service jail jailname start"
command to start / stop your jails.
Your mixing legacy rc.conf jail method with jail.conf method. All
ways use the jail(8) command itself to start/stop your jails. If you
do this in a script then you can check the jail resulting return code
to determine if the jail start/stop failed. But there is no
information to tell you why it failed. In all most all cases it's
caused by jail.conf parameters syntax coding error or invalid value
content. Really pretty simple to determine cause by looking at the
jail.conf content for the offending vnet jail.
Change your mind set from thinking you have to use the exec.* hooks
to configure the vnet jails netgraph network setup.
Just have individual jail.conf files for each vnet jail with no vnet
interface defined.
Now you can start the jail with just the standard exec.start line and
standard exec.stop line. Once your script has issued the jail(8)
command to start the jail then follow it with all the netgraph
commands to enable its network. The vnet jail it self has no
knowledge of any network connectivity at start up, you can wrap
either bridge/epair or netgraph around it and it don't care.
This was learned the hard way.
Post by Jeff Kletsky
Thanks for the suggestion of trying to use 'ifconfig interface vnet jail'
in the scripts themselves.
I'll get my scripts up once I've got them running again confidently
and can get proper licensing on them.
TL;DR
* Is there a clean way to "catch" failures in jail(8) creation after
exec.prestart completes, such as vnet.interface failing?
* Is there a good way to execute commands in the host environment once
jail(8) brings up the jail, but before exec.start runs?
[...]
Ernie Luzar
2017-02-18 17:38:17 UTC
Permalink
Raw Message
On release 10.3 or maybe 10.2 there were Devin Teske's examples of
vnet/netgraph setups in these directories.
/usr/share/examples/netgraph
/usr/share/examples/jails

11.0 does not have them any more.

If I recall correctly, some of the netgraph commands need the jid
number, which means the vnet jail has to be started. But here "started"
does not have the normal meaning. A vnet jail can be created, jail -c,
and as long as there is no "exec.start sh /etc/rc" command in the
jail.conf definition there is nothing running in the jail. Now your
script has a jid to use and can setup the netghaph world that will allow
the vnet jail to commutate. After all that stuff is complete, a jexec
command can be issued to start services in the jail. "sh /etc/rc" is all
you need to fire up services in the vnet jail. Any userland application
like apache have to be installed the normal way and have its
apache24_enable="YES" statement id the vnet's jail rc.conf file.

Here's a general observation. During the 10.x releases I had a netgraph
vnet script that Devin Teske posted to questions or jail list that
worked. I think it was the same one that was in 10.x
/usr/share/examples/jails directory. But when I tried it on 11.0 it
stopped functioning giving a error on some netctl command. For me
netgraph is just to hard. That is why I use the bridge/epair method.
Post by Jeff Kletsky
Thanks again for your thoughts on this.
I *am* using "pure" jail(8) and jail.conf(5) techniques and have been
for many years now. What I'd like to get to is a robust way to start
jails the way I have been
# jail -c some_jail
and just have it work reliably, especially when there is setup that
needs to be done on the host to enable the smooth running of the jail.
Preferably, *all* configuration of the jail and its connectivity is
done in jail.conf, be it /etc/jail.conf or a jail-specific one. I'm
trying to avoid going back to having to define another service that
wraps a call to jail(8) which would just start the jail with persist
set, then have to set up networking in the wrapper, jexec /etc/rc,
and then deal with shutdown of a jail created with persist set.
I agree that *usually* the reason a jail won't start is
misconfiguration. However, there are other conditions that can occur,
such as lack of resources. If you want to more gracefully respond to
this, the wrapper script would need either to parse the appropriate
jail.conf, or to have all the pertinent information available in
another form. Having individual jail.conf files for each jail at
least makes easier to parse, at the expense of not being able to
define global and regional jail properties that are inherited across
the appropriate jails.
The network has to be up and connected *before* /etc/rc runs,
especially where services in the jail need network interfaces present
to bind to specific addresses, to mount network file systems, or have
access to critical services, such as DNS (for example, nginx will fail
to start if it can't resolve proxy host names).
Since there is no jail vnet or jail ID (number) available, you can't
have jail(8) run needed operations in the jail.conf-declared
* ifconfig interface vnet jail
* ipfw add action proto from src to dst jail prisonID
* ipfw add lookup jail table_name
By the time jail(8) will run the jail.conf declared exec.poststart,
exec.start has already run to completion inside the jail.
How do you handle getting the network up *before* /etc/rc or the
specific service is started in the jail?
I unfortunately suspect you're right that I can't use the existing
jail(8) and jail.conf(5) approach without wrapping the whole thing in
a script. The hooks, even for networking, don't seem to be there.
Jeff
Post by Ernie Luzar
Lets make this simple. Do not use the "service jail jailname start"
command to start / stop your jails.
Your mixing legacy rc.conf jail method with jail.conf method. All
ways use the jail(8) command itself to start/stop your jails. If you
do this in a script then you can check the jail resulting return code
to determine if the jail start/stop failed. But there is no
information to tell you why it failed. In all most all cases it's
caused by jail.conf parameters syntax coding error or invalid value
content. Really pretty simple to determine cause by looking at the
jail.conf content for the offending vnet jail.
Change your mind set from thinking you have to use the exec.* hooks
to configure the vnet jails netgraph network setup.
Just have individual jail.conf files for each vnet jail with no vnet
interface defined.
Now you can start the jail with just the standard exec.start line and
standard exec.stop line. Once your script has issued the jail(8)
command to start the jail then follow it with all the netgraph
commands to enable its network. The vnet jail it self has no
knowledge of any network connectivity at start up, you can wrap
either bridge/epair or netgraph around it and it don't care.
This was learned the hard way.
Post by Jeff Kletsky
Thanks for the suggestion of trying to use 'ifconfig interface vnet
jail'
Post by Ernie Luzar
Post by Jeff Kletsky
in the scripts themselves.
I'll get my scripts up once I've got them running again confidently
and can get proper licensing on them.
TL;DR
* Is there a clean way to "catch" failures in jail(8) creation after
exec.prestart completes, such as vnet.interface failing?
* Is there a good way to execute commands in the host environment once
jail(8) brings up the jail, but before exec.start runs?
[...]
James Gritton
2017-02-18 16:52:17 UTC
Permalink
Raw Message
Post by Jeff Kletsky
...
I've been thinking about that for a while, especially as there isn't a
way to "catch" an execution error in jail(8) itself, such as the vnet
transition failing. (Yes, I'll open an issue on that once I'm convinced
I can't do it with the current jail functionality.)
To be able to call 'ifconfig interface vnet jail' the jail needs to
# ifconfig ngeth3 vnet t2
ifconfig: jail "t2" not found
Further, the network needs to be up and running when services are
started. ntpd, anything that binds to a specific interface (rather
than *), anything that needs DNS (such as nginx providing proxy
services), ...
jail(8) tells me I have the following hooks available
exec.prestart -- jail isn't created yet
exec.start -- runs *in* the jail; typically starts execution
exec.poststart -- runs in the host, after exec.start completes
There isn't a "jail up, but not executing yet" hook in the host
environment that I am aware of.
exec.prestart -- do the setup on the host side
exec.start -- '/bin/true' or 'return 0'-- don't do anything
exec.poststart -- 'ifconfig interface vnet jail'-like things
'jexec jail sh /etc/rc > ${exec.consolelog}'
Is there a better approach that someone out there knows of?
There's nothing better at this point - the ugly solution you mention is
the current best way. The exec.* options come from analogs of the
rc-script days, which precede vnet. The specific "ifconfig interface
vnet jail" thing was handled by the vnet.interface parameter, but it
would be good to have a more general set of exec scripts to run in the
create side post-create but pre-start.

But I'm not sure such a thing will appear. Aside from the cumbersome
naming of something between prestart and start, I can see this blowing
up: there could well be a situation where you want something run in the
host, something in the jail, something else in the host, something else
in the jail. I considered vnet.interface to be the common case, but
there will always be more specific work where the best solution is to
just run a script on the host side.

- Jamie
Loading...