32-bit jail on 64-bit host

Discussion:

Peter Blok

2021-04-22 08:36:13 UTC

Hi,

I have created a 32-bit jail on a 64-bit running 12-STABLE. The jail is also build using the same source.

The jail gives me a 32-bit environment. I’m getting an IP address and I can ping others on the same network segment.

But I can’t set a default route.

route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument

# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS

# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Any idea how to fix this?

I’m using vnet bridge

Peter

Eugene Grosbein

2021-04-22 10:27:27 UTC

Permalink

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The jail is also build using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address and I can ping others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?

You will have to put in jail ABI-compatible userland utilities that talk with a kernel directly.
This means 64 bit versions of binaries like route, ipfw, maybe netstat etc.

You should not assume and use jail as virtual machine, it is not. It is a container for a set of processes
sharing same kernel with other jails. If you need full-blown virtual machine, use bhyve.

Daniel Dettlaff via freebsd-hackers

2021-04-22 10:32:56 UTC

Permalink

If you need to run 32bit software with 64bit base system just try creating 64bit jail with lib32 subsystem present. Then 32bit software should be able to run properly in such jail, but you can't run 32bit jail on 64bit base as Eugene said.

Eugene Grosbein

2021-04-22 11:06:21 UTC

Permalink

Post by Daniel Dettlaff via freebsd-hackers
If you need to run 32bit software with 64bit base system just try creating 64bit jail with lib32 subsystem present. Then 32bit software should be able to run properly in such jail, but you can't run 32bit jail on 64bit base as Eugene said.

Not exactly. It is definitely possible to run 32bit-only jail with 64 host (kernel)
if one does not try use it as distinct machine with its own set of interfaces, routing tables etc.
but with its own IP address(-es) assigned to one of host's interfaces (loopback or other)
and assigned to the jail in question. A jail is a containter managed by its host,
so use it appropriately, manage it at host, not inside a jail and you'll be OK.

Peter Blok

2021-04-22 11:13:35 UTC

Permalink

The goal is to use p5-DBD-Oracle which only works/compiles on i386 on a very light system not able to run bhyve.

I have tried with lib32 but it fails as well. I do not have a lot of time to debug this, so I was hoping I was able to make this work on an i386 jail.

I’ll change the jail.conf and manage it from the outside. Hopefully it works.

Post by Eugene Grosbein

Eugene Grosbein

2021-04-22 11:59:51 UTC

Permalink

Post by Peter Blok
The goal is to use p5-DBD-Oracle which only works/compiles on i386 on a very light system not able to run bhyve.
I have tried with lib32 but it fails as well. I do not have a lot of time to debug this, so I was hoping I was able to make this work on an i386 jail.
I’ll change the jail.conf and manage it from the outside. Hopefully it works.

You do not need vnet-enabled bridged jail for the task. Simple jail will do it just fine.

Ian Lepore

2021-04-22 13:39:40 UTC

Permalink

On Thu, 2021-04-22 at 12:32 +0200, Daniel Dettlaff via freebsd-hackers

Post by Daniel Dettlaff via freebsd-hackers
If you need to run 32bit software with 64bit base system just try
creating 64bit jail with lib32 subsystem present. Then 32bit software
should be able to run properly in such jail, but you can't run 32bit
jail on 64bit base as Eugene said.

That is not what Eugene said, and you CAN run a 32-bit jail on a 64-bit
host; I do so on this machine. As Eugene said, you simply need to copy
a few selected 64-bit binaries into the jail, replacing the 32-bit
version of those programs. That is, install the jail from a 32-bit
build or packages, and then just copy the necessary few binaries from
your host root filesystem into the jail.

It would be nice if there was a list somewhere of which binaries need
to be replaced. I just did it by trial and error... when I ran into
things that didn't work, I tried using a 64-bit copy of that program
and if it worked: problem solved.

-- Ian

Post by Daniel Dettlaff via freebsd-hackers

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The
jail is also build using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address
and I can ping others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?
I’m using vnet bridge
Peter

Peter Blok

2021-04-22 18:13:13 UTC

Permalink

I tried to replace some of the 32-bit binaries before, but it started to require shared libs as well, so I stopped. I’ll give it a shot later.

I now run it with out vnet and it indeed works. I have managed to compile p5-DBD-Oracle which works now.

Because all of my other jails were vnet jails, I didn’t think about doing it the old way without vnet.

Peter

Post by Ian Lepore
On Thu, 2021-04-22 at 12:32 +0200, Daniel Dettlaff via freebsd-hackers

Post by Daniel Dettlaff via freebsd-hackers
If you need to run 32bit software with 64bit base system just try
creating 64bit jail with lib32 subsystem present. Then 32bit software
should be able to run properly in such jail, but you can't run 32bit
jail on 64bit base as Eugene said.

That is not what Eugene said, and you CAN run a 32-bit jail on a 64-bit
host; I do so on this machine. As Eugene said, you simply need to copy
a few selected 64-bit binaries into the jail, replacing the 32-bit
version of those programs. That is, install the jail from a 32-bit
build or packages, and then just copy the necessary few binaries from
your host root filesystem into the jail.
It would be nice if there was a list somewhere of which binaries need
to be replaced. I just did it by trial and error... when I ran into
things that didn't work, I tried using a 64-bit copy of that program
and if it worked: problem solved.
-- Ian

Post by Daniel Dettlaff via freebsd-hackers

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The
jail is also build using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address
and I can ping others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?
I’m using vnet bridge
Peter

Chris

2021-04-22 19:27:05 UTC

Permalink

Post by Peter Blok
I tried to replace some of the 32-bit binaries before, but it started to
require
shared libs as well, so I stopped. I’ll give it a shot later.

I'm not sure which binaries you might need either. But could /rescue bring
you
any closer?

--Chris

Post by Peter Blok
I now run it with out vnet and it indeed works. I have managed to compile
p5-DBD-Oracle which works now.
Because all of my other jails were vnet jails, I didn’t think about doing it
the
old way without vnet.
Peter

Post by Ian Lepore
On Thu, 2021-04-22 at 12:32 +0200, Daniel Dettlaff via freebsd-hackers

Post by Daniel Dettlaff via freebsd-hackers
If you need to run 32bit software with 64bit base system just try
creating 64bit jail with lib32 subsystem present. Then 32bit software
should be able to run properly in such jail, but you can't run 32bit
jail on 64bit base as Eugene said.

That is not what Eugene said, and you CAN run a 32-bit jail on a 64-bit
host; I do so on this machine. As Eugene said, you simply need to copy
a few selected 64-bit binaries into the jail, replacing the 32-bit
version of those programs. That is, install the jail from a 32-bit
build or packages, and then just copy the necessary few binaries from
your host root filesystem into the jail.
It would be nice if there was a list somewhere of which binaries need
to be replaced. I just did it by trial and error... when I ran into
things that didn't work, I tried using a 64-bit copy of that program
and if it worked: problem solved.
-- Ian

Post by Daniel Dettlaff via freebsd-hackers

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The
jail is also build using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address
and I can ping others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?
I’m using vnet bridge
Peter

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

Peter Blok

2021-04-23 08:41:36 UTC

Permalink

Chris, that was the right hint.

I copied host /rescue/route to the jail in /amd64. Changed the dhclient-script to use /amd64/route and it worked.

Using dynamic linked version failed with /libexec/ld-elf.so.1.

Peter

Post by Peter Blok
I tried to replace some of the 32-bit binaries before, but it started to require
shared libs as well, so I stopped. I’ll give it a shot later.

I'm not sure which binaries you might need either. But could /rescue bring you
any closer?
--Chris

Post by Ian Lepore
On Thu, 2021-04-22 at 12:32 +0200, Daniel Dettlaff via freebsd-hackers

Post by Daniel Dettlaff via freebsd-hackers
If you need to run 32bit software with 64bit base system just try
creating 64bit jail with lib32 subsystem present. Then 32bit software
should be able to run properly in such jail, but you can't run 32bit
jail on 64bit base as Eugene said.

That is not what Eugene said, and you CAN run a 32-bit jail on a 64-bit
host; I do so on this machine. As Eugene said, you simply need to copy
a few selected 64-bit binaries into the jail, replacing the 32-bit
version of those programs. That is, install the jail from a 32-bit
build or packages, and then just copy the necessary few binaries from
your host root filesystem into the jail.
It would be nice if there was a list somewhere of which binaries need
to be replaced. I just did it by trial and error... when I ran into
things that didn't work, I tried using a 64-bit copy of that program
and if it worked: problem solved.
-- Ian

Post by Daniel Dettlaff via freebsd-hackers

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The
jail is also build using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address
and I can ping others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?
I’m using vnet bridge
Peter

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers <https://lists.freebsd.org/mailman/listinfo/freebsd-hackers>

Chris

2021-04-24 01:50:10 UTC

Permalink

Post by Peter Blok
Chris, that was the right hint.
I copied host /rescue/route to the jail in /amd64. Changed the
dhclient-script to
use /amd64/route and it worked.
Using dynamic linked version failed with /libexec/ld-elf.so.1.

Heh. Not surprising. :-)
If it were me. I'd probably put /rescue at the beginning of my $PATH. Go
a long way to eliminating potential stress. ;-)

--Chris

Post by Peter Blok
Peter

Post by Peter Blok
I tried to replace some of the 32-bit binaries before, but it started to require
shared libs as well, so I stopped. I’ll give it a shot later.

I'm not sure which binaries you might need either. But could /rescue bring you
any closer?
--Chris

Post by Ian Lepore
On Thu, 2021-04-22 at 12:32 +0200, Daniel Dettlaff via freebsd-hackers

Post by Daniel Dettlaff via freebsd-hackers
If you need to run 32bit software with 64bit base system just try
creating 64bit jail with lib32 subsystem present. Then 32bit software
should be able to run properly in such jail, but you can't run 32bit
jail on 64bit base as Eugene said.

That is not what Eugene said, and you CAN run a 32-bit jail on a 64-bit
host; I do so on this machine. As Eugene said, you simply need to copy
a few selected 64-bit binaries into the jail, replacing the 32-bit
version of those programs. That is, install the jail from a 32-bit
build or packages, and then just copy the necessary few binaries from
your host root filesystem into the jail.
It would be nice if there was a list somewhere of which binaries need
to be replaced. I just did it by trial and error... when I ran into
things that didn't work, I tried using a 64-bit copy of that program
and if it worked: problem solved.
-- Ian

Post by Daniel Dettlaff via freebsd-hackers

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The
jail is also build using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address
and I can ping others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?
I’m using vnet bridge
Peter

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
<https://lists.freebsd.org/mailman/listinfo/freebsd-hackers>

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

Chris

2021-04-22 15:12:37 UTC

Permalink

Post by Peter Blok
Hi,
I have created a 32-bit jail on a 64-bit running 12-STABLE. The jail is also
build
using the same source.
The jail gives me a 32-bit environment. I’m getting an IP address and I can
ping
others on the same network segment.
But I can’t set a default route.
route add default 192.168.1.1
route: writing to routing socket: Invalid argument
add net default: gateway 192.168.1.1 fib 0: Invalid argument
# netstat -rn
Routing tables
(0) (0) UH
(0) (0) U
(0) (0) UHS
(0) (0) UH
(0) (0) U
(0) (0) UHS
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_websip: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:88:d7:20:99:80
hwaddr 02:80:ad:6e:79:0b
inet 192.168.1.205 netmask 0xffffff00 broadcast 192.168.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Any idea how to fix this?
I’m using vnet bridge

I do it. I don't think vnet or a bridge is necessary or perhaps even
desirable
in this situation. All my 32/64bit jails access the hosts net out of
localhost
(127.0.0.2-N) and I use pf(4) to redirect the packets.
I have a static block of internet facing addresses. So change yours
accordingly
pf.conf(5)
EXT_ADDR="W.X.Y.Z"
...
set skip on { lo0, lo1 }
...
nat pass on re0 from { lo1 } to any -> $EXT_ADDR
rdr pass on re0 proto tcp from any to { lo1 } -> $EXT_ADDR
...
block in
pass out
...
I add an entry in the hosts hosts(5) file, and in the jails hosts(5) for
accounting purposes. The jails resolve.conf(5) file looks like this
nameserver 127.0.0.1
nameserver 127.0.0.2
options timeout:1 attempts:1 rotate

And all gets it done for me.

HTH

--Chris

Post by Peter Blok
Peter
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers

Dewayne Geraghty

2021-04-24 03:06:35 UTC

Permalink

Post by Chris
I do it. I don't think vnet or a bridge is necessary or perhaps even
desirable
in this situation. All my 32/64bit jails access the hosts net out of
localhost
(127.0.0.2-N) and I use pf(4) to redirect the packets.
I have a static block of internet facing addresses. So change yours
accordingly
pf.conf(5)
EXT_ADDR="W.X.Y.Z"
...
set skip on { lo0, lo1 }
...
nat pass on re0 from { lo1 } to any -> $EXT_ADDR
rdr pass on re0 proto tcp from any to { lo1 } -> $EXT_ADDR
...
block in
pass out
...
I add an entry in the hosts hosts(5) file, and in the jails hosts(5) for
accounting purposes. The jails resolve.conf(5) file looks like this
nameserver 127.0.0.1
nameserver 127.0.0.2
options timeout:1 attempts:1 rotate
And all gets it done for me.
HTH
--Chris

Peter

Peter, I use a similar setup to Chris, though with ipfw. ;)

Jails have a few subtleties. They inherit much of the network of the
base. So you only need to think about the IP's assigned to the jail and
their assignment order. However one particular gotcha

The jail will use the first IP address that's set in jail.conf
effectively becoming your default route for the jail. And I recall that
localhost will also latch onto that IP address, so if its internet
facing, you'll need to think about the implications.

I'm a little paranoid so I use:
- /etc/hosts to define localhost to be something other than the
default. Some applications/ports behave properly IF they use localhost
for their unix sockets, rather than 127.0.0.1. (ie test what you need
and become good friends with tcpdump)
- consider carefully your firewall rules not just internet facing but
also over lo0 :)

And to reiterate what many have said, running i386 and amd64 on an amd64
platform is fun, as there are less machines to maintain when you need,
as in our use-case, to test the operation of software for 32bit targets.
(Though we just perform a buildworld with TARGET_ARCH=i386
CPU_TYPE=PRESCOTT with the appropriate destination.)

I don't think your setup requires the complexity or additional
processing from bridging or vnets.