I spent a lot of time messing with all of this, that's why I deceided to share my notes.
My aim was to collect sFlow samples and to agregate them (sfacct), store them in a TSDB (influxDB) and graph network traffic (grafana).
There are several way to make some crypto :
- To create encrypted image jails, use the -c switch and either pass bde or eli and follow the instructions on screen:
ezjail-admin create -c eli -i 16G example.com 10.0.0.3
Also note, that crypto jails would block the boot process (unless the passphrase is provided via a file or some fetch magic via stdin). So they are being marked as attachblocking and not started during boot time. You need to start them using ezjail-admin startcrypto.
Further reading : http://erdgeist.org/arts/software/ezjail/
- you can use gbde or geli to encrypt partition : https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/disks-encrypt...
- You can also mount an image file (vnode) that you have prepared to be an encrypted filesystem.
Creation of the image :
dd if=/dev/zero of=/path/to/your/image.file bs=1M count=xxx mdconfig -at vnode -f /path/to/your/image.file geli init /dev/md0 geli attach /dev/md0 newfs /dev/md0.eli
Attach/mount the encrypted image :
mdconfig -at vnode -f /path/to/your/image.file geli attach /dev/md0 mount -o noatime /dev/md0.eli /dir/to/mount/under
Further reading : http://erdgeist.org/arts/software/ezjail/
pkg install openbgpd
touch /usr/local/etc/bgpd.conf
chmod 0600 /usr/local/etc/bgpd.conf
echo 'bgpd_flags=""' >> /etc/rc.conf
edit your config :
vi /usr/local/etc/bgpd.conf
service bgpd start
Some useful commands :
List the sessions and states :
bgpctl show summary
Detailed informations on your neighbors:
bgpctl show neighbor
Your configured/announced net :
bgpctl show network
List of routes for a prefix:
bgpctl show rib <network> all
WARNING : This is probably a bad idea to use OpenVPN in a Jail (you will lose *a lot* of time to configure it) but it work's.
I really recommand to read the sources a the bottom of this articles, they were great help.
First things, prepare the host :
ifconfig tun create
echo 'cloned_interfaces="tun0"' >> /etc/rc.conf
Create you Jail (eg named openvpn) with ezjail (cf my previous posts) and change some settings in /usr/local/etc/ezjail/openvpn (you MUST use ifconfig, *_ip_multi0 is NOT supported) :
#export jail_services_devfs_ruleset="devfsrules_jail"
export jail_services_devfs_ruleset="5"
export jail_services_exec_prestart0="/sbin/ifconfig tun0 inet 192.168.40.1/32 192.168.40.2"
export jail_services_exec_prestart1="/sbin/route add -net 192.168.40.0/24 192.168.40.2"
This is what you should put in /etc/devfs.rules in order to allow openVPN to bind on tun interface :
# Rules for jail <VPN> [devfsrules_jail_openvpn=5] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path tun0 unhide
If you have a Warning like this in /var/log/messages :
WARNING: devfs_init_rulesets: could not read rules from /etc/devfs.rules
You sould try to copy the default file, and then clean it and copy only your few lines again (yes, it worked for me...)
cp /etc/defaults/devfs.rules /etc/devfs.rules
echo '' > /etc/devfs.rules
Now you can start the Jail and install OpenVPN in the jail
ezjail-admin start openvpn
ezjail-admin console openvpn
pkg install openvpn
Here is the config I made for the OpenVPN server in the jail /usr/local/etc/openvpn/openvpn.conf (the 2 last lines are really important):
port 1194
proto udp
dev tun
ca /etc/ssl/CA/cacert.pem
cert /etc/ssl/CA/certs/srv.crt
key /etc/ssl/CA/certs/srv.key
crl-verify /etc/ssl/CA/crls/crl.pem
dh /etc/ssl/CA/certs/dh2048.pem
server 192.168.40.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
cipher BF-CBC
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3
mute 20
push "redirect-gateway def1"
push "dhcp-option DNS 208.67.222.222"
# needed in jail
ifconfig-noexec
route-noexec
I have followed this french page in order to setup SSL and OpenVPN : http://blog.nicolargo.com/2009/04/installation-dun-serveur-vpn-sous-free...
Enable OpenVPN at boot in the jail (no, you dont need to enable, neither set, the gateway in the jail) :
echo 'openvpn_enabled="YES"' >> /etc/rc.conf
From the host, you can now restart the jail, check that the tun interface of YOUR host (NOT THE JAIL) has the IP, verify OpenVPN is listening :
ezjail-admin restart openvpn
ifconfig tun0
sockstat -4
BTW, did you remember to redirrct UDP port 1194 to the jail, like I explained in my previons ticket? If yes, did you thought to enable NAT for VPN IP? change your /etc/rc.conf and reload conf (with pfctl -vf /etc/pf.conf) :
rdr on em0 proto udp to port 1194 -> 10.11.12.1 port 1194
nat on em0 from lo1:network -> (em0)
nat on em0 from 192.168.40.0/24 -> (em0)
If you have some troubles getting an IP on the tun interface, you may (but should not need) try to :
sysrc devfs_load_rulesets=YES ; service devfs restart ; ezjail-admin restart
echo 'devfs_load_rulesets="YES"' >> /etc/rc.conf
Sources : https://forums.freebsd.org/threads/openvpn-server-in-jail-using-a-tun-de...
https://github.com/junovitch/my-freebsd-build/blob/master/openvpn-jail-H...
https://forums.freebsd.org/threads/freebsd-update-9-2-10-ezjail-config.4...
We will use ezjail to make it easier :
pkg install ezjail
ezjail-admin install
service ezjail start
Create your first jail and connect to it (we will NAT the jail in this example):
ezjail-admin create myjailname 'lo1|10.11.12.1'
ezjail-admin start myjailname
ezjail-admin list
ezjail-admin console myjailname
Allow raw sockets (needed for ping, but not recommanded) by editing /usr/local/etc/ezjail/myjailname:
export jail_myjailname_parameters="allow.raw_sockets=1"
Enable features on skeleton of jails :
echo sshd_enable="YES" >> /usr/jails/newjail/etc/rc.conf
Enable NAT and port redirection in /etc/pf.conf:
ext_if = "em0" int_if = "lo1" rdr on $ext_if proto udp to port 1194 -> 10.11.12.1 port 1194 nat on $ext_if from $int_if:network -> ($ext_if)
Start PF and Check/Load PF config :
service pf onestart
pfctl -vf /etc/pf.conf
pfctl -e
Enable at startup in /etc/rc.conf :
gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
ezjail_enable="YES"
more about PF : https://www.freebsd.org/doc/handbook/firewalls-pf.html
more about ezjail : https://www.freebsd.org/doc/handbook/jails-ezjail.html
If you choose to Install FreeBSD 10 on Proxmox, you MUST set CPU to QEMU64 or it won't boot (yes, it will be ok for install, but not for boot!)
If you choosed to install ssh and ntp, check they are enabled in /etc/rc.conf :
sshd_enable="YES" ntpd_enable="YES"
remember to modify theses lines in /etc/ntp.conf to avoid beeing a bot in a DDoS amplification attack :
#restrict default kod nomodify notrap nopeer noquery
#restrict -6 default kod nomodify notrap nopeer noquery
#
# Alternatively, the following rules would block all unauthorized access.
restrict default ignore
restrict -6 default ignore
and "service ntpd restart"
PermitRootLogin yes
PasswordAuthentication no
and "service sshd restart"
freebsd-update fetch freebsd-update install
pkg search tcpdump
pkg install tcpdump
pkg info
pkg info tcpdump
pkg update
pkg version
pkg upgrade
(more about packages install and ports : https://www.freebsd.org/doc/handbook/ports-finding-applications.html )
gpart show
pkg install freecolor
freecolor -o -m
To use the BSD ports, first fetch and extract the list :
portsnap fetch portsnap extract portsnap fetch update
Find a port (if you know the name) :
#whereis nmap nmap: /usr/ports/security/nmap
If you dont know the exact name, you can search :
# cd /usr/ports # make search name=lsof
Compile & install (if the file pkg-message exists, you should read it!) :
cd /usr/ports/security/nmap make install clean
or simply install a port with portmaster :
pkg install portmaster portmaster -d sysutils/password-store
update all your ports :
portmaster -a
To add another default gateway to a dedicated interface, in this example I want 2 interfaces and one default GW for each IP (my 2nd interface has IP 10.10.10.10)
ip route add default via 10.10.10.1 table 100 ip rule add from 10.10.10.10 table 100 ip rule list ip route list table local ip route list table 100
source & more : http://lartc.org/howto/lartc.rpdb.html