Per apspera ad adstra
Jump directly to

Setting up an iSCSI SAN Server


SAN (NAS?)

In most larger computing environments, storage is provided by specialized nodes in the network, often over a specialized network: the Storage Area Network, or SAN.
The term "SAN" is often confused with "NAS" or Network Attached Storage. Both offer a way to store and retrieve bytes in some centralized device, but there are some differences.
A NAS is a single device that offers (shared) filesystems to clients in the network. Such filesystems might be NFS or SMB ("Windows shares") filesystems, they are usually shared between lots of clients and are accessible over the 'normal' 100Mb or 1Gb TCP/IP network.
A SAN offers storage devices to clients over a specialized storage network. The network itself is usually much faster than the standard network using 1/10/../300 GBit fibrechannel connections. It tends to use redundant cards, routers and cabling and also tend to be a lot more expensive.
Where a NAS device needs to take care of locking individual files and making sure only authorized and authenticated clients can access and modify data, a SAN server only offers raw storage and lets the client worry about details as filesystem type, locking files and authorizations.

Just because SAN's are usually built on high-end specialized network gear, doesn't mean it can't run on a cheap, slow home network. In the end, serving up a virtual disk is just a matter of speaking the right protocol.

iSCSI

The protocol in this case is SCSI (Small Computer System Interface). When using a computer with local SCSI disks, an adapter sends SCSI commands to talk to these devices. In a networked situation, such as a SAN, the iSCSI protocol is used ("Internet SCSI").

See this wikipedia page for more information on iSCSI.

In short, iSCSI targets offer virtual storage ("server"), iSCSI initiators request and use this storage ("client"). Since the clients are usually servers it helps to use the terms "target" and "initiator".

In this article I set up an iSCSI target and an iSCSI initiator from scratch using Slackware Linux. The target side is implemented using LIO, the initiator side is standard Linux.


iSCSI Target - LIO - Linux-iSCSI.org

The kernel support for iSCSI target was adapted from work done by Rising Tide Systems (RTS) and included in the kernel starting in version 2.6.38. RTS offers the "targetcli" software to configure and use this functionality.
See linux-iscsi.org for a more correct description of the relationship between iSCSI, RTS and the kernel.

Targetcli is the open-sourced (aGPL) version of RTS's enterprise software. For out SAN-on-a-Budget, this version of targetcli is sufficient.

Installing targetcli is easy, once you know all the kernel options to set, what dependencies to install, what scripts to modify and what files to move where. Below is what I did to get it installed and working.

iSCSI Target - Kernel options

To enable the builtin kernel support for iSCSI targets, enable the following options:

  Device Drivers
        SCSI device support
                <M>  SCSI target support        
	<M>  Generic Target Core Mod (TCM) and ConfigFS Infrastructure
		<M>  TCM/IBLOCK Subsystem Plugin for Linux/BLOCK
		<M>  TCM/FILEIO Subsystem Plugin for Linux/VFS
                < >  TCM/pSCSI Subsystem Plugin for Linux/SCSI
		<M>  TCM/Virtual SAS target and Linux/SCSI LDD fabric
                               loopback module
		<M>  Linux-iSCSI.org iSCSI Target Mode Stack

This will generate kernel modules

  scsi_tgt
  target_core_iblock
  target_core_file
  tcm_loop
  iscsi_target_mod
  target_core_mod
Recompile and install your kernel and reboot.

Standard Python Packages

Targetcli is basically a set of Python scripts (and one or two executables). It uses a number of standard Python packages that need to be installed first. You can tell if these packages are installed by checking the

  /usr/lib/python2.7/site-packages

directory. There should be an .egg or .egg-info file for each of these packages:

ipaddr source ipaddr-2.1.10.tar.gz
netiface source netifaces-0.8.tar.gz
setuptools source setuptools-0.6c11.tar.gz
configobj source configobj-4.7.2.tar.gz
epydoc source epydoc-3.0.1.tar.gz
simpleparse source SimpleParse-2.1.1a2.tar.gz

Installing a Python package is done by running its setup.py script as root. For example:

$ tar xzf example-1.0.0.tar.gz
$ cd example-1.0.0
$ su - 
$ python setup.py install

Install any missing packages in this order:

ipaddr,
setuptools,
netifaces,
SimpleParse,
epydoc,
configobj

Targetcli Software

Download the targetcli software from RTS (local packages are snapshots from 14-5-2013):

targetcli source targetcli.tar.gz
rtslib source rtslib.tar.gz
configshell source configshell.tar.gz
lio-utils source lio-utils.tar.gz

Install the first two Python packages in this order:

  rtslib,
  configshell

rtslib: populate /var/target/fabric

Copy the 'fabricspecs' from the rtslib package to /var/target/fabric (if targetcli doesn't show "iscsi", you forgot this):

$ su
$ mkdir -p /var/target/fabric
$ cp rtslib/specs/iscsi.spec /var/target/fabric
$ cp rtslib/specs/loopback.spec /var/target/fabric

lio-utils: correct get-py-modules-path.py

The third file contains two more Python scripts that must be installed. One of its scripts tries to correct a configuration problem in Python that isn't present in Slackware. Correct the error:

$ tar xzf lio-utils.tar.gz
$ cd lio-utils
$ vi get-py-modules-path.py
Use vi to replace this text:
# Fix up get_python_lib() path for Python v2.6 on Ubuntu 9.10 and SLES 11
if re.search('python2.6', str):
      print str.replace('/usr','/usr/local')
else:
      print str;
with this text:
print str;

lio-utils: install /usr/sbin scripts lio and tcm

Now install the lio and tcm scripts in /usr/sbin:

$ su
$ cd lio-utils/tcm-py
$ python setup.py install
$ ./install.sh

$ cd ../..

$ cd lio-utils/lio-py
$ python setup.py install 
$ ./install.sh

lio-utils: install iscsi-name

Compile and install the lio-utils tools (iscsi-name)

$ cd lio-utils/tools
$ make
$ su
$ make install

lio-utils: /etc/rc.d/rc.target

Daemons are used to provide communication between the kernel and userland. The lio-utils/scripts/rc.target script can be used to automatically stop and start this daemon on startup and shutdown. Copy the file to /etc/rc.d:

$ su
$ cp lio-utils/scripts/rc.target /etc/rc.d/rc.target
$ chmod 750 /etc/rc.d/rc.target

Make sure the script gets called by adding it to the boot-sequence. I added it to /etc/rc.d/rc.local:

# Start LIO target
if [ -x /etc/rc.d/rc.target ] ; then
  /etc/rc.d/rc.target start
fi

lio-utils: populate /etc/target

Make sure the (empty for now) configuration files are available in the correct location:

$ su
$ mkdir -p /etc/target
$ cp lio-utils/conf/lio_start.default /etc/target/lio_start.sh
$ cp lio-utils/conf/tcm_start.default /etc/target/tcm_start.sh

Start daemon

Now that everything is in place, start the target daemon:

$ su
$ /etc/rc.d/rc.target start

Loading target_core_mod/ConfigFS core:   [OK]
Calling ConfigFS script /etc/target/tcm_start.sh for target_core_mod:   [OK]
Calling ConfigFS script /etc/target/lio_start.sh for iscsi_target_mod:   [OK]

targetcli

Now, install the targetcli package in the same way as the other packages:

$ tar xzf targetcli.tar.gz
$ su
$ cd targetcli
$ python setup.py install

Start targetcli to see the current (empty) configuration. When first started, targetcli will create directories in /sys/kernel/config/target. If they are not already loaded, the appropriate kernel modules are loaded and initialized.
$ su -
$ targetcli
targetcli GIT_VERSION (rtslib GIT_VERSION)
Copyright (c) 2011 by RisingTide Systems LLC.
All rights reserved.
Loaded tcm_loop kernel module.
Created '/sys/kernel/config/target/loopback'.
Loaded iscsi_target_mod kernel module.
Created '/sys/kernel/config/target/iscsi'.
/> ls
o- / ............................................................. [...]
  o- backstores .................................................. [...]
  | o- fileio ....................................... [0 Storage Object]
  | o- iblock ....................................... [0 Storage Object]
  | o- pscsi ........................................ [0 Storage Object]
  | o- rd_dr ........................................ [0 Storage Object]
  | o- rd_mcp ....................................... [0 Storage Object]
  o- iscsi .................................................. [0 Target]
  o- loopback ............................................... [0 Target]
/>  

Targetcli is now correctly installed.


iSCSI Target - Create 10MB iSCSI disk from local file

Using LIO, local storage can be served up as iSCSI device to clients. There are four forms of local storage that can be used:
  1. a file on the local filesystem (FILEIO)
  2. a local volume (disk, diskpartition or LVM LV)(BLOCKIO)
  3. an actual SCSI drive (pSCSI)
  4. a virtual disk that only exists in server memory (rd_mcp / rd_dr)
In practice only (2) will be used. This method offers maximum flexibility at near maximum performance.

Of course, nothing prevents you from configuring a server to use virtual iSCSI disks from other SAN servers and offering them as iSCSI disks yourself.

In this example, mainly because my laptop has no unused volumes to play with, I will create a iSCSI target based on a file as backing store (1).

In order, I will:

  • Create a backing store based on a file
  • Create an iSCSI target
  • Create a target portal group (TPG) for that target. A TPG defines a set of exported SCSI devices, what backing stores they are mapped to and the external IP address and port where they can be accessed. In this TPG authentication is turned off.
  • Assign the backing store as LUN in this SCSI target
  • Define the Access control list (ACL) for this target. We will allow access to initiator "iqn.2013-06.nl.san:1234567890"
  • Map the internal LUNs in this TPG to external LUNs. Initially, (external) LUN 0 is created and mapped to internal LUN 0. For fun, remap the external LUN to LUN 2.

$ su -
$ targetcli
targetcli GIT_VERSION (rtslib GIT_VERSION)
Copyright (c) 2011 by RisingTide Systems LLC.
All rights reserved.
/> /backstores/fileio create disk1 /lio/disk1.img 10M
Generating a wwn serial.
Not using buffered mode.
Created fileio disk1.
/> /iscsi create
Created target iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc.
Selected TPG Tag 1.
Successfully created TPG 1.
/> cd
  -- use arrow to move to tpgt1 --
/iscsi/iqn.20...tpgt1/> set parameter AuthMethod=None
Parameter AuthMethod is now 'None'.
/iscsi/iqn.20...tpgt1/> set attribute authentication=0
Parameter authentication is now '0'.
/iscsi/iqn.20...tpgt1/> portals/ create
Using default IP port 3260
Automatically selected IP address 10.0.0.142.
Successfully created network portal 10.0.0.142:3260.
/iscsi/iqn.20...cf13ebc/tpgt1> luns/ create /backstores/fileio0/disk1
Selected LUN 0.
Successfully created LUN 0.
/iscsi/iqn.20...cf13ebc/tpgt1> acls/ create iqn.2013-06.nl.san:1234567890
Successfully created Node ACL for iqn.2013-06.nl.san:1234567890
Created mapped LUN 0.
/iscsi/iqn.20...cf13ebc/tpgt1> acls/iqn.2013-06.nl.san:1234567890 create 2 0
Created Mapped LUN 2.
/iscsi/iqn.20...cf13ebc/tpgt1> acls/iqn.2013-06.nl.san:1234567890 delete 2
Delete Mapped LUN 0.

/iscsi/iqn.20...cf13ebc/tpgt1> cd /
/> ls
o- / ................................................................. [...]
  o- backstores ...................................................... [...]
  | o- fileio ........................................... [1 Storage Object]
  | | o- disk1 .................................. [/lio/disk1.img activated]
  | o- iblock ........................................... [0 Storage Object]
  | o- pscsi ............................................ [0 Storage Object]
  | o- rd_dr ............................................ [0 Storage Object]
  | o- rd_mcp ........................................... [0 Storage Object]
  o- iscsi ...................................................... [1 Target]
  | o- iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc ...... [1 TPG]
  |   o- tpgt1 ................................................... [enabled]
  |     o- acls .................................................... [1 ACL]
  |     | o- iqn.2013-06.nl.san:1234567890 ................. [1 Mapped LUNs]
  |     |   o- mapped_lun2 ..................................... [lun0 (rw)]
  |     o- luns .................................................... [1 LUN]
  |     | o- lun0 .......................... [fileio/disk1 (/lio/disk1.img)]
  |     o- portals .............................................. [1 Portal]
  |       o- 10.0.0.142:3260 .......................................... [OK]
  o- loopback ................................................... [0 Target]

We have now created an iSCSI device

  • of 10 MB,
  • which is mapped to the local file /lio/disk1.img,
  • accessible at 10.0.0.142:3260 with LUN = 2,
  • available for an initiator with id "iqn.2013-06.nl.san:1234567890".

Now save the configuration:

/> saveconfig
WARNING: Saving henk current configuration to disk will overwrite your 
    boot settings.
The current target configuration will become the default boot config.
Are you sure? Type 'yes': yes
Making backup of loopback/ConfigFS with timestamp: 
    2013-05-14_17:54:40.099138
Successfully updated default config /etc/target/loopback_start.sh
Making backup of LIO-Target/ConfigFS with timestamp: 
    2013-05-14_17:54:40.099138
Generated LIO-Target config: 
    /etc/target/backup/lio_backup-2013-05-14_17:54:40.099138.sh
Making backup of Target_Core_Mod/ConfigFS with timestamp: 
    2013-05-14_17:54:40.099138
Generated Target_Core_Mod config: 
    /etc/target/backup/tcm_backup-2013-05-14_17:54:40.099138.sh
Successfully updated default config /etc/target/lio_start.sh
Successfully updated default config /etc/target/tcm_start.sh
/> exit


More information on working with targetcli:
 deluiz.com
 Part 1: http://www.youtube.com/watch?v=BkBGTBadOO8
 Part 2: http://www.youtube.com/watch?v=mKjBsgOlYmE
 Part 3: http://www.youtube.com/watch?v=f6cgZotqUj0

iSCSI Initiator

Linux has kernel support for using iSCSI devices. As with iSCSI targets, the kernel needs to be configured through a userland application. I will be using the software from open-iscsi.org.

iSCSI Initiator - Kernel options

To enable the builtin kernel support for iSCSI initiators, enable the following options:
  Device Drivers
	SCSI device support
		<*>  SCSI low-level drivers
			<M>  iSCSI initiator over TCP/IP

This will generate kernel modules

  iscsi_tcp
  libiscsi
  scsi_transprt_iscsi
Compile and install the kernel and reboot.

iSCSI Initiator - Configuration software

The open-iscsi software can be downloaded from www.open-iscsi.org. You can either get the most recent git-version or the stable release. Download the software, compile and install:


Git version:

  $ git clone git://github.com/mikechristie/open-iscsi.git
  $ cd open-iscsi
  $ get checkout -b my_iscsi_branch origin/iscsi

Stable release:

  $ wget http://www.open-iscsi.org/bits/open-iscsi-2.0-873.tar.gz
  $ tar xzf open-iscsi-2.0-873.tar.gz
  $ cd open-iscsi

Compile:

  $ make 
  $ su
  $ make install

This creates and installs (among others)

  • /usr/iscsid - iSCSI daemon
  • /usr/iscsiadmin - management application
  • /usr/iscsistart - boot'tool'
Also, a number of configuration files are created at /etc/iscsi.

Two changes are needed to make sure the initiator is started at boot. One is to start the daemon in /etc/rc.local, the other is to change the default node.startup setting so nodes will log on automatically.

/etc/rc.d/rc.local

Add this line to connect to all registered iSCSI devices:

# Log in to all discovered iSCSI devices
/sbin/iscsiadm -m node -L all

/etc/iscsi/iscsid.conf

Apply this change (change node.startup from "manual" to "automatic"):
-#node.startup="automatic"
+node.startup="automatic"

-node.startup="manual"
+#node.startup="manual"

/etc/iscsi/initiatorname.iscsi

The contents of this file identifies the Initiator to the target. Change the contents to something meaningfull and unique within your storage network.
iqn.2013-06.nl.san:1234567890

TWA overload: this is the iqn of the acl in the tpg you created in your iSCSI target.


iSCSI Initiator: Initiate: use 10 MB iSCSI disk

The Initiator needs to initiate any connection between storage server and client. In the example storage server that was created when configuring the iSCSI target software, a small iSCSI disk was made available.
To connect to a drive on some Target, the Initiator must first discover if there is an iSCSI target (st="send target") offered on the Target's IP addres (10.0.0.142 in this case):
(the two commands do the same. Pick one):

$ iscsiadm --mode discovery --type st --portal 10.0.0.142
$ iscsiadm -m discovery -t st -p 10.0.0.142
10.0.0.142:3260,1 iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc

This confirms there is a iSCSI target at that address at default port 3260. One target portal group (1) is discovered, named "iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc". Now, connect to all LUNs in that portal group assigned to this initiator.
Use iscsiadm -m node -P <level> to show details on the current session.

$ iscsiadm -m node -L all
Logging in to [iface: default, 
               target: iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc, 
               portal: 10.0.0.142,3260] (multiple)
Login to [iface: default, 
          target: iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc,
          portal: 10.0.0.142,3260] successful.

$ iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.0-873
Target: iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc
        Current Portal: 10.0.0.142:3260,1
        Persistent Portal: 10.0.0.142:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2013-06.nl.san:1234567890
                Iface IPaddress: 10.0.0.142
                Iface HWaddress: 
                Iface Netdev: 
                SID: 8
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: 
                password: ********
                username_in: 
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 262144
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 13 State: running
                scsi13 Channel 00 Id 0 Lun: 2
                        Attached scsi disk sdb          State: running

Iscsiadm says an iSCSI device is now added as disk "sdb". Running 'dmesg' shows a new scsi device of 10 MB was detected and added as a disk:

$ dmesg
...
[33436.237835] Loading iSCSI transport class v2.0-870.
[33437.249979] iscsi: registered transport (tcp)
[33566.026146] scsi6 : iSCSI Initiator over TCP/IP
[33566.286841] scsi 6:0:0:0: Direct-Access   LIO-ORG  FILEIO    4.0  PQ: 0 ANSI: 5
[33566.286991] sd 6:0:0:0: Attached scsi generic sg2 type 0
[33566.295605] sd 6:0:0:0: [sdb] 20481 512-byte logical blocks: (10.4 MB/10.0 MiB)
[33566.296003] sd 6:0:0:0: [sdb] Write Protect is off
[33566.296006] sd 6:0:0:0: [sdb] Mode Sense: 43 00 00 08
[33566.296173] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled,
                                 doesn't support DPO or FUA
[33566.298337]  sdb: unknown partition table
[33566.299810] sd 6:0:0:0: [sdb] Attached SCSI disk
...

Disconnect the iSCSI devices again:

$ iscsiadm -m node -U all
Logging out of session [sid: 7,
                        target: iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc,
                        portal: 10.0.0.142,3260]
Logout of [sid: 7, 
           target: iqn.2003-01.org.linux-iscsi.henk.x8664:sn.52a0ccf13ebc,
           portal: 10.0.0.142,3260] successful.

$ iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.0-873
iscsiadm: No active sessions.