================================================================= Linux* Open-iSCSI ================================================================= January 20, 2005 Contents ======== - 1. In This Release - 2. Introduction - 3. Installation - 4. Open-iSCSI daemon - 5. Open-iSCSI Configuration Utility - 6. Configuration - 7. Getting Started - 8. TBD - Appendix A. SendTargets snapshot. 1. In This Release ================== This file describes the Linux* Open-iSCSI Initiator. The software was tested on AMD Opteron (TM) and Intel Xeon (TM). The latest development release is available at: http://www.open-iscsi.org For questions, comments, contributions send e-mail to: open-iscsi@googlegroups.com 1.1. Features - highly optimized and very small-footprint data path; - multiple outstanding R2Ts; - persistent configuration database; - SendTargets discovery; - CHAP; - PDU header Digest; - multiple sessions; - multi-connection sessions. For the most recent list of features please refer to: http://www.open-iscsi.org/cgi-bin/wiki.pl/Roadmap 2. Introduction =============== Open-iSCSI project is a high-performance, transport independent, multi-platform implementation of RFC3720 iSCSI. Open-iSCSI is partitioned into user and kernel parts. The kernel portion of Open-iSCSI is a from-scratch code licensed under GPL. The kernel part implements iSCSI data path (that is, iSCSI Read and iSCSI Write), and consists of three loadable modules: scsi_transport_iscsi.ko, libiscsi.ko and iscsi_tcp.ko. User space contains the entire control plane: configuration manager, iSCSI Discovery, Login and Logout processing, connection-level error processing, Nop-In and Nop-Out handling, and (in the future:) Text processing, iSNS, SLP, Radius, etc. The user space Open-iSCSI consists of a daemon process called iscsid, and a management utility iscsiadm. 3. Installation =============== As of today, the Open-iSCSI Initiator requires a host running the Linux operating system with kernel version 2.6.16, or later. See http://www.open-iscsi.org/cgi-bin/wiki.pl/Supported_Kernels for a more information about which kernels Open-iSCSI supports. You need to enable "Cryptographic API" under "Cryptographic options" in the kernel config. You also need to enable "CRC32c CRC algorithm" if you use header or data digests. They are the kernel options, CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C, respectively. By default the kernel source found at /lib/modules/`uname -a`/build will be used to compile the open-iscsi modules. To specify a different kernel to build against use: make KSRC= or cross-compilation: make KSRC= KARCH="ARCH=um" To compile on SUSE Linux you'll have to use make KSRC=/usr/src/linux \ KBUILD_OUTPUT=/usr/src/linux-obj// where is the kernel configuration to use (eg. 'smp'). If you choose to install the Debian packages instead of building from source, please read the file /usr/share/doc/linux-iscsi/README.debian for information on how to build kernel modules against your specific kernel. For Red Hat/Fedora and Debian distributions open-iscsi can be installed by typing "make install". This will copy iscsid and iscsiadm to /usr/sbin, the init script to /etc/init.d, and the kernel modules: iscsi_tcp.ko, libiscsi.ko and scsi_transport_iscsi to /lib/modules/`uname -r`/kernel/drivers/scsi/ overwriting existing iscsi modules. 4. Open-iSCSI daemon ==================== The daemon implements control path of iSCSI protocol, plus some management facilities. For example, the daemon could be configured to automatically re-start discovery at startup, based on the contents of persistent iSCSI database (see next section). For help, run: ./iscsid --help Usage: iscsid [OPTION] -c, --config=[path] Execute in the config file (/etc/iscsid.conf). -f, --foreground run iscsid in the foreground -d, --debug debuglevel print debugging information -u, --uid=uid run as uid, default is current user -g, --gid=gid run as gid, default is current user group -h, --help display this help and exit -v, --version display version and exit 5. Open-iSCSI Configuration Utility =================================== Open-iSCSI persistent configuration is implemented as a DBM database available on all Linux installations. The database contains two tables: - Discovery table (discovery.db); - Node table (node.db). The regular place for iSCSI database files: /var/db/iscsi/*.db The iscsiadm utility is a command-line tool to manage (update, delete, insert, query) the persistent database. The utility presents set of operations that a user can perform on iSCSI nodes, sessions, connections, and discovery records. Note that some of the iSCSI Node and iSCSI Discovery operations do not require iSCSI daemon (iscsid) loaded. For help, run: ./iscsiadm --help Usage: iscsiadm [OPTION] -m, --mode specify operational mode op = -m discovery --type=[type] --portal=[ip:port] --login perform [type] discovery for target portal with ip-address [ip] and port [port]. Initiate Login for each discovered target if --login is specified -m discovery display all discovery records from internal persistent discovery database -m discovery --portal=[ip:port] --login perform discovery based on portal in database -m discovery --portal=[ip:port] --op=[op] [--name=[name] --value=[value]] perform specific DB operation [op] for specific discovery portal. It could be one of: [new], [delete], [update] or [show]. In case of [update], you have to provide [name] and [value] you wish to update -m node display all discovered nodes from internal persistent discovery database -m node --targetname=[name] --portal=[ip:port] [--login|--logout] -m node --targetname=[name] --portal=[ip:port] --op=[op] [--name=[name] \ --value=[value]] perform specific DB operation [op] for specific portal on target. It could be one of: [new], [delete], [update] or [show]. In case of [update], you have to provide [name] and [value] you wish to update -m session display all active sessions and connections -m session --sid=[sid] [--logout] perform operation for specific session with session id sid or display statistics if no operation specified. -d, --debug debuglevel print debugging information -V, --version display version and exit -h, --help display this help and exit Usage examples (using the one-letter options): 1) SendTargets iSCSI Discovery: ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 2) iSCSI Login: ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 -l 3) iSCSI Logout: ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 -u 4) Changing iSCSI parameter: ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \ -o update -n node.cnx[0].iscsi.MaxRecvDataSegmentLength -v 65536 5) Adding custom iSCSI Node: ./iscsiadm -m node -o new -p 192.168.0.1:3260 6) Removing iSCSI Node: ./iscsiadm -m node -o delete -T iqn.2005-03.com.max -p 192.168.0.4:3260 7) Display iSCSI Node configuration: ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 or ./iscsiadm -m node -o show -T iqn.2005-03.com.max -p 192.168.0.4:3260 8) Show all node records: ./iscsiadm -m node 9) Show all records in discovery database: ./iscsiadm -m discovery 10) Display discovery record setting: ./iscsiadm -m discovery -p 192.168.0.4:3260 11) Display session statistics: ./iscsiadm -m session -r 1 --stats 6. Configuration ================ The default configuration file is /etc/iscsid.conf. This file contains only configuration that could be overwritten by iSCSI Discovery, or manualy updated via iscsiadm utility. Its OK if this file does not exist in which case compiled-in default configuration will take place for newer discovered Target nodes. See the man page and the example file for the current syntax. The manpages for iscsid, iscsiadm are in the doc subdirectory and can be installed in the appropriate man page directories and need to be manually copied into e.g. /usr/local/share/man8. 7. Getting Started ================== There are three steps needed to set up a system to use iscsi storage: 1. automate iscsi startup using the init script. 2. discover targets. 3. automate target logins for future system reboots. 1. automate iscsi startup using the init script ----------------------------------------------- Red Hat or Fedora: ----------------- To start open-iscsi in Red Hat/Fedora you can do: service open-iscsi start To get open-iscsi to automatically start at run time you may have to run: chkconfig --level open-iscsi on Where are the run levels. And, to automatically mount a file system during startup you must have the partition entry in /etc/fstab marked with the "_netdev" option. For example this would mount a iscsi disk sdb: /dev/sdb /mnt/iscsi ext3 _netdev 0 0 SUSE or Debian: --------------- Otherwise, if there is a initd script for your distro in etc/initd that gets installed with "make install" /etc/init.d/open-iscsi start will usually get you started. Other: ------ If there is no initd script, you must start the tools by hand. First load the iscsi modules with: modprobe -q iscsi_tcp after that start iSCSI daemon process: ./iscsid or alternatively, start it with debug enabled and with output redirected to the current console: ./iscsid -d8 -f & and use configuration utility to add/remove/update Discovery records, iSCSI Node records or monitor active iSCSI sessions (see above or the iscsiadm man files). ./iscsiadm To login: ./iscsiadm -m node -T targetname -p ip:port -l where targetname is the name of the target amd ip_address:port is the address and port of the portal of a discovered or manually added iSCSI Target Node (for iscsiadm usage examples see previous sections). 2. discover targets ------------------- Once iscsi is up, you can perform discovery to targets using: iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 While discovery targets are kept in the discovery.db, they are usefull only for re-discovery. The discovered targets (a.k.a. nodes) are stored as records in the xxx.db. The discovered targets are not logged into yet. Rather than logging into the discovered nodes (making LUs from those nodes available as storage), it is better to automate the login to the nodes we need. 3. automate target logins for future system reboots --------------------------------------------------- Note: this may only work for Red Hat, Fedora and SUSE configurations To automate login to a node, use the following with the record ID of the node discovered in the discovery above: iscsiadm -m node -T targetname -p ip:port --op update -n node.conn[0].startup -v automatic Or to set the "node.conn[0].statup" attribute to "startup" as default for all sessions add the following to the /etc/iscsid.conf: node.conn[0].startup = automatic To login to all the automated nodes, simply restart the iscsi service e.g /etc/init.d/open-iscsi restart 8. TBD ====== To be completed: - Kernel tracing and Troubleshooting - Immediate and not-so-immediate plans - Useful scripts - White paper on Open-iSCSI design Appendix A. SendTargets iSCSI Discovery session snapshot. ========================================================= -bash-2.05b# ./iscsiadm -m discovery -tst -p 10.16.16.223:3260 [02f611] 10.16.16.223:3260,1 iqn.2002-07.com.ttechnologies.target.a [01acd1] 17.1.1.223:3260,1 iqn.2002-07.com.ttechnologies.target.a -bash-2.05b# -bash-2.05b# ./iscsiadm -m node [02f611] 10.16.16.223:3260,1 iqn.2002-07.com.ttechnologies.target.a [01acd1] 17.1.1.223:3260,1 iqn.2002-07.com.ttechnologies.target.a -bash-2.05b# -bash-2.05b# ./iscsiadm -m discovery -tst -p 10.16.16.227:3260 [02fb91] 10.16.16.227:3260,1 iqn.2001-04.com.example:storage.disk2.sys1.xyz -bash-2.05b# -bash-2.05b# ./iscsiadm -m node [02f611] 10.16.16.223:3260,1 iqn.2002-07.com.ttechnologies.target.a [02fb91] 10.16.16.227:3260,1 iqn.2001-04.com.example:storage.disk2.sys1.xyz [01acd1] 17.1.1.223:3260,1 iqn.2002-07.com.ttechnologies.target.a