IBM provided a system with 256 physical devices, commected via fiber to a switch. The switch was then connected to 4 fiber channel controllers ( QLA2300 ) , which caused the operating system to discover and configure 1024 disk devices. The devices were partitioned into 5 partitions, which allowed > 1024 disk partitions to be mounted and accessed.
In this configuration >4096 disk partitions were available to test. Linux would allows us to mount all these partitions, but the results of such mount operations would be indeterminate, due to the overlay. Disktest was used to test IO to the raw partitions, read tests were run to 4096 disks.
For file system operations, the four partitions on each drive we mkfs's and mounted. Per Badari, maximum IO for this configuration was ~3 GBit/sec. Most of the filesystem IO operations caused substantial IO wait. The iozone and reaim tests were used to drive small amounts of IO to each partition. No significant problems were exposed by the testing, the system was not configured for very large io. With all disks divided into the maximum 15 partitions ( one extended partition was not mountable ) 3584 disk partitions were created and mounted. ( 3587 total devices mounted )
Scripts available here were created to count, configure and run some IO operations against the disks. ( See Tools/Tarballs ).
Other
SuSE's hwscan program takes a very long time to finish ( 10-15 minutes ) and uses up a bunch of CPU. Kernel complains about depreciated SCSI ioctl in use.
hotplug agent programs ( scsi.agent, etc ) also produce a big load spike at inital boot, but run quite quickly
No resource bottlenecks found other than the above long startup time.
Was able to saturate the fiber with 100-150 spindles of disk, depending on the test.
I am uncertain as to the ability of the measurement tools - iostat did not report correctly.
setup
We used several very simple approaches to create the disk layout. Most commonly, we created a flat text file for input to a simple script.