Facilities

This is the third cluster at CFU and the newest and largest cluster installed. It is designed for simulation and fast processing of the vast amount of data acquired for synthetic aperture imaging. Often, 16 Gbytes of data is acquired per 3 seconds of measurements.

This massive amount of data has to be processed and with the old cluster it may take weeks for just one data set. With the new cluster, it should be possible to do in one day. 

The cluster was fully installed with cooling in 2005.

Processors

The cluster consists of 50 Dell 1750 servers each housing two Intel Xeon 3.06 GHz processors for a total of 100 processors.

Storage

Each computer has 2 Gbytes of internal RAM, a total of 100 Gbytes of RAM in the full cluster. The computers also house a fast 10,000 RPM, 146 Gbytes SCSI disk with an access time of 4.7 ms and a peak transfer rate of 320 Mbytes/sec. The combined disk space is 7.3 Tbytes (7,300,000,000,000 bytes). This is equivalent to 1,550 DVD films.

The disk space can be accessed through NFS or via the Parallel Virtual File System (PVFS) developed at Clemson University and Argonne National Laboratory. The cluster is also connected to CFU's two RAID towers housing 3 Tbytes and 1 Tbytes. Thus, the combined storage at CFU is 10.3 Tbytes.

Network

Each computer is equipped with two 1 Gbit/second Ethernet connections that are switched through a HP Procure 2848 and a 2824 switch. Therefore, all computers can communicate with all other computers at a speed of 1 Gbit/second, and 50 Gbit/second can be routed through the system.

Performance

Each Xeon processor in the system has a theoretical peak performance of 6 Gflops, the theoretical maximum performance being 0.6 Tflops. The world's fastest computer, situated in Japan, has a peak performance of 35 Tflops and the size of a football field.  

A Matlab version 6.5 executed benchmark using Linpack gives a rating of 1.57 Gflops (1.570.000.000 double precision multiplications per second) per CPU, resulting in a cluster performance of 157 Gflops. This is 16.5 times faster than CFU's cluster 1.

The calculated rating compared to the Top 500 computers in the world can be performed using the NCSA Tungsten computer (the world's fifth fastest computer). This uses the same Dell server as Cluster 2 and gives a rating Rmax of 393 and an Rpeak of 612. No computer in Denmark is on the Top 500 list, but Cluster 2 is probably among the 5-10 fastest computers in Denmark.

The performance of the system can also be compared to the first computers used in the Ultrasound group: the Apollo DN3000 from 1987. It had a Linpack performance of 72 kflops. Hence, the new cluster is 2.2 million times faster than the DN3000. This means, that a program running for a full year on the DN3000 would take 14.6 seconds on the new cluster.

The parallel file system can use all disks in parallel. Theoretically, 5 Gbytes of data per second can be either stored or read by the cluster.

The CFU storage cluster is used for storage the massive amount of data from SARUS and for processing the data.

It consist of 12 Fujitsu-Siemens TX 200 machines each with two 6 core Intel Xenon processors and a very fast RAID system with 10 TBytes of data.

The total amount is 120 Tbytes and four 10 Gbit Ethernet links connects it to the SARUS scanner. Each disk system can store 400 Mbytes/sec, which makes it possible to store several Gbytes of data per second.

The processing is controlled by the batch system SLURM and the cluster can process 288 independent jobs at the same time. 

The RASMUS (Remotely Accessible Software configurable Multi-channel Ultrasound Sampling) system consists of four distinct modules: The transmitters, the analog amplifiers (Rx/Tx amplifiers), receivers, and the sync/master unit.

RASMUS seen from the front with the receivers on the top, transmitters in the middle and power supplies in the bottom. Two 19 inch racks house the eight transmitters and the eight receivers, respectively. Each of the racks also house a slot PC running Linux controlling the setup and operation of the boards. A separate enclosure is used for the analogue front-end, which is shielded from the digital electronics. Linear laboratory power supplies are also used to supply the front-end to keep the noise low.

Each transmitter board has 16 channels consisting of a 128 ksamples RAM connected to a 40 MHz, 12 bits DAC. The RAM is controlled by an FPGA (Field Programmable Gate Array), where the individual waveforms are selected as a memory start address and a transmit delay. The waveform as well as the delay can change for each emission ensuring full flexibility in the transmissions. The length of the waveforms can be set and waveform durations up to 100 microseconds can be emitted. The system houses eight boards for a total of 128 independent emission channels.

The receiver board layout without RAM modules to reveal the processing electronics. Board size is 53 cm by 36.5 cm. It samples and processes 8 analogue signals selected from 16 inputs through a 2-to-1 multiplexer. Each of the input signals is sampled at 40 MHz and 12 bits into one of the two SRAMs. One SRAM is used for ampling, the second for transferring data to the Focus FPGA associated with each channel. The data is then processed in the Focus FPGA using parameters from the 128 MBytes Focusing RAM, and the result is passed on to the Sum FPGA. The processing can either be a dynamic receive focusing or the data can be passed unaltered for later storage. The Sum FPGA can either store the data in the 2Gbytes storage RAM or it can sum all eight channels with the result from the cascade bus and pass it on to the next receiver board through the cascade bus. The last receiver board in the system then transmits the focused signal to the ADSP SHARC for transmission to the display PC.

The storage RAM can contain more than three seconds of real time data for each channel, which can later be accessed from the PC's controlling the system. All boards have a size of 53 by 36.5 cm, and the receiver is manufactured using 12 layers.

The experimental ultrasound system SARUS is a platform designed in house on which new ultrasound imaging methods can be implemented and evaluated.

It consists of 1024 independent transmit channels and 1024 independent receive channels, allowing it to fully utilize two-dimensional ultrasound transducers. The sampling is performed at 70 MHz and 12 bits, so the system can continuously sample data at a rate of 140 GB/s.

It houses more than 320 GB of RAM and communicates between boards on 64 1 GBit Ethernet links. Data can be processed in real time on the 320 large FPGAs each housing 160 18 bits multipliers running at 1/2 GHz for a combined output of 20,480 billion multiplications/s. Each FPGA can send more than 10 Gbit of data/s to provide real-time images for live preview of the scanned area, using synthetic aperture imaging technique to achieve very high image quality.

The change between preview mode and experimental imaging with saving of data is quick, so that clinical trials can be conveniently implemented. Data can also be stored for off-line processing through four 10 Gbit Ethernet links on the CFU storage cluster.

Up to 4 users can simultaneously conduct experiments using the SARUS hardware.

The Onda acoustic intensity measurement system is designed to measure ultrasound intensities from diagnostic and therapeutic ultrasound devices.

It includes a scanning tank with a positioning controller and external devices as oscilloscope, function generator, and hydrophones. The positioning system can control the step motor with the accuracy of 11.08 μm per step and six axises can be used to position the object.

With the software Soniq 5.0, we can control all the components in one interface. The signal from the hydrophone is sampled by the oscilloscope and stored by the attached computer.

The Onda system can obtain a range of information from the signal such as pressure, pulse intensity integral, average voltage, peak or negative voltage and the complete waveform.

The system can easily make 1D, 2D or 3D scans and the user can choose any plane of interest.

After the measurement, the system is able to create an automatic report with all measurement parameters and results. The original data can also be saved in an excel format to be used later, where the waveforms can be found at each point.

This system will ensure that measurements conducted at CFU follow FDA regulatory limits.