Computational resources
The computational resources accessible by our lab for its researches come from five different sources:
1. Jülich Supercomputing Centre (JSC)
The JSC at the Forschungszentrum Jülich provides us with a yearly allocation of computational time on its two largest computational facilities:
JUGENE - An IBM Blue Gene/P machine which is currently the second most powerful supercomputer in Europe and the first in Germany (see for example the Top500 website):
• 72 Racks with 32 nodecards x 32 compute nodes (total 73728)• Compute node: 4-way SMP processor
• Processor type: 32-bit PowerPC 450 core 850 MHz
• Processors: 294912
• Overall peak performance: 1 Petaflops
• Linpack: 825.5 Teraflops
• Main RAM memory: 2 Gbytes per node (aggregate 144 TB)
• I/O Nodes: 600
• Networks:
- Three-dimensional torus (compute nodes)
- Global tree / Collective network (compute nodes, I/O nodes)
- 10 Gigabit ethernet / Functional network (I/O Nodes)
• Power Consumption: max. 35 kW per rack
• 2 Service nodes IBM p6 550:
• Total number of processors: 8
• Processor type: Power6 4.2 GHz
• Total amount of RAM memory: 128 GB
• Operating System: SuSE Linux Enterprise (SLES 10)
• 2 Login nodes IBM p6 550:
• Total number of processors: 8
• Processor type: Power6 4.2 GHz
• Total amount of RAM memory: 32 GB
• Operating System: SuSE Linux Enterprise (SLES 10)
• Internet address: jugene.fz-juelich.de
Here a brief overview of the system and a first introductory guide for the users
JUROPA - A Sun Constellation, NovaScale R422-E2 which is currently the second most powerful supercomputer in Germany (see for example the Top500 website):
• 2208 compute nodes• Compute node: 2 Intel Xeon X5570 (Nehalem-EP) quad-core processors
- 2.93 GHz
- SMT (Simultaneous Multithreading)
• Total cores: 17664
• Overall peak performance: 207 Teraflops
• Linpack: 183.5 Teraflops
• Main RAM memory: 24 GB memory per node (DDR3, 166 MHz)
• Networks: IB QDR with non-blocking Fat Tree topology
- Infiniband QDR with non-blocking Fat Tree topology
- Sun Data Center Switch 648
- 10 Gigabit ethernet / Functional network (I/O Nodes)
• 2 Management nodes Bull NovaScale R423-E2:
• Total number of processors: 16
• Processor type: Nehalem-EP quad-core 2.93 GHz
• Total amount of RAM memory: 48 GB DDR3
• Operating System: SuSE Linux Enterprise (SLES 11)
• 6 GPFS Gateway nodes Bull NovaScale R423-E2:
• Total number of processors: 48
• Processor type: Nehalem-EP quad-core 2.93 GHz
• Total amount of RAM memory: 144 GB DDR3
• Operating System: SuSE Linux Enterprise (SLES 11)
• 12 Login nodes Bull NovaScale R423-E2:
• Total number of processors: 96
• Processor type: Nehalem-EP quad-core 2.93 GHz
• Total amount of RAM memory: 288 GB DDR3
• Operating System: SuSE Linux Enterprise (SLES 11)
• Internet address: jugene.fz-juelich.de
Here a brief overview of the system and a first introductory guide for the users
2. Local GRS clusters
German Research School offers to its students and researchers the exclusive use of two local clusters:
iff105 - An Intel Nehalem based cluster:
• 36 compute nodes• Compute node: 2 Intel Xeon X5570 (Nehalem) quad-core processors
- 2.93 GHz
- SMT (Simultaneous Multithreading)
• Total cores: 288
• Main RAM memory: 24 GB memory per node (DDR3, 166 MHz)
• Networks: IB QDR 40 Gbps
- Infiniband QDR 40 Gbps
- 1 Gigabit ethernet / Functional network (I/O Nodes)
• 1 Storage node
• Total number of processors: 4
• Processor type: Xeon E5504 (Nehalem) quad-core 2.00 GHz
• Total amount of RAM memory: 48 GB DDR3 (166 MHz)
• Total disk space: 14 TB
• Bus interface: Serial ATA 300 MB/s
• Networks:
- Infiniband DDR 20 Gbps
- 1 Gigabit ethernet / Functional network (I/O Nodes)
• Operating System: CentOS 5.3 (Final)
Here you can find some useful scripts for the cluster (and also for other machines)
grsjuc - An Intel Nehalem based cluster:
• 16 compute nodes• Compute node: 2 Intel Xeon X5570 (Nehalem) quad-core processors
- 2.93 GHz
- SMT (Simultaneous Multithreading)
• Total cores: 128
• Main RAM memory: 24 GB memory per node (DDR3, 166 MHz)
• Networks: IB QDR 40 Gbps
- Infiniband QDR 40 Gbps
- 1 Gigabit ethernet / Functional network (I/O Nodes)
• 1 Storage node
• Total number of processors: 4
• Processor type: Xeon E5504 (Nehalem) quad-core 2.00 GHz
• Total amount of RAM memory: 48 GB DDR3 (166 MHz)
• Total disk space: 18 TB
• Bus interface: Serial ATA 300 MB/s
• Networks:
- Infiniband DDR 20 Gbps
- 1 Gigabit ethernet / Functional network (I/O Nodes)
• Operating System: CentOS 5.3 (Final)
Here you can find some useful scripts for the cluster (and also for other machines)
Computational biophysics group at GRS also has its own workstation for 3-D visualization tasks:
• CPU: Intel Xeon E5550 (Nehalem) quad-core processors- 2.66 GHz
- SMT (Simultaneous Multithreading)
• Total amount of RAM memory: 12 GB DDR3 (166 MHz)
• Hard Disk: 500 GB 7200 rpm (SATA-2)
• Graphics card: NVIDIA Quadro FX48100 1.5GB
• Infrared emitter and eyewear: NVIDIA GeForce 3D Vision
• Video output:
- Projector: ACER H5360 DLP 720P
- Monitor: 24" 120Hz LCD (1920 x 1080)
3. External resources
Members of GRS frequently apply also for external computational resources on other "large scale facilities" on the other European computational centers (for example, the TGCC in Paris or the HLRS in Stuttgard). Among many successful grants obtained for cpu time, we mention here some of the most recent:
1 - The very recent PRACE first call that granted us 48.117.842 core-hours for 2011
2 - The PRACE early access call that granted us 40.468.480 core-hours for the first months of 2011
3 - Two projects that have been granted access to the DEISA supercomputing infrastructure, benefiting of the resource allocation of:
• 600.000 standardized cpu-hours for 2010• 400.000 standardized cpu-hours for 2009
4. Center for Computing and Communication of the RWTH Aachen University
Since GRS is a joint graduate school of RWTH Aachen University and Forschungszentrum Jülich, any employee and student of GRS can access the HPC RWTH Compute Cluster (RZ-cluster) of RWTH Aachen University. Moreover, as member of JARA consortium we have exclusive access options on that machine. RWTH Compute Cluster consists of several different systems managed with help of the Batchsystem Sun Grid Engine (SGE). The user is able to submit a job in a transparent way. GridEngine will carry the job to process as soon as possible to one of the machines by regarding special needs like main store, CPU-time, etc.
Recently, the main cluster has been updated with a new Bull HPC cluster which is currently the forth most powerful supercomputer in Germany (see for example the Top500 website):
• 1358 Westmere EP nodes• Compute node: 2 Intel Xeon X5675 six-core processors
- 3.06 GHz
- SMT (Simultaneous Multithreading)
• Total cores: 16296
• Main RAM memory: 24-96 GB memory per node (DDR3)
• Networks: IB QDR 40 Gbps
- Infiniband QDR 40 Gbps
- 1 Gigabit ethernet / Functional network (I/O Nodes)
• 326 Nehalem EX nodes
• Compute node: 4 Intel Xeon X7550 eight-core processors
- 2.00 GHz
- SMT (Simultaneous Multithreading)
• Total cores: 10432
• Main RAM memory: 64-256 GB memory per node (DDR3)
• Networks: IB QDR 40 Gbps
- Infiniband QDR 40 Gbps
- 1 Gigabit ethernet / Functional network (I/O Nodes)
• Storage
• 1.5 PByte parallel high perfomance file system designed for high throughput
• 1.5 PByte NFS file system
Here the RWTH Compute Cluster User's Guide which collects the most important information for the efficient use of the central computing facilities, as well as references to further information.
5. JARA-HPC Partition
The JARA-HPC partition consists of contingents of the high-performance computers and supercomputers installed at RWTH Aachen University (RWTH Compute Cluster described in the previous point) and Forschungszentrum Jülich (JUQUEEN). The partition will be successively established in the course of 2012; by the end of 2012 it will have reached its total computing power of about 600 TFlops.
-
RWTH Compute Cluster
The RWTH Compute Cluster currently comprises approximately 2,500 computers with different configurations; 1,500 of these belong to the new Bull Cluster described above. At present, all computing nodes are equipped with processors of the x86_64 architecture and run on the operating systems Linux and Windows. All in all, a third of the cluster with a computing power of 100 TFlops will be made available for users of the JARA-HPC partition. -
JUQUEEN
The new FZJ's supercomputer JUQUEEN is an IBM Blue Gene/Q system that will be installed and run at JSC in the course of 2012. In April 2012, the first two racks will be installed; four more will be added in June followed by another considerable extension in October. About a third of JUQUEEN's total computing power, approx. 500 TFlops, will be successively made available for the JARA-HPC partition in the course of 2012.
• >6 Racks with 32 nodecards x 32 compute nodes (total >6144)
• Compute node: 16 processor cores SMP processor for the execution of the user applications
1 processor core for the operating system
• Processor core type: 64-bit PowerPC A2 core 1.6 GHz
Each processor core can execute 4 processes/threads simultaneously (fourfold simultaneous
multithreading, SMT) and has a quad floating point unit (FPU), thus enabling the simultaneous
execution of 4 fused multiply-add (FMA) operations with real numbers in double precision.
• Processors cores: >104448
• Overall peak performance: 204.8 GFlops per processor
• Linpack: XXX Teraflops
• Main RAM memory: 16 Gbytes SDRAM-DDR3 per node (1333 MTps)
• I/O Nodes: 16-way SMP processor; configurable in 8, 16 or 32 I/O nodes per rack
• Networks:
- Five-dimensional torus 40 GBps; 2.5 μsec latency (worst case)
- Collective network — part of the 5D Torus; collective logic operations supported
- Global Barrier/Interrupt — part of 5D Torus, PCIe x8 Gen2 based I/O
- 1 GB Control Network — System Boot, Debug, Monitoring
• Power Consumption: max. 100 kW per rack
• Cooling: 90% water cooling (18-25 ºC)
10% air cooling