Regarding the use of the new computing system → 日本語

New Computing system miho (official name 「原子核物理学計算機」) officially started operation on 1st of September.

The operation of old computing system saho ended on 1st of September.

Operation of new user management system started on 28th of August.

• From this point on user management for saho and miho became separate.
• The old management system cannot be used anymore. You can access the new user management system from User management URL. ( SYSTEM A or SYSTEM B )
• Rewriting of the .htaccess on www-server has been completed.
• The operation of new user management system has started. We have yet to fix all known problems with the system. We are aware that there are bugs and that it might be bit hard understand. We think it's working well enough so we changed to it. For all of you who use it, we would like to hear your opinion. We would appreciate it if you could cooperate.

The beta release of the new computing system started from 31th of July.

The alpha release of the new computing system started from 17th of July.

Manual

• Current manual can be found here. The new changes won't be added quickly so please refer to this site.
• Material of the lecture held on 8-DEC-2016.

Details/Updates

• The newer updates are found at Japanese page written in English.
• 20151011 Benchmark results updated.
• cp command at ftp-1,2
  • We measured I/O speed vs File size dependency. PRELIMINARY. Detail (Excel)
  • The ftp-1,2 have 2 NIC of 10 Gbps each. One is for storage, another is for general network.
  • Total performance of ftp-1,2 is measured at least 8 Gbps by other measurement.
  • But single cp command performance is 1.7 Gbps at most. This means multiple cp is effective.
  • For maxumum performance, the file size should be greater than 1 GB.
  • For optimum performance, the file size should be greater than 10 MB.
• 20150928 how to use　texlive2015.
  • If you use Japanese text, set the encoding to UTF-8 in the source file. In the manual that can be found in wiki, there is a mistake which tells to use EUC.
  • Add /usr/local/texlive/2015/bin/x86_64-linux to your path.
  • compiler → platex
  • preview → pxdvi
  • dvi → PDF　dvipdfm, dvipdfmx, dvipdft
  • dvi → PS　dvips
  • We also intended to install the same tex as on saho, but we were unable to. Please use the texlive2015.
• 20150920　Benchmark results updated.
  • File I/O : We measured I/O speed vs Data size / File size dependency. PRELIMINARY. Detail (Excel)
• 20150911　The file transfer servers ftp-1 and ftp-2 can now be logged in from inside-network of the RCNP.
• 20150911　emacs is now re-made with --without-gsettings. Please inform us if trouble might be occurred by this setting.
• 20150911　git2.5.0 is installed at /usr/local/git in addition to git1.7.1 by RedHat.
• 20150911　python problem fixed. Can someone test this? pip pep8 pyflakes Sphinx virtualenv
• 20150911　CLHEP library is available at /usr/local/CLHEP on miho, aino and batch nodes.
• 20150906　Network speed.
  • We intended that we can transfer files between this system to others by 10 Gbps.
  • We don't intended that batch and miho can do high performance I/O.
  • But we will prepare high performance batch nodes a few.

  Type of computer	Serice	Storage	Comments
  miho-1, 2	1 Gbps	none
  login-1, 2	1 Gbps	none
  batch (normal)	1 Gbps	1 Gbps
  batch (high Speed)	1 Gbps	10 Gbps	in preparation
  ftp-1, 2	10 Gbps	10 Gbps	1 Gbps now
  aino-1, 2	10 Gbps	10 Gbps	1 Gbps now
  fs-1, 2	10 Gbps	10 Gbps

• 20150906　Benchmark results.
  • File I/O : We measured I/O speed vs Data size dependency. PRELIMINARY. Detail (Excel)
• 20150906　Caution for the protection codes.

  Please take a special care for the protection code for the "others". To permit for the "others" means that the super-com users that don't have RCNP account can access to the files.

  kind	normal user account	group account
  user	means yourself	means group account itself
  group	means all RCNP users (miho users)	means group account members
  others	in addtion to all RCNP users, include super computer user who don't have RCNP account	in addtion to all RCNP users, include super computer user who don't have RCNP account

• 20150902 There might be a problem with sphinx. Currently investigating.

The beta release of the new computing system will start from 31th(Friday) of July.

• All users can participate, for the purpose of having the system tested.
• Big problems should be solved before the operation starts officially on 1st of September. Even small problems must be reported. We hope for the co-operation of all of the users.
• Problem in file migration or
• important feature not working or
• same code gives different result on the new system than it does on the old system or
• feature important to you not working or
• the way of using the system is different from the old system (there are also parts that are intentionally changed) or
• things that should be improved or
• Other things that might be a concern
• We don't know how long it takes to complete a requests or implement a fix so please report any problems as soon as you notice them.
• ！！IMPORTANT！！　If you think we should change or know something, you have to let us know immediately.
• The man power that the contractor has working now during the construction is incomparable to the man power that will be available during normal operation. If we need to go and add/fix something later the time spent will be incomparable. Thing that would take now just one week might take multiple weeks or just not be possible at all afterwards. Probability of the "not be possible at all" will also be quite high.
• No matter if you think its important request, trivial matter, or thing you might think is unnecessary (in other words just let us know first, even if we don't implement it, it's not a problem) without exception please let us know.

Individual updates/information

• 20150831　When having trouble or when asking for consultation regarding the batch system, please provide the information listed below, to your best extent.
• If the job in question is known, provide the ID number of the job.
• stdout and stdError of the job. Letting us know the location of the said logs is also okay.
• Explain what kind of problem you are having.
• 20150831　Because of misconfiguration on batch node's settings, sometimes jobs terminate soon after starting.
• 20150831　To allow high I/O jobs to perform efficiently, we are planning a system that will allow allocating only one high I/O job on a single node.
  • Every node has 1 Gbps NIC for IO on the network attached storage.
  • Each node has 24 cores, so at most 24 cores can be entered.
  • If more than one high IO job is performed on a single node, performance will decrease. We want to control jobs in a way that only one high IO job can be on a single node.
  • The decision if a job is high IO or not will be done by the user, and controlled with a command line option. Even if a user carelessly doesn't assign high IO job as high IO job it will work, but in this case it will cause trouble for other users and waste the user's time too. We ask for your cooperation to isolate high IO jobs.
  • When sending a job with high IO, refer to example below.

        $ qsub -l select=1:ncpus=1:？？？？？？？？　　still in preparation.
    

• 20150827　arpack is planned for installation.
• 20150827　when using mew, set emacs settings in following way.

emacs 24.5.1	(setq load-path (cons (expand-file-name "/usr/local/share/emacs-24.5/site-lisp/mew") load-path))	unconfirmed
emacs 23.1.1	(setq load-path (cons (expand-file-name "/usr/local/share/emacs/site-lisp/mew") load-path))	likely to work

• 20150827　emacs 24.5.1 can be used normally. Old RedHat version can be found -> /usr/bin/emacs.
• 20150827　git's newest version is planned for installation.
• 20150824　python package not working. Currently looking for the problem. pip pep8 pyflakes Sphinx virtualenv
• 20150824　Regarding the change of malformed Japanese file names.
  • There might be files which name's can't be read because their names are malformed. When accessing through the old cifs servers file names can be read, but when using fs-1 or 2 the names might be malformed. This is caused by encoding mismatch, and will be fixed if you change the encoding.
  • Below is example script for changing the file names. Use it for reference when performing the changes.

  #!/bin/sh
  IFS_SAVE=$IFS
  IFS=$'\n'
  for file_name in `find .`
  do
    code=`echo "${file_name}" | nkf --guess`
    if [ "Shift_JIS (LF)" == "${code}" ]; then
      utf8_name=`echo "${file_name}" | iconv -f SJIS -t UTF-8`
      if [ "${file_name}" != "${utf8_name}" ]; then
  #     echo "******************************************************"
  #     echo "${file_name}" "${utf8_name}"
        mv -v "${file_name}" "${utf8_name}"
      fi
    fi
  done
  IFS=$IFS_SAVE
  

  • Shows the files in current directory.
  • If file names is encoded with SJIS,
  • generates UTF-8 name
  • uses mv command to change the name.
  
  
• 20150824　Don't output large data for standard output and error output. In that case, you should redirect them to other file systems where you can write large files. We consider some restriction for that use.
• 20150823　Benchmark results.
  • Linpack : We achived 424 Gflopsb using 24 cores, which is 81% of the theoretical performance. Detail
  • Memory-core dependency : We measured memory-core dependecy for large problem size. Detal (Excel)
• 20150821　miho-2 will be used for validation tests. From 11:00 on 25th (Tuesday) of August until 28th if you change the password be aware that, miho-2 will be idependent during this time, and passwords won't be updated.
• 20150821　On august 25th and 26th a security maintenance will be performed.
  • 25th 06:000 ~ 09:00　miho-1 can't be used.
  • 　　 　09:00 ~ 12:00 miho-2 can't be used.
  • 　　 　11:30 ~ 14:00　login-2 can't be used.
  • 　　　 14:00 ~ 17:00　ftp-2 can't be used.
  • 　　　 17:00 ~ 20:00　batch nodes 1-20 can't be used.
  • 　　　 17:30 ~ 20:30　large quantities of ram can't be used.
  • 26th　09:00 ~ 18:00　batch nodes 21-45 can't be used.
  • 　　　　09:00 ~ 12:00　aino-1 can't be used.
  • 　　　　12:00 ~ 15:00　ftp-1 can't be used.
  • 　　　　16:30 ~ 19:00　login-1 can't be used.
  • Maintenance will also be performed on others systems which are not directly used by users. The systems will be updated in a way that at least one of the two should be functioning.
• 20150821　miho configuring printers finished. aino not yet.。Text to PS converter is paps.
• 20150821　cernlib /usr/local/cern/2006b-epel/lib64 missing files, etc. have been added.
• 20150821　emacs latest version is separately planned for installation.
• 20150820　emacs acting oddly. Updating to latest version necessary?
• 20150820　python packet installation completed. Can someone test this? pip pep8 pyflakes Sphinx virtualenv
• 20150820 -static option/flag on　ifort causes LD error, and can't be used. -fast option also includes -static, so the same error occurs. We know that on 64-bit adding /usr/lib/x86_64-redhat-linux5E/lib64 to LIBRARY_PATH will fix the error, but we have yet to confirm this is the correct way to fix the problem. Using this method will also cause other compilers to show errors.
• 20150819　ifort -fast option causes ld error both 32 and 64 bit mode. This is a known problem. The work around to be announced.
• 20150819　xurlrunner installation completed. Adding /usr/local/xulrunner-i686, xulrunner-x86_64 to PATH necessary.
• 20150819　python package planned for installation. pip pep8 pyflakes Sphinx virtualenv
• 20150819　tmux installation completed. Setting LD_LIBRARY_PATH to /usr/local/lib is necessary.
• 20150818　on miho-1 nslookup fixed.
• 20150817 on miho-1 nslookup can't be used. miho-2 might work.(miho-2は引ける)
• 20150817 Setting up printers is not yet complete.
• 20150816　Disk quota system is operational, but currently it doesn't limit space usage. Currently only working function is showing how much of the quota is used.
  • mmlsquota -u <username>
  • mmlsquota -g <group name>
  • mmrepquota --block-size auto -a | grep <username>
  • mmrepquota --block-size auto -a | grep <group name>
  • mmrepquota --block-size auto -a | grep FILESET

• 20150816　Removal of _ (underscore) from directory names.

  Official folders with _ in their name's have been corrected, for now the folders can be also accessed with the old names.

  • /np1a_cagra → /np1a/cagra
  • /np1b_v07　 → /np1b/v07

  The change of names was done properly, but when running commands warnings about "fileset names" might come up. Until now this was done in file-system names, but because / can 't be used in names _ will come out like it used to.
  ディレクトリ名はすっきりと整理できましたが、コマンドのあちこちで「ファイルセット名」というのが 出てきます。これは従来のファイルシステム名に相当するものですが、/ を名前に使用できませんので、 そこでは相変わらず _ が出てきます。上記のようにきちんと対応していますので、適宜読み替えて下さい。

• 20150816 To show disk usage of all file-systems (even gpfs file-system filesets)add -a flag to df command.

  df -a
df -ah -t nfs -t none 　← Hide unnecessary information, and show capacities in easier format.

  • 新システムでの容量割当ては旧システムの端数切上げ程度にしてあります。
  • On the new system's disks there is more space. The long term plan is not yet finalized, but if you run out of space, it's possible to allocate more space, so please let us know.
• 20150816　/np1b/v07 file move (copy) finished. Problems with single files will be taken care by Hotta.(堀田)
• 20150815　xulrunner in preparations.
• 20150815　The new system's (miho) disk was mounted under /new/ on old system (saho).
• mmlsquota -u <username>、mmlsquota -g <group name>
• set a="expr `qstat -Q | wc -l` - 2" ; qstat -Q | head -n 2 ; qstat -Q | tail -n `$a` | sort
• 20150812　Use of login servers is possible now, please use them freely. login-1.rcnp.osaka-u.ac.jp and login-2... .
• 20150812　On fs-2 cifs service has been started. There are two file servers fs-1,2 for redundancy, both have same features and performance. It dosen't matter which one you use. If you have problem with one, please use the other one, and don't forgot to notify us about the problem.
• 20150812　convmv has been installed on all nodes. If kanjis in filenames don't show up properly you can try to use this command to fix them. It doesn't work in all cases.
• 20150812　nkf has been installed on all nodes./LI>
• 20150812　About locales (language, encoding)
  • Default locale (language) on miho is LANG=en_US.
  • Encoding for Japanese character (kanjis) is UTF-8.
  • You can change your own locale to LANG=ja_JP.UTF-8 if you so desire.
  • If you transfer files to or between computing systems, we recommend using UTF-8 encoding. If you use different encoding, your file names might get malformed when used with other computing systems. This concerns both file name and file's content.
  • This applies to transferring files with WinSCP, transferring files with other similar applications, when using cifs to transfer files and etc..
  • We recommend not using kanjis in file names.
• 20150812　If Japanese file names are malformed, it seems that we will be able to provide tools to change this. Please wait for a moment.
• 20150811　On the so called old installation, I will soon perform packet installation for the missing packets.
• 20150811　Following folder paths will be changed accordingly on 15th of August. /home/homeN → /home、/np1a_cagra → /np1a/cagra、/np1b_v07 → /np1b/v07.
• 20150811　script for initializing root

  shell	location for initialization script.	method to initialize	notes
  csh	/usr/local/root/v5.34.32.Linux-slc6-x86_64-gcc4.4/bin/thisroot.csh	cd /usr/local/root/ ; source bin/thisroot.csh	uses the standard version of the system
  bash	/usr/local/root/v5.34.32.Linux-slc6-x86_64-gcc4.4/bin/thisroot.sh	execute the script on the left	uses separately installed version.

• 20150811　Japanese file and folder names are malformed. How can we fix this?
• 20150811　texlive2015 installation complete. Can someone test this?
  • Add /usr/local/texlive/2015/bin/x86_64-linux to path.
  • Same Tex installation is planned for saho, but not yet finished. If there are no problems texlive2015 might be better. Let us know your opinion.
• 20150811　cifs is for now running on fs-1. fs-2 in preparation. Folders served are from miho. The files do not correspond to saho. Small folder structure changes are planned.
• 20150811　irb for ruby seems to be running.
• 20150810　To fix the problem with missing packets, we installed all packages from DVD that was available to miho-2 and b45 (one batch node). If there are no problems we are planning on installing the packages to rest of the nodes. Someone give it a try.
• 20150810　vncserver installation completed. Someone try it out.
• 20150809　Simplyfying links for home folder.

Currently the status of saho's home folders structure is complex. We want to make it simpler. For backwards compatibility links will be preserved, but the use of them is not recommended. Change won't affect users, so changes will be done as seen appropriate.

use	path	miho after 15th of August	miho currently	saho
recommended	/home/USERNAME	Real location	→ /home/homeN/USRENAME	→ /home/N/USERNAME → /homeN/USERNAME
Will stop working some day	/homeN/USERNAME	→ /home/USERNAME	no（not compatible with saho）	Real location.
	/home/N/USERNAME	→ /home/USERNAME	→ /home/homeN/USRENAME	→ /homeN/USERNAME
	/home/homeN/USERNAME	→ /home/USERNAME	Real location	no

• 20150808　Batch queue correction.By misconfiguration only 24 cores were usable, this is now fixed.
• 20150807　Memory allocation is possible up to 159 GB, but in this case you can only use two nodes. Rest of the 45 nodes can only allocate up to 64 GB. You don't have to specify nodes, as long as you specify the memory amount, the nodes will be assigned automatically.
• 20150807　Install cernlib 2006b-epel. Update the data for 5 items on table below.
• 20150807　vncserver planned for installation → completed
• 20150807　libX11 fixed. Missing modules added.
• 20150807　irb for ruby have been installed. Someone test this → Seems to work.
• 20150807　Default value for environment variable TERM is xterm. When using TeraTerm to access, you might have to change the environment variable to vt200 or something similar.
• 20150807　cernlib not stable

path	version	state
/usr/local/cern/2006/	http://cernlib.web.cern.ch/cernlib/download/2006_source/	？？
/usr/local/cern/2006b/	http://cernlib.web.cern.ch/cernlib/download/2006b_i686-slc5-gcc43-opt/	？？
/usr/local/cern/2006b-epel/	http://linuxsoft.cern.ch/epel/6/x86_64/　　2006-34.el6.x86_64	？？

• 20150807　Be careful of environment variable TERM. It might be different because of users settings, application or different from saho. .
• 20150807　xbae planned for installation
• 20150807　libXt, libXaw installation completed. Someone please test this.
• 20150806　libX11 link is not working. Other module missing.
• 20150806　xbae, libXt, libXaw planned for installation
• 20150806　tmux planned for installation
• 20150806　/usr/bin/gnuplot exists
• 20150806　gv is missing
• 20150806　irb for ruby is missing → installed → seems to work
• 20150806　Shell list

Shell path	version
/bin/tcsh, /bin/csh	tcsh-6.17-24.el6.x86_64
/bin/bash, /bin/sh	bash-4.1.2-15.el6_4.x86_64
/bin/ksh	ksh-20120801-10.el6.x86_64
/bin/zsh	zsh-4.3.10-7.el6.x86_64
/bin/dash	dash-0.5.5.1-4.el6.x86_64

• These shells can be started after login. Only shells supported at the time of login are tcsh and bash.
• 20150806　libX11.so, libXm.so, libXt.so 32-bit version installation completed. Someone please test this.
• 32-bit verision　/usr/lib
• 64-bit version/usr/lib64
• 20150806　g77 became usable on batch.
• 20150806　Mounting directories to supercomputer completed.

Type	formal path	path on miho	Comment
Home directory	/sc/rcnp/home/rcnpNNNN/	/home.super/NNNN/
temporary use	/ext/rcnp/short/rcnpNNNN/	/short.super/NNNN/
for data	/sc/rcnp/work/rcnpNNNN/	/work.super/NNNN/
for data	/sc/rcnp/work2/rcnpNNNN/	/work2.super/NNNN/
for data	/sc/rcnp/work3/rcnpNNNN/	/work3.super/NNNN/

• Formal path will become same as path on super computer. However this will change if the path changes on super computer.
• "path on miho" can be only used on miho. Path on miho won't change even if it changes on the super computer.
• 20150805　successor devices.

server	old system	new system	comment
ssh-login servers	saho-a,b	miho-1,2	please use
online station	hozumi-a,b	aino-1,2	please use
login server	login-a,b	login-1,2	in preparation →please use
file transfer server	ftp-a,b	ftp-1,2	in preparation→please use
cifs server	saho-a,b	fs-1,2	in preparation →please use

• 20150805　libX11.so, libXm.so, libXt.so 32 bit is planned for installation and 64 bit is already installed.
• 20150804　gcc 4.1.2 if planned for installation. saho and similar server's installation is of 4.4.7 is complete.
• 20150804　zsh is planned for installation.
• 20150804　gdb is planned for installation.
• 20150804　ncurses, ncurses-devel is planned for installation.
• 20150804　mounting disks on supercomputer in preparation.
• 20150804　/usr/local/cern/2006/include is in different location. If not fixed...
• 20150804　geant4 was installed in /usr/local/geant4.
• 20150803　bison is planned for installation.
• 20150803　DNS is functioning weirdly?　→　fixed
• 20150803　gcc was added to batch server. → gfortran worked。g77 not available.
• 20150803　Old systems libraries etc are now stored in /usr/local/old/ . The purpose is only to stored the old files, so the files were only copied there. If you wish to use old version of the libs, please send a request for that.
• 20150803　cifs server not yet working. I want to get it running soon.
• 20150803　In /usr/local/root should be root5. The file can't be seen with regular user's permission. root6 is planned for installation.
• 20150803　Storage location for old libraries will be published later.
• 20150802　Compiling with batch not possible now. Free compilers will be added. Intel compilers will be added little bit sooner.

The alpha release of the new computing system will start from 17th(Friday) of July.

• Release which only gives specific users access, for the purpose having them test the system.
• A lot of people people have already asked to help. If you wish to help contact Mr. Togawa.
• α release manual(notes, excel file (Japanese))

Details/Updates

• 20150730　root can be found in /usr/local/root
• 20150730　cernlib can be found in /usr/local/cern/2006
• 20150724　Homefolder is now /home/username . No digits anymore in the path. しばらくは旧 path も有効。
• 20150724　Cernlib will be first installed on miho-2.
• 20150724　When β release happens dotfile is planned to be moved to 000_DotFiles_20150731 (Date for this might chance)
• 20150722　The directory structure change → Created link from　/np1a/cagra to /np1a_cagra . Eventually /np1a_cagra will be removed and only /np1a/cagra will be left.
• 20150722　All disks mounted. Rsync is still copying the files so be careful when using.
• 20150722　Directory path change → from　/np1a/cagra to　→ ??
• 20150722　Directory structure change 　→ Created link from /np1b/v07 to /np1b_v07. Eventually /np1b_v07 will be removed and only /np1b/v07 will be left.
• 20150722　Hurry with installation of Cernlib →
• 20150720 Dotfile was moved to 000_DotFiles_20150720 (the date part may differ)
• 20150720　default umask is 077
• 20150719　Directory path change → from　np1b_v07 to np1b/v07 WHEN?
• 20150719　Directory path change → from　np1a_cagra to np1a/cagra WHEN?
• 20150719　treatment of dotfile　→
• 20150719　How to check disk quota→
• 20150719　How to check disk space used （df）　→
• 20150719　Make 077 default umask →
• 20150719　Automatic removal of /tmp and /tmp-common　→
• 20150719　/tmp-common/ can be used.
• 20150717　miho-1,2 are not registered in dns. →
• 20150717　Simple bash script was successfully run.
• 20150717　ifort 32bit successfully completed simple compile.
• 20150717　ifort 64bit successfully completed simple compile.
• 20150717　gfortran successfully completed simple compile.