forked from NERSC/cosmoflow-sims
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
98 lines (55 loc) · 3.53 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
MUSIC:
to compile:
cd MUSIC
module unload darshan
module load cray-fftw gsl cray-hdf5
make
to run for three cosmological parameters:
cd OmSiNs
python runCode.py
--------------------
pycola:
First step: make a conda env with the correct python modules in it. I called this "cola", could obviously be anything you like.
conda create -n cola python=2.7 numpy
source activate cola
pip install --global-option=build_ext --global-option=-L/usr/common/software/fftw/3.3.4/hsw/gnu/lib --global-option=-I/usr/common/software/fftw/3.3.4/hsw/gnu/include pyfftw
conda install scipy
conda install h5py
conda install matplotlib
conda install cython
to compile:
module load python
source activate NAME_OF_YOUR_CONDA_ENV
python setup.py build_ext --inplace
to run:
module load python
source activate NAME_OF_YOUR_CONDA_ENV
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/common/software/fftw/3.3.4/hsw/gnu/lib/
Need a MUSIC file to run over as input file.
pycola will need to run one job for every input file. I used taskfarmer to bundle all the jobs into units that would run effectively on Cori.
In pycola/OmSiNs/taskfarmer,
- writeTasks.py makes the list of tasks that you will feed into taskfarmer.
- taskwrapper.sh actually contains the tasks to be run. *You will need to change the path in this file to point to your own directory, and to activate your own conda env*
- taskfarmer.sh is the batch file to submit. Usual options apply. You'll at least need to edit to poitn to the correct dir for your installation.
This will run the file:
cosmoflow-sims/pycola/pycola-OmSiNs-template.py
which looks at all the hdf5 files in the directory cosmoflow-sims/MUSIC/OmSiNs/ that were produced by the MUSIC code you just ran. It'll run the pycola code (the N-body simulation code) over each of those hdf5 files. The output is put into the pycola/OmSiNs directory.
Note that when I was running this, I'd delete the tfin files after each taskfarmer run, as there seemed to be some confusion if I tried to use the in-built taskfarmer checks for which jobs had completed. Instead I had the code check for completion of the task list.
--------------------
Reformatting the NBody files
This code does two things
1) makes a histogram of particle counts out of the list of particle coordinates
2) Splits that histogram into 8 sub-volumes
just run :
cosmoflow-sims/pycola> python reformatNBody.py 0
The argument for this defines which file in the pycola output dir you start with. I put this in so I could run multiple versions of this code at the same time over the same output dir.
The output of this code is a directory inside the OmSiNs dir that has the name of the 3 cosmo params behind the simulation. Inside that dir are 8 npy files, each of which is a histogram of particle counts.
--------------------
Making the list of files for input into the CosmoFlow IO code
The next stage of formatting involves putting these sub-volume files into a format required by the CosmoFlow IO code, for when it makes the TFRecord files you'll actually use in the tensorflow training. For this we need a list of cosmologies, and we need to re-name all the dirs we just made to numbers that correspond to the position of this simulation on the list.
I know this is not a very sensible solution.
First run :
cosmoflow-sims/pycola/OmSiNs/makeList.py
to make the list, then run:
cosmoflow-sims/pycola/OmSiNs/copyFiles.py
to make new dirs with new names. Note that I'm copying them, rather than moving the dir, because I am cautious and didn't want to have to re-make all these files again.