Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

gfortran: internal compiler error: Killed (program f951)

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

I was trying to install wrfchem on hpc but I encountered this problem
gfortran: internal compiler error: Killed (program f951)
I realy cant figure our what was wrong

i will appreciate your support
Juli
 

Attachments

  • compile.log
    382.1 KB · Views: 42
Thank you so much for your reply to my request
but how can I provide more memory to the compile job?
because I am working on hpc remotely

Thank you
Juli
 
Hi Juli,

What is the command you use to compile? And do you run the command in an interactive terminal or submit it as a job? For example, if you are submitting a job with the SLURM scheduler, you can add an option such as:

#SBATCH --mem-per-cpu=16G

Jordan
 
Yes I compile on interactive terminal for example I configure
$ ./configure
and I also compile with
$ ./compile em_real >& compile.log

for your information when I used this command to check up the memory then I have the following displayed:
$ free -m
total used free shared buff/cache available
Mem: 128826 3206 41932 2083 83687 115755
Swap: 16383 148 16235

Thank you
 
Hi,

It seems you are submitting the compile on a login node, which may have limited resources. Do you have an option to submit the compile as a job on a compute node? Or can you access a compute node for an interactive job?

Jordan
 
I must sincerely say thank you for your time.
To answer the firsts question on whether i have option to submit the compile as a job on a compute node or not

I am just a new user of CHPC but I read through the docuymentation I think is possible. The CHPC cluster uses PBSPro as its job scheduler. for example i came accross this scheduler sample script on wiki page of chpc http://wiki.chpc.ac.za/quick:start

#!/bin/bash
#PBS -l select=10:ncpus=24:mpiprocs=24:nodetype=haswell_reg
#PBS -P PRJT1234
#PBS -q normal
#PBS -l walltime=4:00:00
#PBS -o /mnt/lustre3p/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.out
#PBS -e /mnt/lustre3p/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.err
#PBS -m abe
#PBS -WMail_Users=youremail@ddress
ulimit -s unlimited
. /apps/chpc/earth/WRF-3.7-impi/setWRF
cd /mnt/lustre3p/users/USERNAME/WRF_Tests/WRFV3/run2km_100
rm wrfout* rsl*
nproc=`cat $PBS_NODEFILE | wc -l`
echo nproc is $nproc
cat $PBS_NODEFILE
time mpirun -np $nproc wrf.exe > runWRF.out

qsub example.job

probably if you tell me what to do I could modify this script for the compiling job
thank you

Juli
 
Hi Juli,

I can provide some information, but other info will need to be gathered from your HPC administrator (e.g., accounts you have access to).

That said, here is a simple example script (from https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/running-jobs/pbs-pro-job-script-examples)

note that the first line must be "#!/bin/bash" if you are using the bash shell (see other examples if you are using tcsh/csh)
Things in bold are my comments.

------------
#!/bin/bash
### Job Name
#PBS -N JOB_NAME_OF_YOUR_CHOICE_HERE
### Project code
#PBS -A name_of_account_you_have_access_to_run_jobs_on: you may be able to see this with "saccount_params"
#PBS -l walltime=01:00:00 how_long_the_job_should_run
#PBS -q queue_name (generally is "batch")
### Merge output and error files (next two lines do this)
#PBS -j oe
#PBS -k eod
### Select 2 nodes with 36 CPUs each for a total of 72 MPI processes -- Here you only need one so can probably delete this line or change all values to 1. but you will need it later to run WRF-Chem
#PBS -l select=2:ncpus=36:mpiprocs=36
### Send email on abort, begin and end
#PBS -m abe
### Specify mail recipient
#PBS -M email_address

cd /location/of/top/level/wrf/dir
# Compile with separate log for errors
./compile em_real > log_comp.txt 2> errors_comp.txt

--------

Please let me know if you have more questions.

Jordan
 
greetings and thank you for all the assistance previously
I have edited the compile job script you sent to me last time and I have run it
but it seems nothing compile yet
the script thus
l#!/bin/bash
### Job Name
#PBS -N compile_wrfjob
### Project_code
#PBS -P ERTH0904
###PBS -A ERTH0904 (chpc prefer -P for -A)
#PBS -l walltime=01:00:00
### added -0 and -e from previous example in chpc
#PBS -o /mnt/lustre/users/jakinyoola/WRFCHEMSETUP/compile12.out
#PBS -e /mnt/lustre/users/jakinyoola/WRFCHEMSETUP/compile12.err
#PBS -q normal
### Merge output and error files (next two lines do this)
#PBS -j oe
#PBS -k eod
### Select 2 nodes with 36 CPUs each for a total of 72 MPI processes
#PBS -l select=2:ncpus=36:mpiprocs=36
### Send email on abort, begin and end
#PBS -m abe
### Specify mail recipient
#PBS -M omoakinyoola@gmail.com
cd /mnt/lustre/users/jakinyoola/WRFCHEMSETUP/WRF-4.1.2
### Compile with separate log for errors
./compile em_real > log_comp.txt 2> errors_comp.txt

I have this Bellow when I check with >qstat -u (USERNAME)
Screenshot from 2021-11-16 23-48-22.png

what you think I need to do
thank you
 
Hi,

It looks like your job is waiting to run, but it is likely because you are requesting more processors than you need.

Change this line:

#PBS -l select=2:ncpus=36:mpiprocs=36

to:

#PBS -l select=1:ncpus=1:mpiprocs=1

since you only need one to compile.

Jordan
 
there was an error flag bellow when I changed from
#PBS -l select=2:ncpus=36:mpiprocs=36
to:
#PBS -l select=1:ncpus=1:mpiprocs=1
qsub: Job violates queue and/or server resource limits

but the former submitted the job still

with regards
Juli
 
Top