[mfasheh@ca-build8 multi_node_mmap]$ mpirun -np 6 n1-4 ./multi_mmap -w regular -r mmap -i 1000 -b 4096 /ocfs2/mmap_test8

ca-test27 (rank 2): Write  via write block 2 of 'c'. My pid = 7470
ca-test28 (rank 3): Expect via mmap  block 2 of 'c'. My pid = 7559
ca-test26 (rank 1): Expect via mmap  block 2 of 'c'. My pid = 7031
ca-test26 (rank 5): Expect via mmap  block 2 of 'c'. My pid = 7032
ca-test25 (rank 0): Expect via mmap  block 2 of 'c'. My pid = 6411
ca-test25 (rank 4): Expect via mmap  block 2 of 'c'. My pid = 6412
ca-test28 (rank 3): Write  via write block 3 of 'd'. My pid = 7559
ca-test26 (rank 1): Expect via mmap  block 3 of 'd'. My pid = 7031
ca-test27 (rank 2): Expect via mmap  block 3 of 'd'. My pid = 7470
ca-test26 (rank 5): Expect via mmap  block 3 of 'd'. My pid = 7032
ca-test25 (rank 0): Expect via mmap  block 3 of 'd'. My pid = 6411
ca-test25 (rank 4): Expect via mmap  block 3 of 'd'. My pid = 6412
ca-test25 (rank 4): Write  via write block 4 of 'e'. My pid = 6412
ca-test27 (rank 2): Expect via mmap  block 4 of 'e'. My pid = 7470
ca-test26 (rank 1): Expect via mmap  block 4 of 'e'. My pid = 7031
ca-test28 (rank 3): Expect via mmap  block 4 of 'e'. My pid = 7559
ca-test26 (rank 5): Expect via mmap  block 4 of 'e'. My pid = 7032
ca-test25 (rank 0): Expect via mmap  block 4 of 'e'. My pid = 6411
ca-test26 (rank 5): Write  via write block 5 of 'f'. My pid = 7032
ca-test28 (rank 3): Expect via mmap  block 5 of 'f'. My pid = 7559
ca-test26 (rank 1): Expect via mmap  block 5 of 'f'. My pid = 7031
ca-test27 (rank 2): Expect via mmap  block 5 of 'f'. My pid = 7470
ca-test25 (rank 0): Expect via mmap  block 5 of 'f'. My pid = 6411
ca-test25 (rank 4): Expect via mmap  block 5 of 'f'. My pid = 6412
ca-test25 (rank 0): Write  via write block 0 of 'b'. My pid = 6411
ca-test27 (rank 2): Expect via mmap  block 0 of 'b'. My pid = 7470
ca-test28 (rank 3): Expect via mmap  block 0 of 'b'. My pid = 7559
ca-test26 (rank 1): Expect via mmap  block 0 of 'b'. My pid = 7031
ca-test26 (rank 5): Expect via mmap  block 0 of 'b'. My pid = 7032
ca-test25 (rank 4): Expect via mmap  block 0 of 'b'. My pid = 6412
ca-test26 (rank 1): Write  via write block 1 of 'c'. My pid = 7031
ca-test26 (rank 5): Expect via mmap  block 1 of 'c'. My pid = 7032
ca-test27 (rank 2): Expect via mmap  block 1 of 'c'. My pid = 7470
ca-test28 (rank 3): Expect via mmap  block 1 of 'c'. My pid = 7559
ca-test25 (rank 0): Expect via mmap  block 1 of 'c'. My pid = 6411
ca-test25 (rank 4): Expect via mmap  block 1 of 'c'. My pid = 6412
ca-test27 (rank 2): Write  via write block 2 of 'd'. My pid = 7470
ca-test26 (rank 1): Expect via mmap  block 2 of 'd'. My pid = 7031
ca-test25 (rank 0): Expect via mmap  block 2 of 'd'. My pid = 6411
ca-test26 (rank 1): Verification failed for block 2.
ca-test26 (rank 1): First 10 expected chars: "dddddddddd"
ca-test26 (rank 1): First 10 actual chars: "cccccccccc"
ca-test28 (rank 3): Expect via mmap  block 2 of 'd'. My pid = 7559
ca-test26 (rank 5): Expect via mmap  block 2 of 'd'. My pid = 7032
ca-test25 (rank 4): Expect via mmap  block 2 of 'd'. My pid = 6412
ca-test28 (rank 3): Write  via write block 3 of 'e'. My pid = 7559
-----------------------------------------------------------------------------
One of the processes started by mpirun has exited with a nonzero exit
code.  This typically indicates that the process finished in error.
If your process did not finish in error, be sure to include a "return
0" or "exit(0)" in your C code before exiting the application.

PID 6411 failed on node n1 (139.185.49.25) with exit status 1.
-----------------------------------------------------------------------------
MPI_Recv: process in local group is dead (rank 3, comm 3)
Rank (3, MPI_COMM_WORLD): Call stack within LAM:
Rank (3, MPI_COMM_WORLD):  - MPI_Recv()
Rank (3, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (3, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (3, MPI_COMM_WORLD):  - main()
MPI_Recv: process in local group is dead (rank 1, SSI:coll:smp:local comm for CID 0)
Rank (4, MPI_COMM_WORLD): Call stack within LAM:
Rank (4, MPI_COMM_WORLD):  - MPI_Recv()
Rank (4, MPI_COMM_WORLD):  - MPI_Bcast()
Rank (4, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (4, MPI_COMM_WORLD):  - main()


dmesg from ca-test26

ocfs2: dumping nopage trace buffer (curr = 16)
(7034,1):print_trace:114 ocfs2: [0] Inode 6258150, index 0,pid: 7031, data aaaaaaaaaaaaaaa
(7034,1):print_trace:114 ocfs2: [1] Inode 6258150, index 0,pid: 7032, data aaaaaaaaaaaaaaa
(7034,1):print_trace:114 ocfs2: [2] Inode 6258150, index 1,pid: 7032, data bbbbbbbbbbbbbbb
(7034,1):print_trace:114 ocfs2: [3] Inode 6258150, index 2,pid: 7031, data ccccccccccccccc
(7034,1):print_trace:114 ocfs2: [4] Inode 6258150, index 2,pid: 7031, data ccccccccccccccc
(7034,1):print_trace:114 ocfs2: [5] Inode 6258150, index 2,pid: 7032, data ccccccccccccccc
(7034,1):print_trace:114 ocfs2: [6] Inode 6258150, index 2,pid: 7032, data ccccccccccccccc
(7034,1):print_trace:114 ocfs2: [7] Inode 6258150, index 3,pid: 7031, data ddddddddddddddd
(7034,1):print_trace:114 ocfs2: [8] Inode 6258150, index 3,pid: 7032, data ddddddddddddddd
(7034,1):print_trace:114 ocfs2: [9] Inode 6258150, index 4,pid: 7032, data eeeeeeeeeeeeeee
(7034,1):print_trace:114 ocfs2: [10] Inode 6258150, index 4,pid: 7031, data eeeeeeeeeeeeeee
(7034,1):print_trace:114 ocfs2: [11] Inode 6258150, index 5,pid: 7031, data fffffffffffffff
(7034,1):print_trace:114 ocfs2: [12] Inode 6258150, index 0,pid: 7031, data bbbbbbbbbbbbbbb
(7034,1):print_trace:114 ocfs2: [13] Inode 6258150, index 0,pid: 7032, data bbbbbbbbbbbbbbb
(7034,1):print_trace:114 ocfs2: [14] Inode 6258150, index 1,pid: 7032, data ccccccccccccccc
(7034,1):print_trace:114 ocfs2: [15] Inode 6258150, index 2,pid: 7032, data ddddddddddddddd