[mfasheh@ca-build8 multi_node_mmap]$ mpirun -np 6 n1-4 ./multi_mmap -w regular -r mmap -i 1000 -b 4096 /ocfs2/mmap_test2

test log:

ca-test27 (rank 2): Write  via write block 2 of 'f'
ca-test26 (rank 1): Expect via mmap  block 2 of 'f'
ca-test28 (rank 3): Expect via mmap  block 2 of 'f'
ca-test26 (rank 5): Expect via mmap  block 2 of 'f'
ca-test25 (rank 0): Expect via mmap  block 2 of 'f'
ca-test25 (rank 4): Expect via mmap  block 2 of 'f'
ca-test28 (rank 3): Write  via write block 3 of 'g'
ca-test27 (rank 2): Expect via mmap  block 3 of 'g'
ca-test26 (rank 1): Expect via mmap  block 3 of 'g'
ca-test26 (rank 5): Expect via mmap  block 3 of 'g'
ca-test25 (rank 0): Expect via mmap  block 3 of 'g'
ca-test25 (rank 4): Expect via mmap  block 3 of 'g'
ca-test25 (rank 4): Write  via write block 4 of 'h'
ca-test27 (rank 2): Expect via mmap  block 4 of 'h'
ca-test26 (rank 1): Expect via mmap  block 4 of 'h'
ca-test28 (rank 3): Expect via mmap  block 4 of 'h'
ca-test26 (rank 5): Expect via mmap  block 4 of 'h'
ca-test25 (rank 0): Expect via mmap  block 4 of 'h'
ca-test26 (rank 5): Write  via write block 5 of 'i'
ca-test27 (rank 2): Expect via mmap  block 5 of 'i'
ca-test28 (rank 3): Expect via mmap  block 5 of 'i'
ca-test26 (rank 1): Expect via mmap  block 5 of 'i'
ca-test25 (rank 0): Expect via mmap  block 5 of 'i'
ca-test25 (rank 4): Expect via mmap  block 5 of 'i'
ca-test25 (rank 0): Write  via write block 0 of 'e'
ca-test27 (rank 2): Expect via mmap  block 0 of 'e'
ca-test25 (rank 4): Expect via mmap  block 0 of 'e'
ca-test28 (rank 3): Expect via mmap  block 0 of 'e'
ca-test26 (rank 1): Expect via mmap  block 0 of 'e'
ca-test26 (rank 5): Expect via mmap  block 0 of 'e'
ca-test26 (rank 1): Write  via write block 1 of 'f'
ca-test27 (rank 2): Expect via mmap  block 1 of 'f'
ca-test26 (rank 5): Expect via mmap  block 1 of 'f'
ca-test25 (rank 0): Expect via mmap  block 1 of 'f'
ca-test28 (rank 3): Expect via mmap  block 1 of 'f'
ca-test25 (rank 4): Verification failed for block 1.
ca-test25 (rank 4): First 10 expected chars: "ffffffffff"
ca-test25 (rank 4): First 10 actual chars: "eeeeeeeeee"
ca-test25 (rank 4): Expect via mmap  block 1 of 'f'
ca-test27 (rank 2): Write  via write block 2 of 'g'
MPI_Recv: process in local group is dead (rank 3, comm 3)
Rank (3, MPI_COMM_WORLD): Call stack within LAM:
Rank (3, MPI_COMM_WORLD):  - MPI_Recv()
Rank (3, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (3, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (3, MPI_COMM_WORLD):  - main()
MPI_Recv: process in local group is dead (rank 2, comm 3)
Rank (2, MPI_COMM_WORLD): Call stack within LAM:
Rank (2, MPI_COMM_WORLD):  - MPI_Recv()
Rank (2, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (2, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (2, MPI_COMM_WORLD):  - main()
MPI_Recv: process in local group is dead (rank 1, SSI:coll:smp:local comm for CID 0)
Rank (5, MPI_COMM_WORLD): Call stack within LAM:
Rank (5, MPI_COMM_WORLD):  - MPI_Recv()
Rank (5, MPI_COMM_WORLD):  - MPI_Bcast()
Rank (5, MPI_COMM_WORLD):  - MPI_Barrier()
Rank (5, MPI_COMM_WORLD):  - main()
-----------------------------------------------------------------------------
One of the processes started by mpirun has exited with a nonzero exit
code.  This typically indicates that the process finished in error.
If your process did not finish in error, be sure to include a "return
0" or "exit(0)" in your C code before exiting the application.

PID 5945 failed on node n1 (139.185.49.25) with exit status 1.
-----------------------------------------------------------------------------


dmesg from ca-test25:

ocfs2: dumping nopage trace buffer (curr = 44)
(5947,1):print_trace:110 ocfs2: [0] Inode 6258144, index 0
(5947,1):print_trace:110 ocfs2: [1] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [2] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [3] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [4] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [5] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [6] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [7] Inode 6258144, index 4
(5947,1):print_trace:110 ocfs2: [8] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [9] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [10] Inode 6258144, index 0
(5947,1):print_trace:110 ocfs2: [11] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [12] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [13] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [14] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [15] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [16] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [17] Inode 6258144, index 4
(5947,1):print_trace:110 ocfs2: [18] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [19] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [20] Inode 6258144, index 0
(5947,1):print_trace:110 ocfs2: [21] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [22] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [23] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [24] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [25] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [26] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [27] Inode 6258144, index 4
(5947,1):print_trace:110 ocfs2: [28] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [29] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [30] Inode 6258144, index 0
(5947,1):print_trace:110 ocfs2: [31] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [32] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [33] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [34] Inode 6258144, index 1
(5947,1):print_trace:110 ocfs2: [35] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [36] Inode 6258144, index 2
(5947,1):print_trace:110 ocfs2: [37] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [38] Inode 6258144, index 3
(5947,1):print_trace:110 ocfs2: [39] Inode 6258144, index 4
(5947,1):print_trace:110 ocfs2: [40] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [41] Inode 6258144, index 5
(5947,1):print_trace:110 ocfs2: [42] Inode 6258144, index 0
(5947,1):print_trace:110 ocfs2: [43] Inode 6258144, index 1
ocfs2: page data 1: eeeeeeeeeeeeeee
ocfs2: page data 2: fffffffffffffff