mpirun -np 6 n1-4 ./multi_mmap -w regular -r mmap -i 1000 -b 4096 /ocfs2/mmap_test3 log from run: ca-test27 (rank 2): Expect via mmap block 5 of 'o' ca-test28 (rank 3): Expect via mmap block 5 of 'o' ca-test26 (rank 1): Expect via mmap block 5 of 'o' ca-test25 (rank 0): Expect via mmap block 5 of 'o' ca-test25 (rank 4): Expect via mmap block 5 of 'o' ca-test25 (rank 0): Write via write block 0 of 'k' ca-test27 (rank 2): Expect via mmap block 0 of 'k' ca-test25 (rank 4): Expect via mmap block 0 of 'k' ca-test28 (rank 3): Expect via mmap block 0 of 'k' ca-test26 (rank 1): Expect via mmap block 0 of 'k' ca-test26 (rank 5): Expect via mmap block 0 of 'k' ca-test26 (rank 1): Write via write block 1 of 'l' ca-test27 (rank 2): Expect via mmap block 1 of 'l' ca-test25 (rank 0): Expect via mmap block 1 of 'l' ca-test26 (rank 5): Expect via mmap block 1 of 'l' ca-test28 (rank 3): Expect via mmap block 1 of 'l' ca-test25 (rank 4): Expect via mmap block 1 of 'l' ca-test27 (rank 2): Write via write block 2 of 'm' ca-test28 (rank 3): Expect via mmap block 2 of 'm' ca-test26 (rank 1): Expect via mmap block 2 of 'm' ca-test26 (rank 5): Expect via mmap block 2 of 'm' ca-test25 (rank 0): Expect via mmap block 2 of 'm' ca-test25 (rank 4): Expect via mmap block 2 of 'm' ca-test28 (rank 3): Write via write block 3 of 'n' ca-test27 (rank 2): Expect via mmap block 3 of 'n' ca-test26 (rank 1): Expect via mmap block 3 of 'n' ca-test26 (rank 5): Expect via mmap block 3 of 'n' ca-test25 (rank 0): Expect via mmap block 3 of 'n' ca-test25 (rank 4): Expect via mmap block 3 of 'n' ca-test25 (rank 4): Write via write block 4 of 'o' ca-test27 (rank 2): Expect via mmap block 4 of 'o' ca-test25 (rank 0): Expect via mmap block 4 of 'o' ca-test28 (rank 3): Expect via mmap block 4 of 'o' ca-test26 (rank 1): Expect via mmap block 4 of 'o' ca-test26 (rank 5): Expect via mmap block 4 of 'o' ca-test26 (rank 5): Write via write block 5 of 'p' ca-test25 (rank 0): Expect via mmap block 5 of 'p' ca-test28 (rank 3): Expect via mmap block 5 of 'p' ca-test27 (rank 2): Expect via mmap block 5 of 'p' ca-test26 (rank 1): Expect via mmap block 5 of 'p' ca-test25 (rank 4): Expect via mmap block 5 of 'p' ca-test25 (rank 0): Write via write block 0 of 'l' ca-test28 (rank 3): Expect via mmap block 0 of 'l' ca-test25 (rank 4): Expect via mmap block 0 of 'l' ca-test27 (rank 2): Expect via mmap block 0 of 'l' ca-test26 (rank 1): Expect via mmap block 0 of 'l' ca-test26 (rank 5): Expect via mmap block 0 of 'l' ca-test26 (rank 1): Write via write block 1 of 'm' ca-test28 (rank 3): Expect via mmap block 1 of 'm' ca-test27 (rank 2): Expect via mmap block 1 of 'm' ca-test25 (rank 0): Expect via mmap block 1 of 'm' ca-test26 (rank 5): Expect via mmap block 1 of 'm' ca-test25 (rank 4): Expect via mmap block 1 of 'm' ca-test27 (rank 2): Write via write block 2 of 'n' ca-test25 (rank 0): Expect via mmap block 2 of 'n' ca-test25 (rank 0): Verification failed for block 2. ca-test25 (rank 0): First 10 expected chars: "nnnnnnnnnn" ca-test25 (rank 0): First 10 actual chars: "mmmmmmmmmm" ca-test28 (rank 3): Expect via mmap block 2 of 'n' ca-test26 (rank 1): Expect via mmap block 2 of 'n' ca-test26 (rank 5): Expect via mmap block 2 of 'n' ca-test25 (rank 4): Expect via mmap block 2 of 'n' ca-test25 (rank 4): Verification failed for block 2. ca-test25 (rank 4): First 10 expected chars: "nnnnnnnnnn" ca-test25 (rank 4): First 10 actual chars: "mmmmmmmmmm" ca-test28 (rank 3): Write via write block 3 of 'o' MPI_Recv: process in local group is dead (rank 2, comm 3) Rank (2, MPI_COMM_WORLD): Call stack within LAM: Rank (2, MPI_COMM_WORLD): - MPI_Recv() Rank (2, MPI_COMM_WORLD): - MPI_Barrier() Rank (2, MPI_COMM_WORLD): - MPI_Barrier() MPI_Recv: process in local group is dead (rank 3, comm 3) Rank (3, MPI_COMM_WORLD): Call stack within LAM: Rank (3, MPI_COMM_WORLD): - MPI_Recv() Rank (3, MPI_COMM_WORLD): - MPI_Barrier() MPI_Recv: process in local group is dead (rank 1, SSI:coll:smp:local comm for CID 0) Rank (5, MPI_COMM_WORLD): Call stack within LAM: Rank (5, MPI_COMM_WORLD): - MPI_Recv() Rank (5, MPI_COMM_WORLD): - MPI_Bcast() Rank (5, MPI_COMM_WORLD): - MPI_Barrier() Rank (5, MPI_COMM_WORLD): - main() Rank (3, MPI_COMM_WORLD): - MPI_Barrier() Rank (3, MPI_COMM_WORLD): - main() Rank (2, MPI_COMM_WORLD): - main() ----------------------------------------------------------------------------- One of the processes started by mpirun has exited with a nonzero exit code. This typically indicates that the process finished in error. If your process did not finish in error, be sure to include a "return 0" or "exit(0)" in your C code before exiting the application. PID 23921 failed on node n2 (139.185.49.26) with exit status 1. ----------------------------------------------------------------------------- dmesg on test25: ocfs2: dumping nopage trace buffer (curr = 15) (6038,0):print_trace:114 ocfs2: [16] Inode 6258145, index 3,pid: 6036, data jjjjjjjjjjjjjjj (6038,0):print_trace:114 ocfs2: [17] Inode 6258145, index 4,pid: 6036, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [18] Inode 6258145, index 5,pid: 6036, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [19] Inode 6258145, index 5,pid: 6037, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [20] Inode 6258145, index 0,pid: 6037, data hhhhhhhhhhhhhhh (6038,0):print_trace:114 ocfs2: [21] Inode 6258145, index 1,pid: 6036, data iiiiiiiiiiiiiii (6038,0):print_trace:114 ocfs2: [22] Inode 6258145, index 1,pid: 6037, data iiiiiiiiiiiiiii (6038,0):print_trace:114 ocfs2: [23] Inode 6258145, index 2,pid: 6036, data jjjjjjjjjjjjjjj (6038,0):print_trace:114 ocfs2: [24] Inode 6258145, index 2,pid: 6037, data jjjjjjjjjjjjjjj (6038,0):print_trace:114 ocfs2: [25] Inode 6258145, index 3,pid: 6036, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [26] Inode 6258145, index 3,pid: 6037, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [27] Inode 6258145, index 4,pid: 6036, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [28] Inode 6258145, index 5,pid: 6036, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [29] Inode 6258145, index 5,pid: 6037, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [30] Inode 6258145, index 0,pid: 6037, data iiiiiiiiiiiiiii (6038,0):print_trace:114 ocfs2: [31] Inode 6258145, index 1,pid: 6036, data jjjjjjjjjjjjjjj (6038,0):print_trace:114 ocfs2: [32] Inode 6258145, index 1,pid: 6037, data jjjjjjjjjjjjjjj (6038,0):print_trace:114 ocfs2: [33] Inode 6258145, index 2,pid: 6036, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [34] Inode 6258145, index 2,pid: 6037, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [35] Inode 6258145, index 3,pid: 6036, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [36] Inode 6258145, index 3,pid: 6037, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [37] Inode 6258145, index 4,pid: 6036, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [38] Inode 6258145, index 5,pid: 6036, data nnnnnnnnnnnnnnn (6038,0):print_trace:114 ocfs2: [39] Inode 6258145, index 5,pid: 6037, data nnnnnnnnnnnnnnn (6038,0):print_trace:114 ocfs2: [40] Inode 6258145, index 0,pid: 6037, data jjjjjjjjjjjjjjj (6038,0):print_trace:114 ocfs2: [41] Inode 6258145, index 1,pid: 6037, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [42] Inode 6258145, index 1,pid: 6036, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [43] Inode 6258145, index 2,pid: 6036, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [44] Inode 6258145, index 2,pid: 6037, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [45] Inode 6258145, index 3,pid: 6036, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [46] Inode 6258145, index 3,pid: 6037, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [47] Inode 6258145, index 4,pid: 6036, data nnnnnnnnnnnnnnn (6038,0):print_trace:114 ocfs2: [48] Inode 6258145, index 5,pid: 6036, data ooooooooooooooo (6038,0):print_trace:114 ocfs2: [49] Inode 6258145, index 5,pid: 6037, data ooooooooooooooo (6038,0):print_trace:114 ocfs2: [0] Inode 6258145, index 0,pid: 6037, data kkkkkkkkkkkkkkk (6038,0):print_trace:114 ocfs2: [1] Inode 6258145, index 1,pid: 6036, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [2] Inode 6258145, index 1,pid: 6037, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [3] Inode 6258145, index 2,pid: 6037, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [4] Inode 6258145, index 2,pid: 6037, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [5] Inode 6258145, index 2,pid: 6036, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [6] Inode 6258145, index 2,pid: 6036, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [7] Inode 6258145, index 3,pid: 6036, data nnnnnnnnnnnnnnn (6038,0):print_trace:114 ocfs2: [8] Inode 6258145, index 3,pid: 6037, data nnnnnnnnnnnnnnn (6038,0):print_trace:114 ocfs2: [9] Inode 6258145, index 4,pid: 6036, data ooooooooooooooo (6038,0):print_trace:114 ocfs2: [10] Inode 6258145, index 5,pid: 6036, data ppppppppppppppp (6038,0):print_trace:114 ocfs2: [11] Inode 6258145, index 5,pid: 6037, data ppppppppppppppp (6038,0):print_trace:114 ocfs2: [12] Inode 6258145, index 0,pid: 6037, data lllllllllllllll (6038,0):print_trace:114 ocfs2: [13] Inode 6258145, index 1,pid: 6036, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [14] Inode 6258145, index 1,pid: 6037, data mmmmmmmmmmmmmmm (6038,0):print_trace:114 ocfs2: [15] Inode 6258145, index 3,pid: 6037, data jjjjjjjjjjjjjjj