HDF5 Fortran90 append data as hyperslabs - only first column working - fortran90

I have a Fortran 90 program that creates a large column of integers at the end of each loop. I would like to write this column into an HDF5 file with each new integer column being written as a new column in the HDF5 file. That is, at the end of the first loop, write column 1, then at the end of the second loop, write column 2, etc, until a 2D array with as many columns as there were loops is created.
Each of these integer columns will always be the same length and I know how many columns will be needed in the end. To do this, I create a dataspace large enough to hold the final 2D array. Then I select a hyperslab beginning with offset (0,0), then (0,1) for the next loop, (0,2) for the loop after that, etc., each time writing the data that my program creates.
This isn't quite working, however. Currently, with the offset (0,0) everything works. However, if I try to write to a hyperslab in any other column, I get nonsense values in my output .h5 file.
Am I implementing the hyperslab code correctly?
Here is a MWE which creates a dataspace of (20,10) and writes a column of twenty '5's into the first column of the .h5 file. If voffset is changed from (/0,0/) to (/0,1/), the '5's are not written and instead random numbers will appear in the second column of the .h5 file:
program hdf5_test
use hdf5
implicit none
! HDF5 file and path names
character(len=10), parameter :: outfilename = "OutData.h5" ! Output file name
character(len=8), parameter :: dsetvbinnedname = "velocity"
! Declare HDF5 IDs
integer(HID_T) :: outfile_id ! File identifier
integer(HID_T) :: dsetv_id
integer(HID_T) :: dataspacev_id
! Dataset dimensions in the file
integer(HSIZE_T), dimension(2) :: binned_vdims = (/20,1/) ! Chunk dimensions
integer(HSIZE_T), dimension(2) :: export_binned_vdims = (/20,10/) ! File dimensions
! Data buffers
integer, dimension(20) :: binned_vdata ! Chunk dimension
! Chunk parameters
integer(HSIZE_T), dimension(2) :: voffset, vchunkcount
integer :: rank = 2
integer :: error ! HDF5 error flag
! Initialize FORTRAN interface
call h5open_f(error)
! Create a new file using the default properties.
call h5fcreate_f(outfilename, H5F_ACC_TRUNC_F, outfile_id, error)
! Create the data space for the binned dataset.
call h5screate_simple_f(rank, export_binned_vdims, dataspacev_id, error)
! Create the chunked dataset.
call h5dcreate_f(outfile_id, dsetvbinnedname, H5T_NATIVE_INTEGER, dataspacev_id, dsetv_id, error)
! Select hyperslab
voffset = (/0,0/)
vchunkcount = (/20,1/)
call h5sselect_hyperslab_f(dataspacev_id, H5S_SELECT_SET_F, voffset, vchunkcount, error)
binned_vdata = 5 ! Write a column of '5's
! Write the data to the dataset.
call h5dwrite_f(dsetv_id, H5T_NATIVE_INTEGER, binned_vdata, export_binned_vdims, error, file_space_id=dataspacev_id)
! Close dataspace, dataset, and file
call h5dclose_f(dsetv_id, error)
call h5sclose_f(dataspacev_id, error)
call h5fclose_f(outfile_id, error)
call h5close_f(error)
end program hdf5_test

You haven't specified a memory dataspace. With the Fortran HDF5 API, the default identifier in the absence of an explicit memory dataspace argument is H5S_ALL_F, which means that the memory dataspace will be the same as the provided file dataspace.
Consequently, when you provide an dataspace that is offset into the file, HDF5 writes values from memory that is similarly offset, and this memory is beyond the bounds of your array.
Create another dataspace for the data in memory using a similar call to h5create_simple_f as you have now, and provide that to the h5dwrite_f call as the mem_space_id argument.

As suggested by others, I needed to use a memory dataspace. Working code, provided by Barbara at the HDF Group, is below.
program hdf5_test
use hdf5
implicit none
! HDF5 file and path names
character(len=10), parameter :: outfilename = "OutData.h5" ! Output file name
character(len=8), parameter :: dsetvbinnedname = "velocity"
! Declare HDF5 IDs
integer(HID_T) :: outfile_id ! File identifier
integer(HID_T) :: dsetv_id
integer(HID_T) :: dataspacev_id
integer(HID_T) :: memspace
! Dataset dimensions in the file
integer(HSIZE_T), dimension(2) :: binned_vdims = (/20,1/) ! Chunk dimensions
integer(HSIZE_T), dimension(2) :: export_binned_vdims = (/20,10/) ! File dimensions
! Data buffers
integer, dimension(20,1) :: binned_vdata ! Chunk dimension
! Chunk parameters
integer(HSIZE_T), dimension(2) :: voffset, vchunkcount
integer :: rank = 2,i, j
integer :: error ! HDF5 error flag
! Initialize FORTRAN interface
call h5open_f(error)
! Create a new file using the default properties.
call h5fcreate_f(outfilename, H5F_ACC_TRUNC_F, outfile_id, error)
! Create the data space for the binned dataset.
call h5screate_simple_f(rank, export_binned_vdims, dataspacev_id, error)
! Create the chunked dataset.
call h5dcreate_f(outfile_id, dsetvbinnedname, H5T_NATIVE_INTEGER, dataspacev_id, dsetv_id, error)
! Create the memory space for the selection
call h5screate_simple_f(rank, binned_vdims, memspace, error)
! Select hyperslab
voffset(1) = 0
vchunkcount = (/20,1/)
DO i = 1, binned_vdims(1)
DO j = 1, binned_vdims(2)
binned_vdata(i,j) = 5
END DO
END DO
DO i= 1, export_binned_vdims(2)
voffset(2) = i-1
call h5sselect_hyperslab_f(dataspacev_id, H5S_SELECT_SET_F, voffset, vchunkcount, error)
! Write the data to the dataset.
call h5dwrite_f(dsetv_id, H5T_NATIVE_INTEGER, binned_vdata, binned_vdims, error, memspace, dataspacev_id)
call h5sclose_f(dataspacev_id, error)
call h5dget_space_f(dsetv_id, dataspacev_id, error)
END DO
! Close dataspace, dataset, and file
call h5dclose_f(dsetv_id, error)
call h5sclose_f(memspace, error)
call h5fclose_f(outfile_id, error)
call h5close_f(error)
end program hdf5_test

Related

Calling a Matlab function in Fortran

Using the MATLAB Engine API for Fortran, I am trying to call a simple MATLABfunction from a Fortran code.
I followed the fengdemo example found here.
It worked, so I want to adapt my Fortran code to call a specific Matlab script I wrote.
My MATLAB script call_fortran.m is very simple: it takes x as an entry and multiplies it by 2:
function multiply = call_fortran(x)
multiply = 2*x;
end
I want my FORTRAN code to generate a variable my_x, open a MATLAB session, send the variable to the workspace, apply the function call_fortran and display the result. Using the fengdemo.f code, I wrote :
program main
C Declarations
implicit none
mwPointer engOpen, engGetVariable, mxCreateDoubleMatrix
mwPointer mxGetPr
mwPointer ep, my_x ! ep: variable linked to engOpen, starting a Matlab session, my_x: variable que je veux donner a Matlab
double precision my_x
integer engPutVariable, engEvalString, engClose
integer temp, status
mwSize i
my_x = 6
ep = engOpen('matlab ')
if (ep .eq. 0) then
write(6,*) 'Can''t start MATLAB engine'
stop
endif
C Place the variable my_x into the MATLAB workspace
status = engPutVariable(ep, 'my_x', my_x)
C
if (status .ne. 0) then
write(6,*) 'engPutVariable failed'
stop
endif
! My issue now is to call the correct Matlab script
! nlhs = 1
! plhs = 1
! nrhs = 1
! prhs = 1
! integer*4 mexCallMATLAB(nlhs, plhs, nrhs, prhs, functionName)
So I have my my_x, I send it to MATLAB, but how do I apply the call_fortran.m function and get the new value of my_x?
You have two fundamental errors with this code. You do not use the correct variable type for my_x, and you cannot call mexCallMATLAB( ) from an Engine application (that can only be used in mex routines). Let's fix these.
First, the my_x variable needs to be an mxArray, not a double precision variable. There are various ways to do this, but for a scalar, the easiest way to create this array is as follows:
mwPointer, external :: mxCreateDoubleScalar
mwPointer my_x
my_x = mxCreateDoubleScalar(6.d0)
Then you can pass this to the MATLAB Engine per your current code. To call your function in the MATLAB Engine workspace, you need to evaluate a string there:
integer*4, external :: engEvalString
integer*4 status
status = engEvalString( ep, 'result = call_fortran(my_x)' )
The result should display in the Engine workspace since we did not terminate the string with a semi-colon. If you want to get the result back into your Fortran code, you would need to do something like this:
mwPointer, external :: engGetVariable
mwPointer result
result = engGetVariable( ep, 'result' )
The result inside your Fortran code will be an mxArray. To extract the number there are various ways, but for a scalar it would be easiest to just do as follows (the real*8 is used instead of double precision to match the MATLAB API signature in the doc exactly):
real*8, external :: mxGetScalar
real*8 myresult
myresult = mxGetScalar(result)
To avoid memory leaks, once you are done with the mxArray variables you should destroy them. E.g.,
call mxDestroyArray(my_x)
call mxDestroyArray(result)
Having written all this, are you sure you want to create MATLAB Engine applications, and not mex routines? Mex routines are generally easier to work with and don't involve extra data copies to pass variables back & forth.

Matlab from Fortran - problems transferring big matrix

I have to call Matlab from Fortran and execute a program there. I have a large 3xN (N is around 2500) matrix of data, which needs to be transferred to Matlab. I noticed some discrepancies in the data - the last line of the Fortran matrix becomes first line in Matlab (other lines stay however on their place, shifted down by 1), and this line also looses the first value.
Like this - In Fortran
1.1 1.2 1.3
2.1 2.2 2.3
.....
1999.1 1999.2 1999.3
2000.1 2000.2 2000.3
becomes in Matlab
0.0 2000.2 2000.3
1.1 1.2 1.3
2.1 2.2 2.3
.....
1999.1 1999.2 1999.3
I cant understand what is going wrong somehow.. Spent several hours...
node_xyz_ini = mxCreateDoubleMatrix(M, N, 0) ! M, N - dimensions
call mxCopyReal8ToPtr(CoordSet, mxGetPr(node_xyz_ini), M*N)
I use Octave rather than matlab. With that as a caveat, here is an example of what I use, this for double precision twod arrays:
MODULE IO
use, intrinsic :: iso_c_binding
!! use c_float,c_double, c_double_complex, c_int,c_ptr
implicit none
real (c_double), allocatable :: x(:,:),h(:),f(:)
integer (c_int),allocatable :: t(:,:)
integer (c_int) :: nx,ne
contains
Subroutine Write_Array_RDP(varname,variable)
implicit none
integer (c_int) :: kx,ky,sh(2),ncol,nrow
character(len=7),intent(in) :: varname
character(:),allocatable :: wrtfmt
character(range(ncol)) :: res
real(c_double),intent(in) :: variable(:,:)
open(unit=10,file=varname,form="formatted",status="replace",action="write")
write(10,fmt="(A)")"# created by ?? "
sh=shape(variable)
ncol=sh(2);nrow=sh(1)
write(10,fmt="(A,A)")"# name: ",varname
write(10,fmt="(A)")"# type: matrix"
write(10,fmt="(A,i0)")"# rows: ",nrow
write(10,fmt="(A,i0)")"# columns: ",ncol
write(res,'(i0)') ncol
wrtfmt="("//trim(res)//"(e20.12))"
do ky=1,nrow
write(10,fmt=wrtfmt)(variable(ky,kx),kx=1,ncol)
end do
write(10,*)" "
write(10,*)" "
close(10)
End Subroutine Write_Array_RDP
END MODULE IO
Program Main
use IO
implicit none
real (c_double),allocatable :: DPArray(:,:)
allocate(DPArray(3,3))
DPArray=reshape((/1.0d0,2.0d0,3.0d0,1.0d0,2.0d0,3.0d0,1.0d0,2.0d0,3.0d0/),(/3,3/))
Call Write_Array_RDP('DPArray',DPArray)
End Program Main
I compile and link with 'gfortran name.f90' then run with ./a.out. The file DPArray has been created. Then in Octave :
load DPArray
DPArray
produces the output:
1 1 1
2 2 2
3 3 3
I have found it necessary to recode the Write subroutine for different variable types (Write_Array_CMPLX, Write_Array_INT) etc...

'pie' function in MATLAB gives "undefined function 'cos'" error

I wrote a function, wins_plot, to read the scoreboard from a file and store the player's name, number of plays, wins, & losses. I stored all those using struct. I loop over the file, store each line in line, textscan for everything I need from line, and then iterate i (initially == 1) as I go to expand my array of structures. A snippet from the code to represent what I am saying:
c = textscan(line, '%s %s %d %d %d');
player(i).firstName = c{1};
player(i).lastName = c{2};
player(i).plays = c{3};
player(i).wins = c{4};
player(i).losses = c{5};
After all the file has been scanned and stored, I then write this code to extract the number of wins of each player and store it in X and then finally use the pie function to represent the values in X
for n=1:(i-1)
X(n) = player(n).wins;
end
pie(X);
I get a wall of error after:
Undefined function 'cos' for input arguments of type 'int32'.
Error in pol2cart (line 22) x = r.*cos(th);
Error in pie (line 99)
[xtext,ytext] = pol2cart(theta0 + x(i)*pi,1.2);
Error in wins_plot (line 30) pie(X);
I have no clue what might be wrong. Any help would be greatly appreciated. Please keep in mind that I only just started learning MATLAB today so my knowledge of it is very limited (and I have R2013a). Thank you in advance!
The numbers got read as int32, but when you call pie, it requires them to be double to do the computation. So, when you call pie, try casting the values to double. Try this,
pie(double(X));

Calculating a checksum of a real array in Fortran

I have a large array in Fortran:
real, dimension(N) :: arr
And I need to check if the array is exactly the same in different runtimes of the program. To do this, I wanted to create a checksum of the array to compare. However, I don't know which algorithm to implement. I have looked at Flether's and Adler's algorithm, but have trouble reading the C syntax provided in the examples I found. And also, I don't know how to implement them with Reals instead of chars/integers.
In the C implementations I have found they return:
return (b << 16) | a;
But I don't know how to implement the b << 16 part in Fortran, or if this translates well to reals.
I finally solved the issue by implementing Adler-32 in Fortran:
subroutine test_hash(var)
implicit none
real, dimension(N), intent(in) :: var
integer, dimension(N) :: int_var
integer :: a=1, b=0, i=1, mod_adler=65521, hash = 0
int_var = TRANSFER(var, a, nijk)
do i= 1, NIJK
a = MOD(a + int_var(i), mod_adler)
b = MOD(b+a, mod_adler)
end do
hash = ior(b * 65536, a)
print*, hash
end subroutine test_hash
I ended up using the Fortran intrinsic Transfer function to convert the 32bit reals to 32bit integers, since that's what the algorithm relies on. After this I perform the standard loop. Use the IOR function as suggested by #VladimirF and represented the b<<16 as b * 65536 described by #ja72.
Finally I'll be able to print the hash to the console.
The reason for implementing it this way was because it's faster in use than opening a file, computing the checksum per file. The main reason for this is because there are many variables I need to check which switch often since I'm only using this for debugging purposes.
A modified version of Lars accomplishes the same without a large temporary array. Also, in Fortran, initializing the variable at declaration time implies the "save" attribute, which is not desirable in this case.
function hash_real_asz(var,size_var) result(hash)
implicit none
integer(8) :: hash
real(8), dimension(*), intent(in) :: var
integer, intent(in) :: size_var
integer(4) :: a,b,i,j
integer(4), parameter :: mod_adler = 65521
integer(4), allocatable :: tmp(:)
a = 1
b = 0
do i= 1, size_var
tmp = transfer(var(i), [0]) ! tmp will be an integer array sufficient to hold var(i)
do j = 1,size(tmp)
a = MOD(a+tmp(j), mod_adler)
b = MOD(b+a, mod_adler)
end do
end do
hash = ior(b * 65536, a)
end function

How to read binary file written by Matlab in Fortran?

I want to read an array of double precision values written in a binary file by Matlab into a code in Fortran (compiler gfortran), however my code to read it is not working. Can you please show me the correct way to do it?
Here is my Matlab code, which works.
a=[0.6557 0.0357 0.8491 0.9340 0.6787];
fid=fopen('ft1.bin','w');
fwrite(fid,a,'double');
fclose('all');
fid=fopen('ft1.bin','r');
a2=fread(fid,5,'double');
fclose('all');
a2
Here is my Fortran, code which returns an error when I try to read file ft1.bin
program code1
implicit none
double precision, dimension(5) :: a2
integer :: i
open(1,FILE="ft1.bin",FORM='UNFORMATTED',ACTION='READ')
read(1) a2
close(1)
print *, a2
end program code1
When I try to run it,
gfortran code1.f90 -o bb1
./bb1
At line 8 of file code1.f90 (unit = 1, file = 'ft1.bin')
Fortran runtime error: Unformatted file structure has been corrupted
One has to avoid the record based I/O with ACCESS="STREAM", e.g.,
PROGRAM test
IMPLICIT NONE
INTEGER, PARAMETER :: dp = KIND(1D0)
INTEGER :: funit, io_stat
REAL(dp) :: a(5)
OPEN(NEWUNIT = funit, FILE = 'ft1.bin', STATUS = "OLD", ACCESS = "STREAM", FORM = "UNFORMATTED", IOSTAT = io_stat)
READ(funit, IOSTAT = io_stat) a
WRITE(*, *) a
CLOSE(funit)
END PROGRAM