I am trying to write a Julia macro that transforms this:
[par1!( par2(d1,d2)+par3(d1,d2) ,d1,d2,dfix3) for d1 in DIM1, d2 in DIM2]
(not very inspiring) into something much more readable, like this:
#meq par1!(d1 in DIM1, d2 in DIM2, dfix3) = par2(d1,d2)+par3(d1,d2)
where par1!() is a function to set some multi-dimensional data and par2() is a getData()-type of function.
I am trying to implement it using a macro, but as I am on my first experience with julia marcro, I'm not sure how to "assemble" the final expression from the various pieces..
Here is what I done so far:
macro meq(eq)
# dump(eq)
lhs_par = eq.args[1].args[1]
rhs = eq.args[2]
lhs_dims = eq.args[1].args[2:end]
loop_counters = [d.args[2] for d in lhs_dims if typeof(d) == Expr]
loop_sets = [d.args[3] for d in lhs_dims if typeof(d) == Expr]
loop_wholeElements = [d for d in lhs_dims if typeof(d) == Expr]
lhs_dims_placeholders = []
for d in lhs_dims
if typeof(d) == Expr
push!(lhs_dims_placeholders,d.args[2])
else
push!(lhs_dims_placeholders,d)
end
end
outExp = quote
[$(lhs_par)($(rhs),$(lhs_dims_placeholders ...)) for $(loop_wholeElements ...) ]
end
#show(outExp)
return outExp
end
However the above macro doesn't compile and returns a syntax error (“invalid iteration specification”) due to the for $(loop_wholeElements) part… indeed I don’t know how to treat the expressions in lhs_dims_placeholders and loop_wholeElements in order to “assemble” the expanded expression…
EDIT:
The example posted, with d1, d2 and dfix3, is only a specific case, but the macro should be able to handle whichever dimensions are looped for..
I think the macro up there does that, but I don't know how to build the final expression.. :-(
Instead of manually doing those hard-coded args matching stuff, we could use MacroTools.jl as a handy tool for template matching:
julia> using MacroTools
julia> macro meq(ex)
#capture(ex, f_(d1_ in dim1_, d2_ in dim2_, dfix3_) = body__)
ret = :([$f($(body[]), $d1, $d2, $dfix3) for $d1 in $dim1, $d2 in $dim2])
end
#meq (macro with 1 method)
julia> prettify(#macroexpand #meq par1!(d1 in DIM1, d2 in DIM2, dfix3) = par2(d1,d2)+par3(d1,d2))
:([(Main.par1!)((Main.par2)(lobster, redpanda) + (Main.par3)(lobster, redpanda), lobster, redpanda, Main.dfix3) for lobster = Main.DIM1, redpanda = Main.DIM2])
UPDATE:
The desired final expression is a comprehension, it seems that for some reason Julia couldn't figure out for expr(where $expr #=> XXX in XXX) is a comprehension. The workaround is directly using its lowered form:
julia> using MacroTools
julia> par1(a, b, c, d) = a + b + c + d
par1 (generic function with 1 method)
julia> par2(a, b) = a + b
par2 (generic function with 1 method)
julia> macro meq(ex)
#capture(ex, par_(dims__) = rhs_)
loopElements = []
dimsPlaceholders = []
for d in dims
#capture(d, di_ in DIMi_) || (push!(dimsPlaceholders, d); continue)
# push!(loopElements, x)
push!(loopElements, :($di = $DIMi))
push!(dimsPlaceholders, di)
end
ret = Expr(:comprehension, :($par($(rhs),$(dimsPlaceholders...))), loopElements...)
end
#meq (macro with 1 method)
julia> prettify(#macroexpand #meq par1!(d1 in DIM1, d2 in DIM2, dfix3) = par2(d1,d2)+par3(d1,d2))
:($(Expr(:comprehension, :((Main.par1!)(begin
(Main.par2)(bee, wildebeest) + (Main.par3)(bee, wildebeest)
end, bee, wildebeest, Main.dfix3)), :(bee = Main.DIM1), :(wildebeest = Main.DIM2))))
julia> #meq par1(m in 1:2, n in 4:5, 3) = par2(m,n) + par2(m,n)
2×2 Array{Int64,2}:
18 21
21 24
Note that, the variable scope of d1,d2 in generated expression will be wrong if we use push!(loopElements, x) rather than push!(loopElements, :($di = $DIMi)). Let's wait for someone knowledgeable to give a thorough explanation.
If you do not want to rely on an external package for this, the solution I provided on the Julia discourse should also work
return :([$(Expr(:generator,:($(Expr(:call,lhs_par,rhs,lhs_dims_placeholders...))),loop_wholeElements...))])
The key is to use the :generator constructor to make the loop expression
Also, rhs can be replaced with rhs.args[n] in order to eliminate the quote block and insert the expression directly.
This then produces the exact expression:
:([(par1!(par2(d1, d2) + par3(d1, d2), d1, d2, dfix3) for d1 in DIM1, d2 in DIM2)])
EDIT:
Alright, so I went ahead and tested this:
return Expr(:comprehension,Expr(:generator,Expr(:call,lhs_par,rhs.args[2],lhs_dims_placeholders...),loop_wholeElements...))
end
Then computing the result like this
meq(:(par1!(d1 = 1:2, d2 = 1:2, 3) = par2(d1,d2)+par3(d1,d2))) |> eval
I am interested in getting the digits of a BigFloat in the form of bytes. I get a very strange error that I cannot debug. I provide a minimal example where the error appears.
function floatToBytes(x::BigFloat)
ret = zeros(UInt8, 4)
xs = significand(x)/2
b = UInt8(0)
for i = 1:4
xs *= 256
b = trunc(UInt8, xs)
ret[i] = b
xs -= b
end
return ret
end
println( floatToBytes(BigFloat(0.9921875001164153)) )
println( floatToBytes(BigFloat(0.9960937501164153)) )
What I get when running this is
UInt8[0xfe, 0x00, 0x00, 0x00]
ERROR: LoadError: InexactError()
Stacktrace:
[1] trunc(::Type{UInt8}, ::BigFloat) at ./mpfr.jl:201
etc.
It seems that it doesn't want to turn 255 into a UInt8. I can circumvent the problem by defining the function as
function floatToBytes(x::BigFloat)
ret = zeros(UInt8, 4)
xs = significand(x)/2
b = UInt8(0)
for i = 1:4
xs *= 256
try
b = trunc(UInt8, xs)
catch
b = trunc(UInt8, xs-1)+UInt8(1)
end
ret[i] = b
xs -= b
end
return ret
end
But this is highly unsatisfactory. What is going on here?
The problem looks like a bug in trunc for BigFloat. The problem is the current code does (typemin(T) <= x <= typemax(T)) || throw(InexactError(:trunc, T, x)) which throws an error because x is larger than 255 which is the typemax.
It actually needs to do the trunc in BigFloat domain and then cast to T (and have the cast check for typemax).
I've opened an issue regarding this at: https://github.com/JuliaLang/julia/issues/24041
In the meantime, a solution could be to do:
UInt8(trunc(xs))
i.e. trunc first and cast later. For example:
julia> UInt8(trunc(BigFloat(0.9960937501164153)*256))
0xff
I'm getting warning that I don't understand.
I first run the following code:
type PDF{T <: Real}
𝛑::Vector{Float64} # K
μs::Matrix{T} # D x K
Σs::Vector{Matrix{T}} # K{D x D}
end
type Q{T <: Real}
w::Union{Float64, Vector{Float64}}
μpair::Union{Vector{T}, Matrix{T}}
Σpair::Union{Matrix{T}, Tuple{Matrix{T}, Matrix{T}} } # K{D x D}
end
type Smod{T <: Real}
H::Matrix{T} # D x D
Σs::Vector{Matrix{T}} # K{D x D}
qs::Vector{Q{T}}
end
type Scale{T <: Real}
μ::Vector{T} # D
Σ::Matrix{T} # D x D
end
type Parameters{T <: Real}
scale::Scale{T}
w::Vector{Float64}
maxNumCompsBeforeCompression::Integer
numComponentsAbsorbed::Integer
end
type KDE{T}
pdf::PDF{T}
smod::Smod{T}
params::Parameters{T}
end
And when after this I run the following snippet in IJulia
function initializeKDE{T <: Real}(x::Vector{T})
d = size(x,1)
T = typeof(x)
𝛑 = ones(Float64, 1)
μs = Array(T, d,1)
μs[:,1] = x
Σs = Array(Matrix{T}, 0)
pdf = PDF(𝛑, μs, Σs)
H = zeros(T, d,d)
qs = Array(Q{T}, 0)
smod = Smod(H, Σs, qs)
scale = Scale(x, H)
w = [0.0, 1.0]
maxNumCompsBeforeCompression = min(10, (0.5d^2 + d))
numComponentsAbsorbed = 0
params = Params(scale, w, maxNumCompsBeforeCompression, numComponentsAbsorbed)
kde = KDE(pdf, smod, params)
return kde::KDE
end
I get the following warning:
WARNING: local variable T conflicts with a static parameter in initializeKDE at In[4]:3.
where In[4]:3 corresponds to the 3rd line of the 2nd snippet.
Can anyone explain in human english what this warning is saying?
This is saying that you are trying to use T in two different ways: once as a "static parameter" and once as a local variable.
Firstly, you are using T as the parameter with which you are parametrising the function initializeKDE:
function initializeKDE{T <: Real}(x::Vector{T})
But then you are trying to redefine a new T in that third line:
T = typeof(x)
What are you trying to do here? If you are trying to define T to be the type of the elements that the vector x contains, then you should just delete this line and everything should just work -- T will automatically take the element type (eltype) of the vector that you pass to the initializeKDE function.
I've been trying to run a code using MPI I/O on a large number of cores. The time required for each core to read from and write to a single file (the same for all cores) increases with the number of cores used. I'm currently using 512 cores and this problem is making my project unfeasible. The problem appears, however, even when running on 8 cores; it then takes about 0.2 seconds to read the first real number in the file. On 32 cores it takes more then 30 seconds to write one real number. I'm running it here: https://www.msi.umn.edu/hpc/itasca. The following simple code generates exactly this problem (the counting of the number of elements in the file might seem unnecessary here but it is necessary in my actual code):
PROGRAM MAIN
USE MPI
IMPLICIT NONE
! INITIALIZING VARIABLES
REAL(8) :: A, B
INTEGER :: COUNT_IO, i, j, ST, GO, tag, t, nb_bytes, N, d_each, d_start, d_end, NN
REAL(8) :: time_start, time_end
! VARIABLES RELATED TO MPI
INTEGER :: ierror ! returns error messages from the mpi subroutines
INTEGER :: rank ! identification number of each processor
INTEGER :: nproc ! number of processors
INTEGER, DIMENSION(mpi_status_size):: status
INTEGER(kind= MPI_OFFSET_KIND ) :: offset
INTEGER :: fh ! file handle
! EXECUTABLE
! INITIALIZE THE MPI ENVIRONMENT
CALL MPI_INIT(ierror) ! initialize MPI
CALL MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierror) ! obtain rank for each node
CALL MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,ierror) ! obtain the number of nodes
CALL MPI_TYPE_SIZE(MPI_REAL8,nb_bytes,ierror)
CALL MPI_FILE_OPEN (MPI_COMM_WORLD,"file.dat",MPI_MODE_RDWR+MPI_MODE_UNIQUE_OPEN,MPI_INFO_NULL,fh,ierror)
NN = 2048
DO d_each=1,NN
IF (d_each*nproc>=NN) EXIT
END DO
d_start = rank*d_each+1
d_end = MIN((rank+1)*d_each,NN)
DO t = d_start,d_end
! READING ONE THREAD AT A TIME
tag = 1
GO = 0
IF (rank .gt. 0) THEN
CALL MPI_RECV (GO,1,MPI_INTEGER,rank-1,tag, MPI_COMM_WORLD ,status,ierror)
ENDIF
time_start = MPI_WTIME()
i = 0
ST = 0
COUNT_IO = 0
DO WHILE ((i .lt. 100000) .AND. (ST .eq. 0))
i = i+1
offset = nb_bytes*(i-1)
CALL MPI_FILE_READ_AT (fh,offset,A,1,MPI_REAL8,status,ierror)
IF (status(1) .eq. 0) THEN
COUNT_IO = i
ST = 1
ELSE
COUNT_IO = 0
END IF
ENDDO
N = (COUNT_IO - 1)
IF (N .gt. 0) THEN
offset = 0
CALL MPI_FILE_READ_AT (fh,offset,B,1,MPI_REAL8,status,ierror)
ENDIF
time_end = MPI_WTIME()
PRINT *, 'My rank is', rank, 'Time for read =',time_end-time_start
GO = 1
IF (rank .lt. nproc-1) THEN
CALL MPI_SEND (GO,1, MPI_INTEGER ,rank+1,tag, MPI_COMM_WORLD ,ierror)
ENDIF
CALL MPI_BARRIER(MPI_COMM_WORLD,ierror)
! WRITING ONE THREAD AT A TIME
tag = 2
GO = 0
IF (rank .gt. 0) THEN
CALL MPI_RECV (GO,1,MPI_INTEGER,rank-1,tag, MPI_COMM_WORLD ,status,ierror)
ENDIF
time_start = MPI_WTIME()
i = 0
ST = 0
COUNT_IO = 0
DO WHILE ((i .lt. 100000) .AND. (ST .eq. 0))
i = i+1
offset = nb_bytes*(i-1)
CALL MPI_FILE_READ_AT (fh,offset,A,1,MPI_REAL8,status,ierror)
IF (status(1) .eq. 0) THEN
COUNT_IO = i
ST = 1
ELSE
COUNT_IO = 0
END IF
ENDDO
N = (COUNT_IO - 1)
offset = nb_bytes*N
CALL MPI_FILE_WRITE_AT (fh,offset,0.0D0,1,MPI_REAL8,status,ierror)
time_end = MPI_WTIME()
PRINT *, 'My rank is', rank, 'Time for write =',time_end-time_start
GO = 1
IF (rank .lt. nproc-1) THEN
CALL MPI_SEND (GO,1, MPI_INTEGER ,rank+1,tag, MPI_COMM_WORLD ,ierror)
ENDIF
CALL MPI_BARRIER(MPI_COMM_WORLD,ierror)
ENDDO
CALL MPI_FILE_CLOSE (fh,ierror)
CALL MPI_FINALIZE(ierror)
END PROGRAM MAIN
The main thing to realize here is that you can read in the data in one fell swoop (or, if memory is a problem, in chunks - but it can be in much larger chunks than individual doubles!) and that you don't need to skip to the end of the file one double at a time.
Here's an example which will read in the data in arbitrary chunk sizes, processes the data as you will, and appends some data (in this case, everyone just adds 4 copies of their rank to the end of the file). For simplicity, little python scripts help with writing and displaying test data.
$ ./writedata.py
$ ./readdata.py
[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
15. 16. 17. 18. 19. 20. 21. 22. 23. 24.]
$ mpirun -np 3 ./usepario
rank: 0 got data: 0.000... 24.000
rank: 1 got data: 0.000... 24.000
rank: 2 got data: 0.000... 24.000
$ ./readdata.py
[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 0. 0. 0. 0. 1.
1. 1. 1. 2. 2. 2. 2.]
usepario.f90:
module pario
contains
function openFile(filename)
use mpi
implicit none
integer :: openFile, ierr
character(len=*) :: filename
integer(MPI_OFFSET_KIND) :: off = 0
call MPI_File_open(MPI_COMM_WORLD, filename, &
ior(MPI_MODE_RDWR, MPI_MODE_UNIQUE_OPEN), &
MPI_INFO_NULL, openFile, ierr)
call MPI_File_set_view(openFile, off, &
MPI_DOUBLE_PRECISION, MPI_DOUBLE_PRECISION, &
"native", MPI_INFO_NULL, ierr)
end function openFile
subroutine closeFile(fh)
use mpi
implicit none
integer :: fh, ierr
call MPI_File_close(fh, ierr)
end subroutine closeFile
function filesizedoubles(fh)
use mpi
implicit none
integer :: fh, ierr
integer(MPI_OFFSET_KIND) :: filesize, filesizedoubles
integer :: dblsize
call MPI_File_get_size(fh, filesize, ierr)
call MPI_type_size(MPI_DOUBLE_PRECISION, dblsize, ierr)
filesizedoubles = filesize / dblsize
end function filesizedoubles
subroutine getdatablock(fh, blocksize, datablock, datasize)
use mpi
implicit none
integer :: fh, ierr
integer :: blocksize, datasize
double precision, dimension(:) :: datablock
integer(MPI_OFFSET_KIND) :: fileloc
integer, dimension(MPI_STATUS_SIZE) :: rstatus
! you can also experiment with read_all for non collective/synchronous file
! access
call MPI_File_read(fh, datablock, blocksize, MPI_DOUBLE_PRECISION, &
rstatus, ierr)
call MPI_Get_count(rstatus, MPI_DOUBLE_PRECISION, datasize, ierr)
end subroutine getdatablock
subroutine eachappend(fh, filesize, numitems, newdata)
use mpi
implicit none
integer :: fh, numitems
integer(MPI_OFFSET_KIND) :: filesize
double precision, dimension(:) :: newdata
integer :: rank, ierr
integer(MPI_OFFSET_KIND) :: offset
call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
offset = filesize + rank*numitems
call MPI_File_write_at_all(fh, offset, newdata, numitems, &
MPI_DOUBLE_PRECISION, &
MPI_STATUS_IGNORE, ierr)
end subroutine eachappend
end module pario
program usepario
use mpi
use pario
implicit none
integer :: fileh
integer, parameter :: bufsize=1000, newsize=4
integer(MPI_OFFSET_KIND) :: filesize
double precision, allocatable, dimension(:) :: curdata, newdata
integer :: datasize
integer :: rank, ierr
call MPI_Init(ierr)
call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
allocate(curdata(bufsize))
fileh = openFile("data.dat")
filesize = filesizedoubles(fileh)
do
call getdatablock(fileh, bufsize, curdata, datasize)
!!
!! process data here
!!
!! do i=1,datasize
!! ...dostuff...
!! end do
!!
print '(1X,A,I3,A,F8.3,A,F8.3)', 'rank: ', rank, ' got data: ', curdata(1), '...', curdata(datasize)
if (datasize /= bufsize) exit
end do
deallocate(curdata)
allocate(newdata(newsize))
newdata = rank
call eachappend(fileh, filesize, newsize, newdata)
call closeFile(fileh)
call MPI_Finalize(ierr)
end program usepario
writedata.py:
#!/usr/bin/env python
import numpy
numdoubles = 25
data = numpy.arange(numdoubles,dtype=numpy.float64)
data.tofile("data.dat")
readdata.py:
#!/usr/bin/env python
import numpy
data = numpy.fromfile("data.dat",dtype=numpy.float64)
print data
i am new in matlab and search everything. I am writing a the function. i could not able to understand why this error is comning :"Input argument "b" is undefined." . shall i intialise b =0 ? whereas it is the parameter coming from input console. my code:
function f = evenorodd( b )
%UNTITLED2 Summary of this function goes here
%zohaib
% Detailed explanation goes here
%f = b;%2;
f = [0 0];
f = rem(b,2);
if f == 0
disp(b+ 'is even')
else
disp(b+ 'is odd')
end
console:
??? Input argument "b" is undefined.
Error in ==> evenorodd at 6
f = rem(b,2);
From what I see, this is what you are trying to do:
function f = evenorodd( b )
f = rem(b,2);
if f == 0
fprintf('%i is even\n', b)
else
fprintf('%i is odd\n', b)
end
=======================
>> evenorodd(2);
2 is even
No need to initialize f as [0,0].
In MATLAB, you cant concatenate a number and string with + operator. Use fprintf.
The above function evenorodd takes one argument (integer) and returns 0 or 1.