Confused about get_rays function in NeRF - neural-network

I've been trying to understand NeRF. I finished reading the paper(Tancik) and watched some of the videos. I have been looking at some parts of the code. However, I can't quite wrap my head around what the get_rays function does in terms of the code. Could anybody just run through line-by-line about what each line in the the get_rays function is supposed to do?
def get_rays(H,W , focal, c2w): #c2w is pose
i, j = tf.meshgrid(tf.range(W, dtype=tf.float32), tf.range(H, dtype=tf.float32), indexing='xy')
dirs = tf.stack([(i-W*.5)/focal, -(j-H*.5)/focal, -tf.ones_like(i)], -1)
rays_d = tf.reduce_sum(dirs[..., np.newaxis, :] * c2w[:3,:3], -1)
rays_o = tf.broadcast_to(c2w[:3,-1], tf.shape(rays_d))
return rays_o, rays_d

It creates two lists, rays_o represents points where rays originate (camera centre) and rays_d represents direction vectors of each ray casting through the centre of every pixel of the camera. In this case, all values in the rays_o are the same because the function gets rays from a single camera.

Related

How to get current frame from Animated Tile/Tilemap.animationFrameRate in Unity

I am using tilemaps and animated tiles from the 2dExtras in unity.
My tiles have 6 frames, at speed=2f, and my tilemap frame rate is 2.
New tiles placed always start on frame 1 and then immediately jump to the current frame of the other tiles already placed, the tilemap is keeping every tile at the same pace, which is working as I want.
However I would like the newly placed tiles to start at the frame the others are currently on,(instead of placing a tile that jumps from frame 1 to frame 4) I would like the new tile to start on frame 4
I've found how to pick the frame I want to start on, however I am having trouble retrieving which frame the animation is currently on, so I was wondering how exactly can I access the current frame of animation of a given tilemap ( Or a given tile, I can create a dummy tile and just read the info out of it, how can I get the current frame of an animated tile? )
The animated tilemaps feature seems to lack the feature to retrieve this information, also when I try tilemap.getsprite it always returns the first frame of the sequence(does not return the sprite currently displayed), and there doesn't seem to be any method to poll info from tilemap.animationFrameRate.
I thought another method would be to set a clock and sync it to the rate of the animation but since I can't get the exact framerate duration the clock eventually goes out of sync.
Any help would be appreciated!
I found a way to solve this question. But it's not 100% insurance.
First of all, I used SuperTile2Unity. That doesn't seem to be the point.
private void LateUpdate()
{
// I use this variable to monitor the run time of the game
this.totalTime += Time.deltaTime;
}
private void func()
{
// ...
TileBase[] currentTiles = tilemap.GetTilesBlock(new BoundsInt(new Vector3Int(0, 0, 0), new Vector3Int(x, y, 1)));
Dictionary<string, Sprite> tempTiles = new Dictionary<string, Sprite>();
//I use SuperTiled2Unity. But it doesn't matter, the point is to find animated tile
foreach (SuperTiled2Unity.SuperTile tile in currentTiles)
{
if (tile == null)
{
continue;
}
if (tile.m_AnimationSprites.Length > 1 && !tempTiles.ContainsKey(tile.name))
{
// find animated tile current frame
// You can easily find that the way SuperTile2Unity is used to process animation is to generate a sprite array based on the time of each frame set by Tiled animation and the value of AnimationFrameRate parameter.
// The length of array is always n times of AnimationFrameRate. You can debug to find this.
tempTiles.Add(tile.name, tile.m_AnimationSprites[GetProbablyFrameIndex(tile.m_AnimationSprites.Length)]);
}
}
//...
}
private int GetProbablyFrameIndex(int totalFrame)
{
//According to the total running time and the total length of tile animation and AnimationFrameRate, the approximate frame index can be deduced.
int overFrameTime = (int)(totalTime * animationFrameRate);
return overFrameTime % totalFrame;
}
I have done some tests. At least in 30 minutes, there will be no deviation in animations, but there may be a critical value. If the critical time is exceeded, there may be errors. It depends on the size of AnimationFrameRate and the accumulation mode of totalTime. After all, we don't know when and how the unity deals with animatedTile.
You could try using implementation presented in [1] which looks as follows:
MyAnimator.GetCurrentAnimatorClipInfo(0)[0].clip.length * (MyAnimator.GetCurrentAnimatorStateInfo(0).normalizedTime % 1) * MyAnimator.GetCurrentAnimatorClipInfo(0)[0].clip.frameRate;
[1] https://gamedev.stackexchange.com/questions/165289/how-to-fetch-a-frame-number-from-animation-clip

Copying data between metal textures of different shapes

I am converting two trained Keras models to Metal Performance Shaders. I have to reshape the output of the first graph and use it as input to the second graph. The first graph's output is an MPSImage with "shape" (1,1,8192), and the second graph's input is an MPSImage of "shape" (4,4,512).
I cast graph1's output image.texture as a float16 array, and pass it to the following function to copy the data into "midImage", a 4x4x512 MPSImage:
func reshapeTexture(imageArray:[Float16]) -> MPSImage{
let image = imageArray
image.withUnsafeBufferPointer { ptr in
let width = midImage.texture.width
let height = midImage.texture.height
for slice in 0..<128{
for w in 0..<width{
for h in 0..<height{
let region = MTLRegion(origin: MTLOriginMake(w, h, 0),
size: MTLSizeMake(1, 1, 1))
midImage.texture.replace(region: region, mipmapLevel: 0, slice: slice, withBytes: ptr.baseAddress!.advanced(by: ((slice * 4 * width * height) + ((w + h) * 4)), bytesPerRow: MemoryLayout<Float16>.stride * 4, bytesPerImage: 0)
}
}
}
}
return midImage
}
When I pass midImage to graph2, the output of the graph is a square with 3/4 garbled noise, 1/4 black in the bottom right corner. I think I am not understanding something about the MPSImage slice property for storing extra channels. Thanks!
Metal 2d texture arrays are nearly always stored in a Morton or “Z” ordering of some kind. Certainly MPS always allocates them that way, though I suppose on MacOS there may be a means to make a linear 2D texture array and wrap a MPSImage around it. So, without undue care, direct access of a 2d texture array backing store is going to result in sadness and confusion.
The right way to do this is to write a simple Metal copy kernel. This gives you storage order independence and you don’t have to wait for the command buffer to complete before you can do the operation.
A feature request in Radar might also be warranted. Please also look in the latest macOS / iOS seed to see if Apple recently added a reshape filter for you.

Compare value of property of all instances of class in python

I'm trying to write an object oriented program (as a learning exercise, I know there may be simpler ways to do it) in which beads bounce around a 2D plane bounded by a ring. Each bead is an object defined by a class ball. In setting the initial positions of the balls I need to check that no other ball has already been placed at the same x and y coordinates.
#Class for the beads
class ball:
NumbBalls = 0
#Constructor
def __init__(self,Beads):
self.ball = sphere(pos = vector(0,0,0), radius = radiusBall,color=color.red)
ball.NumbBalls += 1
self.ball.pos = self.createInitialPosition(Beads)
#Assign ball its initial position
def createInitialPosition(self,other):
#CODE to compare self.ball.pos and other.ball.pos which returns a unique position coordinate
#Main program
NumbBeads = 100
Beads = []#Create empty list for all the beads
#create balls in random initial positions
Beads = [ball(Beads) for item in range(NumbBeads)]
I can get this to work if I create two objects bead1 and bead2 and then pass bead1 and bead2 as arguments ie bead2 = ball(bead1) but how do I do this if I am generating a list of beads and want all of them to be compared with self. Hopefully that makes sense.
Perhaps rather than the approach you are currently taking, you should consider something along these lines (of course, with the necessary changes to your class definition/methods):
N = 100
initialPositions = []
while len(initialPositions) <= N:
newPosition = generateRandomPosition()
if newPosition in initialPositions:
pass
else:
initialPositions.append(newPosition)
ballList = [ ball(p) for p in initialPositions ]
In other words, generate a list of initial positions outside of the creation of your objects, and do whatever validation/restriction you need during that creation. Then just create your objects from that list. If N is really large, you might want to consider a dictionary (or a mutable set of some sort) instead of a list for initialPositions, to help speed up the membership testing, but it's still the same concept...

World.QueryAABB giving incorrect results in libgdx

I'm trying to implement mouse selection for my game. When I QueryAABB it looks like it's treating objects much larger than they really are.
Here's what's going on in the image
The blue box is an actor containing a body that I'd like to select
The outline on the blue box is drawn by Box2DDebugRenderer
The mouse selects a region on the screen (white box), this is entirely graphical
The AABB is converted to meters and passed to QueryAABB
The callback was called for the blue box and turned it red
The green outline left behind is a separate body to check if my conversions were correct, this is not used for the actual selection process
It seems to be connected to my meter size, the larger it is, the more inaccurate the result is. At 1 meter = 1 pixel it works perfectly.
Meter conversions
val MetersToPixels = 160f
val PixelsToMeters = 1/MetersToPixels
def toMeters(n: Float) = n * PixelsToMeters
def toPixels(n: Float) = n * MetersToPixels
In the image I'm using MetersToPixels = 160f so the inaccuracy is more visible, but I really want MetersToPixels = 16f.
Relevant selection code
val x1 = selectPos.x
val y1 = selectPos.y
val x2 = getX
val y2 = getY + getHeight
val (l,r) =
if (x2 < x1)
(x2,x1)
else
(x1,x2)
val (b,t) =
if (y2 < y1)
(y2,y1)
else
(y1,y2)
world.QueryAABB(selectCallback, toMeters(l),toMeters(b), toMeters(r),toMeters(t))
This code is inside the act method of my CursorActor class. And selectPos represents the initial point where the use pressed down the left mouse button and getX and getY are Actor methods giving the current position. The next bit sorts them because they might be out of order. Then they are converted to meters because they are all in pixel units.
selectCallback: QueryCallback
override def reportFixture(fixture: Fixture): Boolean = {
fixture.getBody.getUserData match {
case selectable: Selectable =>
selected += selectable
true
case _ => true
}
}
Selectable is a trait that sets a boolean flag internally after the query which helps determines the color of the blue box. And selected is a mutable.HashSet[Selectable] defined inside of CursorActor.
Other things possibly worth noting
I'm new to libgdx and box2d.
The camera is scaled x2
My Box2DDebugRenderer uses the camera's combined matrix multiplied by MetersToPixels
From what I was able to gather, QueryAABB is naturally inaccurate for optimization. However, I've hit a roadblock with libgdx because it doesn't have any publicly visible function like b2testOverlap and from what I understand, there's no plan for there to be one any time soon.
I think my best solution would probably be to use jbox2d and pretend that libgdx's physics implementation doesn't exist.
Or as noone suggested I could add it to libgdx myself.
UPDATE
I decided to go with a simple solution of gathering the vertices from the fixture's shape and using com.badlogic.gdx.math.Intersector against the vertices of the selection. It works I guess. I may stop using QueryAABB all together if I decide to switch to using a sensor for the select box.

Creating An "Autopilot" For Lander in Perl

I'm using Perl to create a simple Lunar Lander game. All of the elements work (i.e. graphical interface, user implemented controls, etc), but I cannot seem to get the "AutoPilot" function to work. This function should fly the lander to a spot that it can land (or a spot designated as a target for landing), and then safely land there. The restrictions placed on landing are the slope of the place the lander lands and the velocity that the lander has when landing. The only file I can change is AutoPilot.pm. I will post the code I am allowed to work with:
package AutoPilot;
use strict;
use warnings;
# use diagnostics;
=head1 Lunar Lander Autopilot
The autopilot is called on every step of the lunar lander simulation.
It is passed state information as an argument and returns a set of course
correction commands.
The lander world takes the surface of the moon (a circle!)
and maps it onto a rectangular region.
On the x-axis, the lander will wrap around when it hits either the
left or right edge of the region. If the lander goes above the maximum
height of the world, it escapes into the space and thus fails.
Similarly, if the lander's position goes below 0 without ever landing
on some solid surface, it "sinks" and thus fails again.
The simulation is simple in the respect that if the langer goes at a high speed
it may pass through the terrain boundary.
The y-axis has normal gravitational physics.
The goal of the autopilot is to land the craft at (or near) the landing
zone without crashing it (or failing by leaving the world).
=head2 Interface in a nutshell
When the simulation is initialized, AutoPilot::Initialize() is called.
Every clock tick, AutoPilot::ComputeLanding() is called by the simulator.
For more explanation, see below.
=cut
# if you want to keep data between invocations of ComputeLanding, put
# the data in this part of the code. Use Initialize() to handle simulation
# resets.
my $call_count = 0;
my $gravity;
my ($x_min, $y_min, $x_max, $y_max);
my ($lander_width, $lander_height, $center_x, $center_y);
my $target_x;
my ($thrust, $left_right_thrust);
my ($max_rotation, $max_left_right_thrusters, $max_main_thruster);
my $ascend_height = 980;
=head1 AutoPilot::Initialize()
This method is called when a new simulation is started.
The following parameters are passed to initialize:
$gravity, a number, describing the gravity in the world
$space_boundaries, a reference to an array with 4 numerical
elements, ($x_min, $y_min, $x_max, $y_max), describing
the world boundaries
$target_x, a number representing the target landing position
$lander_capabilities, a reference to an array with
5 elements,
($thrust, $left_right_thrust, $max_rotation, $max_left_right_thrusters, $max_main_thruster),
describing the capabilities of the lander.
$lander_dimensions, a reference to an array with
4 elements,
($lander_width, $lander_height, $center_x, $center_y),
describing the dimensions of the lander.
=head2 Details
=head3 Dimensions
The dimensions are given in 'units' (you can think of 'units' as meters).
The actual numbers can take any real value, not only integers.
=head4 World dimensions
The lander world is a square region with a lower left corner at
($x_min,$y_min) and an upper right corner at ($x_max, $y_max).
The measurement units of these dimensions will just be called units
(think about units as meters). By definition, $x_max>$x_min and
$y_max>$y_min.
The default values for the lower left and upper right corners
are (-800,0), and (800,1600), respectively.
=head4 Lander dimensions
The lander is $lander_width units wide and $lander_height high.
The coordinates of the lander are always specified with respect to its center.
The center of the lander relative to the lower left corner of the lander bounding box
is given by $center_x, $center_y. Thus, if ($x,$y) are the coordinates of the lander,
($x-$center_x,$y-$center_y) and ($x-$center_x+$lander_width,$y-$center_y+$lander_height)
specify the corners of the bounding box of the lander. (Think of the lander as completely
filling this box.) The significance of the bounding box of the lander is that a collision
occurs if the bounding box intersects with the terrain or the upper/lower edges of the world.
If a collision occurs, as described earlier, the lander might have just landed,
crashed or 'escaped' (and thus the lander failed).
The constraints on these values are: $lander_width>0, $lander_height>0,
$center_x>0, $center_y>0.
The default value for the width is 60 units, for the height it is 50,
for $center_x it is 30, for $center_y it is 25.
=head4 Forces
The gravitational force is:
$g
The thrust exerted by the engine when fired is:
$thrust
The thrust exerted by the left/right thrusters when fired is:
$left_right_thrust
=head4 Limits to the controls
Within a single timestep there are limits to how many degrees the
lander may rotate in a timestep, and how many times the side thrusters,
and main thruster, can fire. These are stored in:
$max_rotation, $max_left_right_thrusters, $max_main_thruster
=head4 Target
The target landing zone that the lander is supposed to land at:
$target_x
which returns
the string "any" if any safe landing site will do, or
a number giving the x-coordinate of the desired landing site.
Note: there is no guarantee that this is actually a safe spot to land!
For more details about how the lander is controlled, see AutoPilot::ComputeLanding.
=cut
sub Initialize {
my ($space_boundaries, $lander_capabilities,$lander_dimensions);
($gravity, $space_boundaries, $target_x, $lander_capabilities, $lander_dimensions) = #_;
($x_min, $y_min, $x_max, $y_max) = #{$space_boundaries};
( $thrust, $left_right_thrust, $max_rotation,
$max_left_right_thrusters, $max_main_thruster) = #{$lander_capabilities};
($lander_width, $lander_height, $center_x, $center_y) = #{$lander_dimensions};
$call_count = 0;
}
=head1 AutoPilot::ComputeLanding()
This method is called for every clock tick of the simulation.
It is passed the necessary information about the current state
and it must return an array with elements, describing the
actions that the lander should execute in the current tick.
The parameters passed to the method describe the actual state
of the lander, the current terrain below the lander and some
extra information. In particular, the parameters are:
$fuel, a nonnegative integer describing the remaining amount of fuel.
When the fuel runs out, the lander becomes uncontrolled.
$terrain_info, an array describing the terrain below the lander (see below).
$lander_state, an array which contains information about the lander's state.
For more information, see below.
$debug, an integer encoding whether the autopilot should output any debug information.
Effectively, the value supplied on the command line after "-D",
or if this value is not supplied, the value of the variable $autopilot_debug
in the main program.
$time, the time elapsed from the beginning of the simulation.
If the simulation is reset, time is also reset to 0.
=head2 Details of the parameters
=head3 The terrain information
The array referred to by $terrain_info is either empty, or
it describes the terrain element which is just (vertically) below the lander.
It is empty, when there is no terrain element below the lander.
When it is non-empty, it has the following elements:
($x0, $y0, $x1, $y1, $slope, $crashSpeed, $crashSlope)
where
($x0, $y0) is the left coordinate of the terrain segment,
($x1, $y1) is the right coordinate of the terrain segment,
$slope is the left to right slope of the segment (rise/run),
$crashSpeed is the maximum landing speed to avoid a crash,
$crashSlope is the maximum ground slope to avoid a crash.
=head3 The state of the lander
The array referred to by $lander_state contains
the current position, attitude, and velocity of the lander:
($px, $py, $attitude, $vx, $vy, $speed)
where
$px is its x position in the world, in the range [-800, 800],
$py is its y position in the world, in the range [0, 1600],
$attitude is its current attitude angle in unit degrees,
from y axis, where
0 is vertical,
> 0 is to the left (counter clockwise),
< 0 is to the right (clockwise),
$vx is the x velocity in m/s (< 0 is to left, > 0 is to right),
$vy is the y velocity in m/s (< 0 is down, > 0 is up),
$speed is the speed in m/s, where $speed == sqrt($vx*$vx + $vy*$vy)
=head2 The array to be returned
To control the lander you must return an array with 3 values:
($rotation, $left_right_thruster, $main_thruster)
$rotation instructs the lander to rotate the given number of degrees.
A value of 5 will cause the lander to rotate 5 degrees counter clockwise,
-5 will rotate 5 degrees clockwise.
$left_right_thruster instructs the lander to fire either the left or
right thruster. Negative value fire the right thruster, pushing the
lander to the left, positive fire the left thruster, pushing to the right.
The absolute value of the value given is the number of pushes,
so a value of -5 will fire the right thruster 5 times.
$main_thruster instructs the lander to fire the main engine,
a value of 5 will fire the main engine 5 times.
Each firing of either the main engine or a side engine consumes
one unit of fuel.
When the fuel runs out, the lander becomes uncontrolled.
Note that your instructions will only be executed up until the
limits denoted in $max_rotation, $max_side_thrusters, and $max_main_thruster.
If you return a value larger than one of these maximums than the
lander will only execute the value of the maximum.
=cut
sub ComputeLanding {
my ($fuel, $terrain_info, $lander_state, $debug, $time) = #_;
my $rotation = 0;
my $left_right_thruster = 0;
my $main_thruster = 0;
# fetch and update historical information
$call_count++;
if ( ! $terrain_info ) {
# hmm, we are not above any terrain! So do nothing.
return;
}
my ($x0, $y0, $x1, $y1, $slope, $crashSpeed, $crashSlope) =
#{$terrain_info};
my ($px, $py, $attitude, $vx, $vy, $speed) =
#{$lander_state};
if ( $debug ) {
printf "%5d ", $call_count;
printf "%5s ", $target_x;
printf "%4d, (%6.1f, %6.1f), %4d, ",
$fuel, $px, $py, $attitude;
printf "(%5.2f, %5.2f), %5.2f, ",
$vx, $vy, $speed;
printf "(%d %d %d %d, %5.2f), %5.2f, %5.2f\n",
$x0, $y0, $x1, $y1, $slope, $crashSpeed, $crashSlope;
}
# reduce horizontal velocity
if ( $vx < -1 && $attitude > -90 ) {
# going to the left, rotate clockwise, but not past -90!
$rotation = -1;
}
elsif ( 1 < $vx && $attitude < 90 ) {
# going to the right, rotate counterclockwise, but not past 90
$rotation = +1;
}
else {
# we're stable horizontally so make sure we are vertical
$rotation = -$attitude;
}
# reduce vertical velocity
if ($target_x eq "any"){
if (abs($slope) < $crashSlope){
if ($vy < -$crashSpeed + 6){
$main_thruster = 1;
if (int($vx) < 1 && int ($vx) > -1){
$left_right_thruster = 0;
}
if (int($vx) < -1){
$left_right_thruster = 1;
}
if (int($vx) > 1){
$left_right_thruster = -1;
}
}
}
else{
if ( $py < $ascend_height) {
if ($vy < 5){
$main_thruster=2;
}
}
if ($py > $ascend_height){
$left_right_thruster = 1;
if ($vx > 18){
$left_right_thruster = 0;
}
}
}
}
if ($target_x ne "any"){
if ($target_x < $px + 5 && $target_x > $px - 5){
print "I made it here";
if (abs($slope) < $crashSlope){
if ($vy < -$crashSpeed + 1){
$main_thruster = 1;
if (int($vx) < 1 && int ($vx) > -1){
$left_right_thruster = 0;
}
if (int($vx) < -1){
$left_right_thruster = 1;
}
if (int($vx) > 1){
$left_right_thruster = -1;
}
}
}
}
if ($target_x != $px){
if ( $py < $ascend_height) {
if ($vy < 5){
$main_thruster=2;
}
}
if ($py > $ascend_height){
$left_right_thruster = 1;
if ($vx > 10){
$left_right_thruster = 0;
}
}
}
}
return ($rotation, $left_right_thruster, $main_thruster);
}
1; # package ends
Sorry about the length of the code...
So, there are a few things I want this autopilot program to do. In order they are:
Stabilize the lander (reduce attitude and horizontal drift to zero if they are nonzero). Once stabilized:
If above a target and the target's segment is safe to land on then descend on it.
Otherwise ascend to the safe height, which is above 1200 units. You can safely assume that there are no objects at this height or higher and also that during straight ascends from its initial position, the lander will not hit anything.
Once at the safe height, the lander can start going horizontally towards to its target, if a target is given, otherwise it should target the first safe landing spot that is can sense by scanning the terrain in one direction. It is important that the lander maintains its altitude while it moves horizontally, because it cannot sense objects next to it and there could be objects anywhere below this height.
Once the target x coordinate is reached and is found to be safe to land on, start a descend.
If the target x coordinate is is reached, but the terrain is unsafe, if a good spot has been seen while moving towards the target, go back to it, otherwise continue searching for a good spot.
Once a good spot is seen, just land on it nice and safe.
Ok, so, I've updated the code. My code is now able to land the lander in all tests (except one, got fed up, the code works close enough) where there is no target. However, I am having huge troubles figuring out how to get the lander to land at a target. Any ideas with my code so far? (actual used code is found in the ComputeLanding subroutine)
Here's a hint: try approaching the problem from the other end.
Landing is almost equivalent to takeoff with time reversed. The only thing that doesn't get reversed is fuel consumption. (Whether that matters depends on whether the simulation counts fuel as part of the lander's mass. In a realistic sim, it should, but at a glance, it looks like yours might not.)
The optimal way (in terms of fuel efficiency) to take off in a rocket is to fire the engines at maximum power until you're going fast enough, then turn them off. The slower you climb, the more fuel you waste hovering.
Thus, the optimal way to land a rocket is to freefall (after a possible initial burn to correct heading) until the last possible instant, and then fire the engines at full power so that you come to a stop just above the landing pad (or hit the pad at whatever velocity you consider acceptable, if that's greater than zero).
Can you calculate what the right moment to turn on the engines would be? (Even if you can't do it directly, you could always approximate it by binary search. Just pick a point and simulate what happens: if you crash, start the burn earlier; if you stop before hitting the surface, start it later.)
(Ps. This seems like a rather silly exercise for a Perl programming course. Yes, you can certainly solve this in Perl, but there's nothing about Perl that would be particularly well suited for this exercise. Indeed, this isn't even fundamentally a programming problem, but a mathematical one — the only programming aspect to it is translating the mathematical solution, once found, into a working program.)
You could use a genetic algorithm for the lander implementation check out this book AI Techniques for game programming. It has exactly what you need with code examples. However, those examples are in c++.