I have NetCDF files (e.g https://data.ceda.ac.uk/neodc/esacci/lakes/data/lake_products/L3S/v1.0/2019 global domain), and I want to extract the data based on a shapefile boundary ( in this case a Lake here - https://www.sciencebase.gov/catalog/item/530f8a0ee4b0e7e46bd300dd) and then save clipped data as a NetCDF file but retain all the original metadata and variables names within the clipped file. This is what I have done far
library(rgdal)
library(sf)
library(ncdf4)
library(terra)
#Read in the shapefile of Lake
Lake_shape <- readOGR("C:/Users/CEDA/hydro_p_LakeA/hydro_p_A.shp")
# Reading the netcdf file using Terra Package function rast
test <- rast("ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20190705-fv1.0.nc")
# List of some of variables names for orginal dataset
head(names(test))
[1] "water_surface_height_above_reference_datum" "water_surface_height_uncertainty" "lake_surface_water_extent"
[4] "lake_surface_water_extent_uncertainty" "lake_surface_water_temperature" "lswt_uncertainty"
#Clipping data to smaller Lake domain using the crop function in Terra Package
test3 <- crop(test, Lake_shape)
#Listing the some variables names for clipped data
head(names(test3))
[1] "water_surface_height_above_reference_datum" "water_surface_height_uncertainty" "lake_surface_water_extent"
[4] "lake_surface_water_extent_uncertainty" "lake_surface_water_temperature" "lswt_uncertainty"
# Writing the crop dataset as netcdf or Raster Layer using the WriteCDF function
filepath<-"Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0"
fname <- paste0( "C:/Users/CEDA/",filepath,".nc")
rnc <- writeCDF(test3, filename =fname, overwrite=T)”
My main issue here when I read in clipped the netCDF file I don’t seem to be able to keep the names of the data variables of the original NetCDF. They are all being renamed automatically when I am saving the clipped dataset as a new netCDF using the writeCDF function.
#Reading in the new clipped file
LakeA<-rast("Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0.nc")
> head(names(LakeA))
[1] "Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0_1" "Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0_2"
[3] "Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0_3" "Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0_4"
[5] "Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0_5" "Lake_A_ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20020501-fv1.0_6"
So is it possible to clone/copy all the metadata variables from the original NetCDF dataset when clipping to the smaller domain/shapefile in R, then saving as NetCDF? Any guidance on how to do this in R would be really appreciated. (NetCDF and R are all new to me so I am not sure what I am missing or have the in-depth knowledge to sort this).
You have a NetCDF file with many (52) variables (sub-datasets). When you open the file with rast these become "layers". Alternatively you can open the file with sds to keep the sub-dataset structure but that does not help you here (and you would need to skip the first two, see below).
library(terra)
f <- "ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20190101-fv1.0.nc"
r <- rast(f)
r
#class : SpatRaster
#dimensions : 21600, 43200, 52 (nrow, ncol, nlyr)
#resolution : 0.008333333, 0.008333333 (x, y)
#extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +datum=WGS84 +no_defs
#sources : ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20190101-fv1.0.nc:water_surface_height_above_reference_datum
ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20190101-fv1.0.nc:water_surface_height_uncertainty
ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-20190101-fv1.0.nc:lake_surface_water_extent
... and 49 more source(s)
#varnames : water_surface_height_above_reference_datum (water surface height above geoid)
water_surface_height_uncertainty (water surface height uncertainty)
lake_surface_water_extent (Lake Water Extent)
...
#names : water~datum, water~ainty, lake_~xtent, lake_~ainty, lake_~ature, lswt_~ainty, ...
#unit : m, m, km2, km2, Kelvin, Kelvin, ...
#time : 2019-01-01
Note that there are 52 layers and sources (sub-datasets). There are names
head(names(r))
#[1] "water_surface_height_above_reference_datum" "water_surface_height_uncertainty"
#[3] "lake_surface_water_extent" "lake_surface_water_extent_uncertainty"
#[5] "lake_surface_water_temperature" "lswt_uncertainty"
And also "longnames" (they are often much longer than the variable names, not in this case)
head(longnames(r))
# [1] "water surface height above geoid" "water surface height uncertainty" "Lake Water Extent"
# [4] "Water extent uncertainty" "lake surface skin temperature" "Total uncertainty"
You can also open the file with sds, but you need to skip "lon_bounds" and "lat_bounds" variables (dimensions)
s <- sds(f, 3:52)
Now read a vector data set (shapefile in this case) and crop
lake <- vect("hydro_p_LakeErie.shp")
rc <- crop(r, lake)
rc
#class : SpatRaster
#dimensions : 182, 555, 52 (nrow, ncol, nlyr)
#resolution : 0.008333333, 0.008333333 (x, y)
#extent : -83.475, -78.85, 41.38333, 42.9 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +datum=WGS84 +no_defs
#source : memory
#names : water~datum, water~ainty, lake_~xtent, lake_~ainty, lake_~ature, lswt_~ainty, ...
#min values : NaN, NaN, NaN, NaN, 271.170, 0.283, ...
#max values : NaN, NaN, NaN, NaN, 277.090, 0.622, ...
#time : 2019-01-01
It can be convenient to save this to a GTiff file like this (or even better to use the filename argument in crop)
gtf <- writeRaster(rc, "test.tif", overwrite=TRUE)
gtf
#class : SpatRaster
#dimensions : 182, 555, 52 (nrow, ncol, nlyr)
#resolution : 0.008333333, 0.008333333 (x, y)
#extent : -83.475, -78.85, 41.38333, 42.9 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +datum=WGS84 +no_defs
#source : test.tif
#names : water~datum, water~ainty, lake_~xtent, lake_~ainty, lake_~ature, lswt_~ainty, ...
#min values : NaN, NaN, NaN, NaN, 271.170, 0.283, ...
#max values : NaN, NaN, NaN, NaN, 277.090, 0.622, ...
What has changed is that the data are now in a file, rather then in memory. And you still have the layer (variable) names.
To write the layers as variables to a NetCDF file you need to create a SpatRasterDataset. You can do that like this:
x <- as.list(rc)
s <- sds(x)
names(s) <- names(rc)
longnames(s) <- longnames(r)
units(s) <- units(r)
Note the use of longnames(r) and units(r) (not rc). This is because r has subdatasets (and each has a longname and a unit) while rc does not.
Now use writeCDF
z <- writeCDF(s, "test.nc", overwrite=TRUE)
rc2 <- rast("test.nc")
rc2
#class : SpatRaster
#dimensions : 182, 555, 52 (nrow, ncol, nlyr)
#resolution : 0.008333333, 0.008333333 (x, y)
#extent : -83.475, -78.85, 41.38333, 42.9 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +datum=WGS84 +no_defs
#sources : test.nc:water_surface_height_above_reference_datum
test.nc:water_surface_height_uncertainty
test.nc:lake_surface_water_extent
... and 49 more source(s)
#varnames : water_surface_height_above_reference_datum (water surface height above geoid)
water_surface_height_uncertainty (water surface height uncertainty)
lake_surface_water_extent (Lake Water Extent)
...
#names : water~datum, water~ainty, lake_~xtent, lake_~ainty, lake_~ature, lswt_~ainty, ...
#unit : m, m, km2, km2, Kelvin, Kelvin, ...
#time : 2019-01-01
So it looks like we have a NetCDF with the same structure.
Note that the current CRAN version of terra drops the time variable if there is only one time step. The development version (1.3-11) keeps the time dimension, even of there is only one step.
You can install the development version with
install.packages('terra', repos='https://rspatial.r-universe.dev')
I want to change the "MIME Type" in the file:
└──╼ $exiftool realshort.mp4
ExifTool Version Number : 12.10
File Name : realshort.mp4
Directory : .
File Size : 98 kB
File Modification Date/Time : 2021:01:19 23:53:01+00:00
File Access Date/Time : 2021:01:19 23:53:01+00:00
File Inode Change Date/Time : 2021:01:19 23:53:01+00:00
File Permissions : rw-r--r--
File Type : MP4
File Type Extension : mp4
MIME Type : video/mp4
Major Brand : MP4 Base Media v1 [IS0 14496-12:2003]
Minor Version : 0.0.0
Compatible Brands : isom, 3gp4
Movie Header Version : 0
Create Date : 2014:11:05 13:51:33
Modify Date : 2014:11:05 13:51:33
Time Scale : 1000
Duration : 1.20 s
Preferred Rate : 1
Preferred Volume : 100.00%
Preview Time : 0 s
Preview Duration : 0 s
Poster Time : 0 s
Selection Time : 0 s
Selection Duration : 0 s
Current Time : 0 s
Next Track ID : 3
Track Header Version : 0
Track Create Date : 2014:11:05 13:51:33
Track Modify Date : 2014:11:05 13:51:33
Track ID : 1
Track Duration : 1.20 s
Track Layer : 0
Track Volume : 0.00%
Image Width : 320
Image Height : 240
Graphics Mode : srcCopy
Op Color : 0 0 0
Compressor ID : avc1
Source Image Width : 320
Source Image Height : 240
X Resolution : 72
Y Resolution : 72
Compressor Name :
Bit Depth : 24
Video Frame Rate : 30.02
Matrix Structure : 1 0 0 0 1 0 0 0 1
Media Header Version : 0
Media Create Date : 2014:11:05 13:51:33
Media Modify Date : 2014:11:05 13:51:33
Media Time Scale : 48000
Media Duration : 1.17 s
Handler Type : Audio Track
Handler Description : SoundHandle
Balance : 0
Audio Format : mp4a
Audio Channels : 1
Audio Bits Per Sample : 16
Audio Sample Rate : 48000
XMP Toolkit : Image::ExifTool 12.10
Media Data Size : 95268
Media Data Offset : 4610
Image Size : 320x240
Megapixels : 0.077
Avg Bitrate : 636 kbps
Rotation : 0
If I do:
exiftool -artist=ii realshort.mp4
i can add artist tag with the value ii
But if I do: exiftool -"mime type"=ii realshort.mp4 it won't work
I looked at: https://libre-software.net/edit-metadata-exiftool/
And also here: How do you change the MIME type of a file from the terminal?
But I can't find any answer
How can I make it work?
You can't change the MIME type. It's not embedded data. It's a tag derived from what the type of file is.
You could edit try editing the .Exiftool_Config file if you have one (see the example config) to override the base definition, but that will only change what exiftool displays. Another program or another computer will output MP4 as the MIME type.
I have a website with an upload profile photos for users.
First Try
When I try to upload a small size image, the upload was succeed.
This file property :
Size : 500 KB
Width : 900
Height : 900
Type : jpg
Second Try
When I try to upload a large iamge, I got error message :
The uploaded file exceeds the maximum allowed size in your PHP configuration file.
This file property :
Size : 7,2 MB
Width : 5200
Height : 3500
Type : JPG
This is my config file in my controller :
$config['upload_path'] = "./assets/gambar/pengguna/";
$config['allowed_types'] = 'gif|jpg|jpeg|png';
$config['max_size'] = 25000;
$config['max_width'] = 7000;
$config['max_height'] = 7000;
I tried to change max_size, max_width, and max_height to 0. But I still get bthe same problem.
Refrence to set it 0 to make it no limit from :
https://www.codeigniter.com/user_guide/libraries/file_uploading.html
How to edit or add exif metadata by graphicsmagick command? I've tried the following:
gm convert -set EXIF:XPKeywords "bird, sleep, moon" bird.jpg bird2.jpg
My command is executing successfully and the image is creating but the exif metadata is not updating.
I don't believe you can do that with GraphicsMagick, and I would suggest exiv2 like this:
exiv2 -M"add Exif.Image.XPKeywords Ascii 'bird,sleep,moon'" image.jpg
Then you can view them with jhead too:
jhead -v image.jpg
JFIF SOI marker: Units: 0 (aspect ratio) X-density=1 Y-density=1
Exif header 50 bytes long
Exif section in Intel order
(dir has 1 entries)
Windows-XP keywords = "bird,sleep,moon"
Approximate quality factor for qtable 0: 92 (scale 16.28, var 1.13)
Approximate quality factor for qtable 1: 92 (scale 16.20, var 0.15)
JPEG image is 2000w * 2000h, 3 color components, 8 bits per sample
File name : image.jpg
File size : 6968929 bytes
File date : 2015:10:20 09:23:24
Resolution : 2000 x 2000
I know that some of this data is viewable in iPhoto, such as latitude, longitude, and altitude, but is there more than this available? More specifically I am wanting to get the direction of the image as well, which is part of the EXIF format. Or does something other than geotagging need to be enabled (something to do with the compass)?
I suggest installing the command line tool exif. You can get it through Mac Ports by executing port install exif as root.
Here's an example of the exif info stored in a photo I took on an iPod Touch:
bash:$ exif Oct\ 9\,\ 2010/IMG_0038.JPG
EXIF tags in 'Oct 9, 2010/IMG_0018.JPG' ('Motorola' byte order):
--------------------+----------------------------------------------------------
Tag |Value
--------------------+----------------------------------------------------------
Manufacturer |Apple
Model |iPod touch
Orientation |right - top
x-Resolution |72.00
y-Resolution |72.00
Resolution Unit |Inch
Software |4.1
Date and Time |2010:10:06 17:43:43
YCbCr Positioning |centered
Compression |JPEG compression
x-Resolution |72.00
y-Resolution |72.00
Resolution Unit |Inch
Exposure Time |1/120 sec.
FNumber |f/2.4
Exposure Program |Normal program
ISO Speed Ratings |320
Exif Version |Exif Version 2.21
Date and Time (origi|2010:10:06 17:43:43
Date and Time (digit|2010:10:06 17:43:43
Components Configura|Y Cb Cr -
Shutter speed |6.91 EV (1/120 sec.)
Aperture |2.53 EV (f/2.4)
Metering Mode |Average
Flash |No flash function
Focal Length |3.9 mm
FlashPixVersion |FlashPix Version 1.0
Color Space |sRGB
PixelXDimension |640
PixelYDimension |480
Sensing Method |One-chip color area sensor
Exposure Mode |Auto exposure
White Balance |Auto white balance
Scene Capture Type |Standard
Sharpness |Normal
North or South Latit|N
Latitude |44.00, 22.12, 0.00
East or West Longitu|W
Longitude |56.00, 23.98, 0.00
GPS time (atomic clo|17:43:41.60
--------------------+----------------------------------------------------------
EXIF data contains a thumbnail (10215 bytes).
I don't think the iPod Touch has the same capabilities as an iPhone regarding GPS.
It looks like an iPhone will have the data 'GPSImg Direction' as well, which sounds like what you want. I don't think you have to do anything special to enable it, as I haven't found much info on this through searching.
Okay, good news :). The direction is stored in GPS Img Direction. Here's what I was able to get from a photo taken from an iphone (with location services and compass turned on) using the ExifTool mentioned by #rwong.
ExifTool Version Number : 8.34
File Name : photo.JPG
Directory : C:/Documents and Settings/user/My Document
s/Downloads
File Size : 349 kB
File Modification Date/Time : 2010:10:19 14:05:39-06:00
File Permissions : rw-rw-rw-
File Type : JPEG
MIME Type : image/jpeg
JFIF Version : 1.01
Exif Byte Order : Big-endian (Motorola, MM)
Image Description : Back Camera
Make : Apple
Camera Model Name : iPhone
Orientation : Horizontal (normal)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Software : 4.0.1
Modify Date : 2010:10:19 14:00:52
Y Cb Cr Positioning : Centered
Exposure Time : 1/146
F Number : 2.4
Exposure Program : Program AE
ISO : 80
Exif Version : 0221
Date/Time Original : 2010:10:19 14:00:52
Create Date : 2010:10:19 14:00:52
Shutter Speed Value : 1/146
Aperture Value : 2.4
Metering Mode : Average
Flash : Off, Did not fire
Focal Length : 3.9 mm
Subject Area : 1295 967 699 696
Flashpix Version : 0100
Color Space : sRGB
Exif Image Width : 1296
Exif Image Height : 968
Sensing Method : One-chip color area
Exposure Mode : Auto
White Balance : Auto
Scene Capture Type : Standard
Sharpness : Hard
GPS Latitude Ref : North
GPS Longitude Ref : West
GPS Time Stamp : 14:00:46.81
GPS Img Direction Ref : True North
GPS Img Direction : 32.52336904
Image Width : 1296
Image Height : 968
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
Aperture : 2.4
GPS Latitude : 45 deg 14' 58.20" N
GPS Longitude : 121 deg 39' 4.80" W
GPS Position : 45 deg 14' 58.20" N, 121 deg 39' 4.80" W
Image Size : 1296x968
Shutter Speed : 1/146
Focal Length : 3.9 mm
Light Value : 10.0
you don't need an 'external' tool to view all the EXIF-infos, with the iPhone-app iMetaPhoto you can show all metainfos (EXIF, TIFF, IPTC, GPS, AUXiliary EXIF,...) and (the best of all) you can modify the most of them, esp. you can modify GPS ImgDirection.