Method to find device's camera resolution iOS - iphone

Whats the best method to find the image resolution going to be captured using setting AVCaptureSessionPresetPhoto.
I am trying to find the resolution before capturing the image.

Here is the a table of all tested camera sessionPreset`s on devices, that support iOS 9
+-----------------------+--------------+---------------------------------+---------------+
| Device | Camera | AVCaptureSessionPreset | Resolution |
+-----------------------+--------------+---------------------------------+---------------+
| iPhone 4S | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 640x480 |
| | | AVCaptureSessionPresetHigh | 640x480 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | not supported |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 3264x2448 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPhone 5/5C/5S/6/6+ | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 1280x960 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 3264x2448 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPhone 6S/6S | | | |
| | FRONT camera | | |
| | | AVCaptureSessionPresetPhoto | 1280x960 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK camera | | |
| | | AVCaptureSessionPresetPhoto | 4032x3024 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPad 2 | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 640x480 |
| | | AVCaptureSessionPresetHigh | 640x480 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | not supported |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 960x720 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
+-----------------------+--------------+---------------------------------+---------------+
| iPad 3 | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 640x480 |
| | | AVCaptureSessionPresetHigh | 640x480 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | not supported |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 2592x1936 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPad 4/Air | | | |
| iPad Mini 1/2/3 | | | |
| iPod 5G | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 1280x960 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 2592x1936 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPad Air 2 | | | |
| iPad Mini 4 | | | |
| iPad Pro | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 1280x960 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 3264x2448 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPod Touch 5 | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 1280x960 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 2592x1936 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
| iPod Touch 6 | | | |
| | FRONT | | |
| | | AVCaptureSessionPresetPhoto | 1280x960 |
| | | AVCaptureSessionPresetHigh | 1280x720 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | not supported |
| | BACK | | |
| | | AVCaptureSessionPresetPhoto | 3264x2448 |
| | | AVCaptureSessionPresetHigh | 1920x1080 |
| | | AVCaptureSessionPresetMedium | 480x360 |
| | | AVCaptureSessionPresetLow | 192x144 |
| | | AVCaptureSessionPreset640x480 | 640x480 |
| | | AVCaptureSessionPreset1280x720 | 1280x720 |
| | | AVCaptureSessionPreset1920x1080 | 1920x1080 |
+-----------------------+--------------+---------------------------------+---------------+
CODE
+ (NSDictionary*) cameraResolutions {
NSString* deviceModel = [self deviceModel];
// iPhone 4S
if ([deviceModel isEqualToString:#"iPhone4,1"]) {
NSString* lFrontCam = #"640x480,480x360,192x144";
NSString* lBackCam = #"3264x2448,1920x1080,1280x720,640x480,480x360,192x144";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
// iPhone 5/5C/5S/6/6+/iPod 6
else if ([deviceModel isEqualToString:#"iPhone5,1"]
|| [deviceModel isEqualToString:#"iPhone5,2"]
|| [deviceModel isEqualToString:#"iPhone5,3"]
|| [deviceModel isEqualToString:#"iPhone5,4"]
|| [deviceModel isEqualToString:#"iPhone6,1"]
|| [deviceModel isEqualToString:#"iPhone6,2"]
|| [deviceModel isEqualToString:#"iPhone7,1"]
|| [deviceModel isEqualToString:#"iPhone7,2"]
|| [deviceModel isEqualToString:#"iPod7,1"]) {
NSString* lFrontCam = #"1280x960,1280x720,640x480,480x360,192x144";
NSString* lBackCam = #"3264x2448,1920x1080,1280x720,640x480,480x360,192x144";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
// iPhone 6S/6S+
else if ([deviceModel isEqualToString:#"iPhone8,1"]
|| [deviceModel isEqualToString:#"iPhone8,2"]) {
NSString* lFrontCam = #"1280x960,1280x720,640x480,480x360,192x144";
NSString* lBackCam = #"4032x3024,1920x1080,1280x720,640x480,480x360,192x144";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
// iPad 2
else if ([deviceModel isEqualToString:#"iPad2,1"]
|| [deviceModel isEqualToString:#"iPad2,2"]
|| [deviceModel isEqualToString:#"iPad2,3"]
|| [deviceModel isEqualToString:#"iPad2,4"]) {
NSString* lFrontCam = #"640x480,480x360,192x144";
NSString* lBackCam = #"1280x720,960x720,480x360,192x144,640x480";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
// iPad 3
else if ([deviceModel isEqualToString:#"iPad3,1"]
|| [deviceModel isEqualToString:#"iPad3,2"]
|| [deviceModel isEqualToString:#"iPad3,3"]) {
NSString* lFrontCam = #"640x480,480x360,192x144";
NSString* lBackCam = #"2592x1936,1920x1080,1280x720,640x480,480x360,192x144";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
// iPad 4/Air/Mini/Mini 2/Mini 3/iPod 5G
else if ([deviceModel isEqualToString:#"iPad3,4"]
|| [deviceModel isEqualToString:#"iPad3,5"]
|| [deviceModel isEqualToString:#"iPad3,6"]
|| [deviceModel isEqualToString:#"iPad4,1"]
|| [deviceModel isEqualToString:#"iPad4,2"]
|| [deviceModel isEqualToString:#"iPad4,3"]
|| [deviceModel isEqualToString:#"iPad4,4"]
|| [deviceModel isEqualToString:#"iPad4,5"]
|| [deviceModel isEqualToString:#"iPad4,6"]
|| [deviceModel isEqualToString:#"iPad4,7"]
|| [deviceModel isEqualToString:#"iPad4,8"]
|| [deviceModel isEqualToString:#"iPod5,1"]) {
NSString* lFrontCam = #"1280x960,1280x720,640x480,480x360,192x144";
NSString* lBackCam = #"2592x1936,1920x1080,1280x720,640x480,480x360,192x144";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
// iPad Air 2/Mini 4/Pro
else if ([deviceModel isEqualToString:#"iPad5,3"]
|| [deviceModel isEqualToString:#"iPad5,4"]) {
NSString* lFrontCam = #"1280x960,1280x720,640x480,480x360,192x144";
NSString* lBackCam = #"3264x2448,1920x1080,1280x720,640x480,480x360,192x144";
NSDictionary* lCamResolutions = #{#"front":lFrontCam,
#"back":lBackCam};
return lCamResolutions;
}
return nil;
}

With the function below, you can programmatically get the resolution from activeFormat before capture begins, though not before adding inputs and outputs:
https://developer.apple.com/library/ios/documentation/AVFoundation/Reference/AVCaptureDevice_Class/index.html#//apple_ref/occ/instp/AVCaptureDevice/activeFormat
private func getCaptureResolution() -> CGSize {
// Define default resolution
var resolution = CGSize(width: 0, height: 0)
// Get cur video device
let curVideoDevice = useBackCamera ? backCameraDevice : frontCameraDevice
// Set if video portrait orientation
let portraitOrientation = orientation == .Portrait || orientation == .PortraitUpsideDown
// Get video dimensions
if let formatDescription = curVideoDevice?.activeFormat.formatDescription {
let dimensions = CMVideoFormatDescriptionGetDimensions(formatDescription)
resolution = CGSize(width: CGFloat(dimensions.width), height: CGFloat(dimensions.height))
if portraitOrientation {
resolution = CGSize(width: resolution.height, height: resolution.width)
}
}
// Return resolution
return resolution
}

There's probably no way to directly get the camera resolution programmatically before capturing an image.
Apple provides a table for various devices and presets:
Capturing Still Images

Can be done in this way:
/* Callback that is called when you activate the capture session. */
- (void) avCaptureInputPortFormatDescriptionDidChangeNotification:(NSNotification *)notification {
CMFormatDescriptionRef formatDescription = nil;
NSArray *ports = [deviceInput ports];
AVCaptureInputPort *usePort = nil;
for (AVCaptureInputPort *port in ports) {
if (usePort == nil || [port.mediaType isEqualToString:AVMediaTypeVideo]) {
usePort = port;
break;
}
}
if (usePort != nil)
formatDescription = usePort.formatDescription;
if (formatDescription != nil) {
/*------------>>>>>>> THIS IS YOUR RESOLUTION */
CMVideoDimensions dimensions = CMVideoFormatDescriptionGetDimensions(formatDescription);
self.frameSize = CGSizeMake(dimensions.width, dimensions.height);
[[NSNotificationCenter defaultCenter] removeObserver:self];
NSLog(#"Capturing with %dx%d", (int)self.frameSize.width, (int)self.frameSize.height);
} else {
NSLog(#"Failed to detect resolution, using default one.");
self.frameSize = CGSizeMake(640.0f, 480.0f);
}
}
- (id) init {
if ((self = [super init])) {
/* Default framesize. */
self.frameSize = CGSizeMake(640.0f, 480.0f);
AVCaptureSession *capSession = [[AVCaptureSession alloc] init];
/* -------->>>>>>> REQUESTED RESOLUTION, CAN BE SOMTHING ELSE */
[capSession setSessionPreset:AVCaptureSessionPresetHigh];
// if (([modelName rangeOfString:#"iPhone 5"].length != 0) || ([modelName rangeOfString:#"iPhone 6"].length != 0)) {
// [capSession setSessionPreset:AVCaptureSessionPresetHigh];
// } else {
// [capSession setSessionPreset:AVCaptureSessionPreset640x480];
// }
//
self.captureSession = capSession;
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(avCaptureInputPortFormatDescriptionDidChangeNotification:)
name:AVCaptureInputPortFormatDescriptionDidChangeNotification object:nil];
}
return self;
}
- (void) dealloc
{
[[NSNotificationCenter defaultCenter] removeObserver:self];
if (self.captureSession.isRunning)
[self.captureSession stopRunning];
self.captureSession = nil;
}
- (void) addRawViewOutput
{
/* We setup the output */
AVCaptureVideoDataOutput *captureOutput = [[AVCaptureVideoDataOutput alloc] init];
/* While a frame is processes in -captureOutput:didOutputSampleBuffer:fromConnection: delegate methods
no other frames are added in the queue.
If you don't want this behaviour set the property to NO */
captureOutput.alwaysDiscardsLateVideoFrames = YES;
/*We specify a minimum duration for each frame (play with this settings to avoid having too many frames waiting
in the queue because it can cause memory issues). It is similar to the inverse of the maximum framerate.
In this example we set a min frame duration of 1/10 seconds so a maximum framerate of 10fps. We say that
we are not able to process more than 10 frames per second.*/
//captureOutput.minFrameDuration = CMTimeMake(1, 10);
/*We create a serial queue to handle the processing of our frames*/
dispatch_queue_t queue;
queue = dispatch_queue_create("com.YourApp.cameraQueue", NULL);
[captureOutput setSampleBufferDelegate:self queue:queue];
// dispatch_release(queue);
// Set the video output to store frame in BGRA (It is supposed to be faster)
NSString *key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber *value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary *videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
[captureOutput setVideoSettings:videoSettings];
// Register an output
[self.captureSession addOutput:captureOutput];
}
- (bool) startWithDevicePosition:(AVCaptureDevicePosition)devicePosition
{
AVCaptureDevice *videoDevice = [self cameraWithPosition:devicePosition];
if (!videoDevice)
return FALSE;
NSError *error;
AVCaptureDeviceInput *videoIn = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:&error];
self.deviceInput = videoIn;
if (!error)
{
if ([[self captureSession] canAddInput:videoIn])
{
[[self captureSession] addInput:videoIn];
}
else
{
NSLog(#"Couldn't add video input");
return FALSE;
}
}
else
{
NSLog(#"Couldn't create video input");
return FALSE;
}
[self addRawViewOutput];
[self resume];
return TRUE;
}
-(void) pause
{
self.paused = YES;
if (self.captureSession.isRunning)
[captureSession stopRunning];
}
-(void) resume
{
if (!self.captureSession.isRunning)
[captureSession startRunning];
self.paused = NO;
}
#pragma mark -
#pragma mark AVCaptureSession delegate
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
/* Lock the image buffer */
CVPixelBufferLockBaseAddress(imageBuffer,0);
/* Get information about the image */
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t size = CVPixelBufferGetDataSize(imageBuffer);
BGRAVideoFrame frame = {(int)width, (int)height, (int)stride, (int)size, baseAddress};
[delegate frameReady:frame];
/* We unlock the image buffer */
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
}

Try this code for get maximum Image resolution for device cameras:
- (CMVideoDimensions) getCameraMaxStillImageResolution:(AVCaptureDevicePosition) cameraPosition {
CMVideoDimensions max_resolution;
max_resolution.width = 0;
max_resolution.height = 0;
AVCaptureDevice *captureDevice = nil;
NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
for (AVCaptureDevice *device in devices) {
if ([device position] == cameraPosition) {
captureDevice = device;
break;
}
}
if (captureDevice == nil) {
return max_resolution;
}
NSArray* availFormats=captureDevice.formats;
for (AVCaptureDeviceFormat* format in availFormats) {
CMVideoDimensions resolution = format.highResolutionStillImageDimensions;
int w = resolution.width;
int h = resolution.height;
if ((w * h) > (max_resolution.width * max_resolution.height)) {
max_resolution.width = w;
max_resolution.height = h;
}
}
return max_resolution;
}
- (void) printCamerasInfo {
CMVideoDimensions res;
res = [self getCameraMaxStillImageResolution:AVCaptureDevicePositionBack];
NSLog(#" Back Camera max Image resolution: %d x %d", res.width, res.height);
res = [self getCameraMaxStillImageResolution:AVCaptureDevicePositionFront];
NSLog(#" Front Camera max Image resolution: %d x %d", res.width, res.height);
}

Related

Can I specify the output in Test2 test

Is there a way to edit the following Test2 test:
perl -MTest2::V0 -MRegexp::Common=URI -wE 'is( shift, match $RE{URI}, "should match a URI" )' foo
Such that the CHECK output is less verbose than this:
# Seeded srand with seed '20200813' from local date.
1..1
not ok 1 - should match a URI
# Failed test 'should match a URI'
# at -e line 1.
# +-----+----+-------------------------------------------------------+-----+
# | GOT | OP | CHECK | LNs |
# +-----+----+-------------------------------------------------------+-----+
# | foo | =~ | (?:(?:(?:file)://(?:(?:(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][ | 1 |
# | | | -a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9 | |
# | | | ]*[a-zA-Z0-9]|[a-zA-Z]))|(?:[0-9]+[.][0-9]+[.][0-9]+[ | |
# | | | .][0-9]+))|localhost)?)(?:/(?:(?:(?:(?:[-a-zA-Z0-9$_. | |
# | | | +!*'(),:#&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:/(?:(?: | |
# | | | [-a-zA-Z0-9$_.+!*'(),:#&=]|(?:%[a-fA-F0-9][a-fA-F0-9] | |
# | | | ))*))*)))))|(?:(?:telnet)://(?:(?:(?:(?:(?:[-a-zA-Z0- | |
# | | | 9$_.+!*'(),;?&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))(?::( | |
# | | | ?:(?:(?:[-a-zA-Z0-9$_.+!*'(),;?&=]|(?:%[a-fA-F0-9][a- | |
# | | | fA-F0-9]))*)))?)#)?(?:(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][- | |
# | | | a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9] | |
# | | | *[a-zA-Z0-9]|[a-zA-Z]))|(?:[0-9]+[.][0-9]+[.][0-9]+[. | |
# | | | ][0-9]+)))(?::(?:(?:[0-9]+)))?)(?:/)?)|(?:(?:https?): | |
# | | | //(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z | |
# | | | 0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z | |
# | | | ])[.]?)|(?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(? | |
# | | | :(?:[0-9]*)))?(?:/(?:(?:(?:(?:(?:(?:[a-zA-Z0-9\-_.!~* | |
# | | | '():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:;(?:(?: | |
# | | | [a-zA-Z0-9\-_.!~*'():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F0 | |
# | | | -9]))*))*)(?:/(?:(?:(?:[a-zA-Z0-9\-_.!~*'():#&=+$,]+| | |
# | | | (?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:;(?:(?:[a-zA-Z0-9\-_ | |
# | | | .!~*'():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F0-9]))*))*))*) | |
# | | | )(?:[?](?:(?:(?:[;/?:#&=+$,a-zA-Z0-9\-_.!~*'()]+|(?:% | |
# | | | [a-fA-F0-9][a-fA-F0-9]))*)))?))?)|(?:(?:gopher)://(?: | |
# | | | (?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9]) | |
# | | | [.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|( | |
# | | | ?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(?:(?:[0-9] | |
# | | | +)))?/(?:(?:(?:[0-9+IgT]))(?:(?:(?:[-a-zA-Z0-9$_.+!*' | |
# | | | (),:#&=]+|(?:%[a-fA-F0-9][a-fA-F0-9]))*))))|(?:(?:pop | |
# | | | )://(?:(?:(?:(?:[-a-zA-Z0-9$_.+!*'(),&=~]+|(?:%[a-fA- | |
# | | | F0-9][a-fA-F0-9]))+))(?:;AUTH=(?:[*]|(?:(?:(?:[-a-zA- | |
# | | | Z0-9$_.+!*'(),&=~]+|(?:%[a-fA-F0-9][a-fA-F0-9]))+)|(? | |
# | | | :[+](?:APOP|(?:(?:[-a-zA-Z0-9$_.+!*'(),&=~]+|(?:%[a-f | |
# | | | A-F0-9][a-fA-F0-9]))+))))))?#)?(?:(?:(?:(?:(?:(?:[a-z | |
# | | | A-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a | |
# | | | -zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|(?:[0-9]+[.][0-9]+[. | |
# | | | ][0-9]+[.][0-9]+)))(?::(?:(?:[0-9]+)))?)|(?:(?:prospe | |
# | | | ro)://(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a- | |
# | | | zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a- | |
# | | | zA-Z]))|(?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(? | |
# | | | :(?:[0-9]+)))?/(?:(?:(?:(?:[-a-zA-Z0-9$_.+!*'(),?:#&= | |
# | | | ]|(?:%[a-fA-F0-9][a-fA-F0-9]))*)(?:/(?:(?:[-a-zA-Z0-9 | |
# | | | $_.+!*'(),?:#&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))*))(? | |
# | | | :(?:;(?:(?:[-a-zA-Z0-9$_.+!*'(),?:#&]|(?:%[a-fA-F0-9] | |
# | | | [a-fA-F0-9]))*)=(?:(?:[-a-zA-Z0-9$_.+!*'(),?:#&]|(?:% | |
# | | | [a-fA-F0-9][a-fA-F0-9]))*))*))|(?:(?:wais)://(?:(?:(? | |
# | | | :(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9])[.])* | |
# | | | (?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|(?:[0- | |
# | | | 9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(?:(?:[0-9]+)))? | |
# | | | /(?:(?:(?:(?:[-a-zA-Z0-9$_.+!*'(),]|(?:%[a-fA-F0-9][a | |
# | | | -fA-F0-9]))*))(?:[?](?:(?:(?:[-a-zA-Z0-9$_.+!*'(),;:# | |
# | | | &=]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))|/(?:(?:(?:[-a-zA- | |
# | | | Z0-9$_.+!*'(),]|(?:%[a-fA-F0-9][a-fA-F0-9]))*))/(?:(? | |
# | | | :(?:[-a-zA-Z0-9$_.+!*'(),]|(?:%[a-fA-F0-9][a-fA-F0-9] | |
# | | | ))*)))?))|(?:(?:tel):(?:(?:(?:[+](?:[0-9\-.()]+)(?:;i | |
# | | | sub=[0-9\-.()]+)?(?:;postd=[0-9\-.()*#ABCDwp]+)?(?:(? | |
# | | | :;(?:phone-context)=(?:(?:(?:[+][0-9\-.()]+)|(?:[0-9\ | |
# | | | -.()*#ABCDwp]+))|(?:(?:[!'E-OQ-VX-Z_e-oq-vx-z~]|(?:%( | |
# | | | ?:2[124-7CFcf]|3[AC-Fac-f]|4[05-9A-Fa-f]|5[1-689A-Fa- | |
# | | | f]|6[05-9A-Fa-f]|7[1-689A-Ea-e])))(?:[!'()*\-.0-9A-Z_ | |
# | | | a-z~]+|(?:%(?:2[1-9A-Fa-f]|3[AC-Fac-f]|[4-6][0-9A-Fa- | |
# | | | f]|7[0-9A-Ea-e])))*)))|(?:;(?:tsp)=(?: |(?:(?:(?:[A-Z | |
# | | | a-z](?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?)(?:[. | |
# | | | ](?:[A-Za-z](?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9] | |
# | | | )?))*))))|(?:;(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7AB | |
# | | | DEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[ | |
# | | | 0-9ACEace]))*)(?:=(?:(?:(?:(?:[!'*\-.0-9A-Z_a-z~]+|%( | |
# | | | ?:2[13-7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9 | |
# | | | A-Fa-f]|7[0-9ACEace]))*)(?:[?](?:(?:[!'*\-.0-9A-Z_a-z | |
# | | | ~]+|%(?:2[13-7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef] | |
# | | | |6[0-9A-Fa-f]|7[0-9ACEace]))*))?)|(?:%22(?:(?:%5C(?:[ | |
# | | | a-zA-Z0-9\-_.!~*'()]|(?:%[a-fA-F0-9][a-fA-F0-9])))|[a | |
# | | | -zA-Z0-9\-_.!~*'()]+|(?:%(?:[01][a-fA-F0-9])|2[013-9A | |
# | | | -Fa-f]|[3-9A-Fa-f][a-fA-F0-9]))*%22)))?))*)|(?:[0-9\- | |
# | | | .()*#ABCDwp]+(?:;isub=[0-9\-.()]+)?(?:;postd=[0-9\-.( | |
# | | | )*#ABCDwp]+)?(?:;(?:phone-context)=(?:(?:(?:[+][0-9\- | |
# | | | .()]+)|(?:[0-9\-.()*#ABCDwp]+))|(?:(?:[!'E-OQ-VX-Z_e- | |
# | | | oq-vx-z~]|(?:%(?:2[124-7CFcf]|3[AC-Fac-f]|4[05-9A-Fa- | |
# | | | f]|5[1-689A-Fa-f]|6[05-9A-Fa-f]|7[1-689A-Ea-e])))(?:[ | |
# | | | !'()*\-.0-9A-Z_a-z~]+|(?:%(?:2[1-9A-Fa-f]|3[AC-Fac-f] | |
# | | | |[4-6][0-9A-Fa-f]|7[0-9A-Ea-e])))*)))(?:(?:;(?:phone- | |
# | | | context)=(?:(?:(?:[+][0-9\-.()]+)|(?:[0-9\-.()*#ABCDw | |
# | | | p]+))|(?:(?:[!'E-OQ-VX-Z_e-oq-vx-z~]|(?:%(?:2[124-7CF | |
# | | | cf]|3[AC-Fac-f]|4[05-9A-Fa-f]|5[1-689A-Fa-f]|6[05-9A- | |
# | | | Fa-f]|7[1-689A-Ea-e])))(?:[!'()*\-.0-9A-Z_a-z~]+|(?:% | |
# | | | (?:2[1-9A-Fa-f]|3[AC-Fac-f]|[4-6][0-9A-Fa-f]|7[0-9A-E | |
# | | | a-e])))*)))|(?:;(?:tsp)=(?: |(?:(?:(?:[A-Za-z](?:(?:( | |
# | | | ?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?)(?:[.](?:[A-Za-z | |
# | | | ](?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?))*))))|( | |
# | | | ?:;(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0 | |
# | | | -9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]) | |
# | | | )*)(?:=(?:(?:(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABD | |
# | | | Eabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0 | |
# | | | -9ACEace]))*)(?:[?](?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[1 | |
# | | | 3-7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa- | |
# | | | f]|7[0-9ACEace]))*))?)|(?:%22(?:(?:%5C(?:[a-zA-Z0-9\- | |
# | | | _.!~*'()]|(?:%[a-fA-F0-9][a-fA-F0-9])))|[a-zA-Z0-9\-_ | |
# | | | .!~*'()]+|(?:%(?:[01][a-fA-F0-9])|2[013-9A-Fa-f]|[3-9 | |
# | | | A-Fa-f][a-fA-F0-9]))*%22)))?))*))))|(?:(?:ftp)://(?:( | |
# | | | ?:(?:(?:[a-zA-Z0-9\-_.!~*'();:&=+$,]+|(?:%[a-fA-F0-9] | |
# | | | [a-fA-F0-9]))*))(?:)#)?(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][ | |
# | | | -a-zA-Z0-9]*)?[a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9 | |
# | | | ]*[a-zA-Z0-9]|[a-zA-Z])[.]?)|(?:[0-9]+[.][0-9]+[.][0- | |
# | | | 9]+[.][0-9]+)))(?::(?:(?:[0-9]*)))?(?:/(?:(?:(?:(?:(? | |
# | | | :[a-zA-Z0-9\-_.!~*'():#&=+$,]+|(?:%[a-fA-F0-9][a-fA-F | |
# | | | 0-9]))*)(?:/(?:(?:[a-zA-Z0-9\-_.!~*'():#&=+$,]+|(?:%[ | |
# | | | a-fA-F0-9][a-fA-F0-9]))*))*))(?:;type=(?:[AIai]))?))? | |
# | | | )|(?:(?:tv):(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)? | |
# | | | [a-zA-Z0-9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]| | |
# | | | [a-zA-Z])[.]?))?)|(?:(?:news):(?:(?:[*]|(?:(?:[-a-zA- | |
# | | | Z0-9$_.+!*'(),;/?:&=]|(?:%[a-fA-F0-9][a-fA-F0-9]))+#( | |
# | | | ?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0-9])[ | |
# | | | .])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z]))|(? | |
# | | | :[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))|(?:[a-zA-Z][-A- | |
# | | | Za-z0-9.+_]*))))|(?:(?:fax):(?:(?:(?:[+](?:[0-9\-.()] | |
# | | | +)(?:;isub=[0-9\-.()]+)?(?:;tsub=[0-9\-.()]+)?(?:;pos | |
# | | | td=[0-9\-.()*#ABCDwp]+)?(?:(?:;(?:phone-context)=(?:( | |
# | | | ?:(?:[+][0-9\-.()]+)|(?:[0-9\-.()*#ABCDwp]+))|(?:(?:[ | |
# | | | !'E-OQ-VX-Z_e-oq-vx-z~]|(?:%(?:2[124-7CFcf]|3[AC-Fac- | |
# | | | f]|4[05-9A-Fa-f]|5[1-689A-Fa-f]|6[05-9A-Fa-f]|7[1-689 | |
# | | | A-Ea-e])))(?:[!'()*\-.0-9A-Z_a-z~]+|(?:%(?:2[1-9A-Fa- | |
# | | | f]|3[AC-Fac-f]|[4-6][0-9A-Fa-f]|7[0-9A-Ea-e])))*)))|( | |
# | | | ?:;(?:tsp)=(?: |(?:(?:(?:[A-Za-z](?:(?:(?:[-A-Za-z0-9 | |
# | | | ]+)){0,61}[A-Za-z0-9])?)(?:[.](?:[A-Za-z](?:(?:(?:[-A | |
# | | | -Za-z0-9]+)){0,61}[A-Za-z0-9])?))*))))|(?:;(?:(?:[!'* | |
# | | | \-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0-9]|4[1-9A-Fa | |
# | | | -f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]))*)(?:=(?:(?: | |
# | | | (?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0-9] | |
# | | | |4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]))*) | |
# | | | (?:[?](?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]| | |
# | | | 3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEac | |
# | | | e]))*))?)|(?:%22(?:(?:%5C(?:[a-zA-Z0-9\-_.!~*'()]|(?: | |
# | | | %[a-fA-F0-9][a-fA-F0-9])))|[a-zA-Z0-9\-_.!~*'()]+|(?: | |
# | | | %(?:[01][a-fA-F0-9])|2[013-9A-Fa-f]|[3-9A-Fa-f][a-fA- | |
# | | | F0-9]))*%22)))?))*)|(?:[0-9\-.()*#ABCDwp]+(?:;isub=[0 | |
# | | | -9\-.()]+)?(?:;tsub=[0-9\-.()]+)?(?:;postd=[0-9\-.()* | |
# | | | #ABCDwp]+)?(?:;(?:phone-context)=(?:(?:(?:[+][0-9\-.( | |
# | | | )]+)|(?:[0-9\-.()*#ABCDwp]+))|(?:(?:[!'E-OQ-VX-Z_e-oq | |
# | | | -vx-z~]|(?:%(?:2[124-7CFcf]|3[AC-Fac-f]|4[05-9A-Fa-f] | |
# | | | |5[1-689A-Fa-f]|6[05-9A-Fa-f]|7[1-689A-Ea-e])))(?:[!' | |
# | | | ()*\-.0-9A-Z_a-z~]+|(?:%(?:2[1-9A-Fa-f]|3[AC-Fac-f]|[ | |
# | | | 4-6][0-9A-Fa-f]|7[0-9A-Ea-e])))*)))(?:(?:;(?:phone-co | |
# | | | ntext)=(?:(?:(?:[+][0-9\-.()]+)|(?:[0-9\-.()*#ABCDwp] | |
# | | | +))|(?:(?:[!'E-OQ-VX-Z_e-oq-vx-z~]|(?:%(?:2[124-7CFcf | |
# | | | ]|3[AC-Fac-f]|4[05-9A-Fa-f]|5[1-689A-Fa-f]|6[05-9A-Fa | |
# | | | -f]|7[1-689A-Ea-e])))(?:[!'()*\-.0-9A-Z_a-z~]+|(?:%(? | |
# | | | :2[1-9A-Fa-f]|3[AC-Fac-f]|[4-6][0-9A-Fa-f]|7[0-9A-Ea- | |
# | | | e])))*)))|(?:;(?:tsp)=(?: |(?:(?:(?:[A-Za-z](?:(?:(?: | |
# | | | [-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?)(?:[.](?:[A-Za-z]( | |
# | | | ?:(?:(?:[-A-Za-z0-9]+)){0,61}[A-Za-z0-9])?))*))))|(?: | |
# | | | ;(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEabde]|3[0-9 | |
# | | | ]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9ACEace]))* | |
# | | | )(?:=(?:(?:(?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13-7ABDEa | |
# | | | bde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f]|7[0-9 | |
# | | | ACEace]))*)(?:[?](?:(?:[!'*\-.0-9A-Z_a-z~]+|%(?:2[13- | |
# | | | 7ABDEabde]|3[0-9]|4[1-9A-Fa-f]|5[AEFaef]|6[0-9A-Fa-f] | |
# | | | |7[0-9ACEace]))*))?)|(?:%22(?:(?:%5C(?:[a-zA-Z0-9\-_. | |
# | | | !~*'()]|(?:%[a-fA-F0-9][a-fA-F0-9])))|[a-zA-Z0-9\-_.! | |
# | | | ~*'()]+|(?:%(?:[01][a-fA-F0-9])|2[013-9A-Fa-f]|[3-9A- | |
# | | | Fa-f][a-fA-F0-9]))*%22)))?))*))))|(?:(?:nntp)://(?:(? | |
# | | | :(?:(?:(?:(?:(?:(?:[a-zA-Z0-9][-a-zA-Z0-9]*)?[a-zA-Z0 | |
# | | | -9])[.])*(?:[a-zA-Z][-a-zA-Z0-9]*[a-zA-Z0-9]|[a-zA-Z] | |
# | | | ))|(?:[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+)))(?::(?:(?:[ | |
# | | | 0-9]+)))?)/(?:(?:[a-zA-Z][-A-Za-z0-9.+_]*))(?:/(?:[0- | |
# | | | 9]+))?))) | |
# +-----+----+-------------------------------------------------------+-----+
Can I do anything to make the output look something like:
# +-----+----+-------------------------------------------------------+-----+
# | GOT | OP | CHECK | LNs |
# +-----+----+-------------------------------------------------------+-----+
# | foo | =~ | $RE{URI} | 1 |
# +-----+----+-------------------------------------------------------+-----+
Seems like it is not possible to format the check message. Looking at the source:
Test2::Tools::Compare::is() calls
Test2::Compare::Delta::diag() which calls
Test2::Compare::Delta::table() to format the output.
You may try ask for a new feature at the GitHub issue tracker.

Spark Dataframe Union giving duplicates

I have a base dataset, and one of the columns is having null and not null values.
so I do:
val nonTrained_ds = base_ds.filter(col("col_name").isNull)
val trained_ds = base_ds.filter(col("col_name").isNotNull)
When I print that out, I get clear separate of rows. But when I do,
val combined_ds = nonTrained_ds.union(trained_ds)
I get duplicate records of rows from nonTrained_ds, and the strange thing is, rows from trained_ds are no longer in the combined ds.
Why does this happen?
the values of trained_ds are:
+----------+----------------+
|unique_no | running_id|
+----------+----------------+
|0456700001|16 |
|0456700004|16 |
|0456700007|16 |
|0456700010|16 |
|0456700013|16 |
|0456700016|16 |
|0456700019|16 |
|0456700022|16 |
|0456700025|16 |
|0456700028|16 |
|0456700031|16 |
|0456700034|16 |
|0456700037|16 |
|0456700040|16 |
|0456700043|16 |
|0456700046|16 |
|0456700049|16 |
|0456700052|16 |
|0456700055|16 |
|0456700058|16 |
|0456700061|16 |
|0456700064|16 |
|0456700067|16 |
|0456700070|16 |
+----------+----------------+
the values of nonTrained_ds are:
+----------+----------------+
|unique_no | running_id|
+----------+----------------+
|0456700002|null |
|0456700003|null |
|0456700005|null |
|0456700006|null |
|0456700008|null |
|0456700009|null |
|0456700011|null |
|0456700012|null |
|0456700014|null |
|0456700015|null |
|0456700017|null |
|0456700018|null |
|0456700020|null |
|0456700021|null |
|0456700023|null |
|0456700024|null |
|0456700026|null |
|0456700027|null |
|0456700029|null |
|0456700030|null |
|0456700032|null |
|0456700033|null |
|0456700035|null |
|0456700036|null |
|0456700038|null |
|0456700039|null |
|0456700041|null |
|0456700042|null |
|0456700044|null |
|0456700045|null |
|0456700047|null |
|0456700048|null |
|0456700050|null |
|0456700051|null |
|0456700053|null |
|0456700054|null |
|0456700056|null |
|0456700057|null |
|0456700059|null |
|0456700060|null |
|0456700062|null |
|0456700063|null |
|0456700065|null |
|0456700066|null |
|0456700068|null |
|0456700069|null |
|0456700071|null |
|0456700072|null |
+----------+----------------+
the values of the combined ds are:
+----------+----------------+
|unique_no | running_id|
+----------+----------------+
|0456700002|null |
|0456700003|null |
|0456700005|null |
|0456700006|null |
|0456700008|null |
|0456700009|null |
|0456700011|null |
|0456700012|null |
|0456700014|null |
|0456700015|null |
|0456700017|null |
|0456700018|null |
|0456700020|null |
|0456700021|null |
|0456700023|null |
|0456700024|null |
|0456700026|null |
|0456700027|null |
|0456700029|null |
|0456700030|null |
|0456700032|null |
|0456700033|null |
|0456700035|null |
|0456700036|null |
|0456700038|null |
|0456700039|null |
|0456700041|null |
|0456700042|null |
|0456700044|null |
|0456700045|null |
|0456700047|null |
|0456700048|null |
|0456700050|null |
|0456700051|null |
|0456700053|null |
|0456700054|null |
|0456700056|null |
|0456700057|null |
|0456700059|null |
|0456700060|null |
|0456700062|null |
|0456700063|null |
|0456700065|null |
|0456700066|null |
|0456700068|null |
|0456700069|null |
|0456700071|null |
|0456700072|null |
|0456700002|16 |
|0456700005|16 |
|0456700008|16 |
|0456700011|16 |
|0456700014|16 |
|0456700017|16 |
|0456700020|16 |
|0456700023|16 |
|0456700026|16 |
|0456700029|16 |
|0456700032|16 |
|0456700035|16 |
|0456700038|16 |
|0456700041|16 |
|0456700044|16 |
|0456700047|16 |
|0456700050|16 |
|0456700053|16 |
|0456700056|16 |
|0456700059|16 |
|0456700062|16 |
|0456700065|16 |
|0456700068|16 |
|0456700071|16 |
+----------+----------------+
This did the trick,
val nonTrained_ds = base_ds.filter(col("primary_offer_id").isNull).distinct()
val trained_ds = base_ds.filter(col("primary_offer_id").isNotNull).distinct()

Emacs Orgmode table $> References does not work

GNU Emacs 24.4.1 org-mode
Here is an org-mode table
#+TBLNAME: revenue
| / | < | | < | | < | | | | | | | | | | | |
| Product | Year_SUM | Month_SUM | Platform | Platform_SUM | adwo | AdMob | adChina | adSage | appfigures | appdriver | coco | Domob | Dianru | Limei | guohead | youmi |
| | | | | | | | | | | | | | | | | |
|---------+----------+-----------+----------+------------------+------+-------+---------+--------+------------+-----------+------+-------+--------+-------+---------+-------|
| Jan | | | iOS | #ERROR | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| | | | Android | =vsum($6..$>);NE | | 1 | | 1 | | 1 | | 1 | | 1 | | 1 |
|---------+----------+-----------+----------+------------------+------+-------+---------+--------+------------+-----------+------+-------+--------+-------+---------+-------|
| | | | | | | | | | | | | | | | | |
#+TBLFM: $5=vsum($6..$>);NE
As you see ,the formula $5=vsum($6..$>);NE can't be calculated! Here is debug info:
Substitution history of formula
Orig: vsum($6..$>)
$xyz-> vsum($6..$>)
#r$c-> vsum($6..$>)
$1-> vsum((0)..$>)
--------^
Error: Expected `)'
But if I replace the formula with $5=vsum($6..$17) and then it works ,I can't figure out where is the problem?
I need some help ,appreciate it!

Comonad example in Scala

What is Comonad, if it's possible describe in Scala syntax.
I found scalaz library implementation, but it's not clear where it can be useful.
Well, monads allow you to add values to them, change them based on a computation from a non-monad to a monad. Comonads allow you to extract values from them, and change them based on a computation from a comonad to a non-comonad.
The natural intuition is that they'll usually appear where you have a CM[A] and want to extract A.
See this very interesting post that touches on comonads a bit casually, but, to me at least, making them very clear.
What follows is a literal translation of code from this blog post.
case class U[X](left: Stream[X], center: X, right: Stream[X]) {
def shiftRight = this match {
case U(a, b, c #:: cs) => U(b #:: a, c, cs)
}
def shiftLeft = this match {
case U(a #:: as, b, c) => U(as, a, b #:: c)
}
}
// Not necessary, as Comonad also has fmap.
/*
implicit object uFunctor extends Functor[U] {
def fmap[A, B](x: U[A], f: A => B): U[B] = U(x.left.map(f), f(x.center), x.right.map(f))
}
*/
implicit object uComonad extends Comonad[U] {
def copure[A](u: U[A]): A = u.center
def cojoin[A](a: U[A]): U[U[A]] = U(Stream.iterate(a)(_.shiftLeft).tail, a, Stream.iterate(a)(_.shiftRight).tail)
def fmap[A, B](x: U[A], f: A => B): U[B] = U(x.left.map(f), x.center |> f, x.right.map(f))
}
def rule(u: U[Boolean]) = u match {
case U(a #:: _, b, c #:: _) => !(a && b && !c || (a == b))
}
def shift[A](i: Int, u: U[A]) = {
Stream.iterate(u)(x => if (i < 0) x.shiftLeft else x.shiftRight).apply(i.abs)
}
def half[A](u: U[A]) = u match {
case U(_, b, c) => Stream(b) ++ c
}
def toList[A](i: Int, j: Int, u: U[A]) = half(shift(i, u)).take(j - i)
val u = U(Stream continually false, true, Stream continually false)
val s = Stream.iterate(u)(_ =>> rule)
val s0 = s.map(r => toList(-20, 20, r).map(x => if(x) '#' else ' '))
val s1 = s.map(r => toList(-20, 20, r).map(x => if(x) '#' else ' ').mkString("|")).take(20).force.mkString("\n")
println(s1)
Output:
| | | | | | | | | | | | | | | | | | | |#| | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| | | |#| | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | |#|#| | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| |#| |#| | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#|#|#|#|#| | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| | | | | | | |#| | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | | | | | |#|#| | | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| | | | | |#| |#| | | | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#| | | | |#|#|#|#| | | | | | | |
| | | | | | | | | | | | | | | | | | | |#| | | |#| | | |#| | | |#| | | | | | |
| | | | | | | | | | | | | | | | | | | |#|#| | |#|#| | |#|#| | |#|#| | | | | |
| | | | | | | | | | | | | | | | | | | |#| |#| |#| |#| |#| |#| |#| |#| | | | |
| | | | | | | | | | | | | | | | | | | |#|#|#|#|#|#|#|#|#|#|#|#|#|#|#|#| | | |
| | | | | | | | | | | | | | | | | | | |#| | | | | | | | | | | | | | | |#| | |
| | | | | | | | | | | | | | | | | | | |#|#| | | | | | | | | | | | | | |#|#| |
| | | | | | | | | | | | | | | | | | | |#| |#| | | | | | | | | | | | | |#| |#|
| | | | | | | | | | | | | | | | | | | |#|#|#|#| | | | | | | | | | | | |#|#|#|#
The scalaz library provides a ComonadStore which extends the property of Comonad. It is defined like this:
trait ComonadStore[F[_], S] extends Comonad[F] { self =>
def pos[A](w: F[A]): S
def peek[A](s: S, w: F[A]): A
def peeks[A](s: S => S, w: F[A]): A =
peek(s(pos(w)), w)
def seek[A](s: S, w: F[A]): F[A] =
peek(s, cojoin(w))
def seeks[A](s: S => S, w: F[A]): F[A] =
peeks(s, cojoin(w))
def experiment[G[_], A](s: S => G[S], w: F[A])(implicit FG: Functor[G]): G[A] =
FG.map(s(pos(w)))(peek(_, w))
}
And a Store( which is analogous to (S => A, S)) has an instance of Comonad. You can look at this question that explains what it is more specifically.
You also have the Coreader and Cowriter Comonads that are the dual of the Reader and Writer Monads, here is an excellent blog post that talks about it in Scala.

Per-table constants are not being substituted

Here is my table
| | Name | Vert | Horz | Area | Cost | USD |
|---+-------------+---------+-------+--------+------+-----|
| $ | $price = 75 | $Hi=2.9 | | | | |
| # | Kitchen | 4.160 | 3.630 | #ERROR | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
#+TBLFM: $5=$4*$Hi*2+$3*$Hi*2
Here is trace output:
Substitution history of formula
Orig: $4*$Hi*2+$3*$Hi*2
$xyz-> $4*(#UNDEFINED_NAME)*2+$3*(#UNDEFINED_NAME)*2
#r$c-> $4*(#UNDEFINED_NAME)*2+$3*(#UNDEFINED_NAME)*2
$1-> (3.630)*(#UNDEFINED_NAME)*2+(4.160)*(#UNDEFINED_NAME)*2
---------^
Error: #'s not allowed in this context
What's wrong? Why $Hi was not substituted?
Ah, found it by myself. Here is wrong row, and the fixed row:
| $ | $price = 75 | $Hi=2.9 | | | | |
| $ | price = 75 | Hi=2.9 | | | | |