Data Fusion Wrangle CSV with empty string - google-cloud-data-fusion

CSV File to import
"Date","Facility","FacilityNo"
"2022-07-27 00:00:00","Goku Nursing and Rehabilitation Center-1078","1078"
"2022-07-27 00:00:00","Gohan Health and Rehabilitation Center-43","433"
"2022-07-27 00:00:00","Trunks Health and Rehabilitation Center-420",""
When opening the above file (located in GCS) in Wrangle I get the following error.
Failed to call connection service with status 500: field FacilityNo cannot be set to a null value
As soon I put a value in that empty field in the 3rd line everything works.
What is the correct way to handle this in DataFusion?

Google Support (ticket #40900774) was able to reproduce the problem and they submitted the issue to the DataFusion product team.
We found a workaround. Import the file as "Text" then use wrangler to parse it as a CSV.

Related

quickfixj Market Data can't get group field

"8=FIX.4.29=11435=W49=TEST56=MYTEST234=552=20180614-21:36:4955=HKGUSD268=2269=0270=1.0271=20269=1270=0.0271=0262=010=179"
My scala processes this with "58=Conditionally Required Field Missing372=X"
Here is the code snapit:
msg: MarketDataIncrementalRefresh
val group = new MarketDataIncrementalRefresh.NoMDEntries()
msg.getGroup(1, group) // this is already throwing out that error
I pasted the fix message to the online parser and seems ok to get all the field back.
My cfg file has UseDataDictionary=N would this matter?
This is fixed by adding the data dictionary and xml files to the configuration. And it seems that this is only required when parsing repeating groups.

Unable to run experiment on Azure ML Studio after copying from different workspace

My simple experiment reads from an Azure Storage Table, Selects a few columns and writes to another Azure Storage Table. This experiment runs fine on the Workspace (Let's call it workspace1).
Now I need to move this experiment as is to another workspace(Call it WorkSpace2) using Powershell and need to be able to run the experiment.
I am currently using this Library - https://github.com/hning86/azuremlps
Problem :
When I Copy the experiment using 'Copy-AmlExperiment' from WorkSpace 1 to WorkSpace 2, the experiment and all it's properties get copied except the Azure Table Account Key.
Now, this experiment runs fine if I manually enter the account Key for the Import/Export Modules on studio.azureml.net
But I am unable to perform this via powershell. If I Export(Export-AmlExperimentGraph) the copied experiment from WorkSpace2 as a JSON and insert the AccountKey into the JSON file and Import(Import-AmlExperiment) it into WorkSpace 2. The experiment fails to run.
On PowerShell I get an "Internal Server Error : 500".
While running on studio.azureml.net, I get the notification as "Your experiment cannot be run because it has been updated in another session. Please re-open this experiment to see the latest version."
Is there anyway to move an experiment with external dependencies to another workspace and run it?
Edit : I think the problem is something to do with how the experiment handles the AccountKey. When I enter it manually, it's converted into a JSON array comprising of RecordKey and IndexInRecord. But when I upload the JSON experiment with the accountKey, it continues to remain the same and does not get resolved into RecordKey and IndexInRecord.
For me publishing the experiment as a private experiment for the cortana gallery is one of the most useful options. Only the people with the link can see and add the experiment for the gallery. On the below link I've explained the steps I followed.
https://naadispeaks.wordpress.com/2017/08/14/copying-migrating-azureml-experiments/
When the experiment is copied, the pwd is wiped for security reasons. If you want to programmatically inject it back, you have to set another metadata field to signal that this is a plain-text password, not an encrypted password that you are setting. If you export the experiment in JSON format, you can easily figure this out.
I think I found the issue why you are unable to export the credentials back.
Export the JSON graph into your local disk, then update whatever parameter has to be updated.
Also, you will notice that the credentials are stored as 'Placeholders' instead of 'Literals'. Hence it makes sense to change them to Literals instead of placeholders.
This you can do by traversing through the JSON to find the relevant parameters you need to update.
Here is a brief illustration.
Changing the Placeholder to a Literal:

JSON schema validation failed: resource: String does not match pattern:^/[^/~!#\$%^|\s`#&*()\-+={}\[\]:;"'<>,?/\|\\]+(/[^/~!#\

I'm trying to embed Jasper Reports into APEX App. I'm able to gather reports from samples (samples on JasperCommunity website), however, when I'm trying to get created by me and colleagues reports - every time I have the same error:
JSON schema validation failed: resource: String does not match pattern: ^/[^/~!#\$%^|\s#&*()\-+={}\[\]:;"'<>,?/\|\\]+(/[^/~!#\$%^|\s#&()-+={}[]:;"'<>,?/\|\]+)$.
In other words I can't get any of our reports apart from the samples (e.g. /public/Samples/Reports/03._Store_Segment_Performance_Report"). I think the case is that the path to the Report is wrong, but I've tried all possible and impossible options and none of them works. Anyone any ideas please ?! Thanks
P.S. APEX 4.2.6, JasperServer 6.0. And finally, I can get samples Reports ONLY under the JasperAdmin user, simple User always got an error - 'Access Denied'. Why ?!?!?!
Sorted.
If anyone interested - when specify the actual path, in Jasper: right click on Report(or Dashboard, AdHocView etc.) and copy the path from it. As it differs from when you point mouse on it (e.g. adding the underscores etc.), and paste this actual path into your javascript code into HTML section in your web app. Thanks

How to save message file in RAW format

One of our client does not know how to save/generate/produce/provide swift messages in its raw format via their swift alliance system, neither do i. By raw format i mean following format
{1:F21XXXXXXXXXXXX7108170239}{4:{177:1112071543}{451:0}}{1:F01XXXXXXXXXXXX7108170239}{2:O9502137111207XXXXXXXXXXXX54153885641112071537N}{4:
:20:XXXXXXXXXXXXXX
:25:XXXXXXXXXX
:28C:00237/00001
:60F:C090513AUD1162994,30
:61:1112061207D110,00NCHKFDD388407 //071211WDM9103
291111 GUJ
:61:1112061207D200,00NCHK3927611471 //071211WDM9104
090911 KAR
:62F:C090513AUD1610881,52
:64:C090513AUD1610881,52
-}{5:{CHK:XXXXXXXXXXXX}}{S:{COP:S}}
Is there any way/interface to integrate with SWIFT and extract these files Or is there any possibility to schedule SWIFT to generate these files Or even manual steps to do it?
Edit 1
Messages are saved by creating a profile from the application and setting connection type to "File Transfer". It will then keep generating files in ISO format to a shared location.
to get output file from swift you need to configure message partner CBSOUT
to do so do the following
enter to swift application SAE or SAA with super user.
click on
Application interface.
from the list right click on
CBSOUT and then enable
to configure CBSOUT double click on it.
Data format :select RJE
output path name : put the folder you want to store the files
output the extension : put the file extension example :out or txt.
session initiation select Auto.
number of message =1
finish.
good lucky.
byeeeeeee
Messages are saved by creating a profile from the application and setting connection type to "File Transfer". It will then keep generating files in ISO format to a shared location.

Error showing at tAccessInput component

While using tAccessInput component in my job, it showing error like,
It may not be a database that your application recognizes, or the file may be corrupt.
But, it all the connections and database/table names are valid in my job.
What can be the problem. and how can i resolve it.
Metioning the exact DBVersion is more important for tAccessInput/tAccessOutput.
Like Access 2003/Access 2007 and associated database file name.