I have a program that writes dates as text in a column called Expression. Is there a way to write a query that will remove the day of the month? I need it to be something that I can share with multiple people to use in multiple databases. Examples of the format of the values stored are 6/2021, 06/2021, and 06/01/2021.
You can use IIf in the query to compare the length of the data, and pad/split as required. Something like:
SELECT
Expression,
IIf(Len([Expression])=6,"0" & [Expression],IIf(Len([Expression])=7,[Expression],IIf(Len([expression])=10,Left(Expression,2) & "/" & right(expression,4)))) AS OutputData
FROM tblFormatDate;
You may need to add a few more IIfs, and also a final false return statement. If it starts to get messy with the data, then you may want to look at creating a custom VBA function that makes the process flow easier to understand.
Regards,
Related
I'm already struggling for days to use dates from excel in a proper way in anylogic..
I've created a database in where the date is formulated as integers in different columns since otherwise excel is messing up the dates (for example year=2021 , month=12 day=5 hour=6 minute=44 second=0 stands for 2021/12/5 6:44:00)
Now I know this can be converted to a date by the function toDate(year, month, day, hour, minutes seconds). But how can I use this integers to create agent with specific parameters from the database in a source and add to a custom population?
The most simple way is to add a column where the function toDate(......) is added in the database but I do not know how to do this (see picture if it is unclear). Or are there other solutions?
One way: use Dynamic Events.
Create one and in the action code, write mySource.inject(1)
In Main, on startup, load all dbase rows and create a DE for each row, below assuming it is only with an hour-column:
(Use the database query wizard to adjust your query).
In your source object, set it to "call of inject() function"
This will work, but it is quite cumbersome, as you can see. Much easier if you get your Excel right and just import the date column clean and well so you can use the Source option "arrival table in database" directly. I know you need regular arrivals, so maybe code that up in Excel to give you these on specific dates...
I am creating a pipeline where the source is csv files and sink is SQL Server.
The date column in CSV file may have values like
12/31/2020
10162018
20201017
31/12/1982
1982/12/31
I do not find the function which checks the format of the date. How do I check the format and convert the above values to yyyy-MM-dd format.
The solution is given by HimanshuSinha-msft
Solved the issues using expression builder in Derived Column in Mapping Data Flow.
coalesce(toDate(Somedate,'MM/dd/yyyy'),toDate(Somedate,'yyyy/MM/dd'),toDate(Somedate,'dd/MM/yyyy'),toDate(Somedate,'MMddyyyy'),toDate(Somedate,'yyyyddMM'),toDate(Somedate,'MMddyyyy'),toDate(Somedate,'yyyyMMdd'))
This coalesce function answer will not actually solve the problem. It just gets rid of the errors. There are plenty of dates that are valid in multiple formats. For example: "2/1/2020" (mm/dd/yyyy) and "1/2/2020" (dd/mm/yyyy). The previous answer just gets rid of errors, but your analyses downstream will be very incorrect.
You need to do an aggregate analysis of which date format best fits the incoming stream, and the route the logic to the respective separate pipeline branches.
You can configure this in the Mapping tab of your copy activity. The datetime format can be specified, but it only supports one format type. If you have a mix of formats like in your example then it will not work.
One option would be to ingest the column into a staging table as a nvarchar. Then in another copy activity use a custom select statement to detect the column format and cast the date as needed. You should be able to do this using a CASE SQL statement in your SELECT from the staging table.
FYI: data type mapping
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping#data-type-mapping
I have a .csv file and I can't read it on Octave. On R I just use the command below and everything is read alright:
myData <- read.csv("myData.csv", stringsAsFactors = FALSE)
However, when I go to Octave it doesn't do it properly with the below command:
myData = csvread('myData.csv',1,0);
When I open the file with Notepad, the data looks something like the below. Note there isn't a comma separating the last column name (i.e. Column3) from the first value (i.e. Value1) and the same thing happens with the last value of the first row (i.e. Value3) and the first value of the second row (i.e Value4)
Column1,Column2,Column3Value1,Value2,Value3Value4,Value5,Value6
The Column1 is meant for date values (with format yyyy-mm-dd hh:mm:ss), I don't know if that has anything to do with the problem.
Alex's answers already explains why csvread does not work for your case. That function only reads numeric data and returns an array. Since your fields are all strings, you need something that reads a csv file into a cell array.
That function is named csv2cell and is part of the io package.
As a separate note, if you plan to make operation with those dates, you may want to convert those dates as strings, into serial date numbers. This will allow you to put your dates in a numeric array which will allow for faster operations and reduced memory usage. Also, the financial package has many functions to deal with dates.
csvread only reads numeric data, so a date does not qualify unfortunately.
In Octave you might want to check out the dataframe package. In Matlab you would do readtable.
Otherwise there are also more primitive functions you can use like textscan.
I'm trying to automate a process with SSIS that exports data into a flat file (.csv) that is then saved to a directory, where it will be scanned and imported by some accounting software. The software (unfortunately) only recognizes dates that are in MM/DD/YYYY fashion. I have tried every which way to cast or convert the data pulled from SQL to be in the MM/DD/YYYY, but somehow the data is always recognized as either a DT_Date or DT_dbDate data type in the flat file connection, and saved down as YYYY-MM-DD.
I've tried various combinations of data conversion, derived columns, and changing the properties of the flat file columns to string in hopes that I can at least use substring operations to get this formatted correctly, but it never fails to save down as YYYY-MM-DD. It is truly baffling. The preview in the OLE DB source will show the dates as "MM/DD/YYYY" but somehow it always changes to "YYYY-MM-DD" when it hits the flat file.
I've tried to look up solutions (for example, here: Stubborn column data type in SSIS flat flat file connection manager won't change. :() but with no luck. Amazingly if I merely open the file in Excel and save it, it will then show dates in a text editor as "MM/DD/YYYY", only adding more mystery to this Bermuda Triangle-esque caper.
If there are any tips, I would be very appreciative.
This is a date formatting issue.
In SQL and in SSIS, dates have one literal string format and that is YYYY-MM-DD. Ignore the way they appear to you in the data previewer and/or Excel. Dates are displayed to you based upon your Windows regional prefrences.
Above - unlike the US - folks in the UK will see all dates as DD/MM/YYYY. The way we are shown dates is NOT the way they are stored on disk. When you open in Excel it does this conversion as a favor. It's not until you SAVE that the dates are stored - as text - according to your regional preferences.
In order to get dates to always display the same way. We need to save them not as dates, but as strings of text. TO do this, we have to get the data out of a date column DT_DATE or DT_DBDATE and into a string column: DT_STR or DT_WSTR. Then, map this new string column into your csv file. Two ways to do this "date-to-string" conversion...
First, have SQL do it. Update your OLE DB Source query and add one more column...
SELECT
*,
CONVERT(VARCHAR(10), MyDateColumn, 101) AS MyFormattedDateColumn
FROM MyTable
The other way is let SSIS do it. Add a Derived Column component with the expression
SUBSTRING([MyDateColumn],6,2) + "/" + SUBSTRING([MyDateColumn],8,2) + "/" + SUBSTRING([MyDateColumn],1,4)
Map the string columns into your csv file, NOT the date columns. Hope this helps.
It's been a while but I just came across this today because I had the same issue and hope to be able to spare someone the trouble of figuring it out. What worked for me was adding a new field in the Derived Column transform rather than trying to change the existing field.
Edit
I can't comment on Troy Witthoeft's answer, but wanted to note that if you have a Date type input, you wouldn't be able to do SUBSTRING. Instead, you could use something like this:
(DT_WSTR,255)(MONTH([Visit Date])) + "/" + (DT_WSTR,255)(DAY([Visit Date])) + "/" + (DT_WSTR,255)(YEAR([Visit Date]))
1) I need to pass a date (billing) to the RFC, but I am not sure how to map using tmap. How to set it up (see screenshots).
2) I need to run this job daily (M-F) and I am not sure how to automate the date input
3) For the date input, I thought of using a joblet, but I can't find it in Talend. Most screenshots shows the Joblets in the same window as job designs and metadata, but I don't have it. Seen Joblet image.
As you might guessed, I am very new to Talend.
Use a tMap, and inside it use the function TalendDate.parseDate("yyyy-MM-dd", sap_data.date) in the expression field where you want the output. Also, note that the output type must be Date. The Date Pattern in the type definition (on the bottom of the tmap) is irrelevant.
Something like that: