i have to queries which I want to merge.
these are two sample columns i want to compare.
i need to find the string from the second list, in the first one. If there is a match, append a list with the matching string to the first list.
Those lists are from two different queries
Those lists are from two different queries
That's fine, you can reference another query using its name. That could be Source1 and Source2 below.
need to find the string from the second list, in the first on
append a list with the matching string to the first list.
I wasn't sure if you meant
forall B in A =>
Text.Combine( {_} & {A}, "_")
I thought you meant to accumulate a list, that's what I did.
let
json1 = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45Wysw1NjIxNTMz0isoKlGK1QGJmJqhiiQnlsRD1SGpAvLj0TSiG1Wc9mH+hI1pWUBxHCowLcNqEUF7oBJA67YZG5sCqQ7jrDSIfCwA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [File = _t]),
Source1 = Table.TransformColumnTypes(json1,{{"File", type text}}),
json2 = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WSkzLyjEyNskyVYrViVYyNjKBMMzMjMD0h/kTtsEYHWBGZi6YgqlTio0FAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Substring = _t]),
Source2 = Table.TransformColumnTypes(json2,{{"Substring", type text}}),
/*
find all matching values in the source text
then merge the matching values with separator
optional params with defaults:
separator: string to join on, "_" by default
comparer: case sensitivity, off by default
*/
SummarizeMatchingText = (source as text, patterns as list, optional options as nullable record) as any =>
let
separator = options[separator]? ?? "_",
comparer = options[comparer]? ?? Comparer.OrdinalIgnoreCase,
select_matching = List.Select(
patterns,
(item) =>
Text.Contains( source, item, comparer)
),
merged = Text.Combine( select_matching, separator)
in
merged,
col_matches = Table.AddColumn(
Source1,
"Matches",
(row) =>
SummarizeMatchingText(
row[File], Source2[Substring],
[ separator = "—" ]
),
Text.Type
),
// summarize multiple steps
Summary = [
Source1 = Source1,
Source2 = Source2,
col_matches = col_matches,
expectEmpty = SummarizeMatchingText(
"sf🐱fj324", Source2[Substring] ),
expectMatch = SummarizeMatchingText(
"cat_im32456.prt", Source2[Substring] )
],
col_matches1 = Summary[col_matches]
in
col_matches1
Related
I have a dataframe column table_name which is having below string value:
tradingpartner.parent_supplier,lookup.store,lab_promo_invoice.tl_cc_mbr_prc_wkly_inv,lab_promo_invoice.mpp_club_card_promotion_funding_view,lab_promo_invoice.supplier_sale_apportionment_cc,tradingpartner.supplier,stores.rpm_zone_location_mapping,lookup.calendar
How to extract DB name and table name from the above string and store it as DB name in one column and tablename in another column.
I want the output as below
One possible solution is to define two different UDF to achieve the goal.
Starting from this input DataFrame, called dfInput:
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|table_name |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|tradingpartner.parent_supplier,lookup.store,lab_promo_invoice.tl_cc_mbr_prc_wkly_inv,lab_promo_invoice.mpp_club_card_promotion_funding_view,lab_promo_invoice.supplier_sale_apportionment_cc,tradingpartner.supplier,stores.rpm_zone_location_mapping,lookup.calendar,sauces,plant|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The first UDF, called dbName, is responsible to take from the input String column all the database names from the String:
def dbNames(k: String): String = {
// this String is the returning value
// containing all the databases from the input string
var dbNames=""
// split the input String by comma
val arrays = k.split(",")
for (str <- arrays){
// if in the input String there is a value like
// database.table take just the database value
if(str.contains(".")) {
val indexOfPoint = str.indexOf(".")
dbNames += str.substring(0, indexOfPoint) + ", "
}
}
// delete last occurence of char ", "
return dbNames.dropRight(2)
}
val dbName = udf[String, String](dbNames)
The second UDF, called tableName, is responsible to take from the input String column all the table names from the String:
def tableNames(k: String): String = {
// this String is the returning value
// containing all the tables from the input string
var tableNames=""
// split the input String by comma
val arrays = k.split(",")
for (str <- arrays){
// if in the input String there is a value like
// database.table take just the table value
// else is intended to be just the table name
if(str.contains(".")) {
val indexOfPoint = str.indexOf(".")
tableNames += str.substring(indexOfPoint+1) + ", "
}
else tableNames += str + ", "
}
// delete last occurence of char ", "
return tableNames.dropRight(2)
}
val tableName = udf[String, String](tableNames)
Then, to obtain the expected output we need to call the UDFs like the following:
val dfOutput = dfInput.withColumn("DBName", dbName(col("table_name")))
.withColumn("Table", tableName(col("table_name")))
dfOutput.show(true)
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|table_name |DBName |Table |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|tradingpartner.parent_supplier,lookup.store,lab_promo_invoice.tl_cc_mbr_prc_wkly_inv,lab_promo_invoice.mpp_club_card_promotion_funding_view,lab_promo_invoice.supplier_sale_apportionment_cc,tradingpartner.supplier,stores.rpm_zone_location_mapping,lookup.calendar,sauces,plant|tradingpartner, lookup, lab_promo_invoice, lab_promo_invoice, lab_promo_invoice, tradingpartner, stores, lookup|parent_supplier, store, tl_cc_mbr_prc_wkly_inv, mpp_club_card_promotion_funding_view, supplier_sale_apportionment_cc, supplier, rpm_zone_location_mapping, calendar, sauces, plant|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
So I have this portion of code :
stringName = "\(param1)\(param2)\(param3)\(param4)\(param5)"
All these params are calculated inside the function in which the stringName is being called.
What I wanna do is output the string name with all these params separated by say a coma, which in theory would be just as easy as separating them by a coma inside the string, however I want the coma to appear only between these params when they exist and they won't all exist in every case.
Because the function I'm creating depends on an array and these params on the array length, so in some cases the returned string will be :
stringName = "param1,param4"
or
stringName = "param2,param3,param5"
etc...
(got a maximum of 8 of these params)
So as sometimes these params end up being empty strings, I don't wanna end up with a string name only made of comas, just want my comas to appear inbetween 2 params when they're not empty strings.
Any idea how to format that ??
Thanks !
Assuming param1,...,param5 are all String, non-Optional, you can write something like this:
let param1: String = ""
let param2: String = "value2"
let param3: String = "value3"
let param4: String = ""
let param5: String = "value5"
let stringName = [param1, param2, param3, param4, param5]
.filter {!$0.isEmpty}
.joined(separator: ",")
print(stringName) //->value2,value3,value5
You can put all your paramXs in an array, filter out the empty ones, and used joined(separator:) to join them:
stringName = [param1, param2, param3, param4, param5, param6, param7, param8]
.filter { !$0.isEmpty }.joined(separator: ",")
An even better way of doing this would be not use 8 different variables, and rewrite your method so that it uses an array from the very beginning.
My Original Code
func getUserByEmpNum(_ id: Int) -> String {
let nameQuery: String = "SELECT fld_str_firstname, fld_str_lastName FROM userView WHERE fld_int_id = \(id);"
var returnStr = ""
do {
let dbQueue = try DatabaseQueue(path: MYCDatabase.pathToDatabase)
try dbQueue.inTransaction(.none) { (db) -> Database.TransactionCompletion in
let returnStrs = try String.fetchAll(db, sql: nameQuery)
// Placing a breakpoint here, returnStrs only has one element?
return .commit
}
} catch {
print (error)
}
return returnStr
}
My Question
In this code if I do a query like select fld_str_firstname from myOwnUserView where fld_int_id = 2; I get one element in my returnStrs array, which is as expected. Then selecting the two fields, as in nameQuery, I still only ever get one string in the returnStrs array.
Why is this, and how do I fit it to get all the selected columns in the response?
String.fetchAll returns an array of Strings extracted from the leftmost selected column, as documented. One string for each fetched row. Not one string for each selected column.
If you want to grab strings from several columns, use Row.fetchAll, which returns an array of database rows. From those rows, you can extract each column you are interested into:
let rows = try Row.fetchAll(db, sql: "SELECT fld_str_firstname, fld_str_lastName FROM ...")
for row in rows {
let firstName: String = row["fld_str_firstname"]
let lastName: String = row["fld_str_lastName"]
}
See this chapter of the documentation for more information about extracting values from database rows.
Since you are reading the name from a single row identified with its id, you may prefer the fetchOne method, which consumes a single database row (see Fetching Methods):
if let row = try Row.fetchOne(db, sql: "SELECT ... WHERE fld_int_id = ?", arguments: [id]) {
let firstName: String = row["fld_str_firstname"]
let lastName: String = row["fld_str_lastName"]
// Use first and last name
} else {
// ID does not exist in the database: do what is appropriate.
}
I am trying to create a type safe data access layer in F# using FSharp.Data.SqlClient type providers to be callable from C#. I have highly complicated SQL queries that are performance critical but there are also several variations.
Given the following (obviously simplified schema):
CREATE TABLE [dbo].[company] (
[id] INT IDENTITY (1, 1) NOT NULL,
[name] VARCHAR (50) NOT NULL)
I have the following F# code:
module CompanyDAL =
open FSharp.Data // requires FSharp.Data.SqlClient package
[<Literal>]
let conn = "Data Source=(localdb)\ProjectsV13;Initial Catalog=TESTDB;Integrated Security=True;Pooling=False;Connect Timeout=30"
[<Literal>]
let baseQuery = "select id, name from company where 1 = 1 "
[<Literal>]
let filterById = "and id = #id"
[<Literal>]
let filterByName = "and name = #name"
[<Literal>]
let queryFilteredById = baseQuery + filterById
type GetCompanyById = SqlCommandProvider<queryFilteredById, conn>
[<Literal>]
let queryFilterbyName = baseQuery + filterByName
type GetCompanyByName = SqlCommandProvider<queryFilterbyName, conn>
type GetAllCompanies = SqlCommandProvider<baseQuery, conn>
type Company = {
Name : string
id : int
}
let getCompanyById (runtimeConn:string) =
async {
use query = new GetCompanyById(runtimeConn)
let! result = query.AsyncExecute(id = 10)
return result
|> Seq.map (fun x ->
{ Name = x.name
id = x.id })
|> Seq.toArray
} |> Async.StartAsTask
let getCompanyByName (runtimeConn:string) =
async {
use query = new GetCompanyByName(runtimeConn)
let! result = query.AsyncExecute(name = "abc" )
return result
|> Seq.map (fun x ->
{ Name = x.name
id = x.id })
|> Seq.toArray
} |> Async.StartAsTask
let getAllCompanies (runtimeConn:string) =
async {
use query = new GetAllCompanies(runtimeConn)
let! result = query.AsyncExecute()
return result
|> Seq.map (fun x ->
{ Name = x.name
id = x.id })
|> Seq.toArray
} |> Async.StartAsTask
Is there any way I can reduce the number of duplication while maintaining the raw performance of type safe raw queries?
I've tried using
type GetCompany = SqlEnumProvider<baseQuery, conn, provider>
Then I can write this code
GetCompany.Items |> Seq.where(fun (id, name) -> name = "xxx") |> printfn "%A"
So I can avoid duplication and pass a generic predicate fun (id, name) -> to the only GetCompany type.
Similarly,
GetCompany.Items |> Seq.where(fun (id, name) -> id = "1")
would work as well.
Note
Anyway, notice that the above will filter in memory. In fact Dynamic SQL is not permitted with this library, because it relies only on static analysis at compile time. If you need to construct queries dynamically, you can revert to raw ADO.NET or other light ORM.
I implemented Sqlite in my project by adding #import <sqlite3.h> in my header.h file and libsqlite3.dylib.
How can I pass an array as parameter to my query, here is what I thought:
var arrayId = [1,2] // array with interested Id
var query:NSString = "Select id from Product where id IN \(arrayId)" // I want to select Products with id that are in the array
Edit:
Does it change if arrayId is NSArray ? Because I also need arrayId as NSArray.
then I proceed with open sqlite database, prepare query and so on.
Thank you in advance.
You can easily combine the array into a string with join function.
var arrayId = [1,2] // array with interested Id
var inExpression = ",".join(map(arrayId) { "\($0)"})
// inExpression = "1,2"
var query = "Select id from Product where id IN (\(inExpression))"
Update for Swift3:
var arrayId = [1,2] // array with interested Id
var inExpression = arrayId.flatMap{ String($0) }.joined(separator: ",")
// inExpression = "1,2"
var query = "SELECT id FROM Product WHERE id IN (\(inExpression))"
You need to accomplish two things: convert your array of Ints to Strings and then implode the array into a string, by joining them with commas (as you would want to do with an IN SQL statement).
Here's a rudimentary function that does just that:
func implode(ints: [Int]) -> String {
// Convert to Strings
let strs = ints.map { String($0) }
// Join Strings with commas
return ",".join(strs)
}
And then in use:
"WHERE id IN (\(implode(arrayId)))"
I'd probably use something like:
var arrayId = [1,2] // array with interested Id
var str = ",".join(arrayId.map { return "\($0)" })
var query = "SELECT id FROM Product WHERE id IN [\(str)]"
Using Swift's own string interpolation to create SQL statements can be risky (as with any language). The sqlite3 library provides parameter binding for this purpose:
if (statement.prepare("SELECT name FROM products WHERE id = ?") != .Ok) {
// Deal with error here
}
// Bind the question mark to your value
statement.bindInt(1, value: 8766)
if (statement.step() == .Row) {
let name = statement.getStringAt(1)
// ...do something with your data from the statement
}
// Done.
statement.finalizeStatement()
EDIT:
For the comment below, you need () brackets, not []:
select id, body from test where id in (1,2);
not
select id, body from test where id in [1,2];