Getting wrong Count of the Combinations Between Two Fields of a File thru OUTFIL - jcl

i need to include this condition:
1) Total no.of records per combination of field1 and field3 (INCLUDE=(1,2,8,3,CH,A)
INPUT FILE: FIELD1 AND FIELD3 have 5 combinations,if you see in example below
field1 field2 field3 field4
AA 00000 123 ABC
AA 00000 123 ABC
AA 00000 456 ABC
BB 00000 123 ABC
BB 00000 123 ABC
BB 00000 789 ABC
AA 00000 567 ABC
OUTPUT FILE: gets 5 rows, one for each combination, gives no.of occurrences for it
FIELD1 FIELD3 COUNT-OF-COMBINATION
AA 123 2
AA 456 1
AA 567 1
BB 123 2
AA 789 1
My method is:
//SYSIN DD *
SORT FIELDS=COPY
OPTION COPY
OUTFIL REMOVECC,NODETAIL,
TRAILER1=(1,2,'ON',8,3,'=',COUNT=(M11,LENGTH=10)))
/*
Answer i got is:
AA ON 123 = 7
which is wrong:
its should have been
AA ON 123 = 2
AA ON 456 = 1
AA ON 567 = 1
BB ON 123 = 2
AA ON 789 = 1

You have:
SORT FIELDS=COPY
OPTION COPY
OUTFIL REMOVECC,NODETAIL,
TRAILER1=(1,2,'ON',8,3,'=',COUNT=(M11,LENGTH=10)))
First problem is you have SORT FIELDS=COPY and OPTION COPY. These mean the same thing. Remove one or the other (I tend to use OPTION COPY).
Next, you have a spare right parenthesis.
Then you are using TRAILER1. There are three types of TRAILERn: 1 is "at the end of the report"; 2 is at the end of a page; 3 is at a control break.
You use TRAILER1, so at the end of your file you get one record, containing file totals.
After that, your positions for the TRAILER1 match the output, not the input file.
Which brings us to the fact that you are not running those Sort control cards with that data. The control cards have a syntax error which means they don't run. Correcting the cards and retaining the TRAILER1 gets AAON567=0000000007.
Which brings us to the control break, which is what you have missing.
You define a control break with SECTIONS. TRAILER3 is part of SECTIONS.
Fixing everything except your output format:
OPTION COPY
OUTFIL REMOVECC,NODETAIL,
SECTIONS=(1,2,
15,3,
TRAILER3=(1,2,
'ON',
15,3,
'=',
COUNT=(M11,
LENGTH=10)))
Which gives you:
AAON123=0000000002
AAON456=0000000001
BBON123=0000000002
BBON789=0000000001
AAON567=0000000001
If you want column headings, look at how to use HEADER3 (HEADER1 and HEADER2 would also work in this simple case). If you want "page totals" look at TRAILER2. If you want file totals, use TRAILER1.

Related

Relational Division Issue to Solve for Total

I have a table of Card holders and the amount of each cards they have.
I'm trying to query to get a result of Full Card Set holders and how many sets they each have.
The data is simular to below:
**Cards **
Card
User
Count
AA
1234
2
BB
1234
2
CC
1234
2
DD
1234
1
AA
4321
1
BB
4321
1
CC
4321
1
AA
3321
3
BB
2221
1
CC
2221
4
**Card Sets **
Card
Set
AA
SetA
BB
SetA
CC
SetA
If looking for "SetA" I would expect below.
**Results **
User
TotalSets
1234
2
4321
1
I know this is possibly a Relational Division solution but not sure how to get the count of sets per Card Holder.
This is what I was using before I needed total sets:
`SELECT
cs.[Set],
tc.User
FROM Cards tc
JOIN (
SELECT *,
Total = COUNT(*) OVER (PARTITION BY CardSets.[Set])
FROM CardSets cs
) ts ON tc.Card = cs.Card
GROUP BY
cs.[Set], tc.User
HAVING COUNT(*) = MIN(cs.Total);`
A blind try, please provide us a create table/insert script for convenience:
select
c.[user]
,cs.[set]
-- if any cs row
,fullsets=
min(case when c.card is null then 0 else 1 end) -- if card missing for a set, set this flag to 0 to absorb results
*
min(c.[count])
from
cardsets cs
left join cards c on c.card=cs.card -- if a card belongs to multiple sets, you'll need to add here: and cs.[set]='SetA'
group by c.[user],cs.[set]

PostgreSQL: Count Number of Occurrences in Columns

BACKGROUND
I have three large tables (employee_info, driver_info, school_info) that I have joined together on common attributes using a series of LEFT OUTER JOIN operations. After each join, the resulting number of records increased slightly, indicating that there are duplicate IDs in the data. To try and find all of the duplicates in the IDs, I dumped the ID columns into a temp table like so:
Original Dump of ID Columns
first_name
last_name
employee_id
driver_id
school_id
Mickey
Mouse
1234
abcd
wxyz
Donald
Duck
2423
heca
qwer
Mary
Poppins
1111
acbe
aaaa
Wiley
Cayote
1234
strf
aaaa
Daffy
Duck
1256
acbe
pqrs
Bugs
Bunny
9999
strf
yxwv
Pink
Panther
2222
zzzz
zzaa
Michael
Archangel
0000
rstu
aaaa
In this overly simplified example, you will see that IDs 1234 (employee_id), strf (driver_id), and aaaa (school_id) are each duplicated at least once. I would like to add a count column for each of the ID columns, and populate them with the count for each ID used, like so:
ID Columns with Counts
first_name
last_name
employee_id
employee_id_count
driver_id
driver_id_count
school_id
school_id_count
Mickey
Mouse
1234
2
abcd
1
wxyz
1
Donald
Duck
2423
1
heca
1
qwer
1
Mary
Poppins
1111
1
acbe
1
aaaa
3
Wiley
Cayote
1234
2
strf
2
aaaa
3
Daffy
Duck
1256
1
acbe
1
pqrs
1
Bugs
Bunny
9999
1
strf
2
yxwv
1
Pink
Panther
2222
1
zzzz
1
zzaa
1
Michael
Archangel
0000
1
rstu
1
aaaa
3
You can see that IDs 1234 and strf each have 2 in the count, and aaaa has 3. After generating this table, my goal is to pull out all records where any of the counts are greater than 1, like so:
All Records with One or More Duplicate IDs
first_name
last_name
employee_id
employee_id_count
driver_id
driver_id_count
school_id
school_id_count
Mickey
Mouse
1234
2
abcd
1
wxyz
1
Mary
Poppins
1111
1
acbe
1
aaaa
3
Wiley
Cayote
1234
2
strf
2
aaaa
3
Bugs
Bunny
9999
1
strf
2
yxwv
1
Michael
Archangel
0000
1
rstu
1
aaaa
3
Real World Perspective
In my real-world work, the JOIN'd table contains 100 columns, 15 different ID fields and over 30,000 records, and the final table came out to be 28 more than the original. This may seem like a small amount, but each of the 28 represent a broken link that we must fix.
Is there a simple way to get the counts populated like in the second table above? I have been wrestling with this for hours already, and have not been able to make this work. I tried some aggregate functions, but they cannot be used in table UPDATE operations.
The COUNT function, when used as an analytic function, can do what you want here, e.g.
WITH cte AS (
SELECT *,
COUNT(employee_id) OVER (PARTITION BY employee_id) employee_id_count,
COUNT(driver_id) OVER (PARTITION BY driver_id) driver_id_count,
COUNT(school_id) OVER (PARTITION BY school_id) school_id_count
FROM yourTable
)
SELECT *
FROM cte
WHERE
employee_id_count > 1
driver_id_count > 1
school_id_count > 1;

How to concatenate columns in update statement

I have this table:
t:([] name:("aaa";"bbb";"ccc";"dddd"); side:(1;2;1;2))
Now I want to add a new column "concatenated", which contains a symbol, which is the concatenation of both values for each row:
I would assume that I have to do this with an each-both adverb, but this here does not work:
update concatenated:((`$name),'(`$side)) from t
How would I have to change this? Thanks.
Your attempt is close the issue with it works if you convert the 'side' column to string format first
I've added two versions one where the concatenation does not merge the 2 values and one where they are merged as a single symbol
q)t:([] name:("aaa";"bbb";"ccc";"dddd"); side:(1;2;1;2))
q)update conc:((`$name),'`$string side) from t
name side conc
------------------
"aaa" 1 aaa 1
"bbb" 2 bbb 2
"ccc" 1 ccc 1
"dddd" 2 dddd 2
q)update conc:(`$name,'string side) from t
name side conc
-----------------
"aaa" 1 aaa1
"bbb" 2 bbb2
"ccc" 1 ccc1
"dddd" 2 dddd2
Hope this helps

how to concatenate or append current value with existing value in Datastage

I need achive below requirement i.e
Input -- at very first time
Order value
1111 aaa
222 bbb
333 ccc
in the target (Insert) I will have
Order value
Order value
1111 aaa
222 bbb
333 ccc
----------Input -- at second time
Order value
1111 Aaa1
222 Bbb2
333 ccc
Out put must be
Order value
1111 aaa Aaa1
222 bbb Bbb2
So on
I need to keep appending change values for the corresponding key column ..
111 aaa aaa1 aaa2 aaa3 ..like this
Please help
You can follow these steps:
Use a CDC stage. Here by default it keeps the following functions: 0 for copy/duplicate record,1 for insert ,b2 for delete, 3 for update.
Now the link which would be carrying the copy record simply connect it to a transformer where declare a stage variable that would be incremented by 1.
Next in the field derivation write as CONCAT(string,'Initcaps(a'),'aa',svarcount,'')

KDB get substring

How can I add a column containing a substring of a another columns containing symbols. So, go from
t:flip `date`sym`pos!(`d1`d1`d1`d2;`aaaA1`bbbA1`aaaA2`aaaA3;1 2 3 1)
date sym pos
d1 aaaA1 1
d1 bbA1 2
d1 aaaA2 3
d2 aaaA3 1
to
t:flip `date`sym`pos`ext!(`d1`d1`d1`d2;`aaaA1`bbbA1`aaaA2`aaaA3;1 2 3 1;`aaa`bbb`aaa`aaa)
date sym pos ext
d1 aaaA1 1 aaa
d1 bbA1 2 bb
d1 aaaA2 3 aaa
d2 aaaA3 1 aaa
EDIT. The substring should always contain the first len(symbol) -2 characters, so in my example above, aaa for aaaAx and bb for bbAx
If the substring you wish to extract is a constant length, you can do something like this following:
q)t:flip `date`sym`pos!(`d1`d1`d1`d2;`aaaA1`bbbA1`aaaA2`aaaA3;1 2 3 1)
q)update ext:`$3#'string sym from t
date sym pos ext
------------------
d1 aaaA1 1 aaa
d1 bbbA1 2 bbb
d1 aaaA2 3 aaa
d2 aaaA3 1 aaa
If that's not the case, please provide some more detail with regards to how the substring to be extracted can be identified
Hope this helps
Jonathon
There can be a clever way of applying this below, but this is what i first came up with.
t:flip `date`sym`pos!(`d1`d1`d1`d2;`aaaA1`bbbA1`aaaA2`aaaA3;1 2 3 1)
t: update ctr: {-2 + count string x} each sym from t;
t:{[x] :update ext:x[`ctr]#string(x[`sym]) from x} each t;
2nd line is applying your logic of: len(symbol) - 2
3rd line is taking 'ctr' number of characters from the original symbol characters.
You didn’t say so, but this is kdb+, so let’s assume:
your table is long
your sym column has duplicates
You don’t need to convert all the symbols to strings and back: only the distinct ones. (In this example, I’ve changed one of the symbols to create a duplicate.)
q)t:flip `date`sym`pos!(`d1`d1`d1`d2;`aaaA1`bbbA1`aaaA2`aaaA1;1 2 3 1)
q)update ext:{nub:distinct x;(`$-2 _'string nub)nub?x}sym from t
date sym pos ext
------------------
d1 aaaA1 1 aaa
d1 bbbA1 2 bbb
d1 aaaA2 3 aaa
d2 aaaA1 1 aaa
The utility .Q.fu applies a function to the distinct items.
q)update ext:.Q.fu[{`$-2 _'string x};sym] from t
date sym pos ext
------------------
d1 aaaA1 1 aaa
d1 bbbA1 2 bbb
d1 aaaA2 3 aaa
d2 aaaA1 1 aaa
This operation would be faster if the sym column were already stored as an enumeration, because the distinct values would then be available without calculation.
Using drop:
q)t:flip `date`sym`pos!(`d1`d1`d1`d2;`aaaA1`bbA1`aaaA2`aaaA3;1 2 3 1)
q)update ext:`$-2_'string sym from t
date sym pos ext
------------------
d1 aaaA1 1 aaa
d1 bbA1 2 bb
d1 aaaA2 3 aaa
d2 aaaA3 1 aaa