Complex sed Command for Insert Command - sed
I have a bunch of php files, which have many insert commands.
In each query, I want to insert a column variable admin_id = '$admin_id',
i.e., if the query is
insert into users (ch_id, num_value) values ('2', '100')
the query should be converted to
insert into users (admin_id, ch_id, num_value) values ($admin_id, '2', '100')
To do this, I have executed the following command
sed -i 's/\(insert.*into.*\) (\(.*values\)/\1 (admin_id, \2/' *.php
and
sed -i "s/\(insert.*into.*\) values (/\1 values ('\$admin_id', /" *.php
The above has worked successfully, but am still facing problem with SQL queries where there is no where in the query, i.e.,
insert into abctable (id,no)
to
insert into tablename (admin_id, id, no)
and
insert into abctable select $column from $tableperiod
to
insert into abctable select $column from $tableperiod where admin_id='$admin_id'
and
insert into abctable select $column from $tableperiod where abc != 'xyz'
to
insert into abctable select $column from $tableperiod where admin_id = '$admin_id' and abc != 'xyz'
How can I insert admin_id in these queries as well?
The queries in php files are executed by passing the query to the function in the following way:
execute_query("insert * from $table order by username");
I can find the queries still which are left to be modified by
executing
grep 'execute_query' *| grep insert| grep -v admin_id > stillleft.txt
I have solved it by using the following command
sed -e "s/\(query.*insert.*select.*where\)/& admin_id='\$admin_id' and /g" -e t \
-e "s/\(query.*insert.*select.*\)\")/\1 where admin_id='\$admin_id\")'/g" -e t \
-e "s/\(query.*insert.*\)(\(.*\)values (/\1(admin_id, \2values ('\$admin_id', /g" -e t \
-e "s/\(query.*insert.*(\)/& admin_id, /g" \
-i *.php
I'm not sure my testcases are right, but I think this could help you:
I changed the first statement, because I think it's easier and it matches the first and the second command of YOUR sed
sed -i 's/\(insert into .* (\)\(.*) values (\)\(.*\)) /\1admin_id, \2\$admin_id, \3/' *.php
The second (the first you are looking for) should work with the following
sed -i 's/\(insert into .* (\)\(.*) \)/\1admin_id, \2/' *.php
And the last two should work with this:
sed -i "s/\(insert into \w* select \$column from \$tableperiod\)/\1 where admin_id='\$admin_id'/" *.php
I hope this works for you, if not, please send a little bit more test data, if tested the commands with the text of your question as input
I you use multiple sed commands, you'll traverse the complete file each time. You can do it in a single pass. Assuming an input file infile that looks like this:
insert into users (ch_id, num_value) values ('2', '100')
insert into abctable (id, no)
insert into abctable select $column from $tableperiod
insert into abctable select $column from $tableperiod where abc != 'xyz'
we can use the following sed script sedscr
/^insert into/ {
s/\(([^)]*)\)(.*)\(([^)]*)\)/(admin_id, \1)\2($admin_id, \3)/
s/^([^(]+)\(([^)]*)\)$/\1(admin_id, \2)/
/\(.*\)/! {
/where/s/$/ and admin_id ='$admin_id'/
/where/!s/$/ where admin_id='$admin_id'/
}
}
It does the following:
if a line starts with insert into, then
for all lines with two pairs of parentheses, insert admin_id in the the first one and $admin_id in the second one
for lines with one pair of parentheses at the end, insert admin_id
if there are no parentheses, then
if there is a "where" clause, append and admin_id = '$admin_id'
else append where admin_id='$admin_id'
This can be called as follows:
$ sed -rf sedscr infile
insert into users (admin_id, ch_id, num_value) values ($admin_id, '2', '100')
insert into abctable (admin_id, id, no)
insert into abctable select $column from $tableperiod where admin_id='$admin_id'
insert into abctable select $column from $tableperiod where abc != 'xyz' and admin_id ='$admin_id'
If you can't use extened regular expressions (-r), the quoting of parentheses has to be inverted (all \( become ( etc.) and the + has to be replaced by \{1,\}.
The cumbersome regexes such as \(([^)]*)\) stand for "between literal parentheses, capture zero or more characters that are not a closing parenthesis" – this enables non-greedy capturing.
Related
How to `read` postgresql `select` output to bash array,one match is one element?
test tables as below: create table test(col1 int, col2 varchar,col3 date); insert into test values (1,'abc','2015-09-10'); insert into test values (1,'abd2','2015-09-11'); insert into test values (21,'xaz','2015-09-12'); insert into test values (2,'xyz','2015-09-13'); insert into test values (3,'tcs','2015-01-15'); insert into test values (3,'tcs','2016-01-18'); Use bash script to get array res of postgresql select. #!/bin/bash res_temp=$(psql -tAq postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF SELECT "col1","col2" FROM "test" WHERE "col2" LIKE '%a%'; EOF ) read res <<< $res_temp #should be 3,but output 1 echo ${#res[#]} for i in "${!res[#]}"; do printf "%s\t%s\n" "$i" "${res[$i]}" done Output as below: 1 0 1|abc Expect output is: 3 0 1|abc 1 1|abd2 2 21|xaz Where is the problem?
This is going wrong because of the read res <<< $res_temp. What do you expect to retrieve from that? I've fixed your script and put an example how to directly create an array (which I think you're trying). I don't have Postgresql running atm, but SQLite does the same. How I created the data: $ sqlite ./a.test sqlite> create table test(col1 int, col2 varchar(100),col3 varchar(100)); sqlite> insert into test values (1,'abc','2015-09-10'); sqlite> insert into test values (1,'abd2','2015-09-11'); sqlite> insert into test values (21,'xaz','2015-09-12'); sqlite> insert into test values (2,'xyz','2015-09-13'); sqlite> insert into test values (3,'tcs','2015-01-15'); sqlite> insert into test values (3,'tcs','2016-01-18'); sqlite> SELECT col1,col2 FROM test WHERE col2 LIKE '%a%'; Your solution #! /bin/bash res_tmp=$(sqlite ./a.test "SELECT col1,col2 FROM test WHERE col2 LIKE '%a%';") read -a res <<< ${res_tmp[#]} echo ${#res[#]} for i in "${!res[#]}"; do printf "%s\t%s\n" "$i" "${res[$i]}" done exit 0 Directly an array #! /bin/bash res=($(sqlite ./a.test "SELECT col1,col2 FROM test WHERE col2 LIKE '%a%';")) echo ${#res[#]} for i in "${!res[#]}"; do printf "%s\t%s\n" "$i" "${res[$i]}" done exit 0 Oh, and output of both: 3 0 1|abc 1 1|abd2 2 21|xaz
Quote strings and dates in psql query results output
Under the section \pset [ option [ value ] ] of the psql docs, I can set various settings to make my query results convenient for me. I can, for example, approach a CSV-like output with: \pset fieldsep ',' \pset footer off \pset format unaligned \pset null 'NULL' Resulting in output like: > WITH foo_tbl(foo,bar,baz) > AS > ( > VALUES > ('foo', NULL, 1), > (NULL, 'bar', 1) > ) > SELECT * FROM foo_tbl; foo,bar,baz foo,NULL,1 NULL,bar,1 This is great, but I'd like strings and dates to be quoted, like this: foo,bar,baz 'foo',NULL,1 NULL,'bar',1 Is this not possible with psql? p.s. I know this kind of thing can be done with SQL clients like DBeaver, but that isn't in the scope of this question.
To generate CSV output, you can use the copy command rather than trying to tweak the output of a regular SELECT statement. copy ( WITH foo_tbl (foo,bar,baz,dt) AS ( VALUES ('foo', NULL, 1, date '2020-01-02'), (NULL, 'bar', 1, date '2020-03-04') ) SELECT * FROM foo_tbl ) to stdout with (format csv, quote '''', header, null 'NULL', force_quote (foo, dt) ); Will generate the following output foo,bar,baz,dt 'foo',NULL,1,'2020-01-02' NULL,bar,1,'2020-03-04' I am not aware of an option that will quote only dates and strings, but not numbers, so using force_quote and specifying the columns to quote is the only way to get them (always). copy (...) to stdout is easier to use than it's psql sibling \copy because it allows multi-line queries. To write everything into a file, you can use the \o command in psql postgres=> \o data.csv postgres=> copy (...) to stdout with (...);
Remove any text between two parameters not working properly
I need to remove any data between , and ( and the "," along with it. I'm currently using the below command. sed -i '/,/,/(/{//!d;s/ ,$//}' test1.txt cat test1.txt CREATE SET TABLE EDW_EXTRC_TAB.AVER_MED_CLM_HDR_EXTRC ,NO FALLBACK , NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT, DEFAULT MERGEBLOCKRATIO ( EXTRC_RUN_ID INTEGER NOT NULL, Current Output CREATE SET TABLE EDW_EXTRC_TAB.AVER_MED_CLM_HDR_EXTRC ,NO FALLBACK ( EXTRC_RUN_ID INTEGER NOT NULL, Expected Output: CREATE SET TABLE EDW_EXTRC_TAB.AVER_MED_CLM_HDR_EXTRC ( EXTRC_RUN_ID INTEGER NOT NULL, What is wrong here ? Any suggestions? Thanks in advance.
Two approaches: -- GNU sed approach: sed -z 's/,[^(]*//' test1.txt -- GNU awk approach: awk -v RS= '{ sub(/,[^(]+/,"",$0) }1' test1.txt The output: CREATE SET TABLE EDW_EXTRC_TAB.AVER_MED_CLM_HDR_EXTRC ( EXTRC_RUN_ID INTEGER NOT NULL,
SED command Issue with values exceeding 9
I need to generate a file.sql file from a file.csv, so I use this command : cat file.csv |sed "s/\(.*\),\(.*\)/insert into table(value1, value2) values\('\1','\2'\);/g" > file.sql It works perfectly, but when the values exceed 9 (for example for \10, \11 etc...) it takes consideration of only the first number (which is \1 in this case) and ignores the rest. I want to know if I missed something or if there is another way to do it. Thank you ! EDIT : The not working example : My file.csv looks like 2013-04-01 04:00:52,2,37,74,40233964,3860,0,0,4878,174,3,0,0,3598,27.00,27 What I get insert into table val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12,val13,val14,val15,val16 values ('2013-04-01 07:39:43', 2,37,74,36526530,3877,0,0,6080, 2013-04-01 07:39:430,2013-04-01 07:39:431, 2013-04-01 07:39:432,2013-04-01 07:39:433, 2013-04-01 07:39:434,2013-04-01 07:39:435, 2013-04-01 07:39:436); After the ninth element I get the first one instead of the 10th,11th etc...
As far I know sed has a limitation of supporting 9 back references. It might have been removed in the newer versions (though not sure). You are better off using perl or awk for this. Here is how you'd do in awk: $ cat csv 2013-04-01 04:00:52,2,37,74,40233964,3860,0,0,4878,174,3,0,0,3598,27.00,27 $ awk 'BEGIN{FS=OFS=","}{print "insert into table values (\x27"$1"\x27",$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16 ");"}' csv insert into table values ('2013-04-01 04:00:52',2,37,74,40233964,3860,0,0,4878,174,3,0,0,3598,27.00,27); This is how you can do in perl: $ perl -ple 's/([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+)/insert into table values (\x27$1\x27,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16);/' csv insert into table values ('2013-04-01 04:00:52',2,37,74,40233964,3860,0,0,4878,174,3,0,0,3598,27.00,27);
Try an awk script (based on #JS웃 solution): script.awk #!/usr/bin/env awk # before looping the file BEGIN{ FS="," # input separator OFS=FS # output separator q="\047" # single quote as a variable } # on each line (no pattern) { printf "insert into table values (" print q $1 q ", " print $2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16 print ");" } Run with awk -f script.awk file.csv One-liner awk 'BEGIN{OFS=FS=","; q="\047" } { printf "insert into table values (" q $1 q ", " $2","$3","$4","$5","$6","$7","$8","$9","$10","$11","$12","$13","$14","$15","$16 ");" }' file.csv
Get line counts of INSERT's by tablename from mysqldump - awk, sed, grep
bash master needed... To compare mysqldumps from multiple dates, I need sql GROUP BY, ORDER BY functionality... but on the command line... Using grep / sed / awk I need to look for all INSERT statements, then export a line count per tablename. I'd really love a byte count per tablename too... A typical line looks like this: INSERT INTO `admin_rule` ... match the INSERT, then match the tablename in ``, counting by unique tablename
How about this little awk snippet: BEGIN { FS="`" } /^INSERT/ { count[$2]+=1; bytes[$2]+=length($0) } END { for(table in count) print table "," count[table] "," bytes[table]; } Edit: test case here: $ cat test.sql INSERT INTO `t1` VALUES('a', 12, 'b'); INSERT INTO `t2` VALUES('test', 'whatever', 3.14); INSERT INTO `t3` VALUES(1, 2, 3, 4); INSERT INTO `t2` VALUES('yay', 'works', NULL); INSERT INTO `t2` VALUES(NULL, 'something' 2.71); INSERT INTO `t3` VALUES(5, 6, 7, 8); INSERT INTO `t5` VALUES('beta', 'gamma'); INSERT INTO `t6` VALUES('this', 'is', 'table', 'six'); $ awk -f t.awk <test.sql t5,1,41 t6,1,54 t1,1,38 t2,3,144 t3,2,72