I've some ibd and frm files. Now I want to import those into mysql database. But its not working. I tried with mysqlfrm but it showing
C:\xampp\mysql\data\example>mysqlfrm --diagnostic -vvv users.frm
# WARNING: Cannot generate character set or collation names without the --server option.
# CAUTION: The diagnostic mode is a best-effort parse of the .frm file. As such, it may not identify all of the components of the table correctly. This is especially true for damaged files. It will also not read the default values for the columns and the resulting statement may not be syntactically correct.
# Reading .frm file for users.frm:
# The .frm file is a TABLE.
# Skipping to header at : 2
# General Data from .frm file:
{'IO_SIZE': 86,
'MYSQL_VERSION_ID': 100334,
'avg_row_length': 0,
'charset_low': 0,
'create_options': 9,
'db_create_pack': 2,
'default_charset': 224,
'default_part_eng': 0,
'extra_size': 16,
'frm_file_ver': 5,
'frm_version': 10,
'key_block_size': 0,
'key_info_length': 33,
'key_length': 505,
'legacy_db_type': 'INNODB',
'length': 12831,
'max_rows': 0,
'min_rows': 0,
'rec_length': 11314,
'row_type': 0,
'table_charset': 224,
'tmp_key_length': 505}
# Skipping to key data at : 56
# Reading key part 0.
# Index (key) Data from .frm file:
{'key_names': ['PRIMARY'],
'keys': [{'algorithm': 0,
'block_size': 0,
'comment': '',
'flags': 0,
'key_length': 8,
'key_parts': [{'field_num': 1,
'key_part_flag': 0,
'key_type': 16896,
'length': 8,
'offset': 2}],
'num_parts': 1}],
'num_key_parts': (1,),
'num_keys': 1}
# Skipping to default data at : 250
# Skipping to keys at : 2e83
# Engine string: InnoDB
# Partition string:
# Skipping to column data at : 2fb4
{'com_length': 64512,
'int_length': 0,
'interval_count': 0,
'interval_parts': 57352,
'n_length': 3840,
'null_fields': 0,
'num_cols': 20,
'pos': 3,
'unknown': 16896}
# Fields per screen = 0
EXCEPTION: unpack requires a string argument of length 1
ERROR: Cannot read column data.
I installed mysqlfrm on windows 10. If anyone know how to solve this, Please let me know. that be a great help.
Thank you
Since you shared users.ibd and user.frm I took a shot and recovered the table on a Windows 11 machine. Actually, on "Ubuntu on Windows", so it will work on native Ubuntu.
First, get Undrop for InnoDB and compile it.
$ git clone https://github.com/twindb/undrop-for-innodb.git
$ cd undrop-for-innodb/
$ sudo apt-get update
$ sudo apt-get install gcc flex bison
$ make
Drop users.zip in the current directory - undrop-for-innodb. Unzip it.
unzip users.zip
Now let's recover a table schema from users.frm. I like dbsake tool. It's much better than mysqlfrm (sorry Oracle, that's true).
$ curl -s http://get.dbsake.net > dbsake
$ chmod u+x dbsake
$ ./dbsake frmdump users.frm > users.sql
Here's your schema:
$ cat users.sql
--
-- Table structure for table `users`
-- Created with MySQL Version 10.3.34
--
CREATE TABLE `users` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`username` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`email` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`token` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`balance` double NOT NULL,
`admin` tinyint(1) NOT NULL,
`ref_id` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`password` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`referral` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`notification` tinyint(1) NOT NULL,
`referral_total` double NOT NULL,
`life_balance` double DEFAULT NULL,
`refer_by` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`notice_type` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`notice_message` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`coupon` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`total_spent` double DEFAULT NULL,
`total_order` double DEFAULT NULL,
`total_refund` double DEFAULT NULL,
`expire_ref` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '90',
`ref_created` timestamp DEFAULT NULL,
`created_at` timestamp DEFAULT NULL,
`updated_at` timestamp DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Now, let's fetch records from users.ibd. First step would be to parse the ibd file and sort InnoDB pages in it.
./stream_parser -f users.ibd
The stream_parser creates pages-users.ibd/FIL_PAGE_INDEX/0000000000000367.page. This is a file with InnoDB pages of the PRIMARY index of the users table. Basically, this is where the table records are.
Second step is to fetch records from 0000000000000367.page.
$ mkdir -p dumps/default
$ ./c_parser -6f pages-users.ibd/FIL_PAGE_INDEX/0000000000000367.page -t users.sql 2> load.sql > dumps/default/users
The command saves records in a file dumps/default/users
$ head dumps/default/users
-- Page id: 3, Format: COMPACT, Records list: Valid, Expected records: (3 3)
-- Page id: 3, Found records: 0, Lost records: YES, Leaf page: NO
-- Page id: 4, Format: COMPACT, Records list: Valid, Expected records: (27 27)
000000000000 80000000000000 users 1 "support" "admin#saveneur.com" "8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918" 49939.250000 1 "admin" "f7315c0bbe5793a0182599b22932d8a77848ab9a0b06e30c37f8be74d058f1c4" "none" 0 0.000000 56503.250000 NULL NULL NULL "" 8148.250000 1315.00000NULL "90" NULL NULL "2023-01-10 14:40:02"
000000000000 80000000000000 users 61 "deliveredswiftly#gmail.com" "deliveredswiftly#gmail.com" "3c469e9d6c5875d37a43f353d4f88e61fcf812c66eee3457465a40b0da4153e0" 6.500000 0 "65d90fc6d307590b14e9e1800d4e8eab" "8aaf6d0ff6e5b015c56fa47060b11d877cc6f7e09d6499f161203ccbe4f3056f" "none" 0 0.000000 110.000000 "none" """" "Delivered1" 8455.500000 1088.000000 NULL "90" NULL "2022-09-12 06:03:08" "2023-01-10 03:34:07"
000000000000 80000000000000 users 62 "Amander11" "butterphlys11#hotmail.com" "3c469e9d6c5875d37a43f353d4f88e61fcf812c66eee3457465a40b0da4153e0" 0.000000 0 "3f9e3767ef3b10a0de4c256d7ef9805d" "1b9588d1fe5cd0ffc3a126c0e53fca995c44bc2884de5c55ad323a9369087951" "none" 0 0.000000 NULL "none" NULL NULL NULL NULL NULL NULL "90" NULL "2022-09-21 04:47:31" "2022-09-21 04:47:31"
000000000000 80000000000000 users 63 "Mrbargainbeast" "mrbargainbeast#gmail.com" "3c469e9d6c5875d37a43f353d4f88e61fcf812c66eee3457465a40b0da4153e0" 0.000000 "2b38c2df6a49b97f706ec9148ce48d86" "75530007195799a0d4569a68ff833792039b526cdeb8eb99b05b1c1284e74d6f" "none" 0 0.000000 NULL "none" NULL NULL NULL NULL NULL NULL "90" NULL "2022-09-21 04:48:38" "2022-09-21 04:48:38"
000000000000 80000000000000 users 64 "partsofus" "info#partsofus.com" "3c469e9d6c5875d37a43f353d4f88e61fcf812c66eee3457465a40b0da4153e0" 0.000000 0 "05ae14d7ae387b93370d142d82220f1b" "36512164c97c2f83df9ccbb395b8446727992fd1482ef24ae369c85d6b890670" "none" 0 0.000000 NULL "none" NULL NULL NULL NULL NULL NULL "90" NULL "2022-09-21 06:33:55" "2022-09-21 06:33:55"
000000000000 80000000000000 users 65 "Finditforu" "finditforu#gmail.com" "3c469e9d6c5875d37a43f353d4f88e61fcf812c66eee3457465a40b0da4153e0" 0.000000 0 "2122c699d5e3d2fa6690771845bd7904" "6e05e801665a5bc864dc9b59154f67a3fcb92283947592a243361afd4a2427c0" "none" 0 0.000000 NULL "none" NULL NULL NULL NULL NULL NULL "90" NULL "2022-09-21 06:52:00" "2022-09-21 06:52:00"
000000000000 80000000000000 users 66 "VibrantReturns" "Vibrantreturns#protonmail.com" "3c469e9d6c5875d37a43f353d4f88e61fcf812c66eee3457465a40b0da4153e0" 0.000000 "0c2bcf2ee48c024117f6a057105ead45" "be1d510e75e07e39d0874d2a971bb61bc1d218288b699289db6b3a7299fdb1c4" "none" 0 0.000000 NULL "none" NULL NULL NULL NULL NULL NULL "90" NULL "2022-09-21 17:19:56" "2022-09-21 17:19:56"
To load this file into a MySQL instance c_parser also generates a helper LOAD statement which I saved in load.sql
$ cat load.sql
SET FOREIGN_KEY_CHECKS=0;
LOAD DATA LOCAL INFILE '/home/aleks/undrop-for-innodb/dumps/default/users' REPLACE INTO TABLE `users` CHARACTER SET UTF8 FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' LINES STARTING BY 'users\t' (`id`, `username`, `email`, `token`, `balance`, `admin`, `ref_id`, `password`, `referral`, `notification`, `referral_total`, `life_balance`, `refer_by`, `notice_type`, `notice_message`, `coupon`, `total_spent`, `total_order`, `total_refund`, `expire_ref`, `ref_created`, `created_at`, `updated_at`);
-- STATUS {"records_expected": 86, "records_dumped": 86, "records_lost": false} STATUS END
The load command should be something like
mysql my_database < load.sql
If you repeat the steps you should get the same result. If something doesn't work - drop me a line in LinkedIn I will send you the dumps/default/users file.
Good luck.
Related
With the following:
engine = sqlalchemy.create_engine(url)
df = pd.DataFrame({
"eid": [1,2],
"f_i": [123, 1231],
"f_i_arr": [[123], [0]],
"f_53": ["2013/12/1","2013/12/1",],
"f_53a": [["2013/12/1"], ["2013/12/1"],],
})
with engine.connect() as con:
con.execute("""
DROP TABLE IF EXISTS public.test;
CREATE TABLE public.test
(
eid integer NOT NULL,
f_i INTEGER NULL,
f_i_arr INTEGER NULL,
f_53 DATE NULL,
f_53a DATE[] NULL,
PRIMARY KEY(eid)
);;
""")
df.to_sql("test", con, if_exists='append')
If I try to insert only column "f_53" (an date) it succeeds.
If I try to add column "f_53a" (a date[]) it fails with:
^
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "f_53a" is of type date[] but expression is of type text[]
LINE 1: ..._53, f_53a, f_i, f_i_arr) VALUES (1, '2013/12/1', ARRAY['201...
^
HINT: You will need to rewrite or cast the expression.
[SQL: 'INSERT INTO test (eid, f_53, f_53a, f_i, f_i_arr) VALUES (%(eid)s, %(f_53)s, %(f_53a)s, %(f_i)s, %(f_i_arr)s)'] [parameters: ({'f_53': '2013/12/1', 'f_53a': ['2013/12/1', '2013/12/1'], 'f_i_arr': [123], 'eid': 1, 'f_i': 123}, {'f_53': '2013/12/1', 'f_53a': ['2013/12/1', '2013/12/1'], 'f_i_arr': [0], 'eid': 2, 'f_i': 1231})]
I have mentioned the dtypes explicitly and it worked for me for postgres.
//sample code
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.dialects import postgresql
df.to_sql('mytable',pgConn, if_exists='append', index=False, dtype={'datetime': sqlalchemy.TIMESTAMP(), 'cur_c':postgresql.ARRAY(sqlalchemy.types.REAL),
'volt_c':postgresql.ARRAY(sqlalchemy.types.REAL)
})
Yes -- is possible to insert [] and [][] types from a dataframe into postgres form a dataframe.
Unlike flat DATE types, which are may be correctly parsed by sql, DATE[] and DATE[][] need to be converted to datetime objects first. Like so.
with engine.connect() as con:
con.execute("""
DROP TABLE IF EXISTS public.test;
CREATE TABLE public.test
(
eid integer NOT NULL,
f_i INTEGER NULL,
f_ia INTEGER[] NULL,
f_iaa INTEGER[][] NULL,
f_d DATE NULL,
f_da DATE[] NULL,
f_daa DATE[][] NULL,
PRIMARY KEY(eid)
);
""")
d = pd.to_datetime("2013/12/1")
i = 99
df = pd.DataFrame({
"eid": [1,2],
"f_i": [i,i],
"f_ia": [None, [i,i]],
"f_iaa": [[[i,i],[i,i]], None],
"f_d": [d,d],
"f_da": [[d,d],None],
"f_daa": [[[d,d],[d,d]],None],
})
df.to_sql("test", con, if_exists='append', index=None)
I am using sql developer for the first time. I cant understand why this error is occuring.
CREATE TABLE TOY_STORE
( TOY_STORE_ID NUMBER(3,0),
TOY_STORE_NAME VARCHAR2(30 BYTE) NOT NULL ENABLE,
CITY VARCHAR2(30 BYTE) DEFAULT 'DELHI',
PHONENUMBER" NUMBER(10,0) NOT NULL ENABLE,
STORE_OPENING_TIME TIMESTAMP (6),
STORE_CLOSING_TIME TIMESTAMP (6),
CHECK (EXTRACT(HOUR FROM CAST (TO_CHAR (STORE_OPENING_TIME, 'YYYY-MON-DD HH24:MI:SS') AS TIMESTAMP)) > 8 || NULL),
CHECK (EXTRACT(HOUR FROM CAST (TO_CHAR (STORE_CLOSING_TIME, 'YYYY-MON-DD HH24:MI:SS') AS TIMESTAMP)) < 21 || NULL);
INSERT INTO TOY_STORE
VALUES(1, 'Kid''s Cave', 'Delhi', 9912312312, '2014-04-01 09:10:12', '2014-04-01 21:42:05');
Following was the error given:
Error report - SQL Error: ORA-01843: not a valid month 01843. 00000 - "not a valid month" *Cause: *Action: Error starting at line : 1 in command - INSERT INTO TOY_STORE VALUES(1, 'Kid''s Cave', 'Delhi', 9912312312, '04-2014-04 09:10:12', '04-2014-04 21:42:05') Error report - SQL Error: ORA-01843: not a valid month 01843. 00000 - "not a valid month"
Your create table as shown in the question has a stray double-quote and is missing a closing parenthesis. Your check constraints are odd:
you are converting a timestamp to a string using a specific format, and then casting back to a timestamp using the session NLS_TIMESTAMP_FORMAT, which will fail for sessions which don't have the setting you expect;
your are concatenating a null onto the hour value you're checking for, e.g. 21 || NULL, which is converting it to a string. Pretty sure you want to allow that to be null so you're using || as or, which isn't correct; but you don't need to explicitly allow for that anyway.
CREATE TABLE TOY_STORE (
TOY_STORE_ID NUMBER(3,0),
TOY_STORE_NAME VARCHAR2(30 BYTE) NOT NULL ENABLE,
CITY VARCHAR2(30 BYTE) DEFAULT 'DELHI',
PHONENUMBER NUMBER(10,0) NOT NULL ENABLE,
STORE_OPENING_TIME TIMESTAMP (6),
STORE_CLOSING_TIME TIMESTAMP (6),
CHECK (EXTRACT(HOUR FROM STORE_OPENING_TIME) > 8),
CHECK (EXTRACT(HOUR FROM STORE_CLOSING_TIME) < 21)
);
You might want to consider naming your constraints; and you might want a lower bound on the closing time and an upper bound on the opening time, or at least make sure opening isn't after closing (unless you're allowing for night opening; in which case you probably wouldn't want those constraints at all).
Then you are inserting using a string literal, not a timestamp. You are again relying on implicit conversion using your client's NLS_TIMESTAMP_FORMAT setting. The SQL Developer default for that is DD-MON-RR HH24.MI.SSXFF, which means with `'2014-04-01 09:10:12`` it will try to map the parts of the string literal to the parts of the format mask and fail; that default model does give ORA-01843.
You should either convert your string with an explicit format mask, using to_timestamp():
INSERT INTO TOY_STORE (TOY_STORE_ID, TOY_STORE_NAME, CITY, PHONENUMBER,
STORE_OPENING_TIME, STORE_CLOSING_TIME)
VALUES(1, 'Kid''s Cave', 'Delhi', 9912312312,
to_timestamp('2014-04-01 09:10:12', 'YYYY-MM-DD HH24:MI:SS'),
to_timestamp('2014-04-01 21:42:05', 'YYYY-MM-DD HH24:MI:SS'));
or use ANSI timestamp literals:
INSERT INTO TOY_STORE (TOY_STORE_ID, TOY_STORE_NAME, CITY, PHONENUMBER,
STORE_OPENING_TIME, STORE_CLOSING_TIME)
VALUES(1, 'Kid''s Cave', 'Delhi', 9912312312,
timestamp '2014-04-01 09:10:12', timestamp '2014-04-01 21:42:05');
... which violates your constraint as expected:
SQL Error: ORA-02290: check constraint (SCHEMA.SYS_C00113492) violated
02290. 00000 - "check constraint (%s.%s) violated"
*Cause: The values being inserted do not satisfy the named check
*Action: do not insert values that violate the constraint.
Notice the SYS_C00113492 name; that's a system-generated name for your constraint. It will be easier to follow what is happening if you name your constraints.
You are still allowed to insert nulls:
INSERT INTO TOY_STORE (TOY_STORE_ID, TOY_STORE_NAME, CITY, PHONENUMBER,
STORE_OPENING_TIME, STORE_CLOSING_TIME)
VALUES(1, 'Kid''s Cave', 'Delhi', 9912312312, null, null);
1 row inserted.
Is the place_id column in a reverse lookup response, a unique column?
[Long story...]
The reason I need to know is, I have made a monumental mistake, I reverse geo-coded 40 million records, but forgot to include osm_type in the saved results.
This is a problem, because I am updating a SQL table, with the results of my reverse geocoding, but because there are multiple osmIds (its only unique within element type) and because I don't know the element type of my updating row, I've landed myself with a big problem!
So now I need a way to figure out a way to match these rows.
I have the place_id saved so this could be my savior? (if it's a unique column?)
Otherwise if it isn't, is there any other way to implicitly infer what element type a nominatim response is? e.g from the presence of another column?
Here is a sample reverse geocoded result:
{
"placeId": "90367351",
"osmId": "109378817",
"boundingBox": "",
"polygonPoints": "",
"displayName": "Bugallon, Calbayog, Samar, Eastern Visayas, 6710, Philippines",
"road": "Bugallon",
"neighbourhood": "",
"suburb": "",
"cityDistrict": "",
"county": "",
"state": "Samar",
"country": "Philippines",
"countryCode": "ph",
"continent": "",
"house": "",
"village": "",
"town": "",
"city": "Calbayog",
"lat": "12.0666148",
"lon": "124.5958354"
}
and here is a sample row from my Sql (two rows with same osmId but different OsmElementType)
Id OsmId OsmKey OsmValue OsmNodeLat OsmNodeLng OsmInnerXml OsmXmlPath OsmXmlCountry OsmXmlFileNumber OsmElementType OsmPlaceId OsmDisplayName OsmRoad OsmNeighbourhood OsmSuburb OsmCityDistrict OsmCounty OsmState OsmCountry OsmCountryCode OsmContinent OsmHouse OsmVillage OsmTown OsmCity OsmBoundingBox OsmPolygonPoints FriendlyUrlTitle ViewsNum NumFavourites Text DateCreated UserIdCsvWhoViewedProfile IpCsvWhoViewedProfile Country_Id Owner_Id
6518 255653806 place town -3.3383462 35.6735367 NULL africa 105 N 769219 Karatu, Arusha, Northern, Tanzania Arusha Tanzania tz Karatu karatu-arusha-northern-tanzania 0 0 Karatu 1900-01-01 00:00:00.000 NULL NULL 170 2
3078707 255653806 landuse residential 0 0 PG5kIHJlZj0iMjYxMzI1OTE3NyIgLz48bmQgcmVmPSIyNjEzMjU5MTc4IiAvPjxuZCByZWY9IjI2MTMyNTkxNzkiIC8+PG5kIHJlZj0iMjYxMzI1OTE4MCIgLz48bmQgcmVmPSIyNjEzMjU5MTc3IiAvPjx0YWcgaz0ibGFuZHVzZSIgdj0icmVzaWRlbnRpYWwiIC8+PHRhZyBrPSJuYW1lIiB2PSLlj7DljJflt7TloZ7pmobntI0iIC8+ NULL asia 124 W NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL 0 0 ?????? 1900-01-01 00:00:00.000 NULL NULL NULL 2
Looks like it is unique:
https://github.com/twain47/Nominatim/blob/master/sql/tables.sql
CREATE UNIQUE INDEX idx_place_id ON placex USING BTREE (place_id);
I am trying to execute following bteq command on linux environment but couldn't load data properly into Teradata DB server. Can someone please advise me to resolve the below issue that I am facing while loading.
BTEQ Command used :
.SET width 64000;
.SET session transaction btet;
.logmech ldap
.logon XXXXXXX/XXXXXXXX,********;
DATABASE corecm;
.PACK 1000
.IMPORT VARTEXT '~' FILE=/v/global/user/application_event_bus_evt
.REPEAT *
USING(APPLICATION_EVENT_ID CHAR(24),BUS_EVT_ID CHAR(24),BUS_EVT_VID BIGINT,BUS_EVT_RESTATE_IN SMALLINT)
insert into corecm.application_event_bus_evt (APPLICATION_EVENT_ID
, BUS_EVT_ID
, BUS_EVT_VID
, BUS_EVT_RESTATE_IN
)
values
( COALESCE(:APPLICATION_EVENT_ID,1)
, COALESCE(:BUS_EVT_ID,1)
, COALESCE(:BUS_EVT_VID,1)
, COALESCE(:BUS_EVT_RESTATE_IN,1)
) ;
.LOGOFF;
.EXIT;
SAMPLE INPUT FILE DELIMITTER "~" [ /v/global/user/application_event_bus_evt ] :
Ckn3gMxLEeOgIQBQVgErYA==~g+GDDtlaY3n7BdUrYshDFA==~1~1
CL1kEcxLEeOgIQBQVgErYA==~qoKoiuGDbClpcGt/z6RKGw==~1~1
oYIVcMxKEeOgIQBQVgErYA==~mfmQiwl7yAteevzJfilMvA==~1~1
5N7ME5bM4xGhM7exj3ykUw==~yFM2FZbM4xGhM7exj3ykUw==~1~0
JLBH4JfM4xGDH9s5+Ds/8w==~doZ/7pfM4xGDH9s5+Ds/8w==~1~0
fGvpoMxKEeOgIQBQVgErYA==~mQUQIK2mY6WIPcszfp5BTQ==~1~1
Table Definition :
CREATE MULTISET TABLE CORECM.APPLICATION_EVENT_BUS_EVT ,NO FALLBACK ,
NO BEFORE JOURNAL,
NO AFTER JOURNAL,
CHECKSUM = DEFAULT,
DEFAULT MERGEBLOCKRATIO
(
APPLICATION_EVENT_ID CHAR(26) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL,
BUS_EVT_ID CHAR(26) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL,
BUS_EVT_VID BIGINT NOT NULL,
BUS_EVT_RESTATE_IN SMALLINT)
UNIQUE PRIMARY INDEX ( APPLICATION_EVENT_ID ,BUS_EVT_ID ,BUS_EVT_VID )
INDEX APPLICATION_EVENT_BUS_EVT_IDX1 ( APPLICATION_EVENT_ID )
INDEX APPLICATION_EVENT_BUS_EVT_IDX2 ( BUS_EVT_ID ,BUS_EVT_VID );
Results set in DB server as,
APPLICATION_EVENT_ID BUS_EVT_ID BUS_EVT_VID BUS_EVT_RESTATE_IN
1 Ckn3gMxLEeOgIQBQVgErYA == g+GDDtlaY3n7BdUrYshD 85,849,873,219,141,958 12,544
2 CL1kEcxLEeOgIQBQVgErYA == qoKoiuGDbClpcGt/z6RK 85,849,873,219,155,783 12,544
3 oYIVcMxKEeOgIQBQVgErYA == mfmQiwl7yAteevzJfilM 85,849,873,219,142,006 12,544
4 5N7ME5bM4xGhM7exj3ykUw == JAf0GpbM4xGhM7exj3yk 85,849,873,219,155,797 12,288
5 JLBH4JfM4xGDH9s5+Ds/8w == Du6T7pfM4xGDH9s5+Ds/ 85,849,873,219,155,768 12,288
6 fGvpoMxKEeOgIQBQVgErYA == mQUQIK2mY6WIPcszfp5B 85,849,873,219,146,068 12,544
If we look at the Data, we can see two issues as,
First two column data length is 24 CHARACTERS ( as per input file ), but the issue is that it been shifted two characters in next column.
Column BUS_EVT_VID and BUS_EVT_RESTATE_IN has wrong data 85,849,873,219,141,958 and 12,544 instead of 1 and 1 respectively (this may be because first two column data got shifted)
I tried following options to resolve the above issue but couldn't resolve the issue,
Modified the Table Definition, i.e. changed datatype to
CHAR(28),CHAR(24),CHAR(26)
Modified the Table Definition column
datatypes to VARCHAR(24), VARCHAR(26)
Modified BTEQ command, i.e. altered datatype in below line,
USING(APPLICATION_EVENT_ID CHAR(24),BUS_EVT_ID CHAR(24),BUS_EVT_VID BIGINT,BUS_EVT_RESTATE_IN SMALLINT)
Thanks in advance.
When you define VARTEXT all input columns must be defined as VARCHAR, but you used CHAR and INT.
This should work, VARCHAR length based on the definition of your target table:
USING(
APPLICATION_EVENT_ID VARCHAR(26),
BUS_EVT_ID VARCHAR(26),
BUS_EVT_VID VARCHAR(19),
BUS_EVT_RESTATE_IN VARCHAR(6)
)
I have two files. One is a file of table creation strings dumped from database, the other is the name of tables, with a "prompt" as a prefix and a "..." as suffix. Just as below:
file A (index):
prompt branch...
prompt branch_param...
prompt branch_pre_param...
prompt business...
prompt business_map...
prompt business_type...
file B (dump):
CREATE TABLE "KS"."BRANCH"
("BRANCH_CODE" CHARACTER(3) NOT NULL DEFAULT '',
"BRANCH_NAME" CHARACTER(40) NOT NULL DEFAULT '',
"PARAM_LEVEL" INTEGER NOT NULL DEFAULT 0
)
DATA CAPTURE NONE
IN "LONG_DATA_TBS";
CREATE TABLE "KS"."BRANCH2BANK"
("BRANCH_CODE" CHARACTER(3) NOT NULL DEFAULT '',
"BANK_CODE" CHARACTER(6) NOT NULL DEFAULT '',
"ACC_COMP_RESULT" CHARACTER(1) NOT NULL DEFAULT ''
)
DATA CAPTURE NONE
IN "SMALL_TBS";
CREATE TABLE "KS"."BRANCH2BOND"
("BRANCH_CODE" CHARACTER(3) NOT NULL DEFAULT '',
"BOND_CODE" CHARACTER(8) NOT NULL DEFAULT '',
"BOND_NAME" CHARACTER(20) NOT NULL DEFAULT '',
"TOTAL_AMT" DECIMAL(19, 4) NOT NULL DEFAULT 0,
"FINANCING_CUST_NO" CHARACTER(10) NOT NULL DEFAULT '',
"SET_DATE" CHARACTER(8) NOT NULL DEFAULT '',
"SET_TIME" CHARACTER(8) NOT NULL DEFAULT '',
"SET_EMP" CHARACTER(6) NOT NULL DEFAULT '',
"SPARE1" CHARACTER(20) NOT NULL DEFAULT '',
"SPARE2" CHARACTER(20) NOT NULL DEFAULT ''
)
DATA CAPTURE NONE
IN "SMALL_TBS";
CREATE TABLE "KS"."BRANCH_PARAM"
("BRANCH_CODE" CHARACTER(3) NOT NULL DEFAULT '',
"PARAM_CODE" CHARACTER(4) NOT NULL DEFAULT '',
"SET_DATE" CHARACTER(8) NOT NULL DEFAULT '',
"SET_TIME" CHARACTER(8) NOT NULL DEFAULT ''
)
DATA CAPTURE NONE
IN "SMALL_TBS";
CREATE TABLE "KS"."BRANCH_RESERVE_CREDIT_STOCK"
("BRANCH_CODE" CHARACTER(3) NOT NULL DEFAULT '',
"SET_TIME" CHARACTER(8) NOT NULL DEFAULT ''
)
DATA CAPTURE NONE
IN "TX_DATA_TBS"
INDEX IN "TX_INDEX_TBS";
I have written a perl implementation, but I think it is much too ugly and inefficient. Is there a better way to improve this?
my code: (rewritten with Richard and lilydjwg's advice) (last version)
#!/usr/bin/perl
use 5.016;
my (%hash,$cont);
open IN,'<',shift;
while(<IN>){
chomp;
$hash{$1}=1 if /prompt (\w+)\.\.\./;
}
close IN;
open IN,'<',shift;
while(<IN>){
chomp;
$cont = (defined $hash{lc $1}?say "prompt $1..." : 0) if /CREATE TABLE "KS"\."(\w+)"/;
say if $cont == 1;
}
close IN;
Presumably it's the repeated reads you don't like.
So - read the CREATE TABLE file once, checking for:
CREATE TABLE "KS"."(\w+)"
Then you can build up the table definition until the next CREATE TABLE at which point you put the table-definition into a hash keyed by the table-name.
Then, read your prompts and grab the definitions one by one from the hash printing them out.
Alternatively, you could just read the CREATE TABLE file into a single string and search+replace the table-name part since that's all you seem to be changing at the moment. The first approach is more flexible though.
Edit:
You could make the defined bit a little clearer perhaps with:
while ($line=<IN>) {
chomp($line);
if (/CREATE TABLE "KS"\."(\w+)"/ && $hash{lc $1}) {
$line = ...
}
say $line;
}
I like to use an explicit variable in my while-loops once I get beyond a couple of lines too.
It seems that file A is relatively small. You can read and build a set (or the like) from it which contains all the table names. Then read and identify on the SQL dump file, for each table creation statement, check if that table name is in your set.
I don't quite know Perl, but this Python code seems what you want:
import sys
tableNames = {x[7:-3] for l in open(sys.argv[1]) if x.startswith('prompt ')}
for l in open(sys.argv[2]):
if l.startswith('CREATE TABLE "KS"."'):
name = l.split('"')[4].lower()
if name in tableNames:
print("prompt {0}...\nCreate table{0}(".format(name))
print(l, end='')