I have two .xlsx files that total 1.6 million rows, and I'm trying to import these things into SQL Developer.
I right click the table name and select "Import Data..." and then select my file and nothing happens. It logs my attempt to open this file in the "File - Log" output
This is two separate attempts to import the same file logged here. When I click one of them, I get the following message:
However, I know that this warning is not true, because my attempts with importing a smaller .xlsx file are successful. So I figured the problem was just the file size is too large, and tried to change the memory available. I went into "C:\Users\User\Documents\sqldeveloper\sqldeveloper\bin" and changed sqldeveloper.conf to change one existing value to
AddVMOption -XX:MaxPermSize=2048M
and added another value
AddVMOption -Xmx2048M
Which helps the Java VM according to this source:
http://codechief.wordpress.com/2008/07/30/configuring-oracle-sql-developer-for-large-files-fix-out-of-memory-errors/
But this did nothing for me, and I still receive the same errors. I am using SQL Developer version 3.2.20.09 but I have also tried this on 4.0.1.14 to the same effect.
Many thanks!
I tried looking into SQL*Loader. Apparently you should be able to right click a table > Import Data > next and there should be an option to generate SQL*Loader files.
Unfortunately, not only did the import wizard not open with my large .xlsx files, the SQL*Loader option was not even present for smaller .xlsx files, or even .xls for that matter.
In the end, I decided to convert my .xlsx files into .csv and import those instead. This worked for all but 4 rows of my 1.6 million, and gave me the insert statement for those 4, of which 2 worked when run with no additional modifications.
In case of large volume of data, add a line "AddVMOption -Xmx4096M" to the sqldeveloper.conf file and even if it is not working change your file to *.csv format instead of *.xls. It will work fine.
Related
When I launch my query into pgAdmin 4 v5's Query Tool I get this type of data representation (this is also what I would like to get in my export file).
Unfortunately this information is transformed when saving it to a .TXT file by clicking the following button and saving it as indicated in the subsequent image.
As you can see below, after double-clicking on the saved TXT document, it added '.0' and wrapped my long character by indicating 'e+29' up to a certain row.
Can you please indicate me how to remove those transformations ?
All,
I found out the above problem is linked with the version of pgAdmin I was using, pgAdmin 4 v5 precisely.
After downloading pgAdmin 4 v6.4 the problem doesn't appear anymore.
I consider this thus as fixed, even if the cause of the problem remains unknown to me.
Thanks for your help.
Brieuc
I want to open a large text file (400MB) which contains 20 millions lines of domain addresses.
This is the file when I open it normally:
But when I open it in Eclipse I get this error:
I tried to change Xms6000M but it's not working!!
Does anyone have a solution to this problem?
Use the current version of Eclipse instead of the outdated one you have.
If required, increase -Xmx in the eclipse.ini file.
We are generating xlsx files using a perl script. Files usually contains thousands of records. This makes spotting errors a very difficult operation.
This process was working since years without problems.
This week we got a request to check a file which contains errors. While opening Excel prompted that the file contains errors and asked whether we want to repair them.
In fact we do not want to recover the data but want to know which part of the file is corrupt. The error should be coming from corrupt data and we are interested to identify these data.
the log message shows the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<recoveryLog xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
<logFileName>error068200_01.xml</logFileName> </br>
<summary>Errors were detected in file 'D:\Temp\20161020\file_name.xlsx'</summary>
<repairedRecords summary="Following is a list of repairs:"><repairedRecord>Repaired Records: Cell information from /xl/worksheets/sheet1.xml part</repairedRecord>
</repairedRecords>
</recoveryLog>
The error should come from corrupt data. Is there any tool/method which helps to spot this corrupt data?
I tried renaming it a zip file, extracting it and opening it via an XML editor but was not able to find any errors in XML file.
We also checked that the different XML file structures are fine.
Thank you and best regards
As expected, the problem was coming from text cells containing numbers having an E in the middle.I used the following steps to identify the erronous cells.
1. Wrote small Java class to read the file. The class was checking the cell type and displaying the value afterwards.The java program generated an Exception at some line "Cannot get a numeric value from a text cell" even If I was correctly checking the cell type before displaying the content.
2. I checked the opened Excel file at that line and found that the cell contains only 'inf'.
3. I opened the file using open office and looked at the same cells. They contain 0.
4. I debugged the program generating the data and found out that these cells contain data like '914E5514'. Seems that E which was interpreted by Excel as an exponent.We changed the program to use the format '#' for that cell and this solved the issue.
Thank you.
Thank you very much, you helped me a lot by saying that 1 particular content item may be the root problem.
My corrupted content was https://www.example.com XYZ ... ASDAS
Solution: www.example.com XYZ ... ASDAS
This is something which cannot be handled by excel. Would be nice to have a list of thing which do not work
I'm using pgadmin 1.18.
I have a copy of a table that I truncated. I simply want to load an import csv file which essentially looks like this:
20151228,12/28/2015,53,12,December,4,2015,1,Monday
20140828,08/28/2014,35,8,August,3,2014,4,Thursday
20150208,02/08/2015,6,2,February,1,2015,7,Sunday
I'm getting an error:
extra data after last expected column CONTEXT: COPY tblname, line 1:
"20151228,12/28/2015,53,12,December,4,2015,1,Monday"
This is the first line it´s trying to import. Any suggestions on how to fix this?
From the comments it appears you were using the wrong function in pgadmin.
If you have an existing table, which you have truncated and wish to load from a CSV file, select the table and then use Tools => Import, select the file and choose format 'CSV'.
There are other options in the import dialog to allow you to skip specified columns, use different quoting options, and specify how to deal with NULL values.
One tip that always trips me up: make sure there is no blank line at the end of the file.
Today I created a new staging table and a BCP .fmt file. I created some test data and attempted to run the BCP utility from the command line:
I've got about 20 different format files and staging tables from previous work and this is the first time I have encountered this error.
How can I fix this error?
Please note, I have added my solution below, but if you have other answers, please add them in. The answer was so quirky/obscure that I think it may help others.
Basically, this one was really strange. In order for it to work, make sure there is an empty line after the last column defined in the format file. I added an extra empty line, resaved the file, and then the BCP utility ran the file successfully. I've indicated the extra line with a red rectangle.