I'm trying to import a CSV into an existing PostgreSQL table using DBeaver Import Tool and I need to transform a numeric value multiplying it by 100.
Someone knows the syntax to be used in the expression?
The official documentation talks about JEXL expressions, but i can't find any example.
I can't post images, but the one in this question is exactly where i need to put the expression.
I was expecting something like:
${column}*100
It seems the columns are exposed as variables so a simple(r) column * 100 should do.
Related
I am trying to get a concrete understanding on how the pandas UDF grouped map is working. While looking at the code here[1], i see that first the arrow object is converted into pandas series and then pd.concat is applied to create the full data frame.
What is confusing for me is , since arrow has support for Tabular format[2] and an API exist for converting table format to pandas in pyarrow[3] , why is that not being used.
I am pretty sure i am missing out on something very basic, any pointers would be useful ?
[1] https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/python/pyspark/sql/pandas/serializers.py#L234-L276
[2] https://arrow.apache.org/docs/cpp/tables.html
[3] https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas
How can I import arrays data into Quicksight from Postgresql? For example {1,2,3,4,5}. I tried to import all data into Quicksight but it doesn't recognize the arrays. However, if I download as csv from Postgresql and then import local csv file to Quicksight, it will recognize the arrays as string.
Have you tried converting the data type using a custom SQL command that will convert the array? https://docs.aws.amazon.com/quicksight/latest/user/adding-a-SQL-query.html
SELECT column_name->'name' AS name
FROM table_name
I achieved that with custom SQL + unnest() on the column with array, difference is that later you will have to keep in mind that some rows are not unique
You can use array_join(array,';'). This will show your array element in quick sight with ; separated like [1;2;3;4;5]
I have a csv file with 108 columns which i try to import in my postgresql table. It is obvious that I don't want to specify every columns in my CREATE TABLE statement. But when I enter
\COPY 'table_name' FROM 'directory' DELIMITER ',' CSV HEADER; this error message shows up: "ERROR: Extra Data after Last Expected Column". When having a few columns I know how to fix this problem but, like I said, i don't want to specified the entire 108 columns. By the way my table does contain any columns at all. Any help on how I could do that? Thx !
When dealing with problems like this, I often cheat. Plenty of tools exist online for converting CSV to SQL, https://www.convertcsv.com/csv-to-sql.htm being one of them.
Copy/paste your CSV, copy/paste the generated SQL. Not the most elegant solution, although will work as a one-off situation.
Now, if you're looking to repeat this process regularly (automated I hope), then Python may be a interesting language to explore to quickly write a script to do this for you, then schedule it at a CRON job or whatever method you prefer for invoking it automatically with the correct input (CSV file).
Please feel free to let me know if I've misunderstood your original question, or if I can provide any more help give me a shout and I'll do my best!
I have a PostgreSQL database. I had to extend an existing, big table with a few more columns.
Now I need to fill those columns. I tought I can create an .csv file (out of Excel/Calc) which contains the IDs / primary keys of existing rows - and the data for the new, empty fields. Is it possible to do so? If it is, how to?
I remember doing exactly this pretty easily using Microsoft SQL Management Server, but for PostgreSQL I am using PG Admin (but I am ofc willing to switch the tool if it'd be helpfull). I tried using the import function of PG Admin which uses the COPY function of PostgreSQL, but it seems like COPY isn't suitable as it can only create whole new rows.
Edit: I guess I could write a script which loads the csv and iterates over the rows, using UPDATE. But I don't want to reinvent the wheel.
Edit2: I've found this question here on SO which provides an answer by using a temp table. I guess I will use it - although it's more of a workaround than an actual solution.
PostgreSQL can import data directly from CSV files with COPY statements, this will however only work, as you stated, for new rows.
Instead of creating a CSV file you could just generate the necessary SQL UPDATE statements.
Suppose this would be the CSV file
PK;ExtraCol1;ExtraCol2
1;"foo",42
4;"bar",21
Then just produce the following
UPDATE my_table SET ExtraCol1 = 'foo', ExtraCol2 = 42 WHERE PK = 1;
UPDATE my_table SET ExtraCol1 = 'bar', ExtraCol2 = 21 WHERE PK = 4;
You seem to work under Windows, so I don't really know how to accomplish this there (probably with PowerShell), but under Unix you could generate the SQL from a CSV easily with tools like awk or sed. An editor with regular expression support would probably suffice too.
Is anyone using the DataFrame package (https://github.com/rothnic/DataFrame)?
I use it because older MATLAB can also use it. However, I just have very basic question:
How to change value in the DataFrame?
In MATLAB's table function, it is straightforward to do it. For example, t{1,{'prior'}} = {'a = 2000'}, where t is a table and I assign a cell to it. I cannot figure out how to do it in DataFrame package.
The DataFrame author seems not maintaining it anymore(?). I hope someone could give more examples of its methods in DataFrame.
Thanks!