django-simple-history is the perfect solution for a special database I created in Django. After I initialized history, records were created, and history started from this point going forward.
But, one of the fields of a historical record has a value with date error. I need to correct this field value, but not create a new historical record. [I'm using another brilliant package django-extensions to make this change from a Jupyter Notebook through an ssh tunnel--wow! That's where the > output below comes from... ]
I used get() to pull the instance from Django:
rec = ReportModel.objects.all().get(cid=3) # records of a sub-group
rec_history = rec.history.all() # history for the sub-group
rec_history[1].effective_date # the offending record
> datetime.date(2022, 5, 16)
Then I tried to assign the correct date and save without history:
rec_history[1].effective_date = date(2018, 1, 1)
rec_history[1].save_without_historical_record()
>---------------------------------------------------------------------------
>AttributeError Traceback (most recent call last)
> Input In [55], in <cell line: 4>()
> 2 rec_history = rec.history.all()
> 3 rec_history[1].effective_date = date(2018, 1, 1)
>----> 4 rec_history[1].save_without_historical_record()
>
>AttributeError: 'HistoricalReportModel' object has no attribute 'save_without_historical_record'
I tried to import the function save_without_historical_record from the package...
from simple_history.models import save_without_historical_record
>ImportError: cannot import name 'save_without_historical_record' from 'simple_history.models'
How can I accomplish this?
Related
I built a job in AWS Glue Studio setting version to Glue 3, that is spark 3 is supported.
Goal is to truncate the date in column "date" to the minute, that is all seconds set to 00.
I found function date_trunc to be used for that but I get error "NameError: name 'date_trunc' is not defined"
The code runs in a custom transform and looks as follows:
def MyTransform (glueContext, dfc) -> DynamicFrameCollection:
df = dfc.select(list(dfc.keys())[0]).toDF()
df_rounded = df.withColumn("date_truncated", date_trunc("minute", col("date")))
dyf_rounded = DynamicFrame.fromDF(df_rounded, glueContext, "rounded")
return(DynamicFrameCollection({"CustomTransform0": dyf_rounded}, glueContext))
how can i make that function work? I assume I have to import that function but I dont see a way to do that in the Studio designer
Consider the following xml document that is stored in a PostgreSQL field:
<E_sProcedure xmlns="http://www.minushabens.com/2008/FMSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" modelCodeScheme="Emo_ex" modelCodeSchemeVersion="01" modelCodeValue="EMO_E_PROCEDURA" modelCodeMeaning="Section" sectionID="11">
<tCatSnVsn_Pmax modelCodeScheme="Emodinamica_referto" modelCodeSchemeVersion="01" modelCodeValue="tCat4" modelCodeMeaning="My text"><![CDATA[1]]></tCatSnVsn_Pmax>
</E_sProcedure>
If I run the following query I get the correct result for Line 1, while Line 2 returns nothing:
SELECT
--Line 1
TRIM(BOTH FROM array_to_string((xpath('//child::*[#modelCodeValue="tCat4"]/text()', t.xml_element)),'')) as tCatSnVsn_Pmax_MEANING
--Line2
,TRIM(BOTH FROM array_to_string((xpath('/tCatSnVsn_Pmax/text()', t.xml_element)),'')) as tCatSnVsn_Pmax
FROM (
SELECT unnest(xpath('//x:E_sProcedure', s.XMLDATA::xml, ARRAY[ARRAY['x', 'http://www.minushabens.com/2008/FMSchema']])) AS xml_element
FROM sr_data as s)t;
What's wrong in the xpath of Line 2?
Your second xpath() doesn't return anything because of two problems. First: you need to use //tCatSnVsn_Pmax as the xml_element still starts with <E_sProcedure>. The path /tCatSnVsn_Pmax tries to select a top-level element with that name.
But even then, the second one won't return anything because of the namespace. You need to pass the same namespace definition to the xpath(), so you need something like this:
SELECT (xpath('/x:tCatSnVsn_Pmax/text()', t.xml_element, ARRAY[ARRAY['x', 'http://www.minushabens.com/2008/FMSchema']]))[1] as tCatSnVsn_Pmax
FROM (
SELECT unnest(xpath('//x:E_sProcedure', s.XMLDATA::xml, ARRAY[ARRAY['x', 'http://www.minushabens.com/2008/FMSchema']])) AS xml_element
FROM sr_data as s
)t;
With modern Postgres versions (>= 10) I prefer using xmltable() for anything nontrivial. It makes passing namespaces easier and accessing multiple attributes or elements.
SELECT xt.*
FROM sr_data
cross join
xmltable(xmlnamespaces ('http://www.minushabens.com/2008/FMSchema' as x),
'/x:E_sProcedure'
passing (xmldata::xml)
columns
sectionid text path '#sectionID',
pmax text path 'x:tCatSnVsn_Pmax',
model_code_value text path 'x:tCatSnVsn_Pmax/#modelCodeValue') as xt
For your sample XML, the above returns:
sectionid | pmax | model_code_value
----------+------+-----------------
11 | 1 | tCat4
I am trying to pipe a webscraped pandas dataframe into a MySql table with mysql.connector but I can't seem to link df values to the %s variable. The connection is good (I can add individual rows) but it just returns errors when I replace the value witht he %s.
cnx = mysql.connector.connect(host = 'ip', user = 'user', passwd = 'pass', database = 'db')
cursor = cnx.cursor()
insert_df = ("""INSERT INTO table"
"(page_1, date_1, record_1, task_1)"
"VALUES ('%s','%s','%s','%s')""")
cursor.executemany(insert_df, df)
cnx.commit()
cnx.close()
This returns "ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()."
If I add any additional oiperations it returns "ProgrammingError: Parameters for query must be an Iterable."
I am very new to this so any help is appreciated
Work around for me was to redo my whole process. I ran sqlalchemy, all the documentation makes this very easy. message if you want the code I used.
I am creating function in which update command is used consecutively two times the first update is working and the second one is not
Have tried execute format () for the second update not working
While running the function the updation2 is not working but when I manually run this updation command the table get updated...
The code is as follows:
update edmonton.weekly_pmt_report
set permit_number = pmt.prnum
from(select permit_details,
split_part(permit_details,'-',1) as prnum
from edmonton.weekly_pmt_report) as pmt
where edmonton.weekly_pmt_report.permit_details =
pmt.permit_details;
execute format('update edmonton.weekly_pmt_report
set address = ds_dt.adr,
job_description = ds_dt.job,
applicant = ds_dt.apnt
from(select split_part(per_num,''-'',1) as job_id,
job_des as job,addr as adr, applic as apnt
from edmonton.descriptive_details ) as ds_dt
where edmonton.weekly_pmt_report.permit_number =
ds_dt.job_id');
That the second update query has value only 400 out of 1000 so the null columns are on the Top, that's why It seemed to be not working...
I am having difficulties migrating a working a working script to PGSQL from SQLite. I am using SQLalchemy. When I run the script, it raises the following errors:
raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) can't adapt 'INSERT INTO cnn_hot_stocks (datetime, list, ticker, price, change, "pctChange") VALUES (%(datetime)s, %(list)s, %(ticker)s, %(price)s, %(change)s, %(pctChange)s)' {'price': Decimal('7.94'), 'list': 'active', 'datetime': datetime.datetime(2012, 6, 23, 11, 45, 1, 544361), 'pctChange': u'+1.53%', 'ticker': u'BAC', 'change': Decimal('0.12')}
The insert call works well when using sqlite engine, but I want to use pgsql to utilize the native Decimal type for keeping financial data correct. I copied the script and just changed the db engine to my postgresql server. Any advice on how to troubleshoot this error would be greatly appreciated for this SQLalchemy newbie... I think I am up a creek on this one! Thanks in advance!
Here are my relevant code segments and table descriptions:
dbstring = "postgresql://postgres:postgres#localhost:5432/algo"
db = create_engine(dbstring)
db.echo = True # Try changing this to True and see what happens
metadata = MetaData(db)
cnn_hot_stocks = Table('cnn_hot_stocks', metadata, autoload=True)
i = cnn_hot_stocks.insert() # running log from cnn hot stocks web-site
def scrape_data():
try:
html = urllib2.urlopen('http://money.cnn.com/data/hotstocks/').read()
markup, errors = tidy_document(html)
soup = BeautifulSoup(markup,)
except Exception as e:
pass
list_map = { 2 : 'active',
3 : 'gainer',
4 : 'loser'
}
# Iterate over 3 tables on CNN hot stock web-site
for x in range(2, 5):
table = soup('table')[x]
for row in table.findAll('tr')[1:]:
timestamp = datetime.now()
col = row.findAll('td')
ticker = col[0].a.string
price = Decimal(col[1].span.string)
change = Decimal(col[2].span.span.string)
pctChange = col[3].span.span.string
log_data = {'datetime' : timestamp,
'list' : list_map[x],
'ticker' : ticker,
'price' : price,
'change' : change,
'pctChange' : pctChange
}
print log_data
# Commit to DB
i.execute(log_data)
TABLE:
cnn_hot_stocks = Table('cnn_hot_stocks', metadata, # log of stocks data on cnn hot stocks lists
Column('datetime', DateTime, primary_key=True),
Column('list', String), # loser/gainer/active
Column('ticker', String),
Column('price', Numeric),
Column('change', Numeric),
Column('pctChange', String),
)
My reading of the documentation is that you have to use numeric instead of decimal.
PostgreSQL has no type named decimal (it's an alias for numeric but not a very full-featured one), and SQL Alchemy seems to expect numeric as the type it can use for abstraction purposes.