Pyspark adding columns to existing dataframe - pyspark

I am trying to add multiple column to to right how can i do that?
Attributes = ["RequestTypePesId","AgentId","UpdatedBy","CauseType","OriginatingSystem"] for i in Attributes: a = df2load.select(i).distinct() b = a.join(b,a.select(i) == b.select(i),"fullouter")
Output should be:
enter image description here


Check out this example:
https://stackoverflow.com/a/71966176/9658895
and since you’re new, you might find this article useful:
“Hello World” of PySpark for Python & Pandas User [Pandas Vs PySpark]
Lastly, before you ask your next question, please make sure that you have followed standard protocol of asking a question. This video might help you.

Related

Issues with "QUERY(IMPORTRANGE)"

Here's my first question on this forum, though I've read through a lot of good answers here.
Can anyone tell me what I'm doing wrong with my attempt to do a query import from one sheet to a column in another?
Here's the formula I've tried, but all my adjustments still get me a parsing error.
=QUERY(IMPORTRANGE("https://docs.google.com/spreadsheets/d/1yGPdI0eBRNltMQ3Wr8E2cw-wNlysZd-XY3mtAnEyLLY/edit#gid=163356401","Master Treatment Log (Responses)!V2:V")"WHERE Col8="'&B2&'")")

Note that importrange is only needed for imports between spreadsheets. If you only import from one sheet into another within the same spreadsheet I would suggest using filter() or query().
Assuming the value in B2 is actually a string (and not a number), you can try
=QUERY(IMPORTRANGE("https://docs.google.com/spreadsheets/d/1yGPdI0eBRNltMQ3Wr8E2cw-wNlysZd-XY3mtAnEyLLY/edit#gid=163356401","Master Treatment Log (Responses)!V2:V"), "WHERE Col8="'&B2&'", 0)
Note the added comma before "WHERE". If you want to import a header row, change 0 to 1.
See if that helps? If not, please share a copy of your spreadsheet (sensitive data erased).

How to fill tables in powerpoint from a cell array in matlab?

I've downloaded saveppt2, jrichter's code and even WritetoWordfromMatlab and tried reading through them to figure it out with no luck. I have something of my own built already so I just need to figure out how to get tables to work.
Whenever I try something like:
myTable.Cell(1,1).TextFrame.Text = 'textstring'
or
myTable.Table.Cell(1,1) = 'textstring'
Or any combination of table / text commands I end up with there being no such property or function as cell for table objects. Every COM/VBA/C library I can find, as well as some code in Python (PandastoPowerPoint from Github) that does what I'm aiming to do says that Table.Cell(row,col) should work. Is this specifically a problem with matlab trying to use (#,#) as a form of indexing?

Try
myTable.Cell(1, 1).Shape.TextFrame.TextRange.Text = "TextString"
or = 'TextString' if that's what matlab prefers.

Thanks Steve R! With a little tweak, I got it to work, finally. So here's the answer:
% add table to existing slide object
myTable = slide.Shapes.addTable(nRows,nCols,x0,y0,rowWidthnRows,colHeightnCols)
myTable.Table.Cell(1, 1).Shape.TextFrame.TextRange.Text = 'TextString'

Extract text from a Wordnik API URL in Google Sheets

Can anyone explain how to import/extract a particular field from the following url into Google Sheets:
Wordnik URL
I'm guessing there is an IMPORTXML query that could do it, but it doesn't have the nodes that IMPORTXML usually uses to import that. Instead the code looks like this:
[{"mi":6.720745180909532,"gram1":"pretty","gram2":"much","wlmi":18.953166108085608},{"mi":6.650496643050408,"gram1":"pretty","gram2":"good","wlmi":18.469078820531266},{"mi":9.839004198061549,"gram1":"pretty","gram2":"darn","wlmi":17.298435816698845},{"mi":7.515791105774376,"gram1":"pretty","gram2":"cool","wlmi":15.515791105774376},{"mi":8.233704272151307,"gram1":"pretty","gram2":"impressive","wlmi":15.210984195651225}]
So if Cell A2 has the URL that produces this as the code, how do I get B2 to give me the text after "gram2" (in this case, "good", "darn", "cool" and "impressive").
Thanks
Tardy

I've come up with a workaround but it is kind of messy. I'd still like an answer to the question, but for reference and in case it's of use to anyone I'll post it here:
Using =IMPORTDATA(E10) (where E10 is the cell with the URL in) gave me an array (I guess based on .csv principles) that I could then manipulate using some other tools, like regexreplace, to get at the relevant bits of text.

importing website to google sheets

I have tried searching everywhere online for a good answer but cannot seem to find anything that matches specifically what i am looking for.
When i use the IMPORTHTML function in google sheets, i end up with data that looks like:
${player.name} (${player.position}, ${team.abbrev}) ${opponent.abbrev} #${opponent_rank} ${minutes} ${pts} ${fgm}-${fga} ${ftm}-${fta} ${p3m}-${p3a} ${treb} ${ast} ${stl} ${blk} ${tov} ${pf} ${fp} $${salary} ${ratio}
the code that i am using looks like this:
=IMPORTHTML("", "table",2)
When I use the same as above (=IMPORTHTML("", "table",2)) only with "0" as my index, it pulls this:
Opp Stats
Player Team Rank Min Pts FGM/A FTM/A 3PM/A Reb Ast Stl Blk Tov Foul FP Cost Value
Basically, I am attempting to pull the table data from this website:
https://www.numberfire.com/nba/fantasy/fantasy-basketball-projections
(because of my rep i cannot post more than two links, however my IMPORTHTML function has the above link input in both functions)
into a google sheet. Please help. any feedback is much appreciated... thanks!

Best advice is to find another Web table you can import. If you do "view source" on the page, you will find that the table content is dynamically populated from a variable named NF_DATA.
You need to create a document script to extract the data you want:
function this_is_test() {
var response = UrlFetchApp.fetch("https://www.numberfire.com/nba/fantasy/fantasy-basketball-projections");
raw_content = response.getContentText();
re = new RegExp('"daily_projections":\\[[^\\]]+','i');
proj = raw_content.match(re);
Logger.log(proj);
}
It will extract all text in-between "daily_projections":[ and ], which is (as of today):
"daily_projections":[{"nba_player_id":"77","nba_game_id":"20015","date":"2016-01-19","nba_team_id":"21","opponent_id":"7","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":36.3,"fgm":"8.8","fga":"17.1","p3m":"1.9","p3a":"4.8","ftm":"6.2","fta":"6.9","oreb":"0.8","dreb":"7.2","ast":"4.7","stl":"1.1","blk":"1.2","tov":"2.7","pf":"1.8","pts":"25.3","ts":"0.628","efg":"0.655","oreb_pct":"2.6","dreb_pct":"21.4","treb_pct":"12.4","ast_pct":"23.4","stl_pct":"1.5","blk_pct":"2.4","tov_pct":"12.1","usg":"27.8","ortg":"122.2","drtg":"101.8","nerd":"22.34","star_street_fp":43.08,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":39.75,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":43.85,"fanduel_salary":9900,"fanduel_ratio":4.43,"draft_kings_fp":46.55,"draft_kings_salary":9900,"draft_kings_ratio":4.7,"fantasy_feud_fp":39.75,"fantasy_feud_salary":153600,"fantasy_feud_ratio":0.26,"fanthrowdown_fp":45.2,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":44.25,"fantasy_aces_salary":7250,"fantasy_aces_ratio":6.1,"draftday_fp":45.25,"draftday_salary":18800,"draftday_ratio":2.41,"fantasy_score_fp":45.6,"fantasy_score_salary":9600,"fantasy_score_ratio":4.75,"draftster_fp":43.75,"draftster_salary":9400,"draftster_ratio":4.65,"yahoo_fp":44.8,"yahoo_salary":52,"yahoo_ratio":0.86,"treb":8},{"nba_player_id":"397","nba_game_id":"20015","date":"2016-01-19","nba_team_id":"21","opponent_id":"7","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35,"fgm":"8.6","fga":"18.0","p3m":"1.3","p3a":"4.1","ftm":"5.9","fta":"7.2","oreb":"1.3","dreb":"5.0","ast":"8.8","stl":"2.0","blk":"0.4","tov":"3.6","pf":"2.2","pts":"24.4","ts":"0.576","efg":"0.592","oreb_pct":"4.6","dreb_pct":"15.3","treb_pct":"10.2","ast_pct":"44.4","stl_pct":"3.0","blk_pct":"0.8","tov_pct":"14.6","usg":"31.3","ortg":"117.8","drtg":"101.2","nerd":"19.75","star_street_fp":44.48,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":41.33,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":46.36,"fanduel_salary":10500,"fanduel_ratio":4.42,"draft_kings_fp":49.13,"draft_kings_salary":10700,"draft_kings_ratio":4.59,"fantasy_feud_fp":41.33,"fantasy_feud_salary":169800,"fantasy_feud_ratio":0.24,"fanthrowdown_fp":47.33,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":46.68,"fantasy_aces_salary":7800,"fantasy_aces_ratio":5.98,"draftday_fp":47.1,"draftday_salary":20500,"draftday_ratio":2.3,"fantasy_score_fp":48.48,"fantasy_score_salary":9900,"fantasy_score_ratio":4.9,"draftster_fp":45.38,"draftster_salary":9500,"draftster_ratio":4.78,"yahoo_fp":47.01,"yahoo_salary":59,"yahoo_ratio":0.8,"treb":6.3},{"nba_player_id":"279","nba_game_id":"20016","date":"2016-01-19","nba_team_id":"11","opponent_id":"24","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":36.7,"fgm":"7.6","fga":"18.1","p3m":"2.5","p3a":"6.9","ftm":"5.5","fta":"6.5","oreb":"1.1","dreb":"5.8","ast":"5.3","stl":"1.8","blk":"0.4","tov":"3.6","pf":"2.4","pts":"22.5","ts":"0.537","efg":"0.610","oreb_pct":"3.3","dreb_pct":"17.6","treb_pct":"10.5","ast_pct":"25.1","stl_pct":"2.5","blk_pct":"0.9","tov_pct":"15.3","usg":"29.1","ortg":"104.1","drtg":"99.2","nerd":"5.26","star_street_fp":38.55,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":34.13,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":39.53,"fanduel_salary":8700,"fanduel_ratio":4.54,"draft_kings_fp":42.93,"draft_kings_salary":9200,"draft_kings_ratio":4.67,"fantasy_feud_fp":34.13,"fantasy_feud_salary":138800,"fantasy_feud_ratio":0.25,"fanthrowdown_fp":41.13,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":39.88,"fantasy_aces_salary":6500,"fantasy_aces_ratio":6.14,"draftday_fp":41.3,"draftday_salary":16600,"draftday_ratio":2.49,"fantasy_score_fp":41.68,"fantasy_score_salary":8400,"fantasy_score_ratio":4.96,"draftster_fp":39.45,"draftster_salary":8000,"draftster_ratio":4.93,"yahoo_fp":40.78,"yahoo_salary":47,"yahoo_ratio":0.87,"treb":6.9},{"nba_player_id":"2137","nba_game_id":"20014","date":"2016-01-19","nba_team_id":"38","opponent_id":"17","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35,"fgm":"8.0","fga":"16.6","p3m":"0.4","p3a":"1.3","ftm":"4.5","fta":"6.0","oreb":"2.6","dreb":"7.8","ast":"2.2","stl":"1.0","blk":"2.2","tov":"1.9","pf":"2.6","pts":"20.8","ts":"0.541","efg":"0.521","oreb_pct":"8.6","dreb_pct":"24.8","treb_pct":"16.6","ast_pct":"11.5","stl_pct":"1.4","blk_pct":"4.9","tov_pct":"9.4","usg":"27.0","ortg":"107.9","drtg":"103.1","nerd":"5.60","star_street_fp":41.05,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":36.55,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":41.08,"fanduel_salary":10300,"fanduel_ratio":3.99,"draft_kings_fp":44.25,"draft_kings_salary":10000,"draft_kings_ratio":4.43,"fantasy_feud_fp":36.55,"fantasy_feud_salary":149400,"fantasy_feud_ratio":0.24,"fanthrowdown_fp":41.8,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":41.6,"fantasy_aces_salary":7400,"fantasy_aces_ratio":5.62,"draftday_fp":40.43,"draftday_salary":18200,"draftday_ratio":2.22,"fantasy_score_fp":42.55,"fantasy_score_salary":9700,"fantasy_score_ratio":4.39,"draftster_fp":41.53,"draftster_salary":9200,"draftster_ratio":4.51,"yahoo_fp":41.28,"yahoo_salary":54,"yahoo_ratio":0.76,"treb":10.4},{"nba_player_id":"362","nba_game_id":"20013","date":"2016-01-19","nba_team_id":"15","opponent_id":"16","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":34.9,"fgm":"7.3","fga":"15.4","p3m":"1.4","p3a":"3.7","ftm":"4.8","fta":"6.0","oreb":"1.7","dreb":"6.1","ast":"2.3","stl":"0.7","blk":"1.0","tov":"1.7","pf":"2.0","pts":"20.6","ts":"0.571","efg":"0.594","oreb_pct":"6.2","dreb_pct":"19.9","treb_pct":"13.3","ast_pct":"12.1","stl_pct":"1.1","blk_pct":"2.2","tov_pct":"8.5","usg":"26.5","ortg":"115.3","drtg":"104.5","nerd":"11.63","star_street_fp":34.93,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":30.85,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":35.11,"fanduel_salary":7800,"fanduel_ratio":4.5,"draft_kings_fp":37.05,"draft_kings_salary":7600,"draft_kings_ratio":4.88,"fantasy_feud_fp":30.85,"fantasy_feud_salary":120400,"fantasy_feud_ratio":0.26,"fanthrowdown_fp":36.2,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":35.5,"fantasy_aces_salary":5900,"fantasy_aces_ratio":6.02,"draftday_fp":35.43,"draftday_salary":13950,"draftday_ratio":2.54,"fantasy_score_fp":36.35,"fantasy_score_salary":7000,"fantasy_score_ratio":5.19,"draftster_fp":35.35,"draftster_salary":7200,"draftster_ratio":4.91,"yahoo_fp":35.81,"yahoo_salary":40,"yahoo_ratio":0.9,"treb":7.8},{"nba_player_id":"2249","nba_game_id":"20014","date":"2016-01-19","nba_team_id":"17","opponent_id":"38","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35.7,"fgm":"7.2","fga":"16.6","p3m":"0.9","p3a":"3.2","ftm":"4.6","fta":"6.3","oreb":"1.3","dreb":"2.9","ast":"2.1","stl":"0.9","blk":"0.5","tov":"2.2","pf":"2.4","pts":"20.2","ts":"0.521","efg":"0.530","oreb_pct":"4.3","dreb_pct":"9.6","treb_pct":"6.9","ast_pct":"10.2","stl_pct":"1.3","blk_pct":"1.0","tov_pct":"10.3","usg":"26.7","ortg":"101.6","drtg":"111.5","nerd":"-7.07","star_street_fp":28.68,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":23.65,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":28.99,"fanduel_salary":6700,"fanduel_ratio":4.33,"draft_kings_fp":30.75,"draft_kings_salary":6900,"draft_kings_ratio":4.46,"fantasy_feud_fp":23.65,"fantasy_feud_salary":113500,"fantasy_feud_ratio":0.21,"fanthrowdown_fp":29.65,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":29.2,"fantasy_aces_salary":4900,"fantasy_aces_ratio":5.96,"draftday_fp":28.43,"draftday_salary":12000,"draftday_ratio":2.37,"fantasy_score_fp":30.3,"fantasy_score_salary":6000,"fantasy_score_ratio":5.05,"draftster_fp":29.23,"draftster_salary":5900,"draftster_ratio":4.95,"yahoo_fp":29.44,"yahoo_salary":29,"yahoo_ratio":1.02,"treb":4.2},{"nba_player_id":"370","nba_game_id":
Note that even this is not complete. You need to somehow map nba_player_id to the appropriate name. Anyway, a lot coding will be involved...

Matlab: Update an excel sheet

I am using Matlab to read a workbook with a bunch of sheets in it.
I do some calculation and have to update one particular column in one sheet. I tried using xlswrite after xlsread, it does not work.
So, my code looks something like:
[~,~,Data] = xlsread('MyFile.xlsx', 'MySheet');
Data(2:end-1,5) = Data(2:end-1,5) + 1.5; %Random operation for illustration only
ret = xlswrite('MyFile.xlsx',Data,'MySheet');
But ret is 0. So, I am not able to achieve replacement process. Can you please help.
Thanks

Based on my own comment:
Please use the second output argument as well an check what message you get:
[status,message] = xlswrite(filename,A,sheet)
Hopefully that is sufficient to find the cause, please let us know if that's the case.
Apparently it was indeed sufficient for the asker.