How to print textual representation of single documents stored in a tm corpus in R? - tm

I was using {tm} package and then generated a corpus using
corpus = Corpus(VectorSource(sample.words))
then I want to check the content in corpus ,but it print this instead of its texts:
> corpus
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 3933
Now I have found some methods to look into corpus,then I started wondering what exactly R print when an object was typed in ?
> class(corpus)
[1] "VCorpus" "Corpus"
> typeof(corpus)
[1] "list"
Why it didn`t like other ordinary lists ,printing its columns and rows?Does this has something to do whit the class attribute?
I`m new in R and not familiar with some basic concepts, thanks for your patience!


The introduction document to the tm package says that you can use , say, writeLines(as.character(mycorpus[[4]])) to get a textual representation of document 4.
You can also use content(myCorpus[[23]]).
To read the intro document, enter browseVignettes() on your R prompt and the search for it on the browser window that will have opened.

Related

Reading documents with r-tm to use with r-mallet

I have this code to fit a topic model with the R wrapper for MALLET:
docs <- mallet.import(DF$document, DF$text, stop_words)
mallet_model <- MalletLDA(num.topics = 4)
mallet_model$loadDocuments(docs)
mallet_model$train(100)
I have used the tm package to read my documents, which are txt files in a directory:
myCorpus <- Corpus(DirSource("data")) # a directory of txt files
The corpus can't be used as input for mallet.import, so how do I get from the tm corpus myCorpus above to the DF to call upon?

RMallet is intended to be a standalone package, so integration with tm isn't great. The requirement for RMallet input is a data frame with one row per document, and a character field containing the text, which it expects to not be tokenized already.

You can use tidy data principles to process your text and get it ready for input into mallet, with one row per document, as described here.
Also, there are tidiers for the mallet package in tidytext, and you can use them to analyze the output of mallet topic modeling:
# word-topic pairs
tidy(mallet_model)
# document-topic pairs
tidy(mallet_model, matrix = "gamma")
# column needs to be named "term" for "augment"
term_counts <- rename(word_counts, term = word)
augment(mallet_model, term_counts)

how to find partial search in Mongodb?

How to find partial search?
Now Im trying to find
db.content.find({$text: {$search: "Customer london"}})
It finds all records matching customer, and all records matching london.
If I am searching for a part of a word for example lond or custom
db.content.find({$text: {$search: "lond"}})
It returns an empty result. How can I modify the query to get the same result like when I am searching for london?

You can use regex to get around with it (https://docs.mongodb.com/manual/reference/operator/query/regex/). However, it will work for following :
if you have word Cooking, following queries may give you result
cooking(exact matching)
coo(part of the word)
cooked(The word containing the english root of the document word, where cook is the root word from which cooking or cooked are derived)
If you would like to go one step further and get a result document containing cooking when you type vooking (missplled V instead of C), go for elasticsearch.
Elasticsearch is easy to setup, has extremely powerful edge-ngram analyzer which converts each words into smaller weightage words. Hence when you misspell, you will still get a document based on score elasticsearch gives to that document.
You can read about it here : https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html

it will always return the empty array for partial words like when you are searching for lond to get this type of text london..
Because it take full words and search same as that they are ..
Not achive same results like :-
LO LON LOND LONDO LONDON
Here you may get help from ELASTIC-SEARCH . It is quite good for full text search when implement with mongoDB.
Refrence : ElasticSearch
Thanks

The find all is to an Array
clientDB.collection('details').find({}).toArray().then((docs) =>
I now used the str.StartWith in a for loop to pick out my record.
if (docs[i].name.startsWith('U', 0)) {
return console.log(docs[i].name);
} else {
console.log('Record not found!!!')
};
This may not be efficient, but it works for now

importing website to google sheets

I have tried searching everywhere online for a good answer but cannot seem to find anything that matches specifically what i am looking for.
When i use the IMPORTHTML function in google sheets, i end up with data that looks like:
${player.name} (${player.position}, ${team.abbrev}) ${opponent.abbrev} #${opponent_rank} ${minutes} ${pts} ${fgm}-${fga} ${ftm}-${fta} ${p3m}-${p3a} ${treb} ${ast} ${stl} ${blk} ${tov} ${pf} ${fp} $${salary} ${ratio}
the code that i am using looks like this:
=IMPORTHTML("", "table",2)
When I use the same as above (=IMPORTHTML("", "table",2)) only with "0" as my index, it pulls this:
Opp Stats
Player Team Rank Min Pts FGM/A FTM/A 3PM/A Reb Ast Stl Blk Tov Foul FP Cost Value
Basically, I am attempting to pull the table data from this website:
https://www.numberfire.com/nba/fantasy/fantasy-basketball-projections
(because of my rep i cannot post more than two links, however my IMPORTHTML function has the above link input in both functions)
into a google sheet. Please help. any feedback is much appreciated... thanks!

Best advice is to find another Web table you can import. If you do "view source" on the page, you will find that the table content is dynamically populated from a variable named NF_DATA.
You need to create a document script to extract the data you want:
function this_is_test() {
var response = UrlFetchApp.fetch("https://www.numberfire.com/nba/fantasy/fantasy-basketball-projections");
raw_content = response.getContentText();
re = new RegExp('"daily_projections":\\[[^\\]]+','i');
proj = raw_content.match(re);
Logger.log(proj);
}
It will extract all text in-between "daily_projections":[ and ], which is (as of today):
"daily_projections":[{"nba_player_id":"77","nba_game_id":"20015","date":"2016-01-19","nba_team_id":"21","opponent_id":"7","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":36.3,"fgm":"8.8","fga":"17.1","p3m":"1.9","p3a":"4.8","ftm":"6.2","fta":"6.9","oreb":"0.8","dreb":"7.2","ast":"4.7","stl":"1.1","blk":"1.2","tov":"2.7","pf":"1.8","pts":"25.3","ts":"0.628","efg":"0.655","oreb_pct":"2.6","dreb_pct":"21.4","treb_pct":"12.4","ast_pct":"23.4","stl_pct":"1.5","blk_pct":"2.4","tov_pct":"12.1","usg":"27.8","ortg":"122.2","drtg":"101.8","nerd":"22.34","star_street_fp":43.08,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":39.75,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":43.85,"fanduel_salary":9900,"fanduel_ratio":4.43,"draft_kings_fp":46.55,"draft_kings_salary":9900,"draft_kings_ratio":4.7,"fantasy_feud_fp":39.75,"fantasy_feud_salary":153600,"fantasy_feud_ratio":0.26,"fanthrowdown_fp":45.2,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":44.25,"fantasy_aces_salary":7250,"fantasy_aces_ratio":6.1,"draftday_fp":45.25,"draftday_salary":18800,"draftday_ratio":2.41,"fantasy_score_fp":45.6,"fantasy_score_salary":9600,"fantasy_score_ratio":4.75,"draftster_fp":43.75,"draftster_salary":9400,"draftster_ratio":4.65,"yahoo_fp":44.8,"yahoo_salary":52,"yahoo_ratio":0.86,"treb":8},{"nba_player_id":"397","nba_game_id":"20015","date":"2016-01-19","nba_team_id":"21","opponent_id":"7","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35,"fgm":"8.6","fga":"18.0","p3m":"1.3","p3a":"4.1","ftm":"5.9","fta":"7.2","oreb":"1.3","dreb":"5.0","ast":"8.8","stl":"2.0","blk":"0.4","tov":"3.6","pf":"2.2","pts":"24.4","ts":"0.576","efg":"0.592","oreb_pct":"4.6","dreb_pct":"15.3","treb_pct":"10.2","ast_pct":"44.4","stl_pct":"3.0","blk_pct":"0.8","tov_pct":"14.6","usg":"31.3","ortg":"117.8","drtg":"101.2","nerd":"19.75","star_street_fp":44.48,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":41.33,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":46.36,"fanduel_salary":10500,"fanduel_ratio":4.42,"draft_kings_fp":49.13,"draft_kings_salary":10700,"draft_kings_ratio":4.59,"fantasy_feud_fp":41.33,"fantasy_feud_salary":169800,"fantasy_feud_ratio":0.24,"fanthrowdown_fp":47.33,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":46.68,"fantasy_aces_salary":7800,"fantasy_aces_ratio":5.98,"draftday_fp":47.1,"draftday_salary":20500,"draftday_ratio":2.3,"fantasy_score_fp":48.48,"fantasy_score_salary":9900,"fantasy_score_ratio":4.9,"draftster_fp":45.38,"draftster_salary":9500,"draftster_ratio":4.78,"yahoo_fp":47.01,"yahoo_salary":59,"yahoo_ratio":0.8,"treb":6.3},{"nba_player_id":"279","nba_game_id":"20016","date":"2016-01-19","nba_team_id":"11","opponent_id":"24","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":36.7,"fgm":"7.6","fga":"18.1","p3m":"2.5","p3a":"6.9","ftm":"5.5","fta":"6.5","oreb":"1.1","dreb":"5.8","ast":"5.3","stl":"1.8","blk":"0.4","tov":"3.6","pf":"2.4","pts":"22.5","ts":"0.537","efg":"0.610","oreb_pct":"3.3","dreb_pct":"17.6","treb_pct":"10.5","ast_pct":"25.1","stl_pct":"2.5","blk_pct":"0.9","tov_pct":"15.3","usg":"29.1","ortg":"104.1","drtg":"99.2","nerd":"5.26","star_street_fp":38.55,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":34.13,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":39.53,"fanduel_salary":8700,"fanduel_ratio":4.54,"draft_kings_fp":42.93,"draft_kings_salary":9200,"draft_kings_ratio":4.67,"fantasy_feud_fp":34.13,"fantasy_feud_salary":138800,"fantasy_feud_ratio":0.25,"fanthrowdown_fp":41.13,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":39.88,"fantasy_aces_salary":6500,"fantasy_aces_ratio":6.14,"draftday_fp":41.3,"draftday_salary":16600,"draftday_ratio":2.49,"fantasy_score_fp":41.68,"fantasy_score_salary":8400,"fantasy_score_ratio":4.96,"draftster_fp":39.45,"draftster_salary":8000,"draftster_ratio":4.93,"yahoo_fp":40.78,"yahoo_salary":47,"yahoo_ratio":0.87,"treb":6.9},{"nba_player_id":"2137","nba_game_id":"20014","date":"2016-01-19","nba_team_id":"38","opponent_id":"17","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35,"fgm":"8.0","fga":"16.6","p3m":"0.4","p3a":"1.3","ftm":"4.5","fta":"6.0","oreb":"2.6","dreb":"7.8","ast":"2.2","stl":"1.0","blk":"2.2","tov":"1.9","pf":"2.6","pts":"20.8","ts":"0.541","efg":"0.521","oreb_pct":"8.6","dreb_pct":"24.8","treb_pct":"16.6","ast_pct":"11.5","stl_pct":"1.4","blk_pct":"4.9","tov_pct":"9.4","usg":"27.0","ortg":"107.9","drtg":"103.1","nerd":"5.60","star_street_fp":41.05,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":36.55,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":41.08,"fanduel_salary":10300,"fanduel_ratio":3.99,"draft_kings_fp":44.25,"draft_kings_salary":10000,"draft_kings_ratio":4.43,"fantasy_feud_fp":36.55,"fantasy_feud_salary":149400,"fantasy_feud_ratio":0.24,"fanthrowdown_fp":41.8,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":41.6,"fantasy_aces_salary":7400,"fantasy_aces_ratio":5.62,"draftday_fp":40.43,"draftday_salary":18200,"draftday_ratio":2.22,"fantasy_score_fp":42.55,"fantasy_score_salary":9700,"fantasy_score_ratio":4.39,"draftster_fp":41.53,"draftster_salary":9200,"draftster_ratio":4.51,"yahoo_fp":41.28,"yahoo_salary":54,"yahoo_ratio":0.76,"treb":10.4},{"nba_player_id":"362","nba_game_id":"20013","date":"2016-01-19","nba_team_id":"15","opponent_id":"16","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":34.9,"fgm":"7.3","fga":"15.4","p3m":"1.4","p3a":"3.7","ftm":"4.8","fta":"6.0","oreb":"1.7","dreb":"6.1","ast":"2.3","stl":"0.7","blk":"1.0","tov":"1.7","pf":"2.0","pts":"20.6","ts":"0.571","efg":"0.594","oreb_pct":"6.2","dreb_pct":"19.9","treb_pct":"13.3","ast_pct":"12.1","stl_pct":"1.1","blk_pct":"2.2","tov_pct":"8.5","usg":"26.5","ortg":"115.3","drtg":"104.5","nerd":"11.63","star_street_fp":34.93,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":30.85,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":35.11,"fanduel_salary":7800,"fanduel_ratio":4.5,"draft_kings_fp":37.05,"draft_kings_salary":7600,"draft_kings_ratio":4.88,"fantasy_feud_fp":30.85,"fantasy_feud_salary":120400,"fantasy_feud_ratio":0.26,"fanthrowdown_fp":36.2,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":35.5,"fantasy_aces_salary":5900,"fantasy_aces_ratio":6.02,"draftday_fp":35.43,"draftday_salary":13950,"draftday_ratio":2.54,"fantasy_score_fp":36.35,"fantasy_score_salary":7000,"fantasy_score_ratio":5.19,"draftster_fp":35.35,"draftster_salary":7200,"draftster_ratio":4.91,"yahoo_fp":35.81,"yahoo_salary":40,"yahoo_ratio":0.9,"treb":7.8},{"nba_player_id":"2249","nba_game_id":"20014","date":"2016-01-19","nba_team_id":"17","opponent_id":"38","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35.7,"fgm":"7.2","fga":"16.6","p3m":"0.9","p3a":"3.2","ftm":"4.6","fta":"6.3","oreb":"1.3","dreb":"2.9","ast":"2.1","stl":"0.9","blk":"0.5","tov":"2.2","pf":"2.4","pts":"20.2","ts":"0.521","efg":"0.530","oreb_pct":"4.3","dreb_pct":"9.6","treb_pct":"6.9","ast_pct":"10.2","stl_pct":"1.3","blk_pct":"1.0","tov_pct":"10.3","usg":"26.7","ortg":"101.6","drtg":"111.5","nerd":"-7.07","star_street_fp":28.68,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":23.65,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":28.99,"fanduel_salary":6700,"fanduel_ratio":4.33,"draft_kings_fp":30.75,"draft_kings_salary":6900,"draft_kings_ratio":4.46,"fantasy_feud_fp":23.65,"fantasy_feud_salary":113500,"fantasy_feud_ratio":0.21,"fanthrowdown_fp":29.65,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":29.2,"fantasy_aces_salary":4900,"fantasy_aces_ratio":5.96,"draftday_fp":28.43,"draftday_salary":12000,"draftday_ratio":2.37,"fantasy_score_fp":30.3,"fantasy_score_salary":6000,"fantasy_score_ratio":5.05,"draftster_fp":29.23,"draftster_salary":5900,"draftster_ratio":4.95,"yahoo_fp":29.44,"yahoo_salary":29,"yahoo_ratio":1.02,"treb":4.2},{"nba_player_id":"370","nba_game_id":
Note that even this is not complete. You need to somehow map nba_player_id to the appropriate name. Anyway, a lot coding will be involved...

Solr search error when dealing with Arabic string

I'm struggling with Solr search Arabic for several days and made some experiment. Here is the simple reflection of the problem.
After I store some Arabic sentence (now only 1 word السوري ) into database and have Solr index it, then query it by q=*:*&wt=python,(if no wt part, it was garbled chars) the response is:
'\u00d8\u00a7\u00d9\u201e\u00d8\u00b3\u00d9\u02c6\u00d8\u00b1\u00d9\u0160'
The actual word I store there for index is coding in another way:
'\xd8\xa7\xd9\x84\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a'
As you can tell, there is a one-to-to corresponding from \xd8↔\u00d8. But I don't know what is the name of this coding, thus I cannot convert it. And when I do the search as: <>/select/?q=السوري&wt=python,the response is:
{'responseHeader':{'status':0,'QTime':0,'params':{'wt':'python','q':u'\u0627\u0644\u0633\u0648\u0631\u064a'}},'response':{'numFound':0,'start':0,'docs':[]}}
No docs found and it seems using a third version for coding u'\u0627\u0644\u0633\u0648\u0631\u064a'. if I take it and encode('utf8') then it convert back to '\xd8\xa7\xd9\x84\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a'.
In summary, when it (السوري) is in my code (python) or in data base (mysql),
it presents as 'form1':
'\xd8\xa7\xd9\x84\xd8\xb3\xd9\x88\xd8\xb1\xd9\x8a'
When it is indexed by Solr, it converts to form2:
'\u00d8\u00a7\u00d9\u201e\u00d8\u00b3\u00d9\u02c6\u00d8\u00b1\u00d9\u0160'
And when I use <>/select/?q=السوري&wt=python, to query from browser (Google chrome), it becomes form3:
'\u0627\u0644\u0633\u0648\u0631\u064a'
(which could convert back to form1 by encode('utf8') But since they are different, the search matches nothing.
Therefore, those three different encode strategy may be the core problem. Could anyone help me figure it out and solve the search problem?
Thanks in advance.

Is there any Open topic Classifier API for demo purposes

Is there an Open Classifier/Categorizer API which returns categories/topics for a given entity string. For example for an input string like : "Barack Obama" it should return topics like : "Politics" similarly for "Albert Einsten" it should emit "Physicist, Nobel Laureate" etc.

try texlexan. it gave me some really good results but with slightly longer input texts.