Query Mongo using list in a file - mongodb

I need to query Mongo for a field where the value is in a list, and then print that value plus a few others. However the list is about 4000 values long, so it is in a file. How could I do that? So something like
db.table.find({field1:{$eq:"abc"},"field2.subvalue":{$in:list.txt}},{"_id":1,"field2.subvalue":1,"field3":1})
Maybe I'm just not using the correct search terms but I can't find anything where you query from values in a file when I search Google.
Thanks in advance!

I second Sorens answer, try loading them into simple JSON format. for example.
file name : testfile
f1 f2 f3 f4 f5 -- Field
v1 v2 v3 v4 v5 --- Row one
x1 x2 x3 x4 x5 --- row two
your collection would look like,
collection name : testCollection
{
f1:v2,f2:v2,f3:v3,f4:v4,f5:v5 <--- These are documents
}
{
f1:x1,f2:x2,f3:x3,f4:x4,f5:v5
}
And so on and then query from MongoDB / program
Please dont do this manually for 4k fields , that's a nightmare :).
Google around for some java or python code and have that do it for you.
Good luck

Related

OpenEdge Progress 4GL Query returns (MISSING) after % sign

DEFINE TEMP-TABLE tt_pay_terms NO-UNDO
FIELD pt_terms_code LIKE payment_terms.terms_code
FIELD pt_description LIKE payment_terms.description.
DEFINE VARIABLE htt AS HANDLE NO-UNDO.
htt = TEMP-TABLE tt_pay_terms:HANDLE.
FOR EACH platte.payment_terms
WHERE (
active = true
AND system_id = "000000"
)
NO-LOCK:
CREATE tt_pay_terms.
ASSIGN
pt_terms_code = payment_terms.terms_code.
pt_description = payment_terms.description.
END.
htt:WRITE-JSON("FILE", "/dev/stdout", FALSE).
I have written this query and it returns data like this
[pt_terms_code] => 0.4%!N(MISSING)ET46
[pt_description] => 0.4%! (MISSING)DAYS NET 46
While I believe (from using a SQL query) that the data should be
0.4%45NET46
0.4% 45 DAYS NET 46
I'm making an assumption that the % is probably some special character (as I've run into similar issues in the past). I've tried pulling all the data from the table, and I get the same result, (ie, not creating a temp table and populating it with all the only the two fields I want).
Any suggestions around this issue?
I'm still very new to 4gl, so the above query might be terribly wrong. All comments and criticisms are welcome.
I suspect that if you try this:
FOR EACH platte.payment_terms NO-LOCK
WHERE ( active = true AND system_id = "000000" ):
display
payment_terms.terms_code
payment_terms.description
.
END.
You will see what the query actually returns. (WRITE-JSON is adding a layer after the query.) You will likely discover that your data contains something unexpected.
To my eye the "%" looks more like formatting -- the terms are likely 0.4%.
You then seem to have some issues in the contents of the description field. My guess is that there was a code page mismatch when the user entered the data and that there is gibberish in the field as a result.

importing website to google sheets

I have tried searching everywhere online for a good answer but cannot seem to find anything that matches specifically what i am looking for.
When i use the IMPORTHTML function in google sheets, i end up with data that looks like:
${player.name} (${player.position}, ${team.abbrev}) ${opponent.abbrev} #${opponent_rank} ${minutes} ${pts} ${fgm}-${fga} ${ftm}-${fta} ${p3m}-${p3a} ${treb} ${ast} ${stl} ${blk} ${tov} ${pf} ${fp} $${salary} ${ratio}
the code that i am using looks like this:
=IMPORTHTML("", "table",2)
When I use the same as above (=IMPORTHTML("", "table",2)) only with "0" as my index, it pulls this:
Opp Stats
Player Team Rank Min Pts FGM/A FTM/A 3PM/A Reb Ast Stl Blk Tov Foul FP Cost Value
Basically, I am attempting to pull the table data from this website:
https://www.numberfire.com/nba/fantasy/fantasy-basketball-projections
(because of my rep i cannot post more than two links, however my IMPORTHTML function has the above link input in both functions)
into a google sheet. Please help. any feedback is much appreciated... thanks!
Best advice is to find another Web table you can import. If you do "view source" on the page, you will find that the table content is dynamically populated from a variable named NF_DATA.
You need to create a document script to extract the data you want:
function this_is_test() {
var response = UrlFetchApp.fetch("https://www.numberfire.com/nba/fantasy/fantasy-basketball-projections");
raw_content = response.getContentText();
re = new RegExp('"daily_projections":\\[[^\\]]+','i');
proj = raw_content.match(re);
Logger.log(proj);
}
It will extract all text in-between "daily_projections":[ and ], which is (as of today):
"daily_projections":[{"nba_player_id":"77","nba_game_id":"20015","date":"2016-01-19","nba_team_id":"21","opponent_id":"7","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":36.3,"fgm":"8.8","fga":"17.1","p3m":"1.9","p3a":"4.8","ftm":"6.2","fta":"6.9","oreb":"0.8","dreb":"7.2","ast":"4.7","stl":"1.1","blk":"1.2","tov":"2.7","pf":"1.8","pts":"25.3","ts":"0.628","efg":"0.655","oreb_pct":"2.6","dreb_pct":"21.4","treb_pct":"12.4","ast_pct":"23.4","stl_pct":"1.5","blk_pct":"2.4","tov_pct":"12.1","usg":"27.8","ortg":"122.2","drtg":"101.8","nerd":"22.34","star_street_fp":43.08,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":39.75,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":43.85,"fanduel_salary":9900,"fanduel_ratio":4.43,"draft_kings_fp":46.55,"draft_kings_salary":9900,"draft_kings_ratio":4.7,"fantasy_feud_fp":39.75,"fantasy_feud_salary":153600,"fantasy_feud_ratio":0.26,"fanthrowdown_fp":45.2,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":44.25,"fantasy_aces_salary":7250,"fantasy_aces_ratio":6.1,"draftday_fp":45.25,"draftday_salary":18800,"draftday_ratio":2.41,"fantasy_score_fp":45.6,"fantasy_score_salary":9600,"fantasy_score_ratio":4.75,"draftster_fp":43.75,"draftster_salary":9400,"draftster_ratio":4.65,"yahoo_fp":44.8,"yahoo_salary":52,"yahoo_ratio":0.86,"treb":8},{"nba_player_id":"397","nba_game_id":"20015","date":"2016-01-19","nba_team_id":"21","opponent_id":"7","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35,"fgm":"8.6","fga":"18.0","p3m":"1.3","p3a":"4.1","ftm":"5.9","fta":"7.2","oreb":"1.3","dreb":"5.0","ast":"8.8","stl":"2.0","blk":"0.4","tov":"3.6","pf":"2.2","pts":"24.4","ts":"0.576","efg":"0.592","oreb_pct":"4.6","dreb_pct":"15.3","treb_pct":"10.2","ast_pct":"44.4","stl_pct":"3.0","blk_pct":"0.8","tov_pct":"14.6","usg":"31.3","ortg":"117.8","drtg":"101.2","nerd":"19.75","star_street_fp":44.48,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":41.33,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":46.36,"fanduel_salary":10500,"fanduel_ratio":4.42,"draft_kings_fp":49.13,"draft_kings_salary":10700,"draft_kings_ratio":4.59,"fantasy_feud_fp":41.33,"fantasy_feud_salary":169800,"fantasy_feud_ratio":0.24,"fanthrowdown_fp":47.33,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":46.68,"fantasy_aces_salary":7800,"fantasy_aces_ratio":5.98,"draftday_fp":47.1,"draftday_salary":20500,"draftday_ratio":2.3,"fantasy_score_fp":48.48,"fantasy_score_salary":9900,"fantasy_score_ratio":4.9,"draftster_fp":45.38,"draftster_salary":9500,"draftster_ratio":4.78,"yahoo_fp":47.01,"yahoo_salary":59,"yahoo_ratio":0.8,"treb":6.3},{"nba_player_id":"279","nba_game_id":"20016","date":"2016-01-19","nba_team_id":"11","opponent_id":"24","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":36.7,"fgm":"7.6","fga":"18.1","p3m":"2.5","p3a":"6.9","ftm":"5.5","fta":"6.5","oreb":"1.1","dreb":"5.8","ast":"5.3","stl":"1.8","blk":"0.4","tov":"3.6","pf":"2.4","pts":"22.5","ts":"0.537","efg":"0.610","oreb_pct":"3.3","dreb_pct":"17.6","treb_pct":"10.5","ast_pct":"25.1","stl_pct":"2.5","blk_pct":"0.9","tov_pct":"15.3","usg":"29.1","ortg":"104.1","drtg":"99.2","nerd":"5.26","star_street_fp":38.55,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":34.13,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":39.53,"fanduel_salary":8700,"fanduel_ratio":4.54,"draft_kings_fp":42.93,"draft_kings_salary":9200,"draft_kings_ratio":4.67,"fantasy_feud_fp":34.13,"fantasy_feud_salary":138800,"fantasy_feud_ratio":0.25,"fanthrowdown_fp":41.13,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":39.88,"fantasy_aces_salary":6500,"fantasy_aces_ratio":6.14,"draftday_fp":41.3,"draftday_salary":16600,"draftday_ratio":2.49,"fantasy_score_fp":41.68,"fantasy_score_salary":8400,"fantasy_score_ratio":4.96,"draftster_fp":39.45,"draftster_salary":8000,"draftster_ratio":4.93,"yahoo_fp":40.78,"yahoo_salary":47,"yahoo_ratio":0.87,"treb":6.9},{"nba_player_id":"2137","nba_game_id":"20014","date":"2016-01-19","nba_team_id":"38","opponent_id":"17","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35,"fgm":"8.0","fga":"16.6","p3m":"0.4","p3a":"1.3","ftm":"4.5","fta":"6.0","oreb":"2.6","dreb":"7.8","ast":"2.2","stl":"1.0","blk":"2.2","tov":"1.9","pf":"2.6","pts":"20.8","ts":"0.541","efg":"0.521","oreb_pct":"8.6","dreb_pct":"24.8","treb_pct":"16.6","ast_pct":"11.5","stl_pct":"1.4","blk_pct":"4.9","tov_pct":"9.4","usg":"27.0","ortg":"107.9","drtg":"103.1","nerd":"5.60","star_street_fp":41.05,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":36.55,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":41.08,"fanduel_salary":10300,"fanduel_ratio":3.99,"draft_kings_fp":44.25,"draft_kings_salary":10000,"draft_kings_ratio":4.43,"fantasy_feud_fp":36.55,"fantasy_feud_salary":149400,"fantasy_feud_ratio":0.24,"fanthrowdown_fp":41.8,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":41.6,"fantasy_aces_salary":7400,"fantasy_aces_ratio":5.62,"draftday_fp":40.43,"draftday_salary":18200,"draftday_ratio":2.22,"fantasy_score_fp":42.55,"fantasy_score_salary":9700,"fantasy_score_ratio":4.39,"draftster_fp":41.53,"draftster_salary":9200,"draftster_ratio":4.51,"yahoo_fp":41.28,"yahoo_salary":54,"yahoo_ratio":0.76,"treb":10.4},{"nba_player_id":"362","nba_game_id":"20013","date":"2016-01-19","nba_team_id":"15","opponent_id":"16","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":34.9,"fgm":"7.3","fga":"15.4","p3m":"1.4","p3a":"3.7","ftm":"4.8","fta":"6.0","oreb":"1.7","dreb":"6.1","ast":"2.3","stl":"0.7","blk":"1.0","tov":"1.7","pf":"2.0","pts":"20.6","ts":"0.571","efg":"0.594","oreb_pct":"6.2","dreb_pct":"19.9","treb_pct":"13.3","ast_pct":"12.1","stl_pct":"1.1","blk_pct":"2.2","tov_pct":"8.5","usg":"26.5","ortg":"115.3","drtg":"104.5","nerd":"11.63","star_street_fp":34.93,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":30.85,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":35.11,"fanduel_salary":7800,"fanduel_ratio":4.5,"draft_kings_fp":37.05,"draft_kings_salary":7600,"draft_kings_ratio":4.88,"fantasy_feud_fp":30.85,"fantasy_feud_salary":120400,"fantasy_feud_ratio":0.26,"fanthrowdown_fp":36.2,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":35.5,"fantasy_aces_salary":5900,"fantasy_aces_ratio":6.02,"draftday_fp":35.43,"draftday_salary":13950,"draftday_ratio":2.54,"fantasy_score_fp":36.35,"fantasy_score_salary":7000,"fantasy_score_ratio":5.19,"draftster_fp":35.35,"draftster_salary":7200,"draftster_ratio":4.91,"yahoo_fp":35.81,"yahoo_salary":40,"yahoo_ratio":0.9,"treb":7.8},{"nba_player_id":"2249","nba_game_id":"20014","date":"2016-01-19","nba_team_id":"17","opponent_id":"38","season":"2016","game_play_probability":"1.00","game_start":"1.00","minutes":35.7,"fgm":"7.2","fga":"16.6","p3m":"0.9","p3a":"3.2","ftm":"4.6","fta":"6.3","oreb":"1.3","dreb":"2.9","ast":"2.1","stl":"0.9","blk":"0.5","tov":"2.2","pf":"2.4","pts":"20.2","ts":"0.521","efg":"0.530","oreb_pct":"4.3","dreb_pct":"9.6","treb_pct":"6.9","ast_pct":"10.2","stl_pct":"1.3","blk_pct":"1.0","tov_pct":"10.3","usg":"26.7","ortg":"101.6","drtg":"111.5","nerd":"-7.07","star_street_fp":28.68,"star_street_salary":0,"star_street_ratio":0,"draft_street_daily_fp":23.65,"draft_street_daily_salary":0,"draft_street_daily_ratio":0,"fanduel_fp":28.99,"fanduel_salary":6700,"fanduel_ratio":4.33,"draft_kings_fp":30.75,"draft_kings_salary":6900,"draft_kings_ratio":4.46,"fantasy_feud_fp":23.65,"fantasy_feud_salary":113500,"fantasy_feud_ratio":0.21,"fanthrowdown_fp":29.65,"fanthrowdown_salary":0,"fanthrowdown_ratio":0,"fantasy_aces_fp":29.2,"fantasy_aces_salary":4900,"fantasy_aces_ratio":5.96,"draftday_fp":28.43,"draftday_salary":12000,"draftday_ratio":2.37,"fantasy_score_fp":30.3,"fantasy_score_salary":6000,"fantasy_score_ratio":5.05,"draftster_fp":29.23,"draftster_salary":5900,"draftster_ratio":4.95,"yahoo_fp":29.44,"yahoo_salary":29,"yahoo_ratio":1.02,"treb":4.2},{"nba_player_id":"370","nba_game_id":
Note that even this is not complete. You need to somehow map nba_player_id to the appropriate name. Anyway, a lot coding will be involved...

PigLatin - Removing A Tuple/Field From Relation

This is my first post on StackOverflow, so pardon me in advance if this is too lengthy and/or vague.
I have a Pig relation as follows:
my_relation = LOAD '/path/to/data' USING PigStorage(',')
AS (f1:chararray, f2:chararray, f3:chararray);
Now, I wanted to filter out the field 'f3' from the above relation 'my_relation'. I know I could do it like so:
my_new_relation = FOREACH my_relation GENERATE my_relation.f1, my_relation.f2;
The problem with this method comes when I have a large number of fields/tuples in the 'my_relation' relation. Say, my_relation had 900 fields/tuples, and I wanted all of them but for one. Now, with the above method, I'd have to list out 899 fields/tuples after my 'GENERATE' keyword!
My question: Is there an easy way to filter out a handful of fields/tuples from a relation in PigLatin?
Prior: My prior on Apache Pig and PigLatin in general is very weak (as can be told by the difficulty of this question). I'm still reading through the Pig documentation found here.
Thanks for reading this question! Any/all help is appreciated!
First of all, your syntax is not quite right. If you wanted to just save the fields f1 and f2, you would do it like this:
my_new_relation = FOREACH my_relation GENERATE f1, f2;
As to your question, you can use a project-range expression:
my_new_relation = FOREACH my_relation GENERATE f1 .. f345, f347 .. f900;
Also you can write your own loader where you can specify which all columns you want to load.

MongoDB + Java + Search to fetch field or column names only after searching a string in a document using either full text or regular search

I am am in need of a solution quickly for the following:
Assuming that I have n numbere of documents in collection XYZ similar to below:
XYZ{
NAME: Test
AGE: 33
ADDRESS: ASDF
EDUCATION: BSC
}
If I want to search for a string say: "BSC"
I want the api to search for the string "BSC", and after successful search, it has to return me only the column name( or field name) of the document XYZ: "EDUCATION"
Any quick help for the solution of the above problem would be very helpful.
Thanks.
MongoDB does not allow you full text search, Sorry, I am not able to comment this one, Don't have appropriate points.

exporting MongoDB to CSV using pymongo

I would like write a script to generate a CSV file from my mongoDB database and I would like to know the most convenient version !
first let me begin with the structure of collections.
MyDataBase -> setting
users
fruits
in setting I have something like
setting -> _id
data
_tenant
and the thing I am after, is making a CSV file out of profiles in data
which they have some fields/properties like "name", "address", "postalcode", "email", age and etc. and not neccessary all of these profile have all files/properties and even some of them look like collection (have sub-branches) which I am not interested in at all !
so, my code is python so far is look like these
myquery = db.settings.find() # I am getting everything !
output = csv.writer(open('some.csv', 'wt')) # writng in this file
for items in myquery[0:10]: # first 11 entries
a = list(items['data']['Profile'].values()) # collections are importent as dictionary and I am making them as list
tt = list()
for chiz in a:
if chiz is not None:
tt.append(chiz.encode('ascii', 'ignore')) #encoding
else:
tt.append("none")
output.writerow(tt)
these fields/properties dont have neccessary all fields, and also even some of them are collection(with sub-branch) and will be imported as dictionary ! so, I have to convert them to list and all and all, there are quite few things to take care in such a process and in all doesn't look that straightforward !
My question might be sounds very general but is it a typical way to make such report ?! if not, can you someone make it clear ?!
Yes, I am using the same way.
It is clear and fast, also it works without of any additional libraries.