How to get the GitHub data from various URLs and stored in the single mongoDB - github

I'm trying to get the GitHub data using Talend big data. The thing is, i have multiple URLs,because used each URL to take some values & stored into single mongoDB. The below order only i'm going to try & get the informations,
https://api.github.com/users/sample/repos
https://api.github.com/repos/sample/awesome-ciandcd/commits
https://api.github.com/repos/sample/awesome-ciandcd/contributors
Each URLs are giving the single JSONArray with multiple data format.Please give some suggestion to do this. I've already tried with sub-jobs component. But not get clear job.
My Output Should be like,
{
"gitname": "sample",
"gitType" : "User",
"password": "password",
"repoCount": 3,
"repositoryDetails": [
{
"repoName": "MergeCheckRepository",
"fileCount": 10,
"branchCount": 6,
"releaseCount": 2,
"commitsCount": 10,
"contributorsCount": 3,
"totalPulls": 1,
"mergeCount": 1,
"totalIssues": 12,
"closedIssueCount": 3,
"watchersCount": 1,
"stargazersCount": 4,
"contributorsDetails": [
{
"login": "sample",
"avatarURL": "https://avatars2.githubusercontent.com/u/30261572?v=4",
"contributions": 3
}
],
"commitDetails": [
{
"name": "sample",
"email": "sampletest#test.com",
"date": "2017-07-20T09:09:09Z"
}
]
},
{
"repoName": "Dashboard",
"filecount": 19,
"branchCount": 4,
"releasecount": 2,
"commitsCount": 5,
"contributorsCount": 3,
"totalPulls": 1,
"totalIssues": 2,
"closedIssueCount": 3,
"watchersCount": 1,
"stargazersCount": 4,
"contributorsDetails": [
{
"login": "sample",
"avatarURL": "https://avatars2.githubusercontent.com/u/30261572?v=4",
"contributions": 3
},
{
"login": "Dashboard",
"avatarURL": "https://avatars2.githubusercontent.com/u/30261572?v=4",
"contributions": 3
}
],
"commitDetails": [
{
"name": "sample",
"email": "sampletest#test.com",
"date": "2017-07-14T09:09:09Z"
},
{
"name": "Dashboard",
"email": "prakash.thangasamy#test.com",
"date": "2017-07-19T09:09:09Z"
},
{
"name": "testrepo",
"email": "test.dashboard#test.com",
"date": "2017-07-20T09:09:09Z"
}
]
}
]
}
How to achieve this one with sub-job? Is there any other way to do this?

Related

MongoDb Aggregation separated array based on condition

I'm very new to mongodb. I'm trying to do an aggregation pipeline for lookup kinda like SQL left join. Given the following document's schema:
Mongo playground: https://mongoplayground.net/p/yAIwH5V2yv8
Characters:
{
"_id": 1,
"account_id": 1,
"world_id": 0,
"name": "hello"
}
Inventories:
{
"_id": 7,
"character_id": 2,
"type": "EQUIPPED"
}
Items:
{
"_id": 1,
"inventory_id": 7
}
I want to query for characters and look up inventories as well as items in inventories. I was able to achieve this however I would like to separate the inventories field in characters result document.
Current result:
{
"_id": 2,
"account_id": 1,
"world_id": 0,
"name": "hello",
"inventories: [
{
"_id": 1,
"character_id": "2",
"type": "EQUIPPED",
"items: [...]
}
]
}
What I want is based on the type of inventory I want it to be a separate field of the resulted character document something like this:
{
"_id": 2,
"account_id": 1,
"world_id": 0,
"name": "hello",
"equippedInventory: {
"_id": 1,
"character_id": "2",
"type": "EQUIPPED",
"items: [...]
},
"equipInventory: {
"_id": 2,
"character_id": "2",
"type": "EQUIP",
"items: [...]
},
}
Also, is my pipeline the best way to achieve this?

MongoDB update_one vs update_many - Improve speed

I got a collection of 10000 ca. docs, where each doc has the following format:
{
"_id": {
"$oid": "631edc6e207c89b932a70a26"
},
"name": "Ethereum",
"auditInfoList": [
{
"coinId": "1027",
"auditor": "Fairyproof",
"auditStatus": 2,
"reportUrl": "https://www.fairyproof.com/report/Covalent"
}
],
"circulatingSupply": 122335921.0615,
"cmcRank": 2,
"dateAdded": "2015-08-07T00:00:00.000Z",
"id": 1027,
"isActive": 1,
"isAudited": true,
"lastUpdated": 1662969360,
"marketPairCount": 6085,
"quotes": [
{
"name": "USD",
"price": 1737.1982544180462,
"volume24h": 14326453277.535921,
"marketCap": 212521748520.66168,
"percentChange1h": 0.62330307,
"percentChange24h": -1.08847937,
"percentChange7d": 10.96517745,
"lastUpdated": 1662966780,
"percentChange30d": -13.49374496,
"percentChange60d": 58.25153862,
"percentChange90d": 42.27475921,
"fullyDilluttedMarketCap": 212521748520.66,
"marketCapByTotalSupply": 212521748520.66168,
"dominance": 20.0725,
"turnover": 0.0674117,
"ytdPriceChangePercentage": -53.9168
}
],
"selfReportedCirculatingSupply": 0,
"slug": "ethereum",
"symbol": "ETH",
"tags": [
"mineable",
"pow",
"smart-contracts",
"ethereum-ecosystem",
"coinbase-ventures-portfolio",
"three-arrows-capital-portfolio",
"polychain-capital-portfolio",
"binance-labs-portfolio",
"blockchain-capital-portfolio",
"boostvc-portfolio",
"cms-holdings-portfolio",
"dcg-portfolio",
"dragonfly-capital-portfolio",
"electric-capital-portfolio",
"fabric-ventures-portfolio",
"framework-ventures-portfolio",
"hashkey-capital-portfolio",
"kenetic-capital-portfolio",
"huobi-capital-portfolio",
"alameda-research-portfolio",
"a16z-portfolio",
"1confirmation-portfolio",
"winklevoss-capital-portfolio",
"usv-portfolio",
"placeholder-ventures-portfolio",
"pantera-capital-portfolio",
"multicoin-capital-portfolio",
"paradigm-portfolio",
"injective-ecosystem"
],
"totalSupply": 122335921.0615
}
Im pulling updated version of it and, to aviod duplicates, im doing the following by using 'update_one'
for doc in new_doc_list:
CRYPTO_TEMPORARY_LIST.update_one(
{ "name" : doc['name']},
{ "$set": {
"lastUpdated": doc['lastUpdated']
}
},
upsert=True)
The problem is it's too slow.
I'm trying to figure out how to improve speed by using update_many but can't figure out how to set it up.
I Basically want to update every document x name. Completely change the doc and not the "lastUpdated" field would b even better.
Thanks guys <3

Check if student submitted assignment via Moodle web services

I am trying to use the Moodle API (web services) to get information about (my) assignment submissions. I want to know whether I submitted an attempt for the assignment already or not. I am using the mod_assign_get_assignments function (which doesn't have too much documentation) and the results I get (when looking at the assignments portion of each course) are:
{
"id": 25960,
"cmid": 350053,
"course": 8013502,
"name": "\u05d4\u05d2\u05e9\u05ea \u05ea\u05e8\u05d2\u05d9\u05dc \u05d1\u05d9\u05ea 1",
"nosubmissions": 0,
"submissiondrafts": 0,
"sendnotifications": 0,
"sendlatenotifications": 0,
"sendstudentnotifications": 0,
"duedate": 1617566400,
"allowsubmissionsfromdate": 0,
"grade": 100,
"timemodified": 1615897679,
"completionsubmit": 1,
"cutoffdate": 1617569940,
"gradingduedate": 0,
"teamsubmission": 0,
"requireallteammemberssubmit": 0,
"teamsubmissiongroupingid": 0,
"blindmarking": 0,
"hidegrader": 0,
"revealidentities": 0,
"attemptreopenmethod": "manual",
"maxattempts": 1,
"markingworkflow": 0,
"markingallocation": 0,
"requiresubmissionstatement": 0,
"preventsubmissionnotingroup": 0
...irrelevant configuations
}
The above result is for an assignment I have already submitted.
An example of an assignment I did not submit is:
{
"id": 19764,
"cmid": 268225,
"course": 8013201,
"name": "\u05ea\u05d9\u05d1\u05ea \u05d4\u05d2\u05e9\u05d4 14",
"nosubmissions": 0,
"submissiondrafts": 0,
"sendnotifications": 0,
"sendlatenotifications": 0,
"sendstudentnotifications": 0,
"duedate": 1611693000,
"allowsubmissionsfromdate": 0,
"grade": 100,
"timemodified": 1610972842,
"completionsubmit": 0,
"cutoffdate": 1611694860,
"gradingduedate": 0,
"teamsubmission": 0,
"requireallteammemberssubmit": 0,
"teamsubmissiongroupingid": 0,
"blindmarking": 0,
"hidegrader": 0,
"revealidentities": 0,
"attemptreopenmethod": "manual",
"maxattempts": 1,
"markingworkflow": 0,
"markingallocation": 0,
"requiresubmissionstatement": 0,
"preventsubmissionnotingroup": 0
...irrelevant configuations
}
The only apparent difference between these (that might point to a way to check if I submitted it or not) is the completionsubmit property, but that cannot be the solution because a different assignment that I have submitted has it set to 0 (just like the one I didn't submit).
Does someone have an idea how I can solve this issue?
Thanks in Advance!
EDIT: mod_assign_get_submissions denies my access
{"assignments":[],"warnings":[{"item":"assignment","itemid":myitemname,"warningcode":"1","message":"No access rights in module context"}]}
I looked now into mod_assign_get_submission_status but it seems like it is only able to parse one assignment at a time, does anyone have a way to make this more efficient?
You could try using mod_assign_get_submissions instead to retrieve submissions to assignments. Available since Moodle 2.5
References
Moodle API
Emulated Data For Get Submissions from Moodle
Sample Response
{
"assignments": [
{
"assignmentid": 14,
"submissions": [
{
"id": 7,
"userid": 3,
"attemptnumber": 0,
"timecreated": 1426865031,
"timemodified": 1426865062,
"status": "submitted",
"groupid": 0,
"plugins": [
{
"type": "onlinetext",
"name": "Online text",
"fileareas": [
{
"area": "submissions_onlinetext"
}
],
"editorfields": [
{
"name": "onlinetext",
"description": "Submission comments",
"text": "<p>But I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. <br></p>",
"format": 1
}
]
},
{
"type": "file",
"name": "File submissions",
"fileareas": [
{
"area": "submission_files",
"files": [
{
"filepath": "APDFfile.pdf",
"fileurl": "http://localhost/m/stable_master/webservice/pluginfile.php/247/assignsubmission_file/submission_files/12/somefile.pdf"
},
{
"filepath": "anotherfile.docx",
"fileurl": "http://localhost/m/stable_master/webservice/pluginfile.php/247/assignsubmission_file/submission_files/12/somefile.pdf"
}
]
}
]
},
{
"type": "comments",
"name": "Submission comments"
}
]
},
{
"id": 5,
"userid": 4,
"attemptnumber": 0,
"timecreated": 1426864693,
"timemodified": 1426864740,
"status": "draft",
"groupid": 0,
"plugins": [
{
"type": "onlinetext",
"name": "Online text",
"fileareas": [
{
"area": "submissions_onlinetext",
"files": [
{
"filepath": "/Arte esquemático-Cigüeña.png",
"fileurl": "http://localhost/m/stable_master/webservice/pluginfile.php/245/assignsubmission_onlinetext/submissions_onlinetext/5/Arte%20esquem%C3%A1tico-Cig%C3%BCe%C3%B1a.png"
}
]
}
],
"editorfields": [
{
"name": "onlinetext",
"description": "Submission comments",
"text": "<p>Blah Blah Blah lorem ipsum</p><p><br></p><p><b>Blah Blah Blah lorem ipsum</b><br></p><p><b><br></b></p><p><b><span style=\"font-weight: normal;\"><i>Blah Blah Blah lorem ipsum</i></span><br></b></p><p><b><span style=\"font-weight: normal;\"><i><br></i></span></b></p><p><b><span style=\"font-weight: normal;\"><i><img src=\"##PLUGINFILE##/Arte%20esquem%C3%A1tico-Cig%C3%BCe%C3%B1a.png\" alt=\"\" width=\"734\" height=\"844\" role=\"presentation\" style=\"vertical-align:text-bottom; margin: 0 .5em;\" class=\"img-responsive\"><br></i></span></b></p>",
"format": 1
}
]
},
{
"type": "file",
"name": "File submissions",
"fileareas": [
{
"area": "submission_files",
"files": [
{
"filepath": "somefile.pdf",
"fileurl": "http://localhost/m/stable_master/webservice/pluginfile.php/247/assignsubmission_file/submission_files/12/somefile.pdf"
}
]
}
]
},
{
"type": "comments",
"name": "Submission comments"
}
]
}
]
}
],
"warnings": []
}

Loopback 3 query by Property of a embedded model

I'm using loopback 3 to build a backend with mongoDB.
So i have 2 models: Object and Attachment. Object have a relation Embeds2Many to Attachment.
Objects look like that in mongoDB
[
{
"fieldA": "valueA1",
"attachments": [
{
"id": 1,
"url": "abc.com/image1"
},
{
"id": 2,
"url": "abc.com/image2"
}
]
},
{
"fieldA": "valueA2",
"attachments": [
{
"id": 4,
"url": "abc.com/image4"
},
{
"id": 5,
"url": "abc.com/image5"
}
]
}
]
The question is: how can i get Objects with attachments.id=4 over the RestAPI?
I have try with the where and include filter. But it didn't work. It look like, that this function is not implemented in loopback3, right?
I have found the solution. It only works on Mongodb, Cloudant and Memory database.
{
"filter": {
"where": {
"attachments.id": 4
}
}
}

How to set review for revision using Gerrit REST API

I'm trying to setup Teamcity building and verifying patchsets from Gerrit. The last step should set Verified to -1 if build failed. I'm playing around with Gerrit REST API and I think I found a proper command:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#set-review
The documentation says:
As response a ReviewInfo entity is returned that describes the applied
labels.
My request looks like this:
POST <gerrit-url>/a/changes/I696f00f4968fcb35fa614ce6325499aa15367150/revisions/current/review
{
"message": "Build failed",
"labels": {
"Verified": -1
}
}
As a response I get full revision info:
{
"id": "dev_test~master~<change-id>",
"project": "dev_test",
"branch": "master",
"hashtags": [],
"change_id": "<change-id>",
"subject": "a test",
"status": "NEW",
"created": "2017-04-03 07:53:19.000000000",
"updated": "2017-04-04 08:47:34.000000000",
"submit_type": "MERGE_IF_NECESSARY",
"mergeable": true,
"insertions": 133,
"deletions": 7,
"unresolved_comment_count": 0,
"_number": 381,
"owner": {
"_account_id": 4,
"name": "<my-name>",
"email": "<my-email>",
"username": "<my-username>",
},
"labels": {
"Code-Review": {
"all": [
{
"value": 1,
"date": "2017-04-04 08:47:34.000000000",
"permitted_voting_range": {
"min": -2,
"max": 2
},
"_account_id": 4,
"name": "<my-name>",
"email": "<my-email>",
"username": "<my-username>"
}
],
"values": {
"-2": "This shall not be merged",
"-1": "I would prefer this is not merged as is",
" 0": "No score",
"+1": "Looks good to me, but someone else must approve",
"+2": "Looks good to me, approved"
},
"default_value": 0
},
"Verified": {
"all": [
{
"value": 0,
"permitted_voting_range": {
"min": -1,
"max": 1
},
"_account_id": 4,
"name": "<my-name>",
"email": "<my-email>",
"username": "<my-username>"
}
],
"values": {
"-1": "Fails",
" 0": "No score",
"+1": "Verified"
},
"default_value": 0
}
},
"permitted_labels": {},
"removable_reviewers": [],
"reviewers": {
"REVIEWER": [
{
"_account_id": 4,
"name": "<my-name>",
"email": "<my-email>",
"username": "<my-username>"
}
]
},
"current_revision": "913330441711b067899a664a60c78be518e547b4",
"revisions": {
"913330441711b067899a664a60c78be518e547b4": {
"kind": "REWORK",
"_number": 6,
"created": "2017-04-03 14:08:14.000000000",
"uploader": {
"_account_id": 4,
"name": "<my-name>",
"email": "<my-email>",
"username": "<my-username>"
},
"ref": "refs/changes/81/381/6",
"fetch": {
"ssh": {
"url": "ssh://<url>",
"ref": "refs/changes/81/381/6"
},
"http": {
"url": "http://<url>",
"ref": "refs/changes/81/381/6"
}
}
}
}
}
It's not affected by request. Same response is returned when I send request using GET method or using POST method with invalid JSON in body(!)
Gerrit version is: 2.13.6-3008-gcdc381e
Do I something wrong?
PS. Here is similar question, but it isn't helpful: Gerrit set-review api doesn't work
EDIT:
It seems that I getting response from GET request not POST
I figured it out. It's not gerrit problem. I used http request and our server redirected to https with 301 which the Postman fallowed and returned response for GET request.