It is my first time using the Github API, sorry if this is a stupid question. I did a short search for location:Germany, and got 39,063 users. I want to create a list of all the 39,063 usernames and tried this command:
curl -i https://api.github.com/search/users?q=location%3AGermany | grep login
However this returns only 30 hits.. Could anyone give me some advice, or guide me to the right resources?
You will have to make additional requests for other pages:
Pagination
Requests that return multiple items will be paginated to 30 items by default. You can specify further pages with the ?page parameter. For some resources, you can also set a custom page size up to 100 with the ?per_page parameter. Note that for technical reasons not all endpoints respect the ?per_page parameter, see events for example.
$ curl 'https://api.github.com/user/repos?page=2&per_page=100'
Note that page numbering is 1-based and that omitting the ?page parameter will return the first page.
For more information on pagination, check out our guide on Traversing with Pagination.
Related
Yes, there is this question:
Github API: How to get all repositories written in a given language
however the answer provided only returns 100 results.
So how can I get the list of ALL repositories for a given language,
e.g. for Mathematica
curl https://api.github.com/search/repositories?q=language:mathematica
says there are 8000+ items that I should get, but this returns only top 30...
I have tried since
As suggested by #Bertrand Martel adding
&page=<page>&per_page=100
works.
You just have to request page 1 with 1 result per page to get total results, and then iterate over pages as needed.
I'm trying to list all the users in Jira using the REST API, I'm currently using the search user feature using GET : https://docs.atlassian.com/jira/REST/server/#api/2/user-findUsers
The thing is it says that the result will by default display the 50 first result and that we can expand that result up to 1000. Compared to other features available in the REST API, the pagination here is not specified.
An example is the group member feature : https://docs.atlassian.com/jira/REST/server/#api/2/group-getUsersFromGroup
Thus I did a test and with my test Jira filled with 2 members, tried to get only one result and see if there was some sort of indication referring to the rest of my result.
The response provided will only give the results and no ways to get to know if there was more thatn 1000 (or 1 in my example), it's maybe logical but in the case of an organization with more than 1000 members, listing all the users doing this : http://jira/rest/api/2/user/search?username=.&maxResults=1000&includeInactive=true will only give at most 1000 results.
I'm getting all the users no matter what their name are using . as the matching character.
Thanks for your help!
What you can do, is to calculate manually the number of users.
Let's say you have 98 users in your system.
First search will give you 50 users. Now you have an array and you can get the length of that array which is 50.
Since you do not know if there are 50 or 51 users, you execute another search with the parameter &startAt=50.
This time the array length is 48 instead of 50 and you know that you've reached all the users in the system.
From speaking to Atlassian support, it seems like the user/search endpoint has a bug where it will only ever return the first 1,000 results at most.
One possible other way to get all of the users in your JIRA instance is to use the Crowd API's /rest/usermanagement/1/search endpoint:
curl -X GET \
'https://jira.url/rest/usermanagement/1/search?entity-type=user&start-index=0&max-results=1000&expand=user' \
-H 'Accept: application/json' -u username:password
You'll need to create a new JIRA User Server entry to create Crowd credentials (the username:password parameter above) for your application to use in its REST API calls:
Go to User Management.
Select JIRA User Server.
Add an application.
Enter the application name and password that the application will use when accessing your JIRA server application.
Enter the IP address, addresses, or IP CIDR block of the application, and click Save.
I tried to leverage the solution described in this post using fiddler : How to change change Kibana saved search (Discover) with a REST request?
Also I had a look at: https://discuss.elastic.co/t/export-saved-objects-via-rest-api/72028/2
My problem is that even though the json returned by my get rest request has the right value for the number of definitions, it does not embed them all (only 10 out of 34 search definitions), is there like a index + count option for fetching all of them.
Ok just found a workaround, actually, I should have thought about that before...
Here is what I have done, basically listened to the queries from the browser using fiddler (using the https decrypt option) on:
Settings tab
Saved objects
Searches
It seems that there is a size parameter that I was not really aware of (like I said in my initial post, thought about an index + count / limit like in any db system)
POST my_kibana_url/elasticsearch/.kibana/visualization/_search?size=100 HTTP/1.1
{"query":{"match_all":{}}}
And look at the count value to check whether the sizes of all the requests accumulated is matching that number.
I would like to list public GitHub repositories with the latest create/update/push timestamps (for me any of these is acceptable). Can I achieve this with the GitHub API?
I have tried the following:
Tried using /repositories endpoint, and use the link header to navigate to the last page. However, the link header I receive only has first and next links, whereas I need a last link.
Tried using /search/repositories endpoint. This will work as long as I have a keyword or filter in the q parameter, but it will not accept an empty q parameter.
I got in touch with GitHub support, and there are two solutions to this:
Use binary search on the since parameter of the /repositories endpoint to find the last page.
Cons: may quickly exhaust the API rate limit.
Use the /search/repositories endpoint with an always-true predicate such as stars>=0.
Cons: likely to cause a query timeout/ incomplete results.
When I use /{page_id}/feed?access_token=xxxx, this give me all the posts on the page, both by user and page. I want to limit and control the posts. I want to put constraints like:
Timestamp (that is to get posts after a particular timestamp)
Post id (to get post after a particular post)
Since getting all the posts from feed is irrelevant and in-effective. Is there any way to accomplish this ?
You can use
GET /{page_id}/feed?limit={nr_of_posts_to_return}&since={timestamp}
to be able to limit the number of results and specify the starting timestamp. Have a look at the reference here:
https://developers.facebook.com/docs/graph-api/using-graph-api/v2.0#paging
For your second Use Case you'd need to use the Batch API imho, because with a single Graph API request you can't filter on specific Posts. Instead, you need to use the Batch API to split this in two queries as described here:
https://developers.facebook.com/docs/graph-api/making-multiple-requests/#operations
The request would then look like this:
curl \
-F 'access_token={your_access_token}' \
-F 'batch=[{ "method":"GET","name":"get-post","relative_url":"{your_post_id}?fields=created_time"},{"method":"GET","relative_url":"{your_page_id}/feed?since={result=get-post:$.created_time}&limit={nr_of_posts_to_return}"}]' \
https://graph.facebook.com/
In Graph Explorer, you have to change the HTTP method to Post, then add a new field called batch. Leave the URL blank so far. Paste this as batch value:
[{ "method":"GET","name":"get-post","relative_url":"293088074081904_400071946716849?fields=created_time"},{"method":"GET","relative_url":"293088074081904/feed?since={result=get-post:$.created_time}&limit=1"}]
This works at least for me.
For others looking for a solution, it appears the 'since' done at the 'comment' and 'reply' levels are ignored. Which means this is not a solution for me.
The query Tobi provided will provide all the posts after the first 'since' but every comment and reply in those posts, regardless of that you set their 'since' to.
Further to this, if you wish to search for new comments , regardless of the age of the post, this fails as well. For example:remove the first 'since' and change to limit=1000 and only request comments as a fields using 'since' , this will return the last 1000 posts and all comments for all of those 1000.
That said, thank you Tobi for your time and showing me how to get everything I need in a single function call. I may experiment parsing the complete recordset every time. ( maybe too much traffic though!)