How to export 15000+ Google Group members? - google-groups

There are around 15000+ members in my Google Group and I'd like to export them to CSV file. Google Groups says there are too many to export Is there any other way to export all the 15000+ members?

This is possible using the Directory API. You can see how to set up a project in the Google Developer's Console with the appropriate scopes here. I have a sample of code below that shows how I did this for my email list manager (written in Python3). Note that mine just outputs to a list, but you could write to a csv as well.
import urllib.request, urllib.parse, urllib.error
import json
req = urllib.request.Request("https://www.googleapis.com/admin/directory/v1/groups/%s/members" % (group.replace("#", "%40").replace(".", "%2E")))
req.add_header("Authorization", "Bearer %s" % (authToken))
rawResponse = urllib.request.urlopen(req)
fResponse = json.loads(rawResponse.readall().decode('utf-8'))
if "members" in fResponse:
curMembers = fResponse["members"]
while "nextPageToken" in fResponse:
req = urllib.request.Request("https://www.googleapis.com/admin/directory/v1/groups/%s/members?pageToken=%s" % (group.replace("#", "%40").replace(".", "%2E"), fResponse['nextPageToken']))
req.add_header("Authorization", "Bearer %s" % (authToken))
rawResponse = urllib.request.urlopen(req)
fResponse = json.loads(rawResponse.readall().decode('utf-8'))
if "members" in fResponse:
curMembers.extend(fResponse["members"])
You'll need to define a way to retrieve the oauth2 authentication token (authToken in the above snippet). I do so as below, but there are dedicated libraries for this (just not for Python3 that I know of):
def RenewToken():
"""
This function retrieves an authentication token from Google which will allow the script
to view and edit group membership
"""
clientID = "client_id.apps.googleusercontent.com"
clientSecret = "client_secret"
grantType = "refresh_token"
responseType = "code"
refreshToken = "refresh_token"
requestDict = {"client_secret": clientSecret, #field-value pairs for the request body
"grant_type": grantType,
"refresh_token": refreshToken,
"client_id": clientID}
requestUri = "https://accounts.google.com/o/oauth2/token"
requestString = urllib.parse.urlencode(requestDict)
requestBody = requestString.encode('utf-8')
request = urllib.request.Request(requestUri, data=requestBody)
response = urllib.request.urlopen(request)
responseDict = json.loads(response.readall().decode('utf-8'))
return responseDict['access_token']
You can get client_id, client_secret and refresh_token from the Google developer's console for your project.

you can export all contacts in your google group in a Google Spreadsheet
link : https://developers.google.com/apps-script/reference/groups/
After you can create a trigger to add new contact dynamically, and export you Google spreadsheet on a cvs's file.
you can create this solution quickly and simple.

Related

Pass a token via the addition of a URL query parameter with "token=tokenvalue." in Azure Media Service not working

Trying to stream a video from Azure Media Service into a Xamarin.Form application.
The asset is protected with a JWT token.
I use the following code to generate the token:
private string GetJWT(string PrimaryKey) {
var tokenSigningKey = new SymmetricSecurityKey(Convert.FromBase64String(PrimaryKey));
SigningCredentials cred = new SigningCredentials(tokenSigningKey, SecurityAlgorithms.HmacSha256);
JwtSecurityToken token = new JwtSecurityToken(
issuer: "xxx",
audience: "yyy",
claims: null,
notBefore: DateTime.Now.AddMinutes(-5),
expires: DateTime.Now.AddMinutes(60),
signingCredentials: cred);
JwtSecurityTokenHandler handler = new JwtSecurityTokenHandler();
return handler.WriteToken(token);
}
The URI for the asset is like : https://xxx.streaming.media.azure.net/12222-1565-232323-a5b8-c10e148273ae/Test.ism/manifest(format=m3u8-cmaf,encryption=cbc)
If I use Azure Media Player (https://ampdemo.azureedge.net) to test the URI and the AES token, it works fine. So I guess there is no problem with the token itself...
The documentation (https://learn.microsoft.com/en-us/azure/media-services/latest/security-pass-authentication-tokens-how-to#pass-a-token-via-the-addition-of-a-url-query-parameter-with-tokentokenvalue) says that the following code should work to send the token directly with the url.
I need to do this as with Xamarin.Forms and MediaElement, I can't send the token in a header request. So I need the querystring option...
string armoredAuthToken = System.Web.HttpUtility.UrlEncode(authToken);
string uriWithTokenParameter = string.Format("{0}&token={1}", keyDeliveryServiceUri.AbsoluteUri, armoredAuthToken);
Uri keyDeliveryUrlWithTokenParameter = new Uri(uriWithTokenParameter);
player.Source = keyDeliveryUrlWithTokenParameter; (player is a MediaElement control)
But the video is never loaded.
In my opinion there is an error, it should be {0}?token={1} instead of {0}&token={1}.
But that doesn't work neither.
If I test with VLC the https://xxx.streaming.media.azure.net/12222-1565-232323-a5b8-c10e148273ae/Test.ism/manifest(format=m3u8-cmaf,encryption=cbc)?token=zzzzz, it doesn't work neither.
I presume there is a problem with token in the querystring, as if Azure can't read it.

Textlocal URL shortner API does not work with signed url

I am using the textlocal short url API with the signed URL that I have generated for a Cloud Storage blob. I am getting MalformedSecurityHeader error in the browser when I use the short URL. I tried the same API with other random links and it worked fine. Can I get some help on this.
def shorten_url(apikey, url):
data = urllib.parse.urlencode({'apikey': apikey, 'url': url})
data = data.encode('utf-8')
request = urllib.request.Request("https://api.textlocal.in/create_shorturl/")
f = urllib.request.urlopen(request, data,
context=ssl.create_default_context(cafile=certifi.where()))
fr = f.read()
return(fr)
I was able to resolve this. data = urllib.parse.urlencode({'apikey': apikey, 'url': urllib.parse.quote(url)})

How to call SSRS Rest-Api V1.0 with custom security implemented (NOT SOAP)

I have implemented the custom security on my reporting services 2016 and it displays the login page once the URL for reporting services is typed on browser URL bar (either reports or reportserver)
I am using the following code to pass the Credentials
when i use the code WITHOUT my security extension it works and looks like this
ICredentials _executionCredentials;
CredentialCache myCache = new CredentialCache();
Uri reportServerUri = new Uri(ReportServerUrl);
myCache.Add(new Uri(reportServerUri.GetLeftPart(UriPartial.Authority)),
"NTLM", new NetworkCredential(MyUserName, MyUserPassword));
_executionCredentials = myCache;
when i use the code WITH the security extension it doesnt work and looks like this
ICredentials _executionCredentials;
CredentialCache myCache = new CredentialCache();
Uri reportServerUri = new Uri(ReportServerUrl);
myCache.Add(new Uri(reportServerUri.GetLeftPart(UriPartial.Authority)),
"Basic", new NetworkCredential(MyUserName, MyUserPassword));
_executionCredentials = myCache;
and i get an Exception saying "The response to this POST request did not contain a 'location' header. That is not supported by this client." when i actually use this credentials
Is "basic" the wrong option ?
Have anyone done this ?
Update 1
Well it turns out that my SSRS is expecting an Authorisation cookie
which i am unable to pass (according to fiddler, there is no cookie)
HttpWebRequest request;
request = (HttpWebRequest)HttpWebRequest.Create("http://mylocalcomputerwithRS/Reports_SQL2016/api/v1.0");
CookieContainer cookieJar = new CookieContainer();
request.CookieContainer = cookieJar;
Cookie authCookie = new Cookie("sqlAuthCookie", "username:password");
authCookie.Domain = ".mydomain.mylocalcomputerwithRS";
if (authCookie != null)
request.CookieContainer.Add(authCookie);
request.Timeout = -1;
HttpWebResponse myHttpWebResponse = (HttpWebResponse)request.GetResponse();
That's how I got it (SSRS 2017; api v2.0). I took the value for the "body" from Fiddler:
var handler = new HttpClientHandler();
var httpClient = new HttpClient(handler);
Assert.AreEqual(0, handler.CookieContainer.Count);
// Create a login form
var body = new Dictionary<string, string>()
{
{"__VIEWSTATE", "9cZYKBmLKR3EbLhJvaf1JI7LZ4cc0244Hpcpzt/2MsDy+ccwNaw9hswvzwepb4InPxvrgR0FJ/TpZWbLZGNEIuD/dmmqy0qXNm5/6VMn9eV+SBbdAhSupsEhmbuTTrg7sjtRig==" },
{"__VIEWSTATEGENERATOR", "480DEEB3"},
{ "__EVENTVALIDATION", "IS0IRlkvSTMCa7SfuB/lrh9f5TpFSB2wpqBZGzpoT/aKGsI5zSjooNO9QvxIh+QIvcbPFDOqTD7R0VDOH8CWkX4T4Fs29e6IL92qPik3euu5QpidxJB14t/WSqBywIMEWXy6lfVTsTWAkkMJRX8DX7OwIhSWZAEbWZUyJRSpXZK5k74jl4x85OZJ19hyfE9qwatskQ=="},
{"txtUserName", "User"},
{"txtPassword", "1"},
{"btnLogin","Войти"}
};
var content = new FormUrlEncodedContent(body);
// POST to login form
var response = await httpClient.PostAsync("http://127.0.0.1:777/ReportServer/Logon.aspx", content);
// Check the cookies created by server
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
var cookies = handler.CookieContainer.GetCookies(new Uri("http://127.0.0.1:777/ReportServer"));
Assert.AreEqual("sqlAuthCookie", cookies[0].Name);
// Make new request to secured resource
var myresponse = await httpClient.GetAsync("http://127.0.0.1:777/Reports/api/v2.0/Folders");
var stringContent = await myresponse.Content.ReadAsStringAsync();
Console.Write(stringContent);
As an alternative you can customize SSRS Custom Security Sample quite a bit.
I forked Microsoft's Custom Security Sample to do just what you are describing (needed the functionality at a client long ago and reimplemented as a shareable project on GitHub).
https://github.com/sonrai-LLC/ExtRSAuth
I created a YouTube walkthrough as well to show how one can extend and debug SSRS security with this ExtRSAuth SSRS security assembly https://www.youtube.com/watch?v=tnsWChwW7lA
TL; DR; just bypass the Microsoft example auth check in Login.aspx.cs and put your auth in Page_Load() or Page_Init() event of Login.aspx.cs- wherever you want to perform some custom logging check- and then immediately redirect auth'd user to their requested URI.

How do I get images from a particular project using the OpenAsset REST API?

I asked a while ago about how to get the actual image file from the File noun using the OA REST API.
I'm confused as to how to retrieve images related to a specific project.
Thanks to a helpful nudge from Jason I've discovered that the secret password is filtering using the params.
This still feels a bit unpleasant as it requires 4 trips to the OA server, so I'm probably still missing something that would allow me to concatenate these requests into one.
I wrote a short Ruby programme that will go and get a single image to illustrate the principle:
baseURL = "http://IP.A.DDR.ESS"
password = "password"
username = "login"
require 'net/http'
require "uri"
require 'json'
def ask_oa(request, baseURL, params, username, password, useParams)
#This constructs a REST request, makes it, and returns the JSON response.
uri = URI.parse(baseURL + request)
if useParams
uri.query = URI.encode_www_form(params)
end
puts uri
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
request.basic_auth(username, password)
response = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(request)
}
json_response = JSON.parse(response.body)
puts JSON.pretty_generate(json_response)
return json_response
end
puts "Step 1"
#You'll need to change this for *your* project name
project_name = "M020707"
puts "go and get a project where the project number is "+project_name+":"
request = "/REST/1/Projects"
params = { :textMatching => "exact",
:code => project_name
}
projectJSON = ask_oa(request, baseURL, params, username, password, true)
projectNumber = projectJSON[0]["id"]
#This ID is the ID of the Project. You use this as the Project_id to
#key into the Files list that we access next
puts "Step 2"
puts "go and get the files that have a project_id of "+projectNumber+":"
request = "/REST/1/Files"
params = { :textMatching => "exact",
:project_id => projectNumber
}
fileJSON = ask_oa(request, baseURL, params, username, password, true)
#You'd want a more exciting way of picking a file out of that list
#probably based on the ranking of the files or some other secret knowledge
the_file_I_want = 0
fileID = fileJSON[the_file_I_want]["id"]
#This is a single file, picked from a list of all the files associated
#with the project that we pulled in step 1. Now that we have a file id
#we're back to where we were when I asked my last question on SO.
#To OA, File means the source file, we now need to pick a
#processed/formatted version of it:
puts "Step 3"
puts "go and get the size options for the file with the ID "+fileID+":"
request = "/REST/1/Files/"+fileID+"/Sizes"
params = {}
sizeJSON = ask_oa(request, baseURL, params, username, password, false)
#implement a picker in here to get the format that you want. 0 is just
#the first, and I don't know if the order that they are provided in is
#deterministic
s = sizeJSON[0]
#This gets to the list of actual files that you can now construct a URL from
puts "Step 4"
puts "Request an image with this URL:"
puts baseURL + s["http_root"] + s["relative_path"]
#Now we can just request the file at the end of
#this url and we'll see a picture!

Yammer API - Posting to External Networks

I've used the Yammer API extensively for accessing current users internal network. All API calls have been working correctly (GET's and POST's) with the original token extracted from;
"https://www.yammer.com/oauth2/access_token.json?client_id={App ID}&client_secret={App Secret}&code={Access Code}"
and using the headers; "Authorization : Bearer {Token}" and "Cookie : {Cookies Received from HTML request}.
I've gotten the tokens for all accessible networks using;
"https://www.yammer.com/api/v1/oauth/tokens.json".
Accessing external networks beyond this point has proved troublesome. I changed the header to "Authorization : Bearer {NetworkToken}". While I am able to GET details from external networks, I cannot POST to external networks. I always receive a '401 Unauthorized' response. The 'Unauthorized' requests include deleting messages and liking messages in external networks.
Is there another step between being able to read data from an external network and enabling POST methods?
If I could get any insight into this i'd be extremely grateful!
Cheers!
When accessing external networks, you need to set the authToken to the authToken for that external network.
Step 1 - Get all auth tokens:
yam.platform.request({
url: "oauth/tokens.json",
type: 'GET',
success: function (msg) {
accessTokens = msg;
/....
},
error: function (msg) {
console.log(msg);
error(msg);
}
Step 2: Set the authToken to the correct external network
var currentToken = "";
$.each(accessTokens, function (i,val) {
if (val.network_permalink == $.cookie('networkPermalink')) {
currentToken = val;
}
});
While I was working on a project last month, I used the following way to post message.
The message has to be Byte encrypted in UTF-8 format.
Specify the content type as "application/x-www-form-urlencoded".
So, an example code would be:
HttpWebRequest a = (HttpWebRequest)WebRequest.Create(postUrl);
a.Headers.Add("Authorization", "Bearer" + authToken);
a.Method = "POST";
byte[] message = Encoding.UTF8.GetBytes("body=" + message + "&replied_to_id=" + threadID);
a.ContentType = "application/x-www-form-urlencoded";
a.ContentLength = message.Length;
using (var postStream = request.GetRequestStream())
{
postStream.Write(message, 0, message.Length);
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (var postStreamForResponse = response.GetResponseStream())
{
StreamReader postReader = new StreamReader(postStreamForResponse);
string results = postReader.ReadToEnd();
postReader.Close();
}
I've discovered quite a few inconsistencies quirks with the Yammer API. I've figured out external networks in their totality now. Here are some things that may not be clear;
When doing a POST or DELETE request, do not include the network_permalink in the url! Only include the network_permalink when you're doing a GET request. This was my main issue.
Required request headers;
Content-Type : application/x-www-form-urlencoded
Accept : application/json
Cookie : _workfeed_session_id=(A code that can be extracted from the response from your first request with an auth token)
Authorization : Bearer (Access token for whichever network you wish to access)
Oh and just FYI, to request threads within the 'All Company' group this is the url; https://www.yammer.com/(network_permalink)/api/v1/messages/general.json
Thanks for the answers!