Issues with scheduling... Using tasksheduleR to run R script using rtweet and exporting to postgreSQL table - postgresql

I am trying to schedule an R script to run, using rtweet to hit the Twitter API and then using RPOSTgreSQL to load the data into a table once a day.
I was able to successfully use taskscheduleR to create the task. However, when it runs I get an error that says...
<credentials> oauth_token, oauth_token_secret
---
Requesting token on behalf of user...
Error: API user token required. see http://rtweet.info/articles/auth.html for instructions
Execution halted
Here is my entire code with a few things masked due to API credentials and database info, passwords, etc
#load all packages
library(rtweet)
library(sqldf)
library(dplyr)
library("RPostgreSQL")
#connect to Twitter API
create_token(
app = "masked for stack",
consumer_key = "masked for stack",
consumer_secret = "masked for stack",
access_token = "masked for stack",
access_secret = "masked for stack")
## get user IDs of accounts following
followers=get_followers("masked for stack", n = 1000)
## lookup data on those accounts
followers_data=lookup_users(followers$user_id)
#add the date of the run
followers_data$date=Sys.Date()
#extract columns
twitter_followers=dplyr::select(followers_data,"date","screen_name","name","location","description","followers_count",
"friends_count","listed_count","statuses_count","favourites_count","verified")
# create a connection
# save the password that we can "hide" it as best as we can by collapsing it
pw <- {
"masked for stack"
}
# loads the PostgreSQL driver
drv <- dbDriver("PostgreSQL")
# creates a connection to the postgres database
# note that "con" will be used later in each connection to the database
con <- dbConnect(drv, dbname = "masked for stack",
host = "localhost", port = 5432,
user = "postgres", password = pw)
rm(pw) # removes the password
# writes df to the PostgreSQL database
dbWriteTable(con, "twitter_followers",
value = twitter_followers, append = TRUE, row.names = FALSE)
My code runs perfectly fine doing everything on my own manually. It seems that rtweet doesn't like getting ran by windows scheduler.
Any ideas? `

So after hours of digging I figured out that something was not working well with the scheduleR app. It may have been creating a task in windows scheduler that wouldn't run when my laptop was on AC power I'm not sure. Either way I did a workaround like this.
Created a text file and saved as a .bat extension with the following format:
"C:\Program Files\R\R-3.5.2\bin\x64\R.exe" CMD BATCH file_location.R
From there I scheduled in Windows scheduler. Make sure you go through all the options and don't choose a basic task. If you do a basic one you may not allow it to run on laptop battery power.

Related

Automate some Exchange Online functions

Having that Exchange Online functions are not yet included in MS Graph, and Exchange does not have its own SDK also, the only option I found to automate some functions in Exchange Online is ExchangeOnlineManagement PowerShell.
I used it for a while but Microsoft recently ceased the Basic Authentication. I'm using the Basic Authentication because it does not require any prior package installation.
The modern authentication use Connect-ExchangeOnline and requires the ExchangeOnlineManagement to be installed prior to any call. It's easy to install it on a physical server or a VM, but I'm relying on Azure services, like azure Web app, azure Function app, where I cannot install a package. now the situation is:
Basic authentication stopped working, and it's not an option
anymore. This was the only option to execute exchange PowerShell
commands from azure web app or function app.
In modern authentication, you rely on the ExchangeOnlineManagement
that must be installed prior to any call. since I work on web app, or function
app, I cannot install it.
there is no .net Exchange SDK.
MS Graph lack Exchange functions.
So what is the valid solution to execute Exchange command? Maybe the solution is available but I'm not aware of it.
My case is simple: Every night, Upon the expiration of an Azure user account:
I change some user properties on Azure (Available through MS Graph)
I need to set an auto reply message, so sender can know the email
account is no longer monitored. (NOT AVAILABLE)
You can use BasicAuthToOAuthConversion=true in the connection string and if for example you currently have user creds being passed in switch to using the ROPC flow with those creds and just pass the username and accesstoken where you passing the users password eg
PSCredential pSCredential = new PSCredential("user#blah.onmicrosoft.com", new NetworkCredential("", "pass##").SecurePassword);
string MailboxName = pSCredential.UserName;
string scope = "https://outlook.office365.com/.default";
string ClientId = "a0c73c16-a7e3-4564-9a95-2bdf47383716";
HttpClient Client = new HttpClient();
var TenantId = ((dynamic)JsonConvert.DeserializeObject(Client.GetAsync("https://login.microsoftonline.com/" + MailboxName.Split('#')[1] + "/v2.0/.well-known/openid-configuration").Result.Content.ReadAsStringAsync().Result)).authorization_endpoint.ToString().Split('/')[3];
PublicClientApplicationBuilder pcaConfig = PublicClientApplicationBuilder.Create(ClientId);
pcaConfig.WithTenantId(TenantId);
var TokenResult = pcaConfig.Build().AcquireTokenByUsernamePassword(new[] { scope }, pSCredential.UserName, pSCredential.Password).ExecuteAsync().Result;
System.Security.SecureString secureString = new System.Security.SecureString();
foreach (char c in ("bearer " + TokenResult.AccessToken))
secureString.AppendChar(c);
String WSManURIConnectionString = "https://outlook.office365.com/powershell-liveid?DelegatedOrg=" + MailboxName.Split('#')[1] + "&BasicAuthToOAuthConversion=true";
PSCredential credential = new PSCredential(MailboxName, secureString);
WSManConnectionInfo connectionInfo = new WSManConnectionInfo(new Uri(WSManURIConnectionString), "http://schemas.microsoft.com/powershell/Microsoft.Exchange", credential);
connectionInfo.AuthenticationMechanism = AuthenticationMechanism.Basic;
connectionInfo.SkipCACheck = true;
connectionInfo.SkipCNCheck = true;
connectionInfo.MaximumConnectionRedirectionCount = 4;
Runspace runspace = System.Management.Automation.Runspaces.RunspaceFactory.CreateRunspace(connectionInfo);
runspace.Open();
// Make a Get-Mailbox requst using the Server Argument
Command gmGetMailbox = new Command("get-mailbox");
gmGetMailbox.Parameters.Add("ResultSize", "Unlimited");
Pipeline plPileLine = runspace.CreatePipeline();
plPileLine.Commands.Add(gmGetMailbox);
Collection<PSObject> RsResultsresults = plPileLine.Invoke();
Dictionary<string, PSObject> gmResults = new Dictionary<string, PSObject>();
foreach (PSObject obj in RsResultsresults)
{
Console.WriteLine(obj.Members["WindowsEmailAddress"].Value.ToString());
}
Command gmGetUser = new Command("get-user");
plPileLine.Stop();
plPileLine.Dispose();

why wildfly server is asking me to add users even after adding user?

mngmt-users.properties file. The users are added in the file but when I try to run the localhost it says it's running then if I try to view the admin console it is redirecting to http://localhost:9990/error/index_win.html. That tells the server is running but I could not open admin console.
#
# Properties declaration of users for the realm 'ManagementRealm' which is the default realm
# for new installations. Further authentication mechanism can be configured
# as part of the <management /> in standalone.xml.
#
# Users can be added to this properties file at any time, updates after the server has started
# will be automatically detected.
#
# By default the properties realm expects the entries to be in the format: -
# username=HEX( MD5( username ':' realm ':' password))
#
# A utility script is provided which can be executed from the bin folder to add the users: -
# - Linux
# bin/add-user.sh
#
# - Windows
# bin\add-user.bat
#
#$REALM_NAME=ManagementRealm$ This line is used by the add-user utility to identify the realm name already used in this file.
#
# On start-up the server will also automatically add a user $local - this user is specifically
# for local tools running against this AS installation.
#
# The following illustrates how an admin user could be defined, this
# is for illustration only and does not correspond to a usable password.
#
#admin=2a0923285184943425d1f53ddd58ec7a
tejaswini=25ab658c2861b2e64783aaa9ba95c2e5
aswini#19=388ced81791ddb1760b83dc4ec8b7a61
saisana=ff39d778414ab12d84fc4fa7fdacb634
alekya=d72e9c90345ce4d9290c3a2728b3cd60
prasad=c6c7c67cf343f6862d3b77bae9f61d17
teju=28b9e55b314fd60855a7843b4455dbed
Screen shot of added user
May be u have tried to create specifically application user or management user when u ran the addUser utility,
please refer the below link for steps to register user
https://bgasparotto.com/add-user-wildfly

Modify datasource IP addresses in WebSphere Application Server

I have nearly a hundred data sources in a WebSphere Application Server (WAS) and due to office relocation, the IP of the database servers have changed and I need to update the datasource IP addresses in my WAS too.
Considering it error-prone to update hundred IPs through admin console.
Is there any way that I can make the change by updating config files or running a script? My version of WAS is 7.0.
You should be able to use the WAS Admin Console's built-in "command assistance" to capture simple code snippets for listing datasources and changing them by just completing those operations in the UI once.
Take those snippets and create a new jython script to list and update all of them.
More info on command assistance:
http://www.ibm.com/developerworks/websphere/library/techarticles/0812_rhodes/0812_rhodes.html
wsadmin scripting library:
https://github.com/wsadminlib/wsadminlib
You can achieve this using wsadmin scripting. Covener has the right idea with using the admin console's command assistance to do the update once manually (to get the code) and then dump that into a script that you can automate.
The basic idea is that a datasource has a set of properties nested under it, one of which is the ip address. So you want to write a script that will query for the datasource, find its nested property set, and iterate over the property set looking for the 'ipAddress' property to update.
Here is a function that will update the value of the "ipAddress" property.
import sys
def updateDataSourceIP(dsName, newIP):
ds = AdminConfig.getid('/Server:myServer/JDBCProvider:myProvider/DataSource:' + dsName + '/')
propertySet = AdminConfig.showAttribute(ds, 'propertySet')
propertyList = AdminConfig.list('J2EEResourceProperty', propertySet).splitlines()
for prop in propertyList:
print AdminConfig.showAttribute(prop, 'name')
if (AdminConfig.showAttribute(prop, 'name') == 'ipAddress'):
AdminConfig.modify(prop, '[[value '" + newIP + "']]')
AdminConfig.save();
# Call the function using command line args
updateDataSourceIP(sys.argv[0], sys.argv[1])
To run this script you would invoke the following from the command line:
$WAS_HOME/bin/wsadmin.sh -lang jython -f /path/to/script.py myDataSource 127.0.0.1
**Disclaimer: untested script. I don't know the name of the "ipAddress" property off the top of my head, but if you run this once it will print out all the properties on your ds, so you can get it there
Some useful links:
Basics about jython scripting
Modifying config objects using wsadmin
As an improvement to aguibert's script, to avoid having to provide all 100 datasource names and to update it to correct the containment path of the configuration id, consider this script which will update all datasources, regardless of the scope at which they're defined. As always, backup your configuration before beginning and once you're satisified the script is working as expected, replace the AdminConfig.reset() with save(). Note, these scripts will likely not work properly if you're using connection URLs in your configuration.
import sys
def updateDataSourceIP(newIP):
datasources = AdminConfig.getid('/DataSource:/').splitlines()
for datasource in datasources:
propertySet = AdminConfig.showAttribute(datasource, 'propertySet')
propertyList = AdminConfig.list('J2EEResourceProperty', propertySet).splitlines()
for prop in propertyList:
if (AdminConfig.showAttribute(prop, 'name') == 'serverName'):
oldip = AdminConfig.showAttribute(prop, 'value')
print "Updating serverName attribute of datasource '" + datasource + "' from " + oldip + " to " + sys.argv[0]
AdminConfig.modify(prop, '[[value ' + newIP + ']]')
AdminConfig.reset();
# Call the function using command line arg
updateDataSourceIP(sys.argv[0])
The script should be invoked similarly to the above, but without the datasource parameter, the only parameter is the new hostname or ip address:
$WAS_HOME/bin/wsadmin.sh -lang jython -f /path/to/script.py 127.0.0.1

Using JDBCRealm to authenticate user with Shiro

I am trying to authenticate a servlet running within Tomcat 6 using Shiro.
I have the following shiro.ini file:
[main]
ps = org.apache.shiro.authc.credential.DefaultPasswordService
pm = org.apache.shiro.authc.credential.PasswordMatcher
pm.passwordService = $ps
aa = org.apache.shiro.authc.credential.AllowAllCredentialsMatcher
sm = org.apache.shiro.authc.credential.SimpleCredentialsMatcher
jof = org.apache.shiro.jndi.JndiObjectFactory
jof.resourceName = jdbc/UserDB
jof.requiredType = javax.sql.DataSource
jof.resourceRef = true
realm = org.apache.shiro.realm.jdbc.JdbcRealm
realm.permissionsLookupEnabled = true
realm.credentialsMatcher = $pm
; Note factories are automatically invoked via getInstance(),
; see org.apache.shiro.authc.config.ReflectionBuilder::resolveReference
realm.dataSource = $jof
securityManager.realms = $realm
[urls]
/rest/** = authcBasic
/prot/** = authcBasic
And the following in my database:
mysql> select * from users;
+----------+------------------+----------+----------------------------------------------+--------------------------+
| username | email | verified | password | password_salt |
+----------+------------------+----------+----------------------------------------------+--------------------------+
| admin | a.muys#********* | 1 | ojSiTecNwRF0MunGRvz3DRSgP7sMF9EAR77Ol/2IAY8= | eHp9XedrIUa5sECfOb+KOA== |
+----------+------------------+----------+----------------------------------------------+--------------------------+
1 row in set (0.00 sec)
If I use the SimpleCredentialsManager it authenticates fine against a plaintext password in the users table. Trying to use the PasswordMatcher has been extremely frustrating.
The password and password_salt were obtained via the shiro-tools Hasher utility.
When I try to authenticate against a basic HelloWorld servlet I use for testing (path=rest/hello, context=/ws), I get the following in the logs:
15:35:38.667 [http-8080-2] TRACE org.apache.shiro.util.ClassUtils - Unable to load clazz named [ojSiTecNwRF0MunGRvz3DRSgP7sMF9EAR77Ol/2IAY8=] from class loader [WebappClassLoader
context: /ws
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader#79ddd026
]
(Full log at https://gist.github.com/recurse/5915693 )
It appears to be trying to load my hashed password as a classname. Is this a bug, or a configuration error on my part? If it is a bug, how can I work around it? If it is a configuration error, what am I missing?
First, thanks for providing a lot of information for this question - it makes providing an answer a lot easier.
By looking at your sample database row list, it does not appear that you are storing the output that the PasswordService expects when performing a hashed password comparison. For example:
$ java -jar ~/.m2/repository/org/apache/shiro/tools/shiro-tools-hasher/1.2.2/shiro-tools-hasher-1.2.2-cli.jar -p
Password to hash:
Password to hash (confirm):
$shiro1$SHA-256$500000$uxaA2ngfdxdXpvSWzpuFdg==$hOJZc+3+bFYYRgVn5wkbQL+m/FseeqDtoM5mOiwAR3E=
The String that starts with $shiro1$ is what you would save to the password column in the database. There is no need for a separate salt column as all the information Shiro needs is in the $shiro1$... String.
The DefaultPasswordService uses the same default configuration parameters (SHA-256, 500,000 iterations, etc) so if you use the Hasher CLI tool as I've shown above (no extra hash algorithm config), you don't need to customize the DefaultPasswordService POJO any further. However, if you change the hashing parameters on the CLI, you need to ensure that the same parameters are configured on the DefaultPasswordService bean (and/or its internal HashingService).
If you are still in testing and can change your DB schema, I'd recommend doing that now to have a single password field that stores the $shiro1$... string. Then you use the PasswordService as documented here under Usage:
http://shiro.apache.org/static/current/apidocs/org/apache/shiro/authc/credential/PasswordService.html

Powershell remoting and page file

I wrote a powershell script that connects to a remote machine with the intent of executing a software rollout on said machine. Basically it connects, maps a drive, copies the rollout from the mapped drive to the target machine, then executes a perl script to install the rollout. If I do those steps manually everything works fine. When I try using my script, the perl script fails on the remote machine saying, "The paging file is too small for this operation to complete".
Can someone explain the considerations I need to take into account when operating remotely? I've tried monitoring memory usage and I don't see anything out of the ordinary. Is the page file OS wide or is there some type of per user configuration my script should be setting when it connects?
I can post snippets of my script if needed, but the script is 426 lines so I think it would be overwhelming to post in its entirety.
I found that the remote shells are managed differently than logging onto the box and executing a powershell session. I had to increase the maximum amount of memory available using one of the commands below:
Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 1024
winrm set winrm/config #{MaxMemoryPerShellMB="1024"}
The default is 150MB which didn't cut it in my case. I can't say that I recommend 1GB, I'm just a developer. I tried upping it until I found what worked for me.
I tried this code to run the puppet client as an administrator but the framework still complains with "Access Denied"
Exe (C:\Users\lmo0\AppData\Local\Temp\Microsoft .NET Framework 4 Setup_4.0.30319\Windows6.1-KB958488-v6001-x64.msu) failed with 0x5 - Access is denied. .
using System;
using System.Diagnostics;
namespace RunAsAdmin
{
class Program
{
static void Main(string[] args)
{
Process proc = new Process();
Process p = new Process();
p.StartInfo.FileName = #"powershell.exe";
p.StartInfo.Arguments = #"invoke-command -computername vavt-pmo-sbx24 -ScriptBlock {&'C:\Program Files (x86)\Puppet Labs\Puppet\bin\puppet.bat' agent --test --no-daemonize --verbose --logdest console}";
p.StartInfo.Verb = "runas";
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
p.Start();
while (p.HasExited == false) {
Console.WriteLine(p.StandardOutput.ReadLine());
}
Console.ReadLine();
p.WaitForExit();
p.Close();
}
}
}