I want to call a Nifi custom processor as a REST Api and pass the parameters at run-time through pyspark. And retrieve the results in the response object.
Can anyone please help me in suggesting different approaches for the same.
use the following sequence of processors:
HandleHttpRequest
extract patameters
your other processors...
prepare response
HandleHttpResponse
The steps are:
Configure HandleHttpRequest processor.
Enable the required HTTP methods (GET, POST, DELETE, etc.).
Set the listening port.
Attached the Context Map to a service (the listener).
5. Enable the service and the processor.
Bonus:
If you run Nifi from a Docker container, as I do, you should get the container's IP:
docker inspect <container-name> --format='{{.NetworkSettings.IPAddress}}'
Now, you can run Postman, and the HandleHttpRequest processor will fetch it. For example:
I created a simple template to exemplify this scenario. The HTTP request's body is saved into a directory:
Related
I have created a sample play framework api which has one endpoint.
http://play-demo-broker.cfapps.io/say?number=20
Which just return me number that have passed.
I am able successfully deploy the service. Next want this service to Act like service broker
For same want to register this as by using below command
cf create-service-broker play-demo-broker admin admin http://play-demo-broker.cfapps.io --space-scoped
This command it giving me below error -
The service broker rejected the request. Status Code: 404 Not Found
Not sure what is causing this issue as there not much information available for Play Framework Service broker Setup.
The play framework is implemented above the akka packages. Akka rejects paths that are not implemented.
If I an not mistaken, cf create-service-broker command access the / endpoint. If you implemented only say?number=20 endpoint, then be default all other paths, such as the empty path, are rejected by Akka.
In order to open that endpoint you need to add it into the routes.
For example you can add:
GET / controllers.ControllerName.GetEmptyPath
And implement the GetEmptyPath method in ControllerName
I have deployed NiFi with Kerberos in a cluster and for accessing the UI I am using haproxy. I am able to access NiFi UI through the individual node URL but it's not working with the loadbalncer URL and getting following error
System Error
The request contained an invalid host header
I think it can be fixed by nifi.web.proxy.host and nifi.web.proxy.context.path parameters. I tried with this two parameters but the problem still remains.
This issue was pointed at in NiFi 1.5 NIFI-4761.
To resolve this issue, whitelist the hostname used to access NiFi using the following parameter in the nifi.properties configuration file :
nifi.web.proxy.host = <host:port>
Its a comma-separated list of allowed HTTP Host header values to consider when NiFi is running securely and will be receiving requests to a different host[:port]. For example, when running in a Docker container or behind a proxy (e.g. localhost:18443, proxyhost:443). By default, this value is blank, meaning NiFi should allow only requests sent to the host[:port] that NiFi is bound to.
I have an app which is setup by docker-compose. The app contains docker containers for kafka, postgres, rest api endpoints.
One test case is to post data to endpoints. In the data, there is a field called callback URL. the app will parse the data and send the data to the callback URL.
I am curious whether there is any test framework for similar test cases. and how to verify the callback URL is hit with data?
Docker compose support has been added to endly. In the pipeline workflow for the app (app.yaml), you can add a "deploy" task and start the docker services by invoking docker-compose up.
Once test task is completed and your callback url is invoked, in your validation task, you can check to see if it was invoked with the expected data. For this you can utilize endly's recording feature and replay it to validate the callback request.
Below is an example of an ETL application app.yaml using docker-compose with endly to start the docker services. Hope it helps.
tasks: $tasks
defaults:
app: $app
version: $version
sdk: $sdk
useRegistry: false
pipeline:
build:
name: Build GBQ ETL
description: Using a endly shared workflow to build
workflow: app/docker/build
origin:
URL: ./../../
credentials: localhost
buildPath: /tmp/go/src/etl/app
secrets:
github: git
commands:
- apt-get -y install git
- export GOPATH=/tmp/go
- export GIT_TERMINAL_PROMPT=1
- cd $buildPath
- go get -u .
- $output:/Username/? ${github.username}
- $output:/Password/? ${github.password}
- export CGO_ENABLED=0
- go build -o $app
- chmod +x $app
download:
/$buildPath/${app}: $releasePath
/$buildPath/startup.sh: $releasePath
/$buildPath/docker-entrypoint.sh: $releasePath
/$buildPath/VERSION: $releasePath
/$buildPath/docker-compose.yaml: $releasePath
deploy:
start:
action: docker:composeUp
target: $target
source:
URL: ${releasePath}docker-compose.yaml
In your question below, where is Kafka involved? Both sound like HTTP calls.
1)Post data to endpoint
2)Against send data to the callback URL
One test case is to post data to endpoints. In the data, there is a field called callback URL. the app will parse the data and send the data to the callback URL.
Assuming the callback URL is an HTTP endpoint(e.g. REST or SOAP) with POST/PUT api, then it's better to expose a GET endpoint on the same resource. In that case, when callback POST/PUT is invoked, the server side state/data changes and next, use the GET api to verify the data is correct. The output of the GET API is the Kafka data which was sent to the callback URL(this assumes your 1st post message was to a kafka topic).
You can achieve this using traditional JUnit way using bit of code or via declarative way where you can completely bypass coding.
The example has dockerized Kafka containers to bring up locally and run the tests
This section Kafka with REST APIs explains automated way of testing combination of REST api testing with Kafka data streams .
e.g.
---
scenarioName: Kafka and REST api validation example
steps:
- name: produce_to_kafka
url: kafka-topic:people-address
operation: PRODUCE
request:
recordType: JSON
records:
- key: id-lon-123
value:
id: id-lon-123
postCode: UK-BA9
verify:
status: Ok
recordMetadata: "$NOT.NULL"
- name: verify_updated_address
url: "/api/v1/addresses/${$.produce_to_kafka.request.records[0].value.id}"
operation: GET
request:
headers:
X-GOVT-API-KEY: top-key-only-known-to-secu-cleared
verify:
status: 200
value:
id: "${$.produce_to_kafka.request.records[0].value.id}"
postCode: "${$.produce_to_kafka.request.records[0].value.postcode}"
Idaithalam is a low code Test automation Framework, developed using Java and Cucumber. It leverages Behavior Driven Development (BDD). Tester can create test cases/scripts in simple Excel with API Spec. Excel is a simplified way to create Json based test scripts in Idaithalam. Test cases can be created quickly and tested in minutes.
As a tester, you need to create Excel and pass it to Idaithalam Framework.
First, generate the Json based test scripts(Virtualan Collection) from Excel. During test execution, this test script collection can be directly utilized.
Then it generates Feature files from the Virtualan Collection and its executed.
Lastly, It generates test report in BDD/Cucumber style.
This provide complete testing support for REST APIs, GraphQL, RDBMS DB and Kafka Event messages
Refer following link for more information to set up and execute.
https://tutorials.virtualan.io/#/Excel
How to create test scripts using excel
According to the documentation there are two ways to send log information to the SwisscomDev ELK service.
Standard way via STDOUT: Every output to stdout is sent to Logstash
Directly send to Logstash
Asking about way 2. How is is this achieved, especially how is the input expected?
We're using Monolog in our PHP buildpack based application and using its stdout_handler is working fine.
I was trying the GelfHandler (connection refused), SyslogUdPHandler (no error, but no result), both configured to use VCAPServices logstashHost and logstashPort as API endpoint / host to send logs to.
Binding works, env variables are set, but I have no idea how to send SwisscomDev ELK service Logstash API endpoint compatible log information from our application.
Logstash is configured with a tcp input, which is reachable via logstashHost:logstashPort. The tcp input is configured with its default codec, which is the line codec (source code; not the plain codec as stated in the documentation).
The payload of the log event should be encoded in JSON so that the fields are automatically recognized by Elasticsearch. If this is the case, the whole log event is forwarded without further processing to Elasticsearch.
If the payload is not JSON, the whole log line will end up in the field message.
For your use case with Monolog, I suggest you to use the SocketHandler (pointing it to logstashHost:logstashPort) in combination with the LogstashFormatter which will take care of the JSON encoding with the log events being line delimited.
I am trying to test web service for my project. The Web service accepts a SOAP request and gives appropriate response.
In JMeter I have chosen SOAP/ XML-RPC request. It works completely fine for me and gives me correct response. However, I have more than 100s of web services in my scope of testing and I have to test them in different environments. It is very cumbersome work to change the URL value from the SOAP/ XML-RPC sample to point it to different env. Do we have something like HTTP Request Default for SOAP/XML-RPC requests?
I have also tried a bean shell sampler where I am setting the value of a variable and then retrieve it in the SOAP sampler URL parameter. However it did not work for me. Below is the code.
Bean Shell sampler code:
vars.put("baseURL","http://localhost:9191/ws/soap");
SOAP/ XML-RPS Sampler URL value:
${__BeanShell(vars.get("baseURL"))}
Any suggestions? I read in JMeter docs that this can be done via http sampler, however, I want to avoid using the same if possible.
You should avoid using SOAP/XML-RPC in favor of pure Http Sampler.
Use the "Templates..." (menu) > Building a SOAP Webservice Test Plan:
This way you can use HTTP Request Default if you want.
But note from what you describe, using a CSV Data Set Config would allow you to variabilize the URL.
Use JMeter Properties to set base url like:
in user.properties file (under /bin folder of your JMeter installation) add one line per property:
baseURL=http://localhost:9191/ws/soap
alternatively you can pass the property via -J command line key as:
jmeter -JbaseURL=http://localhost:9191/ws/soap -n -t /path/to/your/testplan.jmx -l /path/to/results.jtl
Refer the defined property in your test plan using __P() function
${__P(baseURL,)}
You can even provide the default value, i.e. if the property is not set via user.properties file or command-line argument - default value will be used:
${__P(baseURL,http://localhost:9191/ws/soap)}
See Apache JMeter Properties Customization Guide for more information on JMeter properties and ways of setting, overriding and using them.