I have tried two methods to fetch data from the APIs.
1 file_get_contents() then decoding it.
Guzzle Http::get.
Both of the methods working fine with external API urls but not working when I use the APIs of this same project. Shows this error, but I also have increased the Operation time in the php.ini file to 3600.
enter image description here
How can I use the localhost APIs in the same project?
Tried external APIs and tried increasing the operation time.
Related
My first post here and I'm new to Data Fusion and I'm with low to no coding skills.
I want to get data from ZohoCRM to BigQuery. Module from ZohoCRM (e.g. accounts, contacts...) to be a separate table in BigQuery.
To connect to Zoho CRM I obtained a code, token, refresh token and everything needed as described here https://www.zoho.com/crm/developer/docs/api/v2/get-records.html. Then I ran a successful get records request as described here via Postman and it returned the records from Zoho CRM Accounts module as JSON file.
I thought it will be all fine and set the parameters in Data Fusion
DataFusion_settings_1 and DataFusion_settings_2 it validated fine. Then I previewed and ran the pipeline without deploying it. It failed with the following info from the logs logs_screenshot. I tried to manually enter a few fields in the schema when the format was JSON. I tried changing the format to csv, nether worked. I tried switching the Verify HTTPS Trust Certificates on and off. It did not help.
I'd be really thankful for some help. Thanks.
Update, 2020-12-03
I got in touch with Google Cloud Account Manager, who then took my question to their engineers and here is the info
The HTTP plugin can be used to "fetch Atom or RSS feeds regularly, or to fetch the status of an external system" it does not seems to be designed for APIs
At the moment a more suitable tool for data collected via APIs is Dataflow https://cloud.google.com/dataflow
"Google Cloud Dataflow is used as the primary ETL mechanism, extracting the data from the API Endpoints specified by the customer, which is then transformed into the required format and pushed into BigQuery, Cloud Storage and Pub/Sub."
https://www.onixnet.com/insights/gcp-101-an-introduction-to-google-cloud-platform
So in the next weeks I'll be looking at Data Flow.
Can you please attach the complete logs of the preview run? Make sure to redact any PII data. Also what is the version of CDF you are using? Is CDF instance private or public?
Thanks and Regards,
Sagar
Did you end up using Dataflow?
I am also experiencing the same issue with the HTTP plugin, but my temporary way to go around it was to use a cloud scheduler to periodically trigger a cloud function that fetches my data from the API and exports them as a JSON to GCS, which can then be accessed by Data Fusion.
My solution is of course non-ideal, so I am still looking for a way to use the Data Fusion HTTP plugin. I was able to make it work to get sample data from public API end-points, but for a reason still unknown to me I can't get it to work for my actual API.
I'm investigating by days with no results about this exception that my NextJS app is currently throwing, in particular when I try to open a single specific URL:
502: BAD_GATEWAY
Code: NO_STATUS_CODE_FROM_FUNCTION
ID: zrh1:4zx5l-1572269318137-64d401b5d058
Here's the screenshot:
Basically, I have on https://lucacattide.dev/about/en a page that this app should open. This is linked to a MongoDB third-party cloud API platform - Squidex - which is responsible to populate the page itself, via GraphQL queries. The app uses Apollo as GraphQL client. The app instead, is hosted on ZEIT.co serverless cloud, with Now 2.0 version.
During the development process, everything works fine. The page loads up and data is fetched in the right way. Notice that for development, I'm working on now-dev environment instead of a custom Express server, in order to reproduce the production one, as suggested by ZEIT itself.
The exception is being thrown on the production environment - the live one on the hosting platform, not on localhost; the main problem is that no errors are being shown on live logs or local development. So I'm literally going mad in inspecting the possible cause.
I've already tried to test the involved page, by splitting it in sections and trying to exclude child components, or focusing the inspection on the GraphQL query. But the first hasn't produced results and the latter works fine in every environment.
As last try, I deleted and re-created the back-end contents related to that page, because in the past I had a similar issue due to an old GraphQL edited schema that didn't reflected its modifications through the API - so in that case I was still receiving 502 errors. But this time it didn't worked.
Anyone could help me to understand what's going on, please?
Thanks everyone in advance
The issue was caused by an incompatibility between the d3-cloud library and the Now environment.
By replacing it with the react-wordcloud one, the error has been solved.
Thanks everyone for your assistance.
I am doing load testing in Soap ui and working on cab booking application so set api like this for example
userSingin ,GetlocationDetails,bookCab,getDriverDetails,acceptRide and completeRide so on.
It will become on loop(on completing of one ride successfully ) so this seniors may happened like 1000 or 10000 times.
The question is how to interlink api do it a single entity and perform load test using soap ui, jmeter or any load testing tools,or by coding please help me sort out problem with simple demo.
SoapUI has some load testing capabilities so given you have a functional test ready you can run it with increased load.
If SoapUI's load testing is not enough (i.e. you cannot create the desired load or find results not too verbose/informative) you can build the same test plan using JMeter's HTTP Request samplers. If you need to send custom Content-Type and SOAPAction headers - it can be done via HTTP Header Manager. See Building a SOAP WebService Test Plan article for more details.
Last but not the least, you can either execute existing SoapUI project using JMeter using Taurus tool, see SoapUI Support for more details on the specific feature and Navigating your First Steps Using Taurus to get familiarized with the tool and its concept.
I'm trying to pull-of some tests for my RESTful api functions.
For this I did the following:
Installed PHPUnit.
Created a new database for testing.
Created a new enviorment (test) and changed the doctrine config for it.
Created a test.
My problem is this:
When performing a request (somedomain.com/api/somemethod) -> the requested page doesn't know i'm performing a test on it -> so the data it uses is the production/development database and not the 'test' db i have created for the tests.
(the script using test db, the requested page uses normal configurations).
Is there a way to solve it without touching or modifying the API code/behavior?.
Thanks.
Since you said you're requesting somedomain.com I can only suspect you're firing requests over HTTP.
Symfony is made to be easily testable and you can perform functional test without ever making a real HTTP request. Instead, it will make a request object and tell it's kernel to handle it as if it were coming from a real client.
There is a chapter in symfony book on this: Functional tests
If you use method described there (using Symfony BrowserKit client and paths instead of complete urls), Symfony will have it's kernel booted in test environment and will handle request like that.
If, however, for any reason you are unable/don't want to do it that way, and want to fire real HTTP requests, I suggest you to make a file in web directory called app_test.php. In that file you should boot the kernel in test environment and make sure your tests are actually hitting that file (instead of app.php or app_dev.php). However, have in mind that this file will be publicly available and as so, it will cause a security hole so make sure to guard it somehow (check app_dev.php for hints). As an idea, you could require specific key to be provided in request header to allow it to pass on. Or if it will be tested from a single machine, you could also guard it by IP, or whatever works for your case.
Ive connected my ipad up so it is now talking to my OData.svc served up on the web. I can do a noraml query of a table without. But as soon as I add a select to that query to cut down the columns being transferred I get a http 400 error. I can get the exact url (http://mywebsite.com/OData.svc/Customers?$top=10&$select=CustomerName) it is accessing and put it into a browser and it works without issue. Any ideas what this could be?
I added a select to one of the queries in the Netflix example and it worked fine, it seems to be something with my OData.svc that isnt allowing selects to be performed from my ipad app (but browser is fine).
Thanks in advance
The $select requires that the request is sent with version 2.0. So its DataServiceVersion header must be set to 2.0;. If it's set to 1.0 the request will fail with 400. You can verify that this is the root cause by using some kind of network tracing software to see the body of the response, which should contain an error message saying that the feature you want to use requires a higher version.
It works from the browser, because the browser doesn't send any version header and in that case the server assumes the higher version the server can understand (so 2.0 in these cases).