I was taking a look Pub/Sub pricing and I couldn't get what would be the price if the subscriber hit the endpoint but doesn't get any message. Because my idea is run a worker 24 hours.
With the pricing model as of today, if your subscriber sends a Pull or StreamingPull request (either directly or through the client libraries), it will not incur a cost if no messages are returned. Billing for subscribers is based on the number of bytes in the messages sent to the subscribers per month.
As described in Cloud Pub/Sub pricing documentation, the pricing is based on the amount of data used in a calendar month. Data volume is computed from message and attribute data for pull, push, and publish operations.
So even if there are no messages available in your subscription, you still perform a pull call and that call has a certain size with a certain price.
Note that the minimum billable volume of data per request (not per message in a request) is 1 KB of data. So even if the pull action is smaller than 1 kB, you are charged for 1 kB. When in one calendar month 50 pull action are executed, each with a size of 500 bytes, the cost for those pull actions is 1 KB * 50 requests * price/GiB in that data volume tier.
How much you are charged for a certain amount of monthly data volume is described in the pricing table in the documentation link above. Notice that there is a certain first amount of data volume for free. The next data volume tiers are not for free.
UPDATE:
I am not able to comment on Kamal's answer here. But taking into account he works on Pub/Sub at Google, I expect his answer to be correct and not mine.
In that case I find the documentation unclear since it emphasizes "per request (not message)".
The minimum billable volume of data per request (not message) is 1 KB of data.
The documentation does not explicitly say you are not billed if there are no messages returned. But according to Kamal, that seems to be the case.
Related
I'm running several batch updates (with about 200 entities each one) and I have a Quantum Leap subscribed to the CB to capture the historical data. It is supposed that each batch update should generate 200 different notifications, but, if I have set a throttle of "1", does this mean I will only receive the first notification and loose the other 199? Is this right? (just watching the QL, it seems to me that I'm loosing many notifications)
Entities are processed one by one during a batch update request. Thus, if you have 200 entities the update in each one triggers a subscription, then you will have 200 notifications being sent.
The throttling effect depends on the case. For instance:
If each entity triggers a different subscription, then they have any effect (as the trottling is evaluated by subscription)
If each entity triggers the same subscription and assuming that all the 200 notifications are send very fast (let's say, less than 1 second) then only the first one will be sent, and the 199 remaining ones will be lost.
In general, we don't recommend to use throttling due to this (and some other :) problems. It uses to be better to implement traffic flow control in the receiver.
I am simplifying my problem with the following scenario:
3 friends share a loyalty card. The card has two restrictions
can be used max 10 times (does not matter which uses the card, ie
friend_a can use it 10 times.
the max money in the card is 200. So with 1 "event" of value = 200 the card is "completed".
I am using a kafka producer that sends events in the kafka cluster like this
{ "name": "friend_1", "value": 10 }
{ "name": "friend_3", "value": 20 }
the events are posted to a topic that is connected with a kafka stream that groups by key and doing aggregation to sum the money spent. That seems to work, however I am facing a "concurrency issue"
Let's imaging the card is used 9 times, so only 1 time remains to be used and the total money spent is 190, that means there are 10 units left to spend.
So friend_2 wants to buy something that costs 11 units (which should not be allowed) and friend_3 wants to buy something that costs 9 units which should be allowed. Friend_3 will modify the state using the card for the 10th time. All other future attempts should not modify something.
So it seems reasonable for the card user to know if the event he sent modified the max used number and the total count. How can I do it in kafka? Using the streams aggregation I can always increase the values, but how can do I know if my action "modified the state" of the card?
UPDATE: the card user should immediately get a negative feedback if the transaction validates a rule.
From how I understand your question there are a few options.
One option is to derive a new stream after the aggregation that you filter() for data that would modify 'the state' of the card, like filtering all events that have > 200 units spent or > 10 uses. This stream can then be used to inform the card user(s) that the card has been spent, e.g. by sending an email. This is approach can be implemented solely with the DSL.
When more flexibility or tighter control is needed, another option is to use the Processor API (which you can integrate with the DSL so most of your code could keep using the DSL), where you implement the aggregation step yourself (working with state stores that you attach to a Transformer or Processor). During the aggregation, you can implement the logic that checks whether the incoming event is valid (in your example: friend_3 with 9 units is valid, friend_2 with 11 units is not). If it is valid, the aggregation increases the card's counters (units and uses), and that's it. If it is invalid, the event is discarded and will not modify the counters, the Transformer/Processor can emit a new event to another stream that tells the card user(s) that something didn't work. You can similarly implement the functionality to inform users that a card has been fully used, that a card is no longer usable, or any other 'state change' of that card.
Also, depending on what you want to do, take a look at the interactive queries feature of Kafka Streams. Sometimes other applications may want to do quick point lookups (queries) of the latest state of something, like the card's state, which can be done with interactive queries via e.g. a REST API.
Hope this helps!
I am currently managing two Google Analytics Management Accounts with many clients and view_ids on each one. The task is to request client data via the Google Analytics Reporting API (v4) and store them to a SQL Backend on a daily basis via an Airflow DAG-structure.
For the first account everything works fine.
Just recently I added the second account to the data request routine.
The problem is that even though both accounts are set to the same "USER-100s" quota limits, I keep getting this error for the newly added account:
googleapiclient.errors.HttpError: <HttpError 429 when requesting https://analyticsreporting.googleapis.com/v4/reports:batchGet?alt=json returned "Quota exceeded for quota group 'AnalyticsDefaultGroup' and limit 'USER-100s' of service 'analyticsreporting.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.">
I already set the quota limit "User-100s" from 100 to the maximum of 1000, as recommended in the official Google guidelines (https://developers.google.com/analytics/devguides/config/mgmt/v3/limits-quotas)
Also I checked the Google API Console and the number of requests for my project number, but I never exceeded the 1000 requests per 100 seconds so far (see request history account 2), while the first account always works(see request history account 1). Still the above error appeared.
Also I could rule out the possibility that the 2nd account's clients simply have more data.
request history account 1
request history account 2
I am now down to a try-except loop that keeps on requesting until the data is eventually queried successfully, like
success = False
data = None
while not success:
try:
data = query_data() # trying to receive data from the API
if data:
success = True
except HttpError as e:
print(e)
This is not elegant at all and bad for maintaining (like integration tests). In addition, it is very time and resource intensive, because the loop might sometimes run indefinitely. It can only be a workaround for a short time.
This is especially frustrating, because the same implementation works with the first account, that makes more requests, but fails with the second account.
If you know any solution to this, I would be very happy to know.
Cheers Tobi
I know this question is here for a while, but let me try to help you. :)
There are 3 standard request limits:
50k per day per project
2k per 100 seconds per project
100 per 100 seconds per user
As you showed in your image (https://i.stack.imgur.com/Tp76P.png)
The quota group "AnalyticsDefaultGroup" refers to your API project and the user quota is included in this limit.
Per your description, you are hitting the user quota and that usually happens when you don't provide the userIP or quotaUser in your requests.
So there is to main points you have to handle, to prevent those errors:
Include the quotaUser with a unique string in every request;
Keep 1 request per second
By your code, I will presume that you are using the default Google API Client for Python (https://github.com/googleapis/google-api-python-client), which don't have a global way to define the quotaUser.
To include the quotaUser
analytics.reports().batchGet(
body={
'reportRequests': [{
'viewId': 'your_view_id',
'dateRanges': [{'startDate': '2020-01-01', 'endDate': 'today'}],
'pageSize': '1000',
'pageToken': pageToken,
'metrics': [],
'dimensions': []
}]
},
quotaUser='my-user-1'
).execute()
That will make to Google API register you request for that user, using 1 of the 100 user limit, and not the same for your whole project.
Limit 1 request per second
If you plan to make a lot of requests, I suggest including a delay between every request using:
time.sleep(1)
right after a request on the API. That way you can keep under 100 requests per 100 seconds.
I hoped I helped. :)
in the Reporting API V4 you can do a batchGet and send up to 5 requests at once.
How does this relate to the quota ? Does it count as one request even if i put multiple ones in the request ?
Limits and quotas
It depends on what limits and quotas you are talking about. Note you can always check the API specific quotas in the Developer Console.
Quota group for the Analytics Reporting API V4:
Each batchGet requests counts as one request against these quotas:
Requests per day per project: 50,000
Requests per 100 seconds per project: 2,000
Requests per 100 seconds per user per project: 100.
Meaning you can put up to 5 requests into each batchGet for a total of 250,000 request per day.
General reporting quotas
There are some quotas general reporting quotas, in which each individual request within a batchGet acts individually against your quota.
10,000 requests per view (profile) per day.
10 concurrent requests per view (profile).
This means if you put 5 requests in a single batchGet and make 2 batchGet requests at the same time you will be at the 10 concurrent requests per view limit, and if you continue to put 5 requests in each batchGet request throughout the day you will only be able to make 2,000 batchGet requests against a single view.
Analytics Reporting API V4 batchGet considerations
A note about the ReportRequest objects within a batchGet method.
Every ReportRequest within a batchGet method must contain the same:
viewId
dateRanges
samplingLevel
segments
cohortGroup
I was doing some latency/performance testing for sending push notifications with Azure Notification Hub by consecutively sending many notifications in a foreach loop. It worked fine for 100 "SendNotification" requests, altough it was relatively slow (14s), but I got a QuotaExceededException for 1000 requests in a row:
[QuotaExceededException: The remote server returned an error: (403)
Forbidden. The request was terminated because the namespace
pushnotification-testing is being throttled. Please wait 60 seconds
and try again. TrackingId:...
Even when I don't wait for 60 seconds as advised, I can again execute 100 consecutive requests, but 1000 requests in a row always fail... Anything slightly above 100 consecutive requests fails most of the time...
I couldn't find any documentation on these limitations. This should be documented somewhere, so I can be sure Azure Notification Hubs will fit my needs.
The answer to this question says
There is a throttling for CRUD operation's rate. Quotas depend on tire
your are but it is not going to be less then 2000 operations per
minute per namespace any way. If quota is exceed then service returns
403.
For me, it seems to be less then 2000 operations. By the way, I'm using "FREE" tier for testing, but I guess we would switch to "STANDARD" for production.
Has anyone similar experiences or knows where to look for more information?
In particular, what are the operation quota limitations per timefram for the different tiers of Azure Notification Hubs?
UPDATE1: It's weird, but I sending 1000 requests in parallel works most of the time, but consecutively it fails on the 101st request.
For my best knowledge for right now NH has following limitations on number of SENDS (not registrations) per namespace per minute per NH machine:
Free tire: 100
Basic tire: 900
Standard tire: 11500
Massive sending in parallel allows to send more because calls are very likely to be routed on different machines.