When choosing the Enterprise Plan for IBM Event Streams, there is a huge cost associated for Base Capacity Unit-Hour which costs more than $5K per month if I put 720 hours in it (assuming 1 month is 720 hours).
This makes it way too expensive and made me wonder if I understood correctly what "Base Capacity Unit-Hour" means.
Just got this from an IBM rep:
The price is 6.85/Base Capacity Unit, and that's by the hour. So broken down we have, 6.85 * 24 * 30 = 4932/month, which makes your estimate correct. Base Capacity Unit covers 150 MB/s(3 brokers) and 2TB storage. If you find you need to scale up, then the rate will increase from there.
Related
In order to simplify the question and hopefully the answer I will provide a somewhat simplified version of what I am trying to do.
Setting up fixed conditions:
Max Oxygen volume permitted in room = 100,000 units
Target Oxygen volume to maintain in room = 100,000 units
Maximum Air processing cycles per sec == 3.0 cycles per second (min is 0.3)
Energy (watts) used per second is this formula : (100w * cycles_per_second)SQUARED
Maximum Oxygen Added to Air per "cycle" = 100 units (minimum 0 units)
1 person consumes 10 units of O2 per second
Max occupancy of room is 100 person (1 person is min)
inputs are processed every cycle and outputs can be changed each cycle - however if an output is fed back in as an input it could only affect the next cycle.
Lets say I have these inputs:
A. current oxygen in room (range: 0 to 1000 units for simplicity - could be normalized)
B. current occupancy in room (0 to 100 people at max capacity) OR/AND could be changed to total O2 used by all people in room per second (0 to 1000 units per second)
C. current cycles per second of air processing (0.3 to 3.0 cycles per second)
D. Current energy used (which is the above current cycles per second * 100 and then squared)
E. Current Oxygen added to air per cycle (0 to 100 units)
(possible outputs fed back in as inputs?):
F. previous change to cycles per second (+ or - 0.0 to 0.1 cycles per second)
G. previous cycles O2 units added per cycle (from 0 to 100 units per cycle)
H. previous change to current occupancy maximum (0 to 100 persons)
Here are the actions (outputs) my program can take:
Change cycles per second by increment/decrement of (0.0 to 0.1 cycles per second)
Change O2 units added per cycle (from 0 to 100 units per cycle)
Change current occupancy maximum (0 to 100 persons) - (basically allowing for forced occupancy reduction and then allowing it to normalize back to maximum)
The GOALS of the program are to maintain a homeostasis of :
as close to 100,000 units of O2 in room
do not allow room to drop to 0 units of O2 ever.
allows for current occupancy of up to 100 people per room for as long as possible without forcibly removing people (as O2 in room is depleted over time and nears 0 units people should be removed from room down to minimum and then allow maximum to recover back up to 100 as more and more 02 is added back to room)
and ideally use the minimum energy (watts) needed to maintain above two conditions. For instance if the room was down to 90,000 units of O2 and there are currently 10 people in the room (using 100 units per second of 02), then instead of running at 3.0 cycles per second (90 kw) and 100 units per second to replenish 300 units per second total (a surplus of 200 units over the 100 being consumed) over 50 seconds to replenish the deficit of 10,000 units for a total of 4500 kw used. - it would be more ideal to run at say 2.0 cycle per second (40 kw) which would produce 200 units per second (a surplus of 100 units over consumed units) for 100 seconds to replenish the deficit of 10,000 units and use a total of 4000 kw used.
NOTE: occupancy may fluctuate from second to second based on external factors that can not be controlled (lets say people are coming and going into the room at liberty). The only control the system has is to forcibly remove people from the room and/or prevent new people from coming into the room by changing the max capacity permitted at that next cycle in time (lets just say the system could do this). We don't want the system to impose a permanent reduction in capacity just because it can only support outputting enough O2 per second for 30 people running at full power. We have a large volume of available O2 and it would take a while before that was depleted to dangerous levels and would require the system to forcibly reduce capacity.
My question:
Can someone explain to me how I might configure this neural network so it can learn from each action (Cycle) it takes by monitoring for the desired results. My challenge here is that most articles I find on the topic assume that you know the correct output answer (ie: I know A, B, C, D, E inputs all are a specific value then Output 1 should be to increase by 0.1 cycles per second).
But what I want is to meet the conditions I laid out in the GOALS above. So each time the program does a cycle and lets say it decides to try increasing the cycles per second and the result is that available O2 is either declining by a lower amount than it was the previous cycle or it is now increasing back towards 100,000, then that output could be considered more correct than reducing cycles per second or maintaining current cycles per second. I am simplifying here since there are multiple variables that would create the "ideal" outcome - but I think I made the point of what I am after.
Code:
For this test exercise I am using a Swift library called Swift-AI (specifically the NeuralNet module of it : https://github.com/Swift-AI/NeuralNet
So if you want to tailor you response in relation to that library it would be helpful but not required. I am more just looking for the logic of how to setup the network and then configure it to do initial and iterative re-training of itself based on those conditions I listed above. I would assume at some point after enough cycles and different conditions it would have the appropriate weightings setup to handle any future condition and re-training would become less and less impactful.
This is a control problem, not a prediction problem, so you cannot just use a supervised learning algorithm. (As you noticed, you have no target values for learning directly via backpropagation.) You can still use a neural network (if you really insist). Have a look at reinforcement learning. But if you already know what happens to the oxygen level when you take an action like forcing people out, why would you learn such a simple facts by millions of evaluations with trial and error, instead of encoding it into a model?
I suggest to look at model predictive control. If nothing else, you should study how the problem is framed there. Or maybe even just plain old PID control. It seems really easy to make a good dynamical model of this process with few state variables.
You may have a few unknown parameters in that model that you need to learn "online". But a simple PID controller can already tolerate and compensate some amount of uncertainty. And it is much easier to fine-tune a few parameters than to learn the general cause-effect structure from scratch. It can be done, but it involves trying all possible actions. For all your algorithm knows, the best action might be to reduce the number of oxygen consumers to zero permanently by killing them, and then get a huge reward for maintaining the oxygen level with little energy. When the algorithm knows nothing about the problem, it will have to try everything out to discover the effect.
Currently I'm building my monitoring services for my e-commerce Server, which mostly focus on CPU/RAM usage. It's likely Anomaly Detection on Timeseries data.
My approach is building LSTM Neural Network to predict next CPU/RAM value on chart trending and compare with STD (standard deviation) value multiply with some number (currently is 10)
But in real life conditions, it depends on many differents conditions, such as:
1- Maintainance Time (in this time "anomaly" is not "anomaly")
2- Sales time in day-off events, holidays, etc., RAM/CPU usages increase is normal, of courses
3- If percentages of CPU/RAM decrement are the same over 3 observations: 5 mins, 10 mins & 15 mins -> Anomaly. But if 5 mins decreased 50%, but 10 mins it didn't decrease too much (-5% ~ +5%) -> Not an "anomaly".
Currently I detect anomaly on formular likes this:
isAlert = (Diff5m >= 10 && Diff10m >= 15 && Diff30m >= 40)
where Diff is Different Percentage in Absolute value.
Unfortunately I don't save my "pure" data for building neural network, for example, when it detects anomaly, I modified that it is not an anomaly anymore.
I would like to add some attributes to my input for model, such as isMaintenance, isPromotion, isHoliday, etc. but sometimes it leads to overfitting.
I also want to my NN can adjust baseline over the time, for example, when my Service is more popular, etc.
There are any hints on these aims?
Thanks
I would say that an anomaly is an unusual outcome, i.e. a outcome that's not expected given the inputs. As you've figured out, there are a few variables that are expected to influence CPU and RAM usage. So why not feed those to the network? That's the whole point of Machine Learning. Your network will make a prediction of CPU usage, taking into account the sales volume, whether there is (or was) a maintenance window, etc.
Note that you probably don't need an isPromotion input if you include actual sales volumes. The former is a discrete input, and only captures a fraction of the information present in the totalSales input
Machine Learning definitely needs data. If you threw that away, you'll have to restart capturing it. As for adjusting the baseline, you can achieve that by overweighting recent input data.
We have RDS PostgreSQL instance with type db.t2.small . And we have some strange moment with cpu credit balance metrics .
CPU Credit Usage not growing but balance is down to zero . Anybody know in what could be problem? (RDS instance working fine without any problems).
I am seeing the same behavior with my T2-micro free tier RDS instance. My hypothesis right now is that the service window is when the instance is getting rebooted or hot swapped, resulting in a new instance with the default baseline number of credits. This makes Saturday night more appealing than Sunday night in order to be sure by the next business day credits re-accumulate.
From the documentation, it looks like CPU credits expire 24 hours after being earned.
CPUCreditUsage
[T2 instances] The number of CPU credits consumed by the instance. One
CPU credit equals one vCPU running at 100% utilization for one minute
or an equivalent combination of vCPUs, utilization, and time (for
example, one vCPU running at 50% utilization for two minutes or two
vCPUs running at 25% utilization for two minutes).
CPU credit metrics are available only at a 5 minute frequency. If you
specify a period greater than five minutes, use the Sum statistic
instead of the Average statistic.
Units: Count
CPUCreditBalance
[T2 instances] The number of CPU credits available for the instance to
burst beyond its base CPU utilization. Credits are stored in the
credit balance after they are earned and removed from the credit
balance after they expire. Credits expire 24 hours after they are
earned.
CPU credit metrics are available only at a 5 minute frequency.
Units: Count
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/rds-metricscollected.html
I found "rate limit" and "burst limit" at Design section of API Designer,
What is the difference of them?
Rate limit can be set at second, minute, hour, day an week time interval.
On the other hand, burst limit can be set only second and minute time interval.
Does it mean same to set 1/1sec rate limit and to set 1/1sec burst limit?
Different Plans can have differing rate limits, both between operations and for the overall limit. This is useful for providing differing levels of service to customers. For example, a "Demo Plan" might enforce a rate limit of ten calls per minute, while a "Full Plan" might permit up to 1000 calls per second.
You can apply burst limits to your Plans, to prevent usage spikes that might damage infrastructure. Multiple burst limits can be set per Plan, at second and minute time intervals.
That said, these two parameters have a different meaning and could be used together. E.g.: I want to permit a total of 1000 calls per hour (rate limit) and a maximum spike of 50 calls per second (burst limit).
Rate limit enforce how many calls (total) are possible for a given time frame. After that the calls are not possible anymore. This is to create staged plans with different limits and charges (like e.g. entry or free, medium, enterprise).
Burst limits are used to manage, e.g., system load by capping the maximum calls for a moment (hence seconds or minutes), to prevent usage spikes. They can be used to make sure the allowed number of API calls (the rate limit) is evenly spread across the set time frame (day, week, month). They can also be used to protect the backend system from overloading.
So you could set a rate limit of 1000 API calls for a week and the burst limit to 100 calls a minute. If there were 10 "heavy" minutes, the entire rate would have been consumed. An user could also use 100+ calls per day to reach the 1000 calls a week.
I'm trying to review for my final and I'm going over example problems given to me by my professor. Can anyone explain to me the concept of how leaky bucket works. Also Here's a review problem my professor gave to me about leaky buckets.
A leaky bucket is at the host network interface. The data rate in the network is 2 Mbyte/s and the data rate from the application to the bucket is 2m5 Mbyte/s
A.) Suppose the host has 250 Mbytes to send onto the network and it sends the data in one burst. What should the minimum capacity of the bucket (in byte) in order that no data is lost?
B.) Suppose the capacity of the bucket is 100M bytes. What is the longest burst time from the host in order that no data is lost?
Leaky bucket symbolizes a bucket with a small hole allowing water (data) to come out at the bottom. Since the top of the bucket has a greater aperture than the bottom, you can put water in it faster that it goes out (so the bucket fills up).
Basically, it represents a buffer on a network between 2 links with different rates.
Problem A
We can compute that sending the data will take 250Mbyte / (2,5Mbyte / s) = 100 s.
During that 100 s, the bucket will have retransmitted (leaked) 100s * 2Mbyte/s = 200Mbytes
So the bucket will need a minimum capacity of 250MB - 200MB = 50MB in order not to lose any data
Problem B
Since the difference between the 2 data rates is 2.5MB/s - 2.0MB/s = 0.5MB/s, it means the bucked fills up by 0.5MB/s (when both links transmit at full capacity).
You can then calculate that the 100MB capacity will be filled after a burst of 100MB / 0.5MB/s = 200s = 3m 20s
Interesting problem - here's my attempt at solving A (no gurantees it's right though!)
So rate in = 2.5, rate out = 2.0, where the rate is in Mbyte/s.
So in 1 second, the bucket will contain 2.5 - 2.0 = 0.5 Mbyte.
1) If the host sends 250 Mbytes. This will take 100 seconds to transfer into the bucket at 2.5 Mbytes/s.
2) If the bucket drains at 2.0 Mbytes/s then it will have drained 100 * 2 = 200 Mbytes.
So I think you need a bucket which is 50 Mbytes capacity.