I have setup a Ceph object storage cluster, and I want to monitor the object's access behavior on each OSD. For example, when the object is written to this OSD? How large it is? How long it takes?
So are there any tools or APIs could help me build such a monitor?
For a particular object that exists, you can setup a client to watch the object, and recieve notification about changes. From there you could issue a stat.
https://github.com/ceph/ceph/blob/master/src/include/rados/librados.hpp#L709
Related
Currently I tried to fetch already rotated logs within the node using --since-time parameter.
Can anybody suggest what is the command/mechanism to fetch already rotated logs within kubernetes architecture using commands
You can't. Kubernetes does not store logs for you, it's just providing an API to access what's on disk. For long term storage look at things like Loki, ElasticSearch, Splunk, SumoLogic, etc etc.
I am doing API deployment to Service Fabric Nodes, and it is by default going to D drive (Temp drive), I would like to change this default behavior and deploy it to another drive or C drive to avoid application loss in case of VMSS deallocation. How can I do this?
You say you want to do this to avoid application loss, however:
SF already replicates your application package to multiple machines when you Register the application package in the Image Store (part of the provisioning/deployment process)
Generally, if you want your application code and config to be safe, keeping it somewhere outside the cluster (wherever you're deploying from, or in blob storage) is usually a better answer.
SF doesn't really support deallocating the VMs out from under it and then bringing them back later. See the FAQ answer here.
So overall I'm not sure that what you're trying to do is the right solution to your real problem, and it looks like you're heading into multiple unsupported scenarios, which usually means there's some misunderstanding.
That all said, of course, it's configurable.
Within a node type, you can specify the dataPath (example here). However, it's not recommended that you change this.
"settings": {
"dataPath": "D:\\\\SvcFab",
},
Is there a way for me to programmatically get notified when Bluemix auto scaling has scaled up or down?
I'm reading streaming data from a queue and would like to make sure the number of instances that I have are balanced and data is partitioned correctly
At present this kind of notification service is not available, only you can do is query the instance scaling history in Web UI. I think this requirement is interesting and should be considered to provide to developer in the future.
This kind of alert isn't available yet but you can write a simple script monitoring output of
cf app (appname)
It returns the number of instances running and the state of each one, with the right combination of awk and grep (or a perl script for example) you could have your own alerter while waiting for this of functionality
IN this article "http://en.wikipedia.org/wiki/Object_storage" ,
It says Lustre is a object based file system and says ceph is a hybrid storage.
I really don't know their differences. ceph also is a distributed file system , block storage , object storage. Anyone know of that , Ceph 's file system and block storage is based object store or not ?
The content of a file stored on the Ceph file system which provides a POSIX API can be retrieved via the librados API which is an Object Store API similar to SWIFT or S3. Although this is why Ceph deserves to be called a UFOS (Unified File and Object Storage) or hybrid storage, it is not a supported use case.
If the Ceph file system implementation changes to modify the names of the objects used to store the content of the files, the user of the librados API will need to know about it and adapt.
An hybrid storage would allow the user to conveniently store an object named foo via the object store API and retrieve it under a similar name (for instance /objectstore/foo) via the POSIX API, without knowing the implementation details.
I have a process that populates data into map with persistence periodically. To be more exact there are two nodes: storage node with persistence enabled and cache maps defined and a lite client node started with 'lite' option and no map defined. Connection between nodes does look good. During testing I found out that only around a half of populated data is actually flown into persistence though all data is in cache. I can confirm this by browsing the cache map and via JMX stats. I can't indicate dependencies on the data or time it is populated.
Could someone please advise where the investigation should start from?
This is only my own fault. I did not provide 'lite' option to the populator properly so data is distributed between nodes and only written on the storage side as lite client does not have any persistence set up. I did not remove the question to prevent someone else from this silly failure.