I have tried to copy a .mp4 file from my local directory to my google cloud bucket,
using:
gsutil cp my_filefile.mp4 gs://my_bucket
This part works as expected, but when i try to limit the bandwidth, using:
trickle -d 10 -u 10 gsutil cp my_filefile.mp4 gs://my_bucket
the uploading happens at the same rate, and not with 10 kb/s. I have read that trickle does not handle static executable files, which the .mp4 appears to be since when running ldd my_file.mp4, in the terminal, it returns not a dynamic executable.
Has anyone experienced the same issue, and if that is the case, how was the problem handled, or am i approaching this issue the wrong way?
UPDATE 1:
Turns out it does not matter what file i use. gsutil still bypasses trickle somehow. I have tested to see if trickle worked with other programs, and it performed as expected, with bandwidth control.
I have also tested gsutil mv and gsutil rsync, with the same results, as with cp. I have also tested the bandwidth throttling on an arm64 system, with the same results.
You should limit the number of thread and process as described in the documentation. Trickle shouldn't been applied in case of multi process.
trickle -d 10 -u 10 gsutil -o "GSUtil:parallel_process_count=1" \
-o "GSUtil:parallel_thread_count=1" cp my_filefile.mp4 gs://my_bucket
Related
I am using Neo4j on a remote server (ubuntu 20.4) and would like to stream data from MongoDB to Neo4j. I followed the instructions here. I tried both ways by using the following approaches:
Use the following command:
sudo wget https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/tag/4.3.0.7/apoc-mongodb-dependencies-4.3.0.7.jar -O /mnt/neo4j/plugins/apoc-mongodb-dependencies-4.3.0.7.jar
Note that the plugins directory has a different path due to mounting. I changed the path in the configuration file accordingly. This should not be causing any problems because I had the same problem before mounting.
Also, I tried to match the same release as the apoc-core file (4.4.0.3) in a separate attempt with no better outcome.
Changing the ownership and read permissions as follows didn't help either:
sudo chown neo4j:neo4j apoc-mongodb-dependencies-4.4.0.3.jar
sudo chmod 755 apoc-mongodb-dependencies-4.4.0.3.jar
Use the following commands:
sudo wget https://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/3.12.11/mongo-java-driver-3.12.11.jar -O /mnt/neo4j/plugins/mongo-java-driver-3.12.11.jar
sudo wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver/3.12.11/mongodb-driver-3.12.11.jar -O /mnt/neo4j/plugins/mongodb-driver-3.12.11.jar
sudo wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/4.7.1/mongodb-driver-core-4.7.1.jar -O /mnt/neo4j/plugins/mongodb-driver-core-4.7.1.jar
sudo wget https://repo1.maven.org/maven2/org/mongodb/bson/4.7.1/bson-4.7.1.jar -O /mnt/neo4j/plugins/bson-4.7.1.jar
Note that I used the latest versions. I tried the versions available in the instructions as well with no difference in the outcome.
Now when restarting the neo4j.service, I no longer can access the cypher-shell nor the browser. In the first case, I get "connection refused", while I get a blank page in the browser case. When I check the status, the service is active and running. But I noticed that it is missing a line compared to when I don't have the dependencies.
Starting...
This instance is ServerId{#}
======== Neo4j 4.4.5 ======== (This line is missing with the dependencies downloaded!)
When I delete the dependencies from the plugins directory and restart, everything goes back to normal and functions as expected. One more thing to note is that apoc-core procedures work just fine!
I don't know if I'm doing something wrong here or if there is some sort of underlying problem!
I'm on a Windows Server 2016 machine. I have run pg_dump.exe on a 3gb postgres 9.4 database using the -Fc format.
When I run pg_restore to a local database (9.6):
pg_restore.exe -O -x -C -v -f c:/myfilename
The command runs for over 24 hours. (Still running)
Similar to this issue: Postgres Restore taking ages (days)
I am using the verbose cli option, which looks to be spitting out a lot of JSON. I'm assuming that's getting inserted into tables. The task manager has the CPU at 0%, using .06MB of memory. Looks like I should add more jobs next time, but this still seems pretty ridiculous.
I prefer using a linux machine, but this is what the client provided. Any suggestions?
pg_restore.exe -d {db_name} -O -x c:/myfilename
Did the trick.
I got rid of the -C and manually created the database prior to running the command. I also realized that connection options should come before other options:
pg_restore [connection-option...] [option...] [filename]
see postgres documentation for more.
I have static assets stored in GCS and I'd like to serve them gzipped (but they were uploaded without compression). Is there any way to set files to be compressed without downloading and re-uploading them in gzipped format?
I tried setting the content-encoding header with gsutil (i.e., gsutil setmeta -h 'Content-Encoding:gzip' <some_object_uri>, but it just led to a "Service Unavailable" on the file (which I assume is from the server attempting to ungzip the file and failing or something like that).
There is no way to compress the objects without downloading them and re-uploading.
However, you can have gsutil do this for you, and if you run it from a Google Compute Engine (GCE) Virtual Machine (VM), you'll only be charged for operation counts, not for bandwidth.
Also, regarding setting the content-encoding header with setmeta, you're right in your interpretation of what happened. You set the metadata on the object to indicate that it contained gzip data, but the contents did not contain a valid gzip stream, so when you try to download it with Accept-Encoding: gzip, the GCS service tries to decompress the stream and fails.
I'd suggest downloading the bucket to the local disk on a GCE VM:
gsutil cp -r gs://bucket /path/to/local/disk
Then, use the -z option to indicate which file extensions to gzip:
gsutil cp -z js,css,html -r /path/to/local/disk gs://bucket
i'm working in a instance at us-central1-a zone and I can't copy a ~200GB file.
i've tried :
gsutil -m cp -L my.log my.file gs://my-bucket/
gsutil -m cp -L my.second.log my.file gs://my-bucket2/
And after several "catch ups" I get the following error:
CommandException: Some temporary components were not uploaded successfully. Please retry this upload.
CommandException: X files/objects could not be transferred.
Any clues?
Thanks
This is a message you'll see if gsutil's parallel composite uploads feature fails to upload at least one of the pieces of the file.
A couple of questions...
Have you already tried performing this upload again, after you saw this message?
If this error persists, could you please provide the stack trace from gsutil -d cp...
If you're consistently seeing this error and need an immediate fix (if this is a bug with parallel uploads), you can set parallel_composite_upload_threshold=0 in the GSUtil section your boto config to disable parallel uploads.
I had the same experience using gsutil. I fixed by installing the crcmod.
First run the command you have issues with using the debug flag, for example:
gsutil -d -m cp gs://<path_to_file_in_bucket>
In the output I can see:
CommandException: Downloading this composite object requires integrity checking with CRC32c, but your crcmod installation isn't using the module's C extension, so the hash computation will likely throttle download performance. For help installing the extension, please see "gsutil help crcmod".
To download regardless of crcmod performance or to skip slow integrity checks, see the "check_hashes" option in your boto config file.
NOTE: It is strongly recommended that you not disable integrity checks. Doing so could allow data corruption to go undetected during uploading/downloading.
You can follow the instructions here from google to install crcmod for your specific os: https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod
I got the same error message. I tried login in to gcloud again with
gcloud auth login
and then I could run the command successfully.
my memcached server have to be "flush_all" per about 12 hours ,unless the app would down soon.
if i ignore some memcached settings? my app based on java/tomcat
i startup memcached by typed "nohup ./memcached -p [port] -m 8192" that all,
i' can't found the configure file of my memcached server.
i installing memcached by typed tar zxf memcached-version.tar.gz
cd memcached-version/
./configure -prefix /usr
make && make install
How much memory do you have on that machine? if you have 8G, what might be happening is that memcached is taking up all the memory available on the machine.
if so, just change the memcached startupline to -m 5000 or so.