I ran into a problem where I saved the server storage snapshot while the node was not stopped. Then when I want to restart the node service using the snapshot, the log is displayed as Truncating freezer table. Then WHEN I used Export to export the specified height, the log showed: Head state missing, Angle. How do I restart my node。
Related
I have a container that goes into a crash-restart loop, and it does it reliably after a log entry Adapter disconnected, stopping.
Currently the container is automatically restarting, but it's not enough to break this crash-restart loop.
If I recreate the container, (ie docker-compose down && docker-compose up), then the problem is resolved.
Is there a recommended method for recreating a container on failure instead of restarting it? Particularly after a specific log event.
We have a ECS fargate cluster that runs the fluentd application for collecting logs and routing them to elasticsearch. Logs are buffered on the disk(file buffer) before being routed to the destination. Since we are using FARGATE we mount the buffer path /var/log/fluentd/buffer/ to EFS.
What we would ideally expect is, the data in the buffer path will be flushed to elasticsearch and the buffer directory will be deleted. However we see a huge number of these buffer directories from tasks that have died and restarted several months ago.
So when a ECS tasks dies and comes back up again (autoscaling) it creates a new path /var/log/fluentd/buffer/ that gets mounted on EFS while also holding on to the path /var/log/fluentd/buffer/. I am not sure if its the EFS that holding on to these and remounting back on the new tasks.
Is there a way to delete these stale directories from EFS and just have paths specific to the running tasks. At a time, we have 5 tasks running in a service.
Any help is appreciated?
New to MDT.
So I am following through the MS step by step guides:
https://learn.microsoft.com/en-us/windows/deployment/windows-10-poc
https://learn.microsoft.com/en-us/windows/deployment/windows-10-poc-mdt
I am at step 28 in (in the second guide):
Deploy Windows 10 in a test lab using Microsoft Deployment Toolkit
Where the deployment wizard has been launched in a VM on the host system and have watched the process continue for an hour. It finally finishes but it does not create the .wim on the the server share as
expected and as referred to in the bootstrap.ini:
Bootstrap.ini
[Settings]
Priority=Default
[Default]
DeployRoot=\\SRV1\MDTBuildLab$
UserDomain=CONTOSO
UserID=MDT_BA
UserPassword=pass#word1
SkipBDDWelcome=YES
I have verified that the share "DeployRoot" exists and can be connected to using the provided credentials and that the share has the correct permissions to create/delete files.
Not sure what I'm missing but my expectation was a .wim should have been created in \srv1\MDTBuildLab$\Captures but there is nothing in that folder.
Just before stopping the deployment wizard reboots several times in quick succession, which to me doesn't appear correct but as I have never witnessed a successful capture I can't say for sure this isn't what's supposed to happen.
I'm not even sure where I can view any log files to figure out why it fails.
Any assistance appreciated!
Further Info:
Activated monitoring. It gets to step 86 of 93. The last thing I see is "Applying WinPE (BD)" or something similar and then it restarts. Then several quick reboots occur (the loading bar appears for a second or two and then reboots) (Which I think are failing) finally it gives up! The process never completes!
When I attempt to mount the client REFW10X64-001.vhdx to check the logs I am greeted with this message
The disk image isn't initialized, contains partitions that aren't recognizable, or contains volumes that haven't been assigned drive letters. Please use the Disk Management snap-in to make sure that the disk, partitions, and volumes are in a usable state.
So it looks like the last step totally screwed the disk! Which would explain the last several boots failing to load anything.
So no errors no warnings, no logs, no finish and no wim generated.
How do I troubleshoot this?
I know this post is old, but the normal behavior would be as follows:
Using the boot image, you boot into WinPE
The task sequence is started and the OS gets applied to the disk
Reboot
Boot into full Windows where the task sequence also continues
Under full Windows, one of the last steps is that WinPE gets applied again
Reboot
Computer boots automatically into WinPE
The wim file gets created (WinPE is running on the RAM disk and the regular C: drive (and any additional drives) is being mirrored into the wim file)
Computer performs the FINISHACTION.
We would need at least BDD.log and smsts.log to further troubleshoot. My guess is that WinPE was not applied correctly.
we are using OpenShift container platform (v3.11) for hosting our java application. We are writing application logs to standard pod console. However when I try to view pod logs or try to save logs to file, I am not getting complete log file instead getting only partial log (looks logs are truncated). I have tried to provide different options while viewing logs (like --since=48h etc..), but none of them worked.
Is there any way I can increase pod console buffer size or write complete log file contents to file.
The better way is configuring log aggrigation via fluentd/elastic (see elk_logging), however there's an option to change docker log driver settings on the node with the running container (see managing_docker_container_logs or docker_logging_configure)
I have a working VM instance that I'm trying to copy to allow redundancy behind google load balancer.
A test run with a dummy instance worked fine, creating a new instance from a snapshot of a running one.
Now, the real "original" instance have a persistent disk attached and this cause a problem in starting up the cloned instance because of the (obviously) missing persistent disk mount.
Logs from serial console output is as:
* Stopping cold plug devices[74G[ OK ]
* Stopping log initial device creation[74G[ OK ]
* Starting enable remaining boot-time encrypted block devices[74G[ OK ]
The disk drive for /mnt/XXXX-log is not ready yet or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery
As I understand there is no way to send any of this key strokes to the instance, is there any other way to overcome this issue? I know that I could unmount the disk before the snapshot, but the workflow I would like to instate is creating period snapshots of production servers, so un-mounting disks every time before performing it would require instance downtime (plus all the unnecessary risks of doing an action that would seem pointless).
Is there a way to boot this type of cloned instances successfully, and attach a new persistence disk afterwards?
Is this happening because the original persistent disk is in use, or the same problem would occur even if the original instance is offline (for example due to a failure in which case I would try to created a new instance from a snapshot)?
One workaround that I am using to get away from the same issue is that I dont't actually unmount the disk rather just comment out the the mount line in /etc/fstab and take the snapshot. This way my instance has no downtime or down disks while snapshoting. (I am using Ubuntu 14.04 as OS if that matters)
Later I fix and uncomment it when I use that snapshot on a new instance.
However you can also look into adding the nofail option in the commented line to get a better solution.
By the way I am doing a similar task building a load balanced setup with multiple webserver nodes. Each being cloned from the said snapshot with extra persistent disks mounted for eg uploads,data and logs etc
I'm a little unclear as to what you're trying to accomplish. It sounds like you're looking to periodically snapshot the data volumes of a production server so you can clone them later.
In all likelihood, you simply need to sync and fsfreeze to before you make your snapshot, rather than just unmounting/remounting it. The GCP documentation has a basic example of this in the Snapshots documentation.