Unreal Engine 4: What is correct way to PAK files, load/mount them and load assets with AssetRegistry in packaged game? - unreal-engine4

I want to do this:
Create and package original game. Then I want to create additional PAK files with new meshes/sounds/animations and blueprints based on blueprint in the original game. Original game should not know anything about additional meshes/animations/etc. So I need to create a smart system with AssetRegistry in the original game that scans all PAK files, load/mount them and with AssetRegistry scan those PAK files for all assets.
What I did to achieve my goal:
I create and package successfully original game for the target platform (windows standalone). Then in the project, I create additional content and cook them for the target platform. I use UnrealPak.exe to create PAK files for additional content. I am able to load/mount PAK file in the original game by placing PAK file in Paks folder and they load/mount at startup of the game (this sentence is based on the LOG file from the original game, I don't know how to check if it is true or not). I am able to load/mount PAK file even with code by using FCoreDelegates::OnMountPak.Execute (this sentence is also based on the LOG file from the original game). So loading/mounting PAK files should work well. But now where is the biggest issue. I want to use AssetRegistry to scan for all assets in all PAK files. I tried everything I came up with. I tried ScanPathsSynchronous method, GetAllAssets method. Only what happens is it loads assets from ORIGINAL GAME PAK FILE. It seems that AssetRegistry doesn't know anything about other PAK files. I tried to tell AssetRegistry where are those files with AddPath method and still doesn't work.
So my example code what I tried is here:
FString path1 = FPaths::ConvertRelativePathToFull(FString("../../../TestPaks/Content/Paks/test.pak"));
FString path2 = FPaths::ConvertRelativePathToFull(FString("../../../TestPaks/Content/Paks/testmaterial.pak"));
bool check1 = false;
bool check2 = false;
if (FCoreDelegates::OnMountPak.IsBound())
{
check1 = FCoreDelegates::OnMountPak.Execute(path1, 0, nullptr); //Number should be 0-4; specifies search order
check2 = FCoreDelegates::OnMountPak.Execute(path2, 0, nullptr); //Number should be 0-4; specifies search order
}
UE_LOG(LogTemp, Warning, TEXT("%s"), *path1);
UE_LOG(LogTemp, Warning, TEXT("%s"), *path2);
FString NewString1 = check1 ? "true" : "false";
FString NewString2 = check2 ? "true" : "false";
UE_LOG(LogTemp, Warning, TEXT("check 1 = %s"), *NewString1);
UE_LOG(LogTemp, Warning, TEXT("check 2 = %s"), *NewString2);
FAssetRegistryModule& AssetRegistryModule = FModuleManager::LoadModuleChecked<FAssetRegistryModule>(FName("AssetRegistry"));
IAssetRegistry& assetRegistry = AssetRegistryModule.Get();
TArray<FString> ContentPaths;
TArray<FAssetData> data;
//assetRegistry.AddPath(path1);
FString contentRelativeDir = TEXT("/Game/Paks");
assetRegistry.AddPath(contentRelativeDir);
assetRegistry.ScanPathsSynchronous({ contentRelativeDir });
//assetRegistry.SearchAllAssets(true);
assetRegistry.GetAllAssets(data, false);
assetRegistry.GetAllCachedPaths(ContentPaths);
for (FString& data : ContentPaths)
{
UE_LOG(LogTemp, Warning, TEXT("GetAllCachedPaths: %s"), *data);
}
FString NewString = FString::FromInt(data.Num());
UE_LOG(LogTemp, Warning, TEXT("%s"), *NewString);
for (int32 i = 0; i < data.Num(); i++)
{
FString s = data[i].AssetName.ToString();
FString ss = data[i].AssetClass.ToString();
UE_LOG(LogTemp, Warning, TEXT("%s | %s"), *s, *ss);
}
I tried a lot of versions of paths and nothing is working. I am in this mess around 2 weeks and I don't have any much more tips on what to do and what will work. So how should this work properly??? I looked in forums here and StackOverflow and there are some solutions, but they don't work anymore.

I have loaded asset from .pak file but I use FStreamableManager instead of AssetRegistry.
I will describe what I do...
1) Mount .pak file: I just need to put additional .pak file to WindowNoEditor/[Project_Name]/Content/Paks/ folder (assumed that you package for window), game engine will auto mount that .pak file.
2) Load/Get asset from that .pak file:
FPakPlatformFile *PakPlatformFile;
FString PlatformFileName = FPlatformFileManager::Get().GetPlatformFile().GetName();
if (PlatformFileName.Equals(FString(TEXT("PakFile"))))
{
PakPlatformFile = static_cast<FPakPlatformFile*>(&FPlatformFileManager::Get().GetPlatformFile());
}
else
{
PakPlatformFile = new FPakPlatformFile;
if (!PakPlatformFile->Initialize(&FPlatformFileManager::Get().GetPlatformFile(), TEXT("")))
{
UE_LOG(LogTemp, Error, TEXT("FPakPlatformFile failed to initialize"));
return;
}
FPlatformFileManager::Get().SetPlatformFile(*PakPlatformFile);
}
TArray<FString> ArrAllMountedPakFile;
PakPlatformFile->GetMountedPakFilenames(ArrAllMountedPakFile);
for (int32 i = 0; i < ArrAllMountedPakFile.Num(); i++)
{
FString PakFileName = ArrAllMountedPakFile[i];
FString PakFilePathFull = FPaths::ConvertRelativePathToFull(PakFileName);
FPakFile PakFile(PakPlatformFile, *PakFilePathFull, false);
TArray<FString> FileList;
FString MountPoint = PakFile.GetMountPoint();
PakFile.FindFilesAtPath(FileList, *MountPoint, true, false, true);
for (int32 i = 0; i < FileList.Num(); i++)
{
FString AssetName = FileList[i];
FString AssetShortName = FPackageName::GetShortName(AssetName);
FString FileName, FileExt;
AssetShortName.Split(TEXT("."), &FileName, &FileExt);
FString NewAssetName = TEXT("/Game/<path_to_asset>/") + FileName + TEXT(".") + FileName;
FSoftObjectPath StrNewAssetRef = NewAssetName;
FStreamableManager AssetLoader;
UObject* NewLoadedObject = AssetLoader.LoadSynchronous(StrNewAssetRef);
if (NewLoadedObject)
{
// do something, cast to compatible type.
}
}
}
Hope that to help you. Cheer !

Related

Decoding delimited frames from byte arrays

I have frames that are delimited by bytes to start and stop the frame (they do not appear in the stream).
I read a chunk from disk or network socket, i then need to pass to a deserializer but only after I have de-framed the packet first.
Frames may span multiple chunks that have been read, note how frame 3 is split across array 1 and array 2.
Rather than reinvent the wheel for this common problem, do any github or similar projects exist?
I am investigating ReadOnlySequenceSegment<T> from https://www.codemag.com/article/1807051/Introducing-.NET-Core-2.1-Flagship-Types-Span-T-and-Memory-T and will post updates as I work out the requirements.
Update
Further to Stephen Cleary link (thank you!!) to https://github.com/davidfowl/TcpEcho/blob/master/src/Server/Program.cs I have the below.
My data is json, so unlike the original question the delimiter tokens will appear in the stream. Therefore I have to count the array delimitator and only declare a frame when i have found the outermost [ and ] characters.
The below code works, and less manual copies done (not sure if still done behind the scenes - code is quite neater using David Fowl approach).
However I am casting to array instead of using buffer.PositionOf((byte)'[') since I was unable to see how I could call the PositionOf with an offset applied (i.e. scan deeper into the frame past previously found delimiter tokens).
Am i using/butchering the library in a brute force way, or is the below good to go with the array cast?
class Program
{
static async Task Main(string[] args)
{
using var stream = File.Open(args[0], FileMode.Open);
var reader = PipeReader.Create(stream);
while (true)
{
ReadResult result = await reader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
while (TryDeframe(ref buffer, out ReadOnlySequence<byte> line))
{
// Process the line.
var str = System.Text.Encoding.UTF8.GetString(line.ToArray());
Console.WriteLine(str);
}
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.Start, buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.CompleteAsync();
}
private static bool TryDeframe(ref ReadOnlySequence<byte> buffer, out ReadOnlySequence<byte> frame)
{
int frameCount = 0;
int start = -1;
int end = -1;
var bytes = buffer.ToArray();
for (var i = 0; i < bytes.Length; i++)
{
var b = bytes[i];
if (b == (byte)'[')
{
if (start == -1)
start = i;
frameCount++;
}
else if (b == (byte)']')
{
frameCount--;
if (frameCount == 0)
{
end = i;
break;
}
}
}
if (start == -1 || end == -1) // no frame found
{
frame = default;
return false;
}
frame = buffer.Slice(start, end+1);
buffer = buffer.Slice(frame.Length);
return true;
}
}
do any github or similar projects exist?
David Fowler has an echo server that uses Pipelines to implement delimited frames.

How to allow users to upload files with Google Form without login?

Where can I find code and instruction on how to allow users to upload files with Google Form without login?
I searched all over here and couldn't find any information.
https://developers.google.com/apps-script/reference
Thanks in advance.
The user will be uploading the files to your drive. So, google needs to verify the user. If there is no verification, someone can fill your drive in no time.
It is for your safety to know who has uploaded, so, login is must.
There's a workaround, I'm in a hurry to write the code now, but if you're interested let me know and I'll edit later.
Basically, you set up a web app with apps script, then you setup a custom HTML form, you'll have to manually collect the file, convert is to base64 then json, then when you catch it in apps script you reverse the process and save it wherever you want in your drive.
Since the user will be executing the script as you, there's no verification required
/*
These functions basically go through a file array and reads the files first as binary string (in second function), then converts the files to base64 string (func 1) before stringifying the files (after putting their base64 content into an object with other metadata attached; mime, name e.t.c);
You pass this stringified object into the body part of fetch(request,{body:"stringified object goes here"})
see next code block for how to read in apps script and save the files to google drive
N.B. The body data will be available under doPost(e){e.postData.contents}
*/
async function bundleFilesForUpload(){
let filesDataObj = [];
let copy = {fileInfo:{"ogname":"","meme":""},fileData:""};
for(let i = 0 ; i < counters.localVar.counters.filesForUploadArr.length ; i++){
let tempObj = JSON.parse(JSON.stringify(copy));
let file = counters.localVar.counters.filesForUploadArr[i];
tempObj.fileInfo.ogname = file.name;
tempObj.fileInfo.meme = file.type;
tempObj.fileData = await readFile(file).then((file)=>{
file = btoa(file);
return file;
}).then((file)=>{
return file;
})
filesDataObj.push(tempObj);
}
return filesDataObj;
}
async function readFile (file){
const toBinaryString = file => new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsBinaryString(file);
reader.onload = () => resolve(reader.result);
reader.onerror = error => reject(error);
});
let parsedFile = null;
parsedFile = await toBinaryString(file);
return parsedFile;
}
/*From doPost downward, we read the file Array convert the base64 to blob and make a file in google drive using the blob and metadata we have, you may also see some sheet code, I'm using sheet as db for this */
//in buit function doPost in Code.gs
doPost(e){
const myDataObj = JSON.parse(e.postData.contents);
mainFileFunc(myDataObj.params[0].dataObj.images);
//the actual object structure might look different from yours, console log around
}
function mainFileFunc(fileArr) {
let myArrObj = [{"madeit":"toFileF"}];
let copy = JSON.parse(JSON.stringify(myArrObj[0]));
//sheet.getRange("A1").setValue(JSON.stringify(fileArr.length));
for(let i=0 ; i < fileArr.length ; i++){
myArrObj.push(copy);
let blob = doFileStuff(fileArr[i].data,fileArr[i].info[0].mime,fileArr[i].id);
myArrObj[i] = uploadFileOne(blob,fileArr[i].id);
myArrObj[i].mime = fileArr[i].info[0].mime;
myArrObj[i].realName = fileArr[i].name;
// sheet.getRange("A"+(i+1)).setValue(myArrObj[i].name);
// sheet.getRange("B"+(i+1)).setValue(myArrObj[i].url);
// sheet.getRange("C"+(i+1)).setValue(myArrObj[i].mime);
// sheet.getRange("D"+(i+1)).setValue(myArrObj[i].size);
}
return myArrObj;
}
function doFileStuff(filedata,filetype,filename){
var data = Utilities.base64Decode(filedata, Utilities.Charset.UTF_8);
var blob = Utilities.newBlob(data,filetype,filename);
return blob;
}
function uploadFileOne(data,filename) {
let myObj = {}
myObj["name"] = "";
myObj["realName"] = "Story_Picture";
myObj["url"] = "";
myObj["mime"] = "";
myObj["size"] = "";
myObj["thumb"] = "nonety";
var folders = DriveApp.getFoldersByName("LadhaWeb");
while (folders.hasNext()) {
var folder = folders.next();
folder.createFile(data);
}
var files = DriveApp.getFilesByName(filename);
while (files.hasNext()) {
var file = files.next();
myObj.name = file.getName();
myObj.url = file.getUrl();
myObj.mime = file.getMimeType();
myObj.size = file.getSize();
}
return myObj;
}
You can view the full frontend code for this project here and the backend here.
Hope this helps someone.

Analyzer creating multiple diagnostics for compilation unit

I am writing a Roslyn Diagnostic to turn on/off option strict. Since there can only be one per file, I am using the compilation for the node to be examined:
context.RegisterSyntaxNodeAction(CompilationUnitCheck, SyntaxKind.CompilationUnit);
I am seeing multiple diagnostics displayed in the error list pane when running the development hive, at times as many as 3, but always at least 2 per file. They show the same location. What could be causing this, and what can I do to fix it?
private void CompilationUnitCheck(SyntaxNodeAnalysisContext context)
{
var orgRoot = (CompilationUnitSyntax) context.Node;
var fileName = System.IO.Path.GetFileNameWithoutExtension(orgRoot.SyntaxTree.FilePath) ;
if ((fileName?.EndsWith("designer", StringComparison.CurrentCultureIgnoreCase)).GetValueOrDefault() ||
"Reference".Equals(fileName, StringComparison.CurrentCultureIgnoreCase))
{
return;
}
if (fileName != "TestFile") return;
var newErrors = fileName == "TestFile";
var location = orgRoot.GetLocation();
string strictMsg = null ?? "Off";
var diagnostic = Diagnostic.Create(Rule, location, strictMsg);
context.ReportDiagnostic(diagnostic);
}
My current workaround is to get the hashcode of the root, and store that in a static list, if it's in the list, I don't check again.

Take all text files in a folder and combine then into 1

I'm trying to merge all my text files into one file.
The problem I am having is that the file names are based on data previously captured in my app. I don't know how to define my path to where the text files are, maybe. I keep getting a error, but the path to the files are correct.
What am I missing?
string filesread = System.AppDomain.CurrentDomain.BaseDirectory + #"\data\Customers\" + CustComboB.SelectedItem + #"\";
Directory.GetFiles(filesread);
using (var output = File.Create("allfiles.txt"))
{
foreach (var file in new[] { filesread })
{
using (var input = File.OpenRead(file))
{
input.CopyTo(output);
}
}
}
System.Diagnostics.Process.Start("allfiles.txt");
my error:
System.IO.DirectoryNotFoundException
HResult=0x80070003
Message=Could not find a part of the path 'C:\Users\simeo\source\repos\UpMarker\UpMarker\bin\Debug\data\Customers\13Dec2018\'.
I cant post a pic, but let me try and give some more details on my form.
I select a combobox item, this item is a directory. then I have a listbox that displays the files in my directory. I then have a button that executes my desires of combining the files. thanks
I finally got it working.
string path = #"data\Customers\" + CustComboB.SelectedItem;
string topath = #"data\Customers\";
string files = "*.txt";
string[] txtFiles;
txtFiles = Directory.GetFiles(path, files);
using (StreamWriter writer = new StreamWriter(topath + #"\allfiles.txt"))
{
for (int i = 0; i < txtFiles.Length; i++)
{
using (StreamReader reader = File.OpenText(txtFiles[i]))
{
writer.Write(reader.ReadToEnd());
}
}
System.Diagnostics.Process.Start(topath + #"\allfiles.txt");
}

Google script: Download web image and save it in a specific drive folder

I need to download an image with GS and save it in a specific drive folder.
I'm able to save the image in the root folder but i cannot save it in a specific folder:
function downloadFile(fileURL,folder) {
var fileName = "";
var fileSize = 0;
var response = UrlFetchApp.fetch(fileURL, {muteHttpExceptions: true});
var rc = response.getResponseCode();
if (rc == 200) {
var fileBlob = response.getBlob()
var folder = DriveApp.getFoldersByName(folder);
if (folder != null) {
var file = DriveApp.createFile(fileBlob);
fileName = file.getName();
fileSize = file.getSize();
}
}
var fileInfo = { "rc":rc, "fileName":fileName, "fileSize":fileSize };
return fileInfo;
}
Question: what have I to add to use the variable "folder"?
I found a lot of examples with "DocList" Class that is not in use anymore
Many thanks
Well, I guess GAS has make a lot of progress on developing its API, the function
createFile(blob) of an object Folder will do the job:
https://developers.google.com/apps-script/reference/drive/folder#createfileblob
// Create an image file in Google Drive using the Maps service.
var blob = Maps.newStaticMap().setCenter('76 9th Avenue, New York NY').getBlob();
DriveApp.getRootFolder().createFile(blob);
It's quite late for the answer but just incase some one runs into the situation.
Are you familiar with this app? It does exactly what you're asking for.
However, if you want to re-create this for your own purposes, I would change your declaration of variable file to read as such:
var file = folder.next().createFile(fileBlob);
when you create your variable folder, the method you use creates a FolderIterator, not a single folder. You have to call the next() method to get a Folder object.
To be precise with your script and avoid saving to an incorrect-but-similarly-named folder, I would recommend passing the folder ID to your script rather than the folder Name. If you pass the folder ID, you could declare folder as:
var folder = DriveApp.getFolderById(folder);
and then continue the script as you have it written. I hope that helps.
Working on similar problem, I came up with the solution below to save a file to a folder. If the folder doesn't exist it creates it, otherwise it saves the file specified by "FOLDER_NAME"
var folderExists = checkFolderExists("FOLDER_NAME");
if (folderExists) {
saveFolder = DriveApp.getFolderById(folderExists);
} else {
saveFolder = DriveApp.createFolder("FOLDER_NAME");
}
// Make a copy of the file in the root drive.
var file = DriveApp.getFileById(sheetID);
// Take the copy of the file created above and move it into the folder:
var newFile = DriveApp.getFolderById(saveFolder.getId()).addFile(file);
// Remove the copy of the file in the root drive.
var docfile = file.getParents().next().removeFile(file);
Further to Eric's answer, I have also provided a utility function that checks if the folder exists. It's reusable in any project.
function checkFolderExists(fName) {
try {
var folderId;
var folders = DriveApp.getFolders();
while (folders.hasNext()) {
var folder = folders.next();
folderName = folder.getName();
if (folderName == fName) {
folderId = folder.getId();
}
}
} catch(e) {
log("Services::checkFolderExists()" + e.toString());
throw e;
}
return folderId;
}