I am trying to read into some existing and unmounted ESE database files. I have been playing around with one .dat file rather successfully. But when I try to open a existing database with PageSize that is equal 32768 I get an error.
Here's my code (without error-handling):
FError := JetSetSystemParameter(&FInstance, nil, JET_paramRecovery, FPagesize, "off");
FError := JetCreateInstance(&FInstance, 'myinstance');
FError := JetInit(&FInstance);
FError := JetBeginSession(FInstance, &FSessionId, nil, nil);
FError := JetAttachDatabase(FSessionId, FFilename, JET_bitDbReadOnly);
It fails at the JetAttachDatabase call which returns an -1213 code. Am I doing something wrong?
I am running a Windows 7 32bit.
The Esent engine uses a certain page size by default. If I'm not mistaken it's 4K. You will have to tell the engine that the database you want to open has a different page size. Use something like that:
FError := JetSetSystemParameter(&FInstance, nil, JET_paramDatabasePageSize, 32768, nil);
If you open up different databases all the time, you might want to have your application checking out and setting the pagesize automatically.
Related
I am trying to configure a FireDAC TFDQuery component so it fetches records on demand in batches of no more than 500, but I need it to report back what is the total record count for the query not just the number of fetched records. The FetchOptions are configured as follows:
FetchOptions.AssignedValues = [evMode, evRowsetSize, evRecordCountMode, evCursorKind, evAutoFetchAll]
FetchOptions.CursorKind = ckForwardOnly
FetchOptions.AutoFetchAll = afTruncate
FetchOptions.RecordCountMode = cmTotal
FetchOptions.RowSetSize = 500
This immediately returns all records in the table not just 500. I have tried setting RecsMax to 500, which works in limiting the fetched records, but RecordCount for the query shows only 500 not the total.
The FireDAC help file states that setting RecordCountMode to `cmTotal' causes FireDAC to issue
SELECT COUNT(*) FROM (original SQL command text).
Either there is a bug or I am doing something wrong!
I cannot see what other properties I can change. I am confused as to the relationship between RowSetSize and RecsMax and din't find the help file clarified.
I have tried playing with the properties of AutoFetchAll (Again confused as to this properties' purpose), but noting that is was set to afAll I set it to afTruncate to see if that would make a difference, but it didn't.
I have tested FetchOptions' fmOnDemand Mode with a FDTable component and a FDQuery component. Both with identical settings for FetchOptions ie RowSetSize=50. 425,000 rows in the dataset fetched over a network server.
FDTable performs as expected. It loads just 50 tuples and does so almost instantly. When pressing Ctrl+End to get to the end of the DBGrid display, it loads just 100 tuples. Whilst scrolling it rarely loads more than 100 tuples. Impact on memory negligible. But it is slow in scrolling.
FDQuery loads 50 tuples, but takes around 35 seconds to do so and consumes over 0.5GB of memory in the process. If you press Ctrl+Home to move to the end of the connected DBGrid it does so virtually instantly and in the process loads the entire table and consumes a further 700MB of memory.
I also experimented with CachedUpdates. The results above where with CachedUpdates off. When on, there was no impact at all on the performance of FDQuery (still poor), but for FDTable it resulted in it loading the entire table at start up, taking over half a minute to do so and consuming 1.2GBs of memory.
It looks like fmOnDemand mode is only practically usable with FDTable with CachedUpdates off and is not suitable for use with FDQuery at all.
The results of my tests using fmOnDemand with postgreSQL and MySQL are basically the same. With FDTable fmOnDemand only downloads what it needs limited to the RowSetSize. With a RowSetSize of 50 it initially downloads 50 tuples and no matter where you scroll to it never downloads more than 111 tuples (though doubtless that is dependent on the size of the connected DBGrid. If you disconnect the FDTable from a data source it initially downloads 50 tuples and if you then navigate to any record in the underlying table it downloads one tuple only and discards all other data.
FDQuery in fmOnDemand downloads only the initial 50 tuples when opened, but if you navigate by RecNo it downloads every tuple in between. I had rather hoped it would use LIMIT and OFFSET commands to get only records that were being requested.
To recreate the test for PostGre you need the following FireDAC components:
object FDConnectionPG: TFDConnection
Params.Strings = (
'Password='
'Server='
'Port='
'DriverID=PG')
ResourceOptions.AssignedValues = [rvAutoReconnect]
ResourceOptions.AutoReconnect = True
end
object FDQueryPG: TFDQuery
Connection = FDConnectionPG
FetchOptions.AssignedValues = [evMode, evRowsetSize]
end
object FDTable1: TFDTable
CachedUpdates = True
Connection = FDConnectionPG
FetchOptions.AssignedValues = [evMode, evRowsetSize, evRecordCountMode]
FetchOptions.RecordCountMode = cmFetched
end
If you wish to recreate it with MYSQL, you will basically need the same FireDAC components, but the FDConnectionneeds to be set as follows:
object FDConnectionMySql: TFDConnection
Params.Strings = (
'DriverID=MySQL'
'ResultMode=Use')
ResourceOptions.AssignedValues = [rvAutoReconnect]
ResourceOptions.AutoReconnect = True
end
You'll need an edit box, two buttons, a checkbox, a timer and a label and the following code:
procedure TfrmMain.Button1Click(Sender: TObject);
begin
if not FDQueryPG.IsEmpty then
begin
FDQueryPG.EmptyDataSet;
FDQueryPG.ClearDetails;
FDQueryPG.Close;
end;
if not FDTable1.IsEmpty then
begin
FDTAble1.EmptyDataSet;
FDTable1.ClearDetails;
FDTable1.Close;
end;
lFetched.Caption := 'Fetched 0';
lFetched.Update;
if cbTable.checked then
begin
FDTable1.TableName := '[TABLENAME]';
FDTable1.Open();
lFetched.Caption := 'Fetched '+ FDTable1.Table.Rows.Count.ToString;
end
else
begin
FDQueryPG.SQL.Text := 'Select * from [TABLENAME]';
FDQueryPG.open;
lFetched.Caption := 'Fetched '+ FDQueryPG.Table.Rows.Count.ToString;
end;
timer1.Enabled:=true;
end;
procedure TfrmMain.Button2Click(Sender: TObject);
begin
if cbTable.Checked then
FDTable1.RecNo := strToInt(Edit1.Text)
else
FDQueryPG.RecNo := strToInt(Edit1.Text);
end;
procedure TfrmMain.cbTableClick(Sender: TObject);
begin
timer1.Enabled := False;
end;
procedure TfrmMain.Timer1Timer(Sender: TObject);
begin
if cbTable.checked then
lFetched.Caption := 'Fetched '+ FDTable1.Table.Rows.Count.ToString
else
lFetched.Caption:='Fetched '+FDQueryPG.Table.Rows.Count.ToString;
lFetched.Update;
end;
I developed a procedure that receives a TStream; but the basic type, to allow the sending of all the types of stream heirs.
This procedure is intended to create one thread to each core, or multiple threads. Each thread will perform detailed analysis of stream data (read-only), and as Pascal classes are assigned by reference, and never by value, there will be a collision of threads, since the reading position is intercalará.
To fix this, I want the procedure do all the work to double the last TStream in memory, allocating it a new variable. This way I can duplicate the TStream in sufficient numbers so that each thread has a unique TStream. After the end of the very thread library memory.
Note: the procedure is within a DLL, the thread works.
Note 2: The goal is that the procedure to do all the necessary service, ie without the intervention of code that calls; You could easily pass an Array of TStream, rather than just a TStream. But I do not want it! The aim is that the service is provided entirely by the procedure.
Do you have any idea how to do this?
Thank you.
Addition:
I had a low-level idea, but my knowledge in Pascal is limited.
Identify the object's address in memory, and its size.
create a new address in memory with the same size as the original object.
copy the entire contents (raw) object to this new address.
I create a pointer to TStream that point to this new address in memory.
This would work, or is stupid?? If yes, how to operate? Example Please!
2º Addition:
Just as an example, suppose the program perform brute force attacks on encrypted streams (just an example, because it is not applicable):
Scene: A 30GB file in a CPU with 8 cores:
1º - TMemoryStream:
Create 8 TMemoryStream and copy the entire contents of the file for each of TMemoryStreams. This will result in 240GB RAM in use simultaneously. I consider this broken idea. In addition it would increase the processing time to the point of fastest not use multithreading. I would have to read the entire file into memory, and then loaded, begin to analyze it. Broke!
* A bad alternative to TMemoryStream is to copy the file slowly to TMemoryStream in lots of 100MB / core (800MB), not to occupy the memory. So each thread looks only 100MB, frees the memory until you complete the entire file. But the problem is that it would require Synchronize() function in DLL, which we know does not work out as I open question in Synchronize () DLL freezes without errors and crashes
2º - TFileStream:
This is worse in my opinion. See, I get a TStream, create 8 TFileStream and copy all the 30GB for each TFileStream. That sucks because occupy 240GB on disk, which is a high value, even to HDD. The read and write time (copy) in HD will make the implementation of multithreaded turns out to be more time consuming than a single thread. Broke!
Conclusion: The two approaches above require use synchronize() to queue each thread to read the file. Therefore, the threads are not operating simultaneously, even on a multicore CPU. I know that even if he could simultaneous access to the file (directly creating several TFileStream), the operating system still enfileiraria threads to read the file one at a time, because the HDD is not truly thread-safe, he can not read two data at the same time . This is a physical limitation of the HDD! However, the queuing management of OS is much more effective and decrease the latent bottleneck efficiently, unlike if I implement manually synchronize(). This justifies my idea to clone TStream, would leave with S.O. all the working to manage file access queue; without any intervention - and I know he will do it better than me.
Example
In the above example, I want 8 Threads analyze differently and simultaneously the same Stream, knowing that the threads do not know what kind of Stream provided, it can be a file Stream, a stream from the Internet, or even a small TStringStream . The main program will create only one Strean, and will with configuration parameters. A simple example:
TModeForceBrute = (M1, M2, M3, M4, M5...)
TModesFB = set of TModeForceBrute;
TService = record
stream: TStream;
modes: array of TModesFB;
end;
For example, it should be possible to analyze only the Stream M1, M2 only, or both [M1, M2]. The TModesFB composition changes the way the stream is analyzed.
Each item in the array "modes", which functions as a task list, will be processed by a different thread. An example of a task list (JSON representation):
{
Stream: MyTstream,
modes: [
[M1, m5],
[M1],
[M5, m2],
[M5, m2, m4, m3],
[M1, m1, m3]
]
}
Note: In analyzer [m1] + [m2] <> [m1, m2].
In Program:
function analysis(Task: TService; maxCores: integer): TMyResultType; external 'mydll.dll';
In DLL:
// Basic, simple and fasted Exemple! May contain syntax errors or logical.
function analysis(Task: TService; maxCores: integer): TMyResultType;
var
i, processors : integer;
begin
processors := getCPUCount();
if (maxCores < processors) and (maxCores > 0) then
processors := maxCores;
setlength (globalThreads, processors);
for i := 0 to processors - 1 do
// It is obvious that the counter modes in the original is not the same counter processors.
if i < length(Task.modes) then begin
globalThreads[i] := TAnalusysThread.create(true, Task.stream, Task.modes[i])
globalThreads[i].start();
end;
[...]
end;
Note: With a single thread the program works beautifully, with no known errors.
I want each thread to take care of a type of analysis, and I can not use Synchronize() in DLL. Understand? There is adequate and clean solution?
Cloning a stream is code like this:
streamdest:=TMemoryStream.create;
streamsrc.position:=0;
streamdest.copyfrom(streamdest);
streamsrc.position:=0;
streamdest.position:=0;
However doing things over DLL borders is hard, since the DLL has an own copy of libraries and library state. This is currently not recommended.
I'm answering my question, because I figured that no one had a really good solution. Perhaps because there is none!
So I adapted the idea of Marco van de Voort and Ken White, for a solution that works using TMemoryStream with partial load in memory batch 50MB, using TRTLCriticalSection for synchronization.
The solution also contains the same drawbacks mentioned in addition 2; are they:
Queuing access to HDD is the responsibility of my program and not of the operating system;
A single thread carries twice the same data in memory.
Depending on the processor speed, it may be that the thread analyze well the fast 50MB of memory; On the other hand, to load memory can be very slow. That would make the use of multiple threads are run sequentially, losing the advantage of using multithreaded, because every thread are congested access to the file, running sequentially as if they were a single thread.
So I consider this solution a dirty solution. But for now it works!
Below I give a simple example. This means that this adaptation may contain obvious errors of logic and / or syntax. But it is enough to demonstrate.
Using the same example of the issue, instead of passing a current to the "analysis" is passed a pointer to the process. This procedure is responsible for making the reading of the stream batch 50MB in sync.
Both DLL and Program:
TLotLoadStream = function (var toStm: TMemoryStream; lot, id: integer): int64 of object;
TModeForceBrute = (M1, M2, M3, M4, M5...)
TModesFB = set of TModeForceBrute;
TaskTService = record
reader: TLotLoadStream; {changes here <<<<<<< }
modes: array of TModesFB;
end;
In Program:
type
{ another code here }
TForm1 = class(TForm)
{ another code here }
CS : TRTLCriticalSection;
stream: TFileStream;
function MyReader(var toStm: TMemoryStream; lot: integer): int64 of object;
{ another code here }
end;
function analysis(Task: TService; maxCores: integer): TMyResultType; external 'mydll.dll';
{ another code here }
implementation
{ another code here }
function TForm1.MyReader(var toStm: TMemoryStream; lot: integer): int64 of object;
const
lotSize = (1024*1024) * 50; // 50MB
var
ler: int64;
begin
result := -1;
{
MUST BE PERFORMED PREVIOUSLY - FOR EXAMPLE IN TForm1.create()
InitCriticalSection (self.CriticalSection);
}
toStm.Clear;
ler := 0;
{ ENTERING IN CRITICAL SESSION }
EnterCriticalSection(self.CS);
{ POSITIONING IN LOT OF BEGIN}
self.streamSeek(lot * lotSize, soBeginning);
if (lot = 0) and (lotSize >= self.stream.size) then
ler := self.stream.size
else
if self.stream.Size >= (lotSize + (lot * lotSize)) THEN
ler := lotSize
else
ler := (self.stream.Size) - self.stream.Position; // stream inicia em 0?
{ COPYNG }
if (ler > 0) then
toStm.CopyFrom(self.stream, ler);
{ LEAVING THE CRITICAL SECTION }
LeaveCriticalSection(self.CS);
result := ler;
end;
In DLL:
{ another code here }
// Basic, simple and fasted Exemple! May contain syntax errors or logical.
function analysis(Task: TService; maxCores: integer): TMyResultType;
var
i, processors : integer;
begin
processors := getCPUCount();
if (maxCores < processors) and (maxCores > 0) then
processors := maxCores;
setlength (globalThreads, processors);
for i := 0 to processors - 1 do
// It is obvious that the counter modes in the original is not the same counter processors.
if i < length(Task.modes) then begin
globalThreads[i] := TAnalusysThread.create(true, Task.reader, Task.modes[i])
globalThreads[i].start();
end;
{ another code here }
end;
In DLL Thread Class:
type
{ another code here }
MyThreadAnalysis = class(TThread)
{ another code here }
reader: TLotLoadStream;
procedure Execute;
{ another code here }
end;
{ another code here }
implementation
{ another code here }
procedure MyThreadAnalysis.Execute;
var
Stream: TMemoryStream;
lot: integer;
{My analyzer already all written using buff, the job of rewriting it is too large, then it is so, two readings, two loads in memory, as I already mentioned in the question!}
buf: array[1..$F000] of byte; // 60K
begin
lot := 0;
Stream := TMemoryStream.Create;
self.reader(stream, lot);
while (assigned(Stream)) and (Stream <> nil) and (Stream.Size > 0) then begin
Stream.Seek(0, soBeginning);
{ 2º loading to memory buf }
while (Stream.Position < Stream.Size) do begin
n := Stream.read(buf, sizeof(buf));
{ MY CODE HERE }
end;
inc(lot);
self.reader(stream, lot, integer(Pchar(name)));
end;
end;
So as seen this is a stopgap solution. I still hope to find a clean solution that allows me to double the flow controller in such a way that access to data is the operating system's responsibility and not my program.
The pagesize of the file I read is 32768. When i set the JET_paramDatabasePageSize to 32768,JetInit returns -1213.Then,i set the JET_paramRecovery to "Off",JetInit succeeds.But,when I use JetAttachDatabase,it returns -550.
Here is my code:
err=JetSetSystemParameter(&instance,sesid,JET_paramDatabasePageSize ,32768 ,NULL);
err=JetCreateInstance(&instance,NULL);
err=JetSetSystemParameter(&instance,sesid,JET_paramRecovery,0,"Off");
err=JetInit(&instance);
err=JetBeginSession(instance,&sesid,NULL,NULL);
err=JetAttachDatabase(sesid,buffer, JET_bitDbReadOnly );
err=JetOpenDatabase ( sesid, buffer, NULL, &dbid, JET_bitDbReadOnly );
What's wrong with it？I am running a Windows 7 32bit.
The page size is global to the process (NOT just the instance) and is persisted in the log files and the database, so changing the page size can be annoyingly tricky.
Is there information in the database that you're trying to access? Or did you just experience this during development?
If you saw this during development, then the easiest thing to do is to blow everything away (del .edb edb) [Assuming that you kept the prefix as "edb"].
Also, are you sure that the database is 32k pages? You can confirm with esentutl.exe -mh <database-name>.
It will be trickier to recover the data if you do care about, and you switched the page size. (I don't know off the top of my head, and I'd have to try a few things out...)
-martin
I am writing an app taking advantage of Lua Socket lib. Here is a simple code snippet that describes my problem.
local com=require("socket");
local socket=com.tcp();
local hello="hi stack overflow";
local myIP="192.168.1.1";
local myPort = 2000;
local err = nil;
-- Main
while 1
if(~err) then
err = socket:send(hello);
else
if(socket:connect("myIP", myPort))) then
err = 1;
end -- second if
end -- first if
waitfor(10); -- wait for 10 sec.
do
(I actually didn't run this particular code but it is identical to the running code in my problem).
When I see that if socket is closed, this code cannot reopen it. I would have guessed once we grab the master TCP object we can open and close as we please.
I can force this code to work by repeating the socket.tcp() call however I suspect that leaves the previous object somewhere in memory and I want to avoid this.
Well, things that are wrong in your code (starting from the top)
It's a bad habbit reusing default module names like socket, it'll confuse people reading your code
While loop syntax is while <condition> do <statements> end
setting err to nil is useless in this case, as it does not exist in the snippet scope
it makes no sense to send to an unconnected master.
negation is done with not instead of ~
you pass the string "myIP" instead of the actual IP
then there's also a function socket.sleep() you might be interested in.
I'm wondering why you complicate the loop this way, and don't just use something like:
require("socket");
local soc=socket.tcp();
local hello="hi stack overflow";
local myIP="192.168.1.1";
local myPort = 2000;
local stat,err=socket:connect(myIP, myPort)
if not stat then
error(err)
else
-- Main
while not err do
err = socket:send(hello)
socket.sleep(10)
end
end
The connect call will never work as you are passing the string "myIP" not the variable myIP which contains your target Ip address.
I am trying to read into some existing and unmounted esent database files (like Windows.edb). I have been playing around with some edb files rather successfully. But when I try to open a database with PageSize that is not equal 8192 I get an error.
Here's my code (without error-handling):
FError := JetSetSystemParameter(#FInstance, nil, JET_paramDatabasePageSize, FPagesize, nil);
FError := JetCreateInstance(#FInstance, 'EDBInstance');
FError := JetInit(#FInstance);
FError := JetBeginSession(FInstance, #FSessionId, nil, nil);
FError := JetAttachDatabase(FSessionId, FFilename, JET_bitDbReadOnly);
It works fine as long as FPageSize = 8192. Any other value (4096, 32768) fails at the JetInit call which returns an -1213 code. If I don't set the proper PageSize value for the database I get the same error at JetAttachDatabase, which I can understand. But the first error that gets returned by JetInit I fail to comprehend. What do I do wrong? I hope Laurion Burchall is reading this! :-)
I am running a Windows 7 64bit.
There are two possibilities:
The database you are trying to open has an 8Kb page size. Use ESENTUTL /M database to see the page size.
The page size is always persisted in the logfiles, which are created by the JetInit call. If you don't clear out those files between runs then you will get a -1213 error when calling JetInit with a different page size.
If you want to open an existing database in a read only way then you should turn recovery off (set JET_paramRecovery to "off"). That will prevent any logfiles from being generated which will avoid a lot of problems.