GmMemoryDump Class Reference

Class storing a memory dump containing several dump items. More...

#include <gmMemoryDump.h>

Collaboration diagram for GmMemoryDump:

Classes
struct	DumpItem
	The structure adding controll information to a GmMemoryDumpItem. More...
struct	FileControlHeader
	The file structure with file control info. More...
struct	FileDataChunkHeader
	The file header for each data chunk. More...
struct	FileInfoBlockHeader
	The structure storing control info for an Info block. More...
struct	FileItemHeader
	The file header for every data item. More...
struct	ItemHeader
	The in memory control header for every data item. More...

Public Member Functions
	GmMemoryDump (GmMemoryDumpMode mode, const GmLogCategory &logger, int compression=0, int compressionMinSize=GM_MEMORY_DUMP_MIN_COMPRESSION, double dumpSizeA=1.0, unsigned dumpSizeB=0)
	Constructor. Defines the adopted dump mode and additional control options. Must be followed by calls to addDumpItem() and a call to init(). More...
virtual	~GmMemoryDump ()
	Destructor. Deletes the dump file if keep was false in the call to init()
void	setPrintStructure (bool mode)
	Enables or disables printing the dump item structure, both when addDumpItem() is called and when a file is initialy loaded.
int	addDumpItem (GmMemoryDumpItem *item, int groupId=-1)
GmMemoryDumpItem *	item (int itemId) const
	Returns the dump item, given the id returned by addDumpItem()
int	findItem (QString name, int typeMask=0, int typeValue=0, int groupId=-1, int start=0)
	Finds an item with the given name, type and group. More...
int	itemGroup (int itemId) const
	Returns the group id of the given item.
int	numItems () const
	Returns the number of added dump items.
void	setItemActive (int itemId, bool active)
	Marks an item as active / inactive. For use with saveActiveItems() / loadActiveItems()
void	setGroupActive (int groupId, bool active)
	Marks all items belonging to the given group as active / inactive. If groupId is -1, sets all items with the given mode. For use with saveActiveItems() / loadActiveItems().
bool	isItemActive (int itemId)
	Returns true if the given item is active.
bool	init (QString dumpFile, bool overwrite, bool keep)
	Initializes the dump object. Must be called after dump items where set by calls to addDumpItem(). Returns true on success, false on error. More...
bool	init ()
	init() overload when always dumping to memory.
bool	save ()
	Saves the contents of all the dump items to either the memory or to the dump file. Can apply compression / reserve grow space if so required on the constructor parameters. Replaces all the data on the dump file. Ignores the active/inactive item status.
bool	load ()
	Loads the contents of all the dump items from either the memory or the dump file. Ignores the active/inactive item status.
bool	saveItem (int itemId, qint64 size=0, qint64 offset=0)
	Saves the data from the given DumpItem only. Returns false on error. More...
bool	loadItem (int itemId, qint64 size=0, qint64 offset=0)
	Loads the data from the given DumpItem only. Returns false on error. More...
bool	saveActiveItems ()
	Saves the set of items that are marked as active. Equivalent to calling saveItem(i) for all i with active flag.
bool	loadActiveItems ()
	Loads the set of items that are marked as active. Equivalent to calling loadItem(i) for all i with active flag.
void	flush ()
	Flushes the dump file. Can be used when using saveItem() to make sure that the data was saved to the disk. Not needed when calling save()
void	clear ()
	Discards stored memory when dumping to memory.
int	adjustGroupId (int id) const
	Returns the adjusted group id, translating the -1 (next group) or -2 (current group) options for addDumpItem()

Static Public Member Functions
static const char *	modeToStr (GmMemoryDumpMode mode)
	Returns the string used to define the given matrix mode.
static int	strToMode (const QString &str)
	Returns the dump mode associated with the given string. Returns -1 if no match was found.

Private Member Functions
bool	fileInit (QString dumpFile, bool overwrite, bool keep)
	init() for file dumping mode. See init() for parameter description
bool	fileInitNew ()
	Writes the control structure to the new file.
bool	fileInitExisting ()
	Checks that the file control structure is compatible with current dump items and inits _dumpItems with header info.
bool	memoryInit ()
	init() for memory dumping mode.
void	fileClose ()
	Closes (and possibly removes) the dump file when dumpingto a file.
bool	fileSaveItem (DumpItem *item, bool update, qint64 size=0, qint64 offset=0)
	Saves the data from dump item to the dump file at the position pointed to by item->_data->_dataBlockPos. More...
bool	memorySaveItem (DumpItem *item, qint64 size=0, qint64 offset=0)
	Saves the (possibly partial) data from dump item to memory.
bool	fileLoadItem (DumpItem *item, qint64 size=0, qint64 offset=0)
	Loads the dump item data from the dump file. Loading can be partial if size > 0.
bool	memoryLoadItem (DumpItem *item, qint64 size=0, qint64 offset=0)
	Loads the dump item data from memory. Loading can be partial if size > 0.
bool	writeOrCheckControlHeader (bool write, qint64 *infoBlockPos)
	Aux function to either write the control header or read it from file and check magic numbers, returning the info block position. More...
bool	saveDataBlock (qint32 blockId, qint64 blockSize, const char *buffer, qint64 size)
	Saves a full data block in a single chunk at the current file position.
bool	saveDataChunck (const FileDataChunkHeader *h, const char *buffer, qint64 size)
	Saves a single data chunk at the current file position. The chunk size is given by the header. The buffer size is given by size. If the size is smaller than the chunk size, the remaining space is filled with zeros.
bool	saveChunkHeader (const FileDataChunkHeader *h)
	Saves the given chunk header.
bool	saveChunkPadding (const FileDataChunkHeader *h, qint64 dataSize)
	Saves the needed "zero" bytes to complete the chunk when the data size is less than the chunk size.
bool	loadChunkHeader (FileDataChunkHeader *h)
	Loads the data from a chunk header from the current file position.
bool	loadChunkData (char *buffer, qint64 offset, qint64 size)
	Loads size bytes from the given offset at the current chunk. The chunk data should start at the current file position (not the header).
bool	updateFileInfoHeader ()
	Updates the file info header with information from _lastChunkBlockId and number of items.
bool	updateFileItemHeader (const ItemHeader *h, const GmMemoryDumpItem *item)
	Updates the contents of the FileItemHeader structure at headerPos.
bool	readData (char *buffer, qint64 size)
	Read size bytes from the dump file, filling buffer. Returns false on error (plus emmits log message)
bool	readData (GmMemoryBufferList &blist, qint64 size)
	Read size bytes from the dump file, filling buffer. Returns false on error (plus emmits log message)
bool	writeData (const char *buffer, qint64 size)
	Write size bytes from buffer to the dump file. Returns false on error (plus emmits log message)
bool	writeData (GmMemoryBufferList &blist, qint64 size)
	Write size bytes from buffer to the dump file. Returns false on error (plus emmits log message)
qint64	blockSize (qint64 dataSize, bool grow)
	Returns the block size given the stored data size, accounting for extra grow space (if grow is true) and aligning to 16 bytes.

Private Attributes
GmMemoryDumpMode	_mode
	Dump mode.
int	_compression
	Compression level for dumped data. From -1 to 9. 0 = no, 9 = max, -1 = zlib default.
int	_minCompressionSize
	The minumum size a buffer must have to be compressed.
double	_dumpSizeA
	Multiplier for extra grow size added to a data item size.
unsigned	_dumpSizeB
	Offset for extra grow size added to a data item size.
QString	_fileName
	Dump file name.
bool	_keepFile
	Should we keep dump files after the object is destroyed?
bool	_printStructure
	Should we print the configured item structure?
QFile *	_file
	The dump file (if the current mode so requires)
bool	_init
	Check if init was called successfully.
qint64	_fileInfoPos
	The position on the file for the Info block header.
qint64	_firstChunkPos
	The position in the dump file for the first dump item chunk.
qint32	_lastChunkBlockId
	The block Id for the last chunk stored in the file.
QVector< DumpItem * >	_dumpItems
	A list with the set of dump items + their controll info.
int	_currentGroupId
	The id of the last added group id.
int	_nextGroupId
	The id of the next item group id (can be different from _currentGroupId+1 if the user explicitly gave a group id on addDumpItem())
const GmLogCategory &	_logger
	Basic logger object for dump messages.

Detailed Description

Class storing a memory dump containing several dump items.

A memory dump object is responsible for storing the contents provided by GmMemoryDumpItems to either disk or to another place in memory. It sees the contents as simple byte blocks, without any semantics.

The set of dump items is fixed and must be configured by calls to addDumpItem() before a call to init() prepares the memory dump object to be used.

Supported operations are to either save or load the full set of dump items. When saving, the dump items are queried to provide the byte blocks to save, and when loading, the recovered byte blocks are sent to the dump items.

There is also support for saving and loading individual dump items or groups of items. When working with individual items, the operation can specify that only a part of the dump item data be saved or loaded.

The dump item interface for getting / setting data is optimized for avoiding intermediate copies of the data.

When loading data from a file saved by a previous execution of GeMA, the set of dump items configured on the memory dump object MUST be the SAME as the one configured when the dump file was saved. In the current implementation, that means that the number of items should be equal and each item should have exactly the same name, type and group. They should also be in the same relative order. As an EXCEPTION, if the file will be used only for loading the data (restore mode), NO UPDATES, it is possible to operate with a partial set of dump items. In that setting, tailored for recovering mesh data before the orchestration, we still expect that the partial set is equivalent and in the same order as part of the dump file configuration. Some basic sanity checks are done when loading a dump file.

The file layout is quite simple and is intended for saving and recovering data on the SAME machine, with the SAME version of GeMA, so it does NOT take extra steps to handle endianness or structure alignment changes from different compilers.

The file starts with a FileControlHeader structure that stores magic numbers and version information for minimum sanity checks along with the location of the "Info block" (currently after this header, but might be moved to the end of the file in the future if we ever decide to support changing the set of saved dump items) and other relevant control data.

Other information is stored in data blocks. Each data block can be split in several non-continuous data chunks. This allows for a dump item to grow without the need of reorganizing the whole file. If dump items never grow in size after being saved for the first time, or if save operations are always done over the full set of dump items (no individual item save operations are used), every data block will have a single data chunk.

The "Info block" is a single chunk data block that stores metadata and control information for the set of dump items stored in the file. Its contents is given by a FileInfoBlockHeader structure followed by a FileItemHeader structure for each dump item.

Each data chunk stores a header (FileDataChunkHeader) followed by its data contents. The header identifies to which data block the chunk belongs along with the position on the file for the next chunk in this block, forming a "linked list" with the block chunks. The chunk can have a size that is greater than the stored data size. This is used for both chunk alignment (16 bytes) and to provide a configurable growing space for dump items (so that a dump item can grow without the need of creating another chunk).

A call to the save() operation always replaces the complete set of file data, so it never creates blocks with multiple chunks and suffers no consequences if the dump item sizes are changed (either growing or shrinking). Calls to saveItem(), on the other hand will update the file with the given data saving it on top of the old one (unless at the first call). If the dump item size has grown, a new data chunk is added if there is no space available on the last chunk of the dump item block. If the size has shrinked, data chunks might be left unused on the file.

To prevent inefficiency due to adding multiple chunks to a dump item, the Memory dump object constructor receives as parameters the A and B coefficients of a linear equation that is used to find the file block size when saving a dump item for the first time or when growing is needed: fileBlockSize = A * dumpItemSize + B. Another alternative is to use one Memory Dump object per dump item. In that case, being the only dump item in the file allows for growing a chunk.

Stored data can be optionally compressed with the standard qCompress() call (uses zlib). In that case, extra memory must be allocated and the information can not be copied directly to and from the dump item. The desired compression level and the minimum buffer size for which compression is applied are parameters to the Memory Dump object contructor. If the dump file was loaded from an existing file (not created), and the desired compress level is different from the file level, the desired level will be applied as each data item is saved. The compression level can also be used when dumping to memory.

IMPORTANT: Compression can NOT be used TOGETHER with partial dump item load / save.

TODO:

• Multiple threads

Constructor & Destructor Documentation

GmMemoryDump()

GmMemoryDump::GmMemoryDump	(	GmMemoryDumpMode	mode,
		const GmLogCategory &	logger,
		int	compression = 0,
		int	compressionMinSize = GM_MEMORY_DUMP_MIN_COMPRESSION,
		double	dumpSizeA = 1.0,
		unsigned	dumpSizeB = 0
	)

Constructor. Defines the adopted dump mode and additional control options. Must be followed by calls to addDumpItem() and a call to init().

Parameters

mode	The dump mode. Controls if dumps go to memory or to a dump file.
logger	The logger used for error messages
compression	Should we enable data compression? A value of 0 means no. A value between 1 and 9 controls the compression level (9 = maximum compression). A value of -1 means compression with the default zlib compression level. See comments on the class description.
compressionMinSize	The minimum size that the dump item must have to be compressed. Must be > 0.
dumpSizeA,dumpSizeB	The "A * size + B" coefficients used for defining a chunk size when dump items are exected to grow in size. See comments on the class description.

Member Function Documentation

fileSaveItem()

bool GmMemoryDump::fileSaveItem ( DumpItem *  item, bool  update, qint64  size = 0, qint64  offset = 0  )

private

Saves the data from dump item to the dump file at the position pointed to by item->_data->_dataBlockPos.

If update is false the data will be saved in a single block, while when update is true, it will be saved over the existing set of data chunks for that item, adding one new chunk if necessary.

If size is greater than 0, a partial update is done with the specified sub-block. Although an update, this can extend the item size, but in that case it MUST span all the new part of the data.

After a call to fileSaveItem the whole dump file is kept consistent with all control headers updated as needed. In memory control variables are also updated.

findItem()

int GmMemoryDump::findItem	(	QString	name,
		int	typeMask = 0,
		int	typeValue = 0,
		int	groupId = -1,
		int	start = 0
	)

Finds an item with the given name, type and group.

The search string CAN be a regular expression understood by QRegExp.

The item type is anded with typeMask and the result must be equal to typeValue, so a mask of 0 with a value of 0 accepts any type. A mask of 0xFF000000 with a value of GM_MESH_DUMP_ITEM accepts any mesh, independently of its remaining type bits. A mask of 0xFFFFFFFF (-1) accepts the exact typeValue given.

The item group must be equal to groupId, unless groupId is -1, which accepts any group.

Returns the itemId of the first item that matches the given criteria or -1 if not found, starting at index start (returned value >= start).

init()

bool GmMemoryDump::init	(	QString	dumpFile,
		bool	overwrite,
		bool	keep
	)

Initializes the dump object. Must be called after dump items where set by calls to addDumpItem(). Returns true on success, false on error.

Can be called multiple times to reinitialize the dump object with the same structure to a different file.

Parameters

dumpFile	The name of the dump file that will be used. Can be empty if always dumping to memory
overwrite	When true, if the dumpFile already exists, it will be immediately overwritten and its data discarded. If false, if the file exists it MUST share the same structure as definied in this object (see the exceptions for RESTORE mode on the class documentation). If not the init() will fail. Overwrite MUST be set to false for reading data saved on previous GeMA executions, so it MUST be false in RESTORE mode.
keep	When true, the dump file will not be deleted when this object is deleted.

Returns

Returns true on success, false on errors.

loadItem()

bool GmMemoryDump::loadItem	(	int	itemId,
		qint64	size = 0,
		qint64	offset = 0
	)

Loads the data from the given DumpItem only. Returns false on error.

The item id is the value returned by the call to addDumpItem(). If size is diferent from zero, only part of the item data will be loaded (defined by the given size and offset).

If the dump item does not supports partial operation (partialOp() returns false) and size / offset are different from 0, the operation fails.

Ignores the active/inactive item status.

saveItem()

bool GmMemoryDump::saveItem	(	int	itemId,
		qint64	size = 0,
		qint64	offset = 0
	)

Saves the data from the given DumpItem only. Returns false on error.

The item id is the value returned by the call to addDumpItem(). If size is diferent from zero, only part of the item data will be saved (defined by the given size and offset). Although an update, this can extend the item size, but in that case it MUST span all the new part of the data.

If the dump item does not supports partial operation (partialOp() returns false) and size / offset are different from 0, the operation fails.

Ignores the active/inactive item status.

writeOrCheckControlHeader()

bool GmMemoryDump::writeOrCheckControlHeader ( bool  write, qint64 *  infoBlockPos  )

private

Aux function to either write the control header or read it from file and check magic numbers, returning the info block position.

Expects that the file pointer is on the begining of the file

---

The documentation for this class was generated from the following files:

• inc/gmMemoryDump.h
• src/gmMemoryDump.cpp