Skip to contents

Rfmalloc: Memory-Mapped File Allocation for R

Source: R/Rfmalloc-package.R
Rfmalloc-package.Rd

Rfmalloc provides experimental memory-mapped file allocation capabilities for R using a patched copy of the fmalloc library. The current package exposes ALTREP file-backed vector allocation for logical, integer, numeric, raw, complex, character, and list vectors with fmalloc payload storage.

Main Functions

open_fmalloc

Open an explicit fmalloc runtime handle.

init_fmalloc

Open and install a default fmalloc runtime.

create_fmalloc_vector

Create vectors using fmalloc.

create_fmalloc_matrix

Create matrix-shaped fmalloc vectors.

create_fmalloc_array

Create array-shaped fmalloc vectors.

create_fmalloc_data_frame

Create data.frames from fmalloc-backed columns.

as_fmalloc_matrix

Convert fmalloc vectors to matrix-shaped objects.

as_fmalloc_array

Convert fmalloc vectors to array-shaped objects.

as_fmalloc_data_frame

Convert fmalloc-backed objects to a data.frame.

list_fmalloc_allocations

List persistent allocation catalog records.

diagnose_fmalloc_runtime

Summarize persistent allocation catalog state and runtime diagnostics.

cleanup_fmalloc

Request cleanup of an fmalloc runtime.

Current Scope

  • ALTREP file-backed allocation for logical, integer, numeric, raw, complex, character, and list vectors. List elements are restricted to NULL or Rfmalloc-backed vectors from the same runtime.

  • Large allocations spanning multiple fmalloc chunks.

  • Multiple runtime handles in one R process.

  • Persistent and scratch runtime modes.

  • Reference serialization for persistent fixed-width atomic and character ALTREP vectors.

  • Fmalloc-backed ALTREP subset copies for vector indexing operations.

  • An in-file allocation catalog for persistent vectors.

  • A C-callable API and installed header for other packages.

  • Native lifetime tracking so runtime mappings outlive reachable vectors allocated from them.

  • Runtime and catalog diagnostics for planning recovery and operational cleanup.

Known Limitations

  • ALTREP-backed dispatch now covers core Ops, Summary, Math, Math2, and matrix rowSums/colSums/rowMeans/colMeans workflows through S3 methods for common vector/matrix usage.

  • Explicit base-fallback boundaries are:

    • rowSums(), colSums(), rowMeans(), and colMeans() when the input is not an exact 2D matrix or dims != 1L; these cases now emit a warning and call the corresponding base:: reducer.

    • Scalar or zero-length results from Summary, Math, and Math2 generics (for example sum(x) returning a single value) are returned as ordinary R scalars by design.

  • Full operator- and method-family coverage is still incomplete for all R generics. Some advanced families may still materialize ordinary R objects in a few edge cases.

Future Work

Future work includes view-based subset representations, catalog compaction and reset tooling, metadata storage for attributes on persisted elements, robust nested-list reference validation, and compaction of recovery metadata.

See also

Useful links:

Author

Maintainer: Sounkou Mahamane Toure sounkoutoure@gmail.com

Other contributors:

  • Kenichi Yasukata (fmalloc) [copyright holder]

  • Wolfram Gloger (ptmalloc3) [copyright holder]

  • Free Software Foundation, Inc. (selected GNU C Library support files) [copyright holder]