The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
The filematrix
package provides functions to create and access large matrices stored in files, not computer memory. Filematrices can be as large as the storage allows. The package has been tested on matrices multiple terabytes in size.
File matrices can be created using functions fm.create
, fm.create.from.matrix
, or fm.create.from.text.file
:
library(filematrix)
fm = fm.create(filenamebase = "fmat", nrow = 200, ncol = 200, type = "double")
The code above creates two files: fmatrix.bmat
which stores the filematrix values, and fmatrix.desc.txt
which stores the filematrix description, such as dimensions, data type, and data type size.
Here is the content of the description file:
cat(readLines("fmat.desc.txt"), sep = "\n")
# Information file for R filematrix object
nrow=200
ncol=200
type=double
size=8
The elements of a filematrix can be read and written to using the same syntax as is used for regular R matrices. Any changes to a filematrix are written to the .bmat
file without extra buffering.
fm[1:3, 1:2] = 1:6
fm[1:4, 1:3]
## [,1] [,2] [,3]
## [1,] 1 4 0
## [2,] 2 5 0
## [3,] 3 6 0
## [4,] 0 0 0
colSums(fm[,1:4])
## [1] 6 15 0 0
Elements of a filematrix can also be accessed as elements of a vector (in which elements proceed sequentially down columns stacked 1:n). Thus, as fm
has nrow(fm)
rows, fm[1,2]
accesses the same element as fm[nrow(fm)+1]
.
fm[1:4]
## [1] 1 2 3 0
fm[nrow(fm)+1:4]
## [1] 4 5 6 0
File matrices can also have row and column names, like regular R matrices.
colnames(fm) = paste0("Col", 1:ncol(fm))
rownames(fm) = paste0("Row", 1:nrow(fm))
The row and column names of the filematrix fm
are stored in fmatrix.nmsrow.txt
and fmatrix.nmscol.txt
respectively.
An open filematrix object can be closed with close
function. This closes the internal file handle (connection). Closing filematrix objects is optional, changes would not be lost if the object is not closed.
close(fm)
An existing filematrix can be opened with fm.open
.
fm = fm.open(filenamebase = "fmat", readonly = FALSE)
To prevent any changes to the values of the filematrix set readonly = TRUE
.
An existing filematrix that would fit in memory can be loaded fully with fm.load
mat = fm.load("fmat")
The values of a filematrix are stored by columns, as with regular R matrices:
matrix(1:12, nrow = 3, ncol = 4)
## [,1] [,2] [,3] [,4]
## [1,] 1 4 7 10
## [2,] 2 5 8 11
## [3,] 3 6 9 12
Thus, access to a filematrix values by columns is much faster than access by rows:
timerow = system.time( { sum(fm[1:10, ]) } )[3]
timecol = system.time( { sum(fm[ ,1:10]) } )[3]
cat("Reading ", nrow(fm)*10, " values from 10 columns takes ", timecol, " seconds", "\n",
"Reading ", ncol(fm)*10, " values from 10 rows takes ", timerow, " seconds", "\n", sep = "")
## Reading 2000 values from 10 columns takes 0 seconds
## Reading 2000 values from 10 rows takes 0.02 seconds
The performance difference may not be observed on small matrices, as in this example. Change the size from 200 x 200 to 10,000 x 10,000 to see the difference (it is at least hundred fold).
Unlike with regular R matrices, columns can be appended to the right side of a filematrix with very little computational overhead.
dim(fm)
## [1] 200 200
fm$appendColumns(nrow(fm):1)
fm$appendColumns(1:nrow(fm))
dim(fm)
## [1] 200 202
fm[nrow(fm)+(-1:0), ncol(fm)+(-1:0)]
## [,1] [,2]
## [1,] 2 199
## [2,] 1 200
If you no longer need a filematrix and want to delete its files from the hard drive and close the object, please use closeAndDeleteFiles()
closeAndDeleteFiles(fm)
sessionInfo()
## R version 3.4.3 (2017-11-30)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 16299)
##
## Matrix products: default
##
## locale:
## [1] LC_COLLATE=C
## [2] LC_CTYPE=English_United States.1252
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United States.1252
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] filematrix_1.3 knitr_1.18
##
## loaded via a namespace (and not attached):
## [1] compiler_3.4.3 backports_1.1.2 magrittr_1.5 rprojroot_1.3-2
## [5] htmltools_0.3.6 tools_3.4.3 yaml_2.1.16 Rcpp_0.12.14
## [9] stringi_1.1.6 rmarkdown_1.8 stringr_1.2.0 digest_0.6.14
## [13] evaluate_0.10.1
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.