I want to run a machine learning task on my Win 10 PC, 16GB RAM, Mathematica 11.3.0, but I am facing the following problems: training set size 10GB CSV file, with 700,000,000 x 2 datasets. Mathematica simply stops during import via Import or ReadList function. My idea is to split the input file into…

I uploaded my data to Mysql and than just created views in MySQL and read them from Mathematica

Good hint, thank you. I am currently trying to import the data into PostgreSQL. This approach means that the Predict function or a neural network would get the Data in a stream with the consequence that not all data could be available in one moment of time (due to limited RAM). Is it possible for …

I don’t know, but I know Mathematica can take advantage of Hadoop and MapReduce.

That is very useful! Thank you very much. It seems that I can solve the problem with a database based stream of data and a neural network. I will make a try.

I suggest to use a generating function for training of large data sets, as it is described here Training on large data sets

Another possibility is to use a Mongo database — as it is described in the same link I gave above.

I am sorry for late response. Well, I have to admit that I am not a MongoDB expert. It is not possible for me to import the entire Csv file in Mongo. The import stops after ca. 1% (5,5 * 10^6) datasets. Now I am trying to parse CSV to Json, and I hope the Mongo import will lead to success with Json. …

I suggest to use a generating function for training of large data sets, as it is described here Arthrozene review I actually just made this account to say thanks for posting this - it really helped me and I thought it deserved some recognition!

Import large data files for Machine Learning?

Questions

rohitnamjoshi (Rohit Namjoshi) January 26, 2019, 9:27pm 5

Is it possible for Mathematica to build a ML model based on a stream
of data?

See my answer to this question .

Topic		Replies	Views
How to import a Dataset ? Questions data-science , import-and-export	4	13844	February 10, 2015
Use NetTrain[net,f] to read in a large set of rules (Is there an example?) Questions data-science , machine-learning , neural-networks	5	9907	December 5, 2018
Import Large Text Files? Questions mathematica , import-and-export	2	7973	January 18, 2015
Problem with big input not accepted Questions high-performance-computing , import-and-export , numerical-computation	11	14705	September 26, 2013
Importing big files (ImportSequential) Ideas computer-science , data-science , education , software-development , mathematica , import-and-export	5	10784	November 28, 2017

Import large data files for Machine Learning?

Related topics