Feeding Data

So, during my time as a Data Scientist, I have come to learn different ways to feed data to a program. sometimes you have to feed an image to a program and sometimes labels, but most of data is always fed as arrays.

Based on your data, sometimes files have to be saved/loaded as 'numpy' arrays from files:

X = np.load('./data/X_train.npy')
y = np.load('./data/Y_train.npy')

If you have arrays that can be loaded into the memory, then you might as well load the arrray from memory instead of a file:

X_train, y_train = X[:8000], y[:8000]
X_test, y_test = X[8000:], y[8000:]

Other than that large data can be saved and accessed in HDF5 formats:

hdf5_path = '/home/a7md/Data/dataset.hdf5'
hdf5_file = h5py.File(hdf5_path, "r")
images = hdf5_file["train_img"]
labels = hdf5_file["train_labels"]