Documentation
2.3 Ingesting Data
Momentum allows ingesting data from a wide variety of sources and formats. The process of ingesting data from different sources is the same, except that the configuration fields vary for different sources.
Though the following steps are for ingesting CSV data, the process remains the same for all other data formats and sources.
- Expand “Ingester” from the left navigation panel, and then click “Ingster Home”.
- Click “Ingest Data” located at the top menu option.
- Select “Delimited File” for CSV upload. Select appropriate ingester type based on the data format or system you need to ingest data from. See Figure 2.4 below.
Figure 2.4: Screenshot showing ingester type selection
- Fill out the form on the next screen.
- Click “Browse Data Files” to launch the screen to select the files to be ingested.
- To select the files, expand the uploaded files, click the directory name, and select the files you want to ingest. If you want to ingest all files from the directory, simply close the popup window without selecting individual files.
- Enter asterisk or * in the file pattern field, and comma “,” in the delimiter field. If the data delimiter is different than comma, enter that.
- Enter max core for parallel ingestion and depending on the data size and number of CPU cores available. For example, 4 cores.
- Submit the form to save the ingester.
- In the next page, select the ingester you just created by clicking the checkbox and click “Run” located at the top menu.
- Click Ingester located at the top to refresh the ingester page to see the running status
- Click “Logs” link corresponding to the ingester to see the running logs – stdout and stderr.
Important Note
- For a file based ingester, the data from the source directory is moved to an archive directory. In other words, after each ingester run, the source directory is emptied.
- For database or external system based ingester, the source data is not moved.