Documentation
< All Topics
Print

2.6 Emitter: Exporting Data to External System

Data created by Momentum components are stored on a distributed data lake. Momentum can efficiently access these data for any processing and analysis. However, there are use cases when the processed data from Momentum needs to be exported to a third-party external system. The emitter allows exporting data from Momentum to various external systems. To keep Momentum data warehouse, Impulse, as an independent system, it is also treated as an external system. In other words, data from the lake needs to be exported to Impulse like any other external system. 

This section explains how to configure an emitter that can be attached to a data pipeline (see Setting Up Data Pipeline section) to automate the data ingestion, transformation and export via emitter to external system. 

To configure an emitter: 

  1. Expand Emitter from the main menu option (left hand side menu panel) and click “Emitter Home”. 
  1. Click “Create New Emitter” located at the top menu bar and fill out the config form as explained below. 
  1. Emitter Name: a user defined unique name to identify your emitter. 
  1. Group Name: this field is used in case of IoT to group multiple devices so that their data is collected within the same group. For all other purposes, give any name. 
  1. Memory per Core: How much RAM for parallel process to be used. 1GB should be sufficient for most cases but should be given more for larger dataset. 
  1. Max Core: Number of CPU cores to be used for parallelism. 1 core for small data should be enough. 
  1. Storage Type: Select the appropriate external storage system where you want your data to be exported to. Depending on the type of the storage system, the form fields will different. In this example, the form fields of “Impulse” storage system is explained. 
  1. URL: Give the hostname of Impulse server. For security, give the local hostname as opposed to public DNS or IP address. Make sure the local hostname or IP address is accessible to Momentum server. 
  1. Port: The default port of Impulse is 18888. Use a different port of your admin has installed Impulse to listen on a different port. Make sure Momentum can access this port. 
  1. Warehouse Alias: You must obtain the warehouse alias. See Creating a Warehouse to create a warehouse and alias to be used in this field. The alias name must exist in the data warehouse. 
  1. Table Name: table name where you want to export data to. If the table name does not exist, it be created. 
  1. Username: the authorized username that has at least write permission to the data warehouse 
  1. Password: The authorized user’s password. 
  1. Primary Partition Column: This must be a datatime column. If there is no datetime column, enter __none__ . 
  1. Partition Column Datetime Format: Specify the datetime format. For example: YYYY-MM-DD hh:mm:sss. The datetime should be as specified by jodatime guidelines, https://www.joda.org/joda-time/key_format.html. If your dataset does not have a datetime column and you specified __none__ in the primary partition column field, enter __none__ here as well. 
  1. Partition Granularity: This field is to define the granularity level of your partition either by allnonesecondminutefifteen_minutethirty_minutehourdayweekmonthquarter or year. 
  1. Missing Datatime Placeholder: If there is no datetime column or the datetime value is null or invalid, the datetime will be replaced by the value provided in this field. Leave it default to replace the datetime to current datetime. 
  1. Save Mode: Select the appropriate mode as:  

Overwrite: delete all previous records and recreate new set of records 
Append: create new rows and append to existing dataset 
Combine and Overwrite: Combine new data with the old ones and then overwrite. 
Incremental: Update existing rows if matching primary key is found else create a new row. 

  1. Partition Overwrite Period Start Date: For SaveMode = Combine & Overwrite, specify the start date of the partition that you want to overwrite with the new data. Otherwise, leave it blank. 
  1. Partition Overwrite Period End Date: For SaveMode = Combine & Overwrite, specify the end date of the partition that you want to overwrite with the new data. Otherwise, leave it blank. 
  1. Status: Select active. 
  1. Click Submit to save the emitter config. 

Emitter does not independently run. It always run as a part of a pipeline. 

Creating emitter

Editing An Existing Emitter 

  1. Expand Emitter menu from the left menu options –> click “Emitter Home”  
  1. On the emitter home page, select (checkbox) the emitter you want to edit. 
  1. Click “Edit” located at the top on the emitter home page. 
  1. Edit the form fields as necessary and submit to save the changes. 
Table of Contents

Lester Firstenberger

Lester is recognized nationally as a regulatory attorney and expert in consumer finance, securitization, mortgage, and banking law.

Lester is recognized nationally as a regulatory attorney and expert in consumer finance, securitization, mortgage, and banking law. In a variety of capacities, over the past 30 years as an attorney, Mr. Firstenberger has represented the interests of numerous financial institutions in transactions valued in excess of one trillion dollars. He was appointed to and served a three-year term as a member of the Consumer Advisory Council of the Board of Governors of the Federal Reserve System. He has extensive governmental relations experience in the US and Canada at both the federal and state and provincial levels.

Shamshad (Sam) Ansari

Shamshad (Sam) Ansari is the founder, president and CEO of Accure. He drives technology innovations and works with a great team of engineers, data scientists, and business drivers at Accure.

Shamshad (Sam) Ansari is the founder, president, and CEO of Accure. He drives technology innovations and works with a great team of engineers, data scientists, and business drivers at Accure. He takes great pride in working with customers and putting together teams for solving their business problems. Sam is the product architect of Momentum, an AI and automation platform for data engineers, scientists, and business analysts.

Sam brings more than 20 years of technology development and management expertise. He developed, deployed and managed several large scale AI projects. He is a domain expert in healthcare systems, protocols, standards and compliances. Sam is a serial entrepreneur and worked with 4 startups. Prior to starting Accure, he worked with Apixio as the principal architect and director of engineering. He had another successful startup Orbit Solutions where he developed healthcare systems that went through an acquisition. He worked with IBM and the US Government at various capacities.

Sam is a distinguished data scientist, inventor and author. He has several technology publications in his name. He has co-authored 4 US Patents in healthcare AI. He is a well respected authority in computer vision and AI and has authored a book, “Building Computer Vision Applications Using Artificial Neural Networks” that is also translated into other languages including Chinese. Sam contributes to academia as well. He mentors graduate students and sponsors Capstone projects. He is also a member of the Advisory Board, Data Analytics Engineering Department at George Mason University.

Sam has a Master’s degree from Indian Institute of Information Technology & Management, Kerala (IIITM-K) and Bachelor’s degree in engineering from Bihar Institute of Technology Sindri (BIT Sindri).

Moghisuddin Raza

Mogishuddin Raza is a technology leader. As the COO of Accure he is having global product delivery responsibility along with overall strategic and operational responsibility.

Mogishuddin Raza is a technology leader. As the COO of Accure he is having global product delivery responsibility along with overall strategic and operational responsibility.

Having extensive background in technology product development and integration, in particular to Enterprise storage, virtualization, cloud computing, high availability & business continuity technology/solutions, and Big Data & related technologies. Has been passionate and evangelizing the usage of Big data technologies using Momentum to implement advanced analytics (descriptive and predictive) to directly impact the business via an intuitive set of use cases.

Having approximately two decades of experience in high-tech industries which includes big MNCs corporate like EMC Corp and Hewlett-Packard to mid-size organization such as Netkraft, Trados Inc driving transformation in strategizing, planning and architecting product engineering, execution and delivery of high quality products releases within budget & time.

Skilled in all aspects of big MNCs as well as company startups and growth including: strategizing, business planning, market research, finance, product development and profit margins & revenue management. Excellent leadership and people motivation skills. Expert in managing cross-functional, cross cultural global team and building strategic partnership in the global virtual matrix team environment.

Overall, a senior software business professional, skilled in the management of people, resources and partnerships which enables building an eco system for a winning organization.