In this article, we will see how to analyze the Twitter data with Azure Event Hub and Azure Stream Analytics. Later, we will save the live data to Cosmos DB using stream output in Azure Stream Analytics. We will see the live data in a browser using ASP. NET MVC application.
Step 1 – Create an event hub for Streaming Analytics input
We are using a Twitter WPF Client application to generate the events with the help of Twitter app and pushes them to an Azure event hub. Azure Event Hubs are used to pull and push data from other sources.
Log in to the Azure Portal.
Create an event hub namespace. One event hub namespace can hold multiple event hubs.
When the namespace has finished deploying, find the event hub namespace in your list of Azure resources.
Name the Event Hub.
Grant access to the event hub in Shared access policies, click the Add button.
Add an SAS Policy name and click the “Create” button.
Now, our SAS policy is created. Please copy the connection string and save carefully. We will use this connection string later in with our Twitter WPF Client application.
The client application gets tweet events directly from Twitter. To do so, it needs permission to call the Twitter Streaming APIs.
It is very easy to create a Twitter application. Please use this URL for that. You need a verified (Verify with your mobile number) Twitter account for this.
After creating the Twitter app, you must create an access token. This token will be used to authenticate our Twitter WPF Client application.
Please download the Twitter WPF Client application from here. I have already attached this zip file in this article too. Run the TwitterWPFClient.exe application. When the application prompts you, enter the following values –
- Twitter Consumer Key
- Twitter Consumer Secret
- Twitter Access Token
- Twitter Access Token Secret
Also, set EventHubName to the event hub name (that is, to the value of the entity path) and set EventHubNameConnectionString to the connection string.
Enter the search group which we must analyze sentiments of from Twitter. I have added flood as search group. We will get more Twitter feed data related to the recent flood in Kerala.
Click the green “Start” button to collect the social sentiment data.
Step 4 – Create a Stream Analytics job
Now that tweet events are streaming in real-time from Twitter, we can set up a Stream Analytics job to analyze these events in the real time.
In the Azure portal.
It’s a good idea to place the job and the event hub in the same region for the best performance and so that you don’t pay to transfer data between regions.
Step 5 – Specify the job input
Step 6 – Create an output sink
We have now defined an event stream, an event hub input to ingest events, and a query to perform a transformation over the stream. The last step is to define an output sink for the job.
Step 7 – Start the Job
Step 8 – Query the data from Cosmos DB using a simple MVC application.
On a final step, we will create a simple ASP. NET MVC application and get the data from Cosmos DB. I have attached this sample project with this article. I have not yet used any pagination in this application. We will get all the data in a single page. I am planning to create an Angular app with proper pagination and more features and that is a matter of another article.
We have covered all the steps for creating Event Hub Namespace, Event Hub, and later, we have created a Twitter application and with help of this Twitter app, we push the data to Event Hub. For pulling the data from Twitter and pushing to Event Hub, we used a Twitter WPF Client application provided by Microsoft. We created a Stream Analytics job with one Input, Output, and Query stream. Finally, we created a simple MVC application to get the data from Cosmos DB and show in the browser.
I hope you enjoyed the article and got a full idea of Sentiment Analysis with Azure Stream Analytics and Event Hub.
TwitterClient (WPF) source code can be found here
MVC Application source code obtained from here