Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Snap 2919 : Implementation of Structured Streaming UI Tab #184

Merged
merged 17 commits into from
Nov 27, 2019
Merged

Snap 2919 : Implementation of Structured Streaming UI Tab #184

merged 17 commits into from
Nov 27, 2019

Conversation

snappy-sachin
Copy link

@snappy-sachin snappy-sachin commented Nov 15, 2019 •
edited
Loading

What changes were proposed in this pull request?

Implementation of the Structure Streaming UI Tab to let users monitor the structured streaming query/application statistics and progress .
Structured Streaming Tab is available both in SnappyData embedded cluster as well as in smart connector application (using Snappy Spark distribution)

Structured Streaming Tab has below capabilities:

  • Listing all Structured Streaming Queries/Applications submitted to SnappyData cluster using submit-job command. Similarly in smart connector this tab will list streaming queries executed in cluster.
  • Allows user selecting queries from left hand side navigation panel, to view details view on right side main query details panel.
  • Query details panel displays selected queries details and statistics, as listed below;
    -- Query Name if provided, Query Id otherwise
    -- Start Date & Time
    -- Up time
    -- Trigger Interval
    -- Batches Processed
    -- Status
    -- Total Input Records
    -- Current Input Rate
    -- Current Processing Rate
    -- Total Batch Processing Time
    -- Avg. Batch Processing Time
  • Query details panel also lists sources of streaming query along with each source details like type, description, input records, input and processing rate
  • Query details panel also displays sink details of streaming query.
  • Query details panel depicts structured streaming queries behaviourial trends using following
    -- Input Records on every batch
    -- Input Rate vs Processing Rate
    -- Processing Time
    -- Aggregation State, if available

Please check JIRA item SNAP-2919 for UI screenshots (https://jira.snappydata.io/browse/SNAP-2919)

How was this patch tested?

  • Tested manually

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review http://spark.apache.org/contributing.html before opening a pull request.

@snappy-sachin
Code changes for SNAP-2919:
333e572 
 - Adding streaming page CSS and JavaScript files.
@snappy-sachin
Intermediate code changes:
8979a4b 
 - Auto-refresh feature.
 - Query stats populating on UI.
 - CSS changes.
@snappy-sachin
- CSS changes
e349c92
@snappy-sachin
Code changes for:
dde2627 
 - CSS changes
 - Display Status in different colors
 - Rounding of float values on UI
 - Display numbers with thousands separator mark.
 - Display time unit as ms for milliseconds
 - Display sources table with basic stats.
 - Display sink description
 - Display Aggregation state (state operator) charts.
@snappy-sachin
CSS and JavaScript code changes for highlighting query selection in q…
8a78e1d 
…uery navigation panel.
@snappy-sachin
Code changes for:
209bc5b 
 - Adding utility function for conversion of duration time in human readable form.
 - JavaScript changes for displaying queries latest input rate and processing rate.
@snappy-sachin
- CSS and JavaScript changes for displaying connecton error message.
03002e5
@snappy-sachin
Code changes for:
af08547 
 - Display aggregation states chart only when applicable
 - Removed updated records/rows trend line from aggregation states chart
 - Fixing few other issues
@snappy-sachin
- JavaScript changes for displaying sink specific details.
13c8af8
@snappy-sachin
Code changes for displaying sink details table.
d13d2e5
@snappy-sachin
Code Changes for SNAP-2919:
518c729 
 - Moving below classes/files from SnappyData Core to Spark
   SnappyStreamingQueryListener.scala,
   StreamingRepository.scala,
   SnappyStreamingApiRootResource.scala,
   StreamsInfoResource.scala,
   streamapi.scala,
   SnappyStreamingTab.scala,
   SnappyStructuredStreamingPage.scala
 - Moving Code for updating UI for Structured Streaming tab from SnappySession to SparkSession
@snappy-sachin
Merge branch 'snappy/branch-2.1' into SNAP-2919
944ab97
@snappy-sachin
Some code cleanup.
e1df601
@snappy-sachin
Merge branch 'snappy/branch-2.1' into SNAP-2919
d373f2d
@snappy-sachin
- Code clean up and code refactoring.
4232573
@snappy-sachin
Code changes:
75c44f8 
 - In spark code, updating QueryStartedEvent signature to include triggerInterval
 - Changes for displaying Processing Threshold line in Stream Processing Time chart on UI
 - Adding Trigger Interval on UI
@snappy-sachin snappy-sachin mentioned this pull request Nov 20, 2019
Merged
@snappy-sachin
Code changes for SNAP-2919 :
e61e559 
  - Feedback Question "Is there a case where onQueryprogress is called without onQueryStarted?"
    Ans: As per discussion, YES it can happen in rare situation but no need to handle such case.
      So removing the counter code implementation (on queryProgressEvent) of handling missed queryStartEvent.
      Instead, now just logging warning message for the same.
  - Adding two configurable parameters in sparks SQLConf.scala
      1) spark.sql.streaming.uiRunningQueriesDisplayLimit :
           To configure how many queries be displayed on structure streaming UI.
      2) spark.sql.streaming.uiTrendsMaxSampleSize :
           To configure how many historic data points be plotted on trends charts on structure streaming UI.
  - Handling of query removal in case of uiRunningQueriesDisplayLimit limit is reached.
      Inactive queries are removed if there is no space for newly added running query.
      If all existing queries are active and uiRunningQueriesDisplayLimit limit is reached then
      newly added query won't be displayed on UI.
  - Fixed issue - Query details panel keeps displaying old inactive query details if that query was
      selected before it was removed from query navigation panel.
  - Code refactoring
@snappy-sachin
Copy link
Author

All feedback taken care.. Hence merging.

@snappy-sachin snappy-sachin merged commit 1cdbfb7 into snappy/branch-2.1 Nov 27, 2019
@snappy-sachin snappy-sachin deleted the SNAP-2919 branch November 27, 2019 12:50
sumwale pushed a commit to sumwale/spark that referenced this pull request Nov 5, 2020
@snappy-sachin @sumwale
[SNAP-2919] Implementation of Structured Streaming UI Tab (TIBCOSoftw…
9fe0264 
…are#184)

Implementation of the Structured Streaming UI Tab which lets users monitor the structured streaming queries/applications statistics and progress .
Structured Streaming Tab is available both in TIBCO ComputeDB/SnappyData embedded cluster as well as in smart connector application (using Snappy Spark distribution)

Structured Streaming Tab has below capabilities:

- Listing all Structured Streaming Queries/Applications submitted to SnappyData cluster using submit-job command. Similarly in smart connector this tab will list streaming queries executed in cluster.
- Allows user selecting queries from left hand side navigation panel, to view details view on right side main query details panel.
- Query details panel displays selected queries details and statistics, as listed below;
  -- Query Name if provided, Query Id otherwise
  -- Start Date & Time
  -- Up time
  -- Trigger Interval
  -- Batches Processed
  -- Status
  -- Total Input Records
  -- Current Input Rate
  -- Current Processing Rate
  -- Total Batch Processing Time
  -- Avg. Batch Processing Time
- Query details panel also lists sources of streaming query along with each source details like type, description, input records, input and processing rate
- Query details panel also displays sink details of streaming query.
- Query details panel depicts structured streaming queries behavioural trends using following
  -- Input Records on every batch
  -- Input Rate vs Processing Rate
  -- Processing Time
  -- Aggregation State, if available
- All statistics displayed on UI are auto updated periodically

- Adding two configurable parameters in sparks SQLConf.scala
      1) spark.sql.streaming.uiRunningQueriesDisplayLimit :
           To configure how many queries be displayed on structure streaming UI.
      2) spark.sql.streaming.uiTrendsMaxSampleSize :
           To configure how many historic data points be plotted on trends charts on structure streaming UI.
sumwale pushed a commit that referenced this pull request Jul 11, 2021
@snappy-sachin @sumwale
[SNAP-2919] Implementation of Structured Streaming UI Tab (#184)
334b4a6 
Implementation of the Structured Streaming UI Tab which lets users monitor the structured streaming queries/applications statistics and progress .
Structured Streaming Tab is available both in TIBCO ComputeDB/SnappyData embedded cluster as well as in smart connector application (using Snappy Spark distribution)

Structured Streaming Tab has below capabilities:

- Listing all Structured Streaming Queries/Applications submitted to SnappyData cluster using submit-job command. Similarly in smart connector this tab will list streaming queries executed in cluster.
- Allows user selecting queries from left hand side navigation panel, to view details view on right side main query details panel.
- Query details panel displays selected queries details and statistics, as listed below;
  -- Query Name if provided, Query Id otherwise
  -- Start Date & Time
  -- Up time
  -- Trigger Interval
  -- Batches Processed
  -- Status
  -- Total Input Records
  -- Current Input Rate
  -- Current Processing Rate
  -- Total Batch Processing Time
  -- Avg. Batch Processing Time
- Query details panel also lists sources of streaming query along with each source details like type, description, input records, input and processing rate
- Query details panel also displays sink details of streaming query.
- Query details panel depicts structured streaming queries behavioural trends using following
  -- Input Records on every batch
  -- Input Rate vs Processing Rate
  -- Processing Time
  -- Aggregation State, if available
- All statistics displayed on UI are auto updated periodically

- Adding two configurable parameters in sparks SQLConf.scala
      1) spark.sql.streaming.uiRunningQueriesDisplayLimit :
           To configure how many queries be displayed on structure streaming UI.
      2) spark.sql.streaming.uiTrendsMaxSampleSize :
           To configure how many historic data points be plotted on trends charts on structure streaming UI.
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@vatsalmevada vatsalmevada vatsalmevada left review comments

@suranjan suranjan suranjan approved these changes

@dshirish dshirish Awaiting requested review from dshirish

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@snappy-sachin @suranjan @vatsalmevada