flinkRESTapi #3

Merged

v-bmonajjemi merged 87 commits from flinkRESTapi into main 2025-12-09 13:58:54 -05:00

Conversation 0 Commits 87 Files changed 48 +2221

v-bmonajjemi commented 2025-12-09 13:58:35 -05:00

Owner

Copy link

Merging Metric Runner into main to split repo.

Merging Metric Runner into main to split repo.

v-bmonajjemi added 86 commits 2025-12-09 13:58:35 -05:00

Added flink code necessary to automatically submit jobs to flink. 629107ab63

Added docker-compose and directories containing job yaml and job python files to be mounted. e361d5491f

Added all updated files and resources necessary to build the pyflink container. c5ee6e06a9

Added all files necessary to build the kafkasubmit container. cdaa6d0080

Updated to allow for external access to kafka topics for writing purposes. 1df826858c

Added basic producer and consumer client-side tool, will expand soon 862971b3ab

Added new test job template that uses Cerberus to validate incoming json. 79abb6f6bc

Fixed issue with Kafka directory mount a7679da21e

Added Cerberus dependency to pyflink container 7d4a7fa54e

Merge branch 'flink' of https://dev.azure.com/DataSerfs/Runbook/_git/Kubernetes%20Development into flink bec09cf443

user can now upload specific json files specified in their directory, poc 7b7db2f77f

Added basictemplate.py file that can send messages to TSDB fa78ede320

Added basictemplate yaml file for associated python job 9ce3b8c861

Merge branch 'flink' of https://dev.azure.com/DataSerfs/Runbook/_git/Kubernetes%20Development into flink 4968c2c653

Added notes on data model. c0f0c0b779

Made changes to only allow specific characters for metric name, and added more fields to be passed through. aa3bf61018

Commit of Threat Model for MetricRunner 087676285d

Updated responses to all generated threats baf8e6fa05

Fixed bug with json loading of Kafka stream data. 49525a92ac

Added code for API that accepts post requests for data to be sent to TSDB. 3f6b4e8c86

Merge branch 'flink' of https://dev.azure.com/DataSerfs/Runbook/_git/Kubernetes%20Development into flink f068a5d9ce

Forgot to commit these changes. b41c28d3bd

Added initial code for continuous deployment of jobs. d7f3150ecb

Adding reference code. ec793794dd

Adding code inside file. 03b35d137b

Adding test code for how template file will be rewritten programmatically for different pipelines. 8a1df4a362

Adding template files in correct locations. bc8e09dc9d

Adding REST api code for creation of pipeline c3cbe31af7

Adding code to make GET, UPDATE, and DELETE for Metric Runner data pipelines. 41dbc04ccb

Added management api to Metric Runner POC. 6c39b76b4f

Added documentation for management api container. 6c4b2ec840

Added code to send message that fails Cerberus validation to dead letter archive. a80b01e420

Adding server to receive dead letter messages and write them to log. 1aa908ea75

Adding dead letter queue server to docker compose. d8f9374ef0

Change to make yaml template file more consistent. 0fa781aeba

Adding code to retrieve pipeline data from outside server and write to mounted local directory. 9070cb0f33

Adding windowing test job and yaml files. def4f5d94d

Now correctly sends data to TSDB. 9782c45f2a

Added documentation about how to use Metric Runner. aebfb722ab

Added yaml files for dj event streams 76c7ed4ea8

Added pyflink files for every pipeline for dj events. 86d2edebc8

Modifications to pyflink file for DJ's ldap base latency event stream. aeb63a79fe

Changed to calculate hourly average latency keyed on machine name. 70b93df757

Committed code needed for dj base latency script. 940b43c651

Added code for dj replication status script data source. 1b2910da44

Added source data code for dj ldap operations per host script. 3c3ef974f4

Committing changes to pipeline code. 42c29fba30

Changed code to send data to metric runner. e3d7cab91d

Added needed changes to pyflink job files, and added memory configuration to flink Taskmanager needed for larger load. 16ce1c496d

Added graylog sending code to front api for metric runner. 77dc69345f

Committing java flink job test version. d54053ef99

Adding initial code for full version of Flink job, with all variables passed as arguments. Had to revert version of Flink to 1.19.3 due to dependency issues. 01221f6acd

Fixed issues with map not working with Flink datastream, changed to Tuple4 to pass all values along for input from Kafka and Window function, still need to fix Keyed window function. aad7a2cbc4

Deleted FlinkTemplateJob.java to correct refactoring on naming conventions. 9acad70621

Added additional fixes for TSDB template job. Feature complete for current scope. a5dea85ea3

Added fix to convert milliseconds to seconds for timestamp in payload to TSDB. e51b9beb26

Merge branch 'flinkRESTapi' of https://dev.azure.com/DataSerfs/Runbook/_git/Kubernetes%20Development into flinkRESTapi ed7346106e

Added various fixes to reassure java compiler with TypeHint. 961b9f705a

Refactored code for management api, should now be production ready. 4d31c6092e

Added MySQL to docker compose file and set all needed env variables for management api. 3b8c1fd567

Minor fix for readability on README. af4624062e

Fixed documentation for active field, removed kafka server field as that will be an internal variable. 81333d97ed

Changed Kafka and Flink Rest API director containers to not use CLI and use Flink REST API and Kafka python module. fa7cb01311

Changed front api that checks incoming data with Cerberus and then sends it to Kafka, to use updated data structure. 5bc99f81ac

Added generic TSDB Flink job to container. 0b7502eaaa

Change to remove Graylog dependencies until Graylog is set up. 5fa6c6eb24

Fix for TSDB flink job location within docker container. 79551c28c2

Made needed changes to docker-compose file. 725da6b086

Fix for full chain cert name. be383e139c

Added needed fixes for mistake between post/put in java flink job and data structure in several management containers. c5e1a6d8ce

Added restart always to all containers in compose. cb260edeee

Removed redundant line. 326fa49b63

Added simple user facing docs. 5a1e8b8c7a

Updated README.md 6048c921b1

Fix for not correctly stopping jobs, need to pass job id not name. b63dab829f

Merge branch 'flinkRESTapi' of https://dev.azure.com/DataSerfs/Runbook/_git/Kubernetes%20Development into flinkRESTapi a0f8e29a74

Added documentation on getting TSDB data sent to Grafana. b564936138

Various minor fixes for Cerberus validation to work correctly. 1fd24299bd

Cleaning up Metric Runner directory. 5b43bf0135

Cleaning up old pyflink job specs. bdebd2fe8b

Removing obsolete java flink job. a0d0b2d1c3

Added needed fields to Metric Runner API for email alerting. a3973c4567

Updated Flink job to send post request to email container. fb9b2f4d41

Added all needed code for deployed email alerting. 3df9650d11

Minor fixes. dde6f5c501

Metric Runner Documentation. 6d3737c5ca

v-bmonajjemi added 1 commit 2025-12-09 13:58:43 -05:00

Merge branch 'main' into flinkRESTapi 4a311afd52

v-bmonajjemi merged commit 006b7c74e2 into main 2025-12-09 13:58:54 -05:00

v-bmonajjemi deleted branch flinkRESTapi 2025-12-09 13:58:55 -05:00

v-bmonajjemi referenced this pull request from a commit 2025-12-09 13:58:56 -05:00

Merge pull request 'flinkRESTapi' (#3) from flinkRESTapi into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Runbook/KubernetesDevelopment!3

No description provided.